Fifth Tuna Data Workshop (TDW-5) April 2011, Noumea, New Caledonia Oceanic Fisheries Programme (OFP) Secretariat of the Pacific Community (SPC) Session 9.2 Auditing data management systems

Presentation outline Focus on logsheet system, no port sampling What is an audit and why is it needed ? Two types of audit : –Systematic –Non-systematic Two components : –Logsheet data collection –Logsheet data management in-country audit schedule Future development of logsheet data management audit

What is an audit A definition of “Audit” “…a formal, often periodic examination and checking of data or procedures to verify their correctness…” “Auditing” is usually associated with the accounting or financial sectors, but the principles can be applied to any type of data, or processes involving the collection of data. For logsheet management data audit, correctness means completeness and accuracy

Auditing tuna fishery data Why is there a need to audit systems ? Improves the quality and coverage of data available at the national level and to the WCPFC Ensures data are standardised (fundamental to users) Ensures every actor of the logsheet chain shares the same knowledge on the coverage status Systems and data will be RECOGNISED for their quality (e.g. “WCPFC-audited” data or data collection system may have a high-level reputation)

Auditing tuna fishery data History… SPC/OFP have already conducted several audits in the past… DATA COLLECTION AUDITS (see Agenda Item 9.1) DATA MANAGEMENT AUDITS “Systematic” AUDITS, for example Observer Debriefing Process Related to production of PROCEDURES DOCUMENTS Review of Logsheet data management system in Fiji Port Sampling review Workshops Regional Observer Coordinator Workshops “Non-systematic” AUDITS, for example Review of Port Sampling during visits Review of Logsheet and Unloadings data collection during visits

Auditing of logsheet data management Particularities of logsheet data Three data sources used : VMS, TUFMAN (if installed) and SPC Regional database VMS : “Trips” generated from detailed VMS data provided by FFA Pros : Very high coverage of Vessel activity – “base” data Cons : VMS “Trip” generation is not perfect, needs to be improved with experience TUFMAN : Logsheet data entered in-country into TUFMAN Pros : Close to fishing companies, data available sooner Cons : Difficulty to highlight missing logsheet SPC : Logsheet sent by countries to SPC Pros : Strong historic and multi-country data Cons : Delay (last link of the chain)

Systemic logsheet data management audit “Systemic” means centralized and common audit Based on data stored at SPC, e.g Logsheets from countries and VMS Vs TUFMAN or other in-country data registration system For the moment, to be done using VMS and SPC data sources To be achieved throughout the year Every quarter, an export will be sent to countries to highlight gaps in the logsheets received at SPC Three main reasons for a gap: Data received in-country from fishing company but not sent to SPC Data not received in-country from fishing company Wrong VMS trip This export will also provide the logsheet coverage per year/gear/fleet as an indicator

Non-Systemic logsheet data collection audit Non-systemic audit will be achieved during a one-week in-country visit. This procedure involves interviews of the stakeholders involved in the logsheet data collection process. Planning needs to be carefully designed prior the auditor visit in order to ensure availability of the logsheet data collection staff. A copy of the Logsheet Audit Workbook will be sent prior to the country prior the audit During in-country visit, and following the logsheet audit workbook, the following aspects (but not restricted to) of data collection will be covered : Existence of data registration system Timeliness and frequency of provision of logsheet data from companies Level of liaison with fishing companies (regarding logsheets) After highlighting gaps, do companies cooperate to fill them Is there a filing system which makes it easy to retrieve hard- copy or scanned logsheet

Non-Systemic logsheet data management audit During in-country visit, and following the logsheet audit workbook, the following aspect of data management will be checked : Data Registration E.g any potential hard-copy logsheet going missing prior to data entry or submission to regional agency Hard-copy data management E.g does the responsibility matrix for hard-copy data management cover all work and match actual responsibilities Database System E.g Are there any database post data-entry checking TUFMAN logsheets will be compared to VMS Highlight gaps, identify missing logsheets to be requested from fishing companies TUFMAN logsheets will be compared SPC-entered Logsheet data A summary report will be produced highlighting the differences (this will be a key output of the audit process) TUFMAN logsheets will be checked against randomly selected hard- copy logsheets

audit plan Four in-country audits per year are scheduled For 2011, FSM has already been audited (March 2011) The three other countries to be audited during 2011 will be –(At this stage) RMI, Cook Islands and Fiji

VMS – Logsheet Reconciliation Can be used to enhance your annual catch estimates for your national fleet… Use the % coverage from the VMS-logsheet reconciliation report to “raise” your catch estimate obtained from TUFMAN … For example, your tuna catch estimate for 2010 based on logsheets entered and reported from TUFMAN is 4,000 t., but you are not sure whether all logsheets have been received. The VMS-logsheet reconciliation shows that the coverage of logsheets in TUFMAN for 2010 for your national fleet is 80% So, the ‘raised’ estimate will be 4,000 t. * 1/0.80 = 5,000 t. Try to do this for your national fleet and compare with other coverage rates you may have determined …

VMS – Logsheet Reconciliation Information regarding the process Different filters for the process : Year : 2008/2009/2010 Gear : LL/PS Country : Fleet/EEZ Data source : VMS with TUFMAN/SPC Data Report : Detailed/Summary Reconciliation based on vessel FFA identification and trip dates All unmapped VMS trip entered the EZ of the designated country

VMS – Logsheet Reconciliation How to read reconciliation detailed files One file per Year/Gear/Country/Flag One line per trip, containing the vessel name, the vms and the logsheet trip dates, and whether the trip was in the EZ or out. Examples of a reconciliation for parameters Year = 2008 Gear = PS Fleet = TW, EZ = KI Data source = VMS and SPC Data Report = Detailed

VMS – Logsheet Reconciliation 3 Different scenarios Mapped trip This trip has been sent to SPC and entered into SPC database It is confirmed by the VMS trip Best situation

VMS – Logsheet Reconciliation 3 Different scenarios Unmapped VMS trip This vessel left a port on May 31 st 2008, and went to another one on June 17 th 2008 During this period, it entered the KI EEZ SPC Database does not have a logsheet for this vessel Need to trace the logsheet

VMS – Logsheet Reconciliation 3 Different scenarios Unmapped VMS trip SPC to produce an export with this type of missing logsheet Run another reconciliation, replacing SPC (regional) by TUFMAN (national) data source Rest of the parameters remain the same The logsheet has not been entered in country, and thus should ask the corresponding fishing company to “fill the gaps”

VMS – Logsheet Reconciliation 3 Different scenarios Unmapped logsheet trip (unlikely to happened) Logsheet has been produced but cannot me mapped with a VMS trip Flagged for further investigation at regional level Feedback will be provided to countries if the situation is not cleared (manual report due to failure of VMS device for instance)

VMS – Logsheet Reconciliation How to read reconciliation summary files Global overview of your EZ or your national fleet Same parameters as for the detailed files Provides the following data : Total number of trips Total number of vms trips Total number of logsheet trips Coverage for both vms and logsheet Also provides this information Per Flag (if EZ selected) or per Country (if Fleet selected) Per Year (if EZ Selected)

VMS – Logsheet Reconciliation Example of a reconciliation summary Summary report for fleet Year = 2009 Gear = PS Fleet = TW Data source = VMS and SPC Database Report = Summary Taiwan PS fleet provided in % of logsheet, for a total number of trip of 314

VMS – Logsheet Reconciliation Example of a reconciliation summary Repartition per fleet for the following parameter Year = 2009 Gear = LL EZ = FSM Data source = VMS and FSM Tufman Report = Summary 1250 LL trips happened in 2009 in FSM EEZ, and 72.64% of these trips have their logsheet entered in FSM Tufman

Future development How could audits be undertaken in the future ? The audit workbook will be improved during first in-country visits before becoming the standard “Systemic” logsheet data audit could be managed by the countries using TUFMAN Authorisation to send VMS data -- under discussion VMS “Trip” data will be sent to countries to be imported using a TUFMAN screen A set of reports will then compare VMS and TUFMAN data sources to provide missing logsheet and information on coverage BENEFITS : No need to wait for the in-country visit to have this report, and will save time during such audit for other tasks

Future development (continued) The proposed TUFMAN post-entry DATA AUDIT Module A specific module to provide a more thorough check of the Data entered into TUFMAN For example … More comprehensive Control totals checking Position checking using distance and a visual aid (a map of the cruise track), and comparison to VMS data More comprehensive FIELD range-checking (e.g. species catch and average weight) The system will offer suggestions for missing fields…

Exercise Each country will be provided with a list of the VMS – Logsheet reconciliation for your country Please review these lists and the summaries Do you have any comments on the coverage ? Can you indicate the fishing companies … … That will readily provide the missing logsheets … That will be difficult to deal with when providing the missing logsheets … where there are logistical problems in getting the missing logsheets