WP 2.4 Evaluation of NMFS Toolbox Assessment Models on Simulated Groundfish Data Sets Comparative Simulation Tests Overview Brooks, Legault, Nitschke, O’Brien, Sosebee, Rago, and Seaver Nothing gives rest but the sincere search for truth. Blaise Pascal (French philosopher)

What did we do? Evaluated 5 NFT stock assessment models for three stocks under 4 scenarios meant to examine potential difficulties in real assessments –AIM, ASPIC, SCALE, VPA, ASAP –GB yt (retro), GB cod (domes), white hake (ageing) PopSim used to generate true conditions and create 100 datasets with the same random errors for all 5 models Evaluated Accuracy and Precision of the 100 point estimates from the models –Did not examine precision of each of the 100 runs 60 scenarios 6000 assessments

Why? Test hypothesis that all models are impacted similarly when presented with the same underlying problem A priori know that some models will not perform well under the test conditions because limiting data to VPA years NOT trying to declare one model “winner” NOT trying to declare any model “bad”

PopSim Primer Age and Length Based Population Simulator User defines Dimensions (Years, Ages, Plus Group Age, Lengths) Initial NAA Recruitment time series (or SRR) Annual Fmult and selectivity Biological Characteristics –M, von B, L-W Fishery Sampling Surveys Sets Template for Stock Assessment Model

Surveys vs Indices Surveys –Are a property of the true population –Catchability defined for all ages and years –Uncertainty added to true values at age and length Indices –Are a property of the model –Sum values from surveys –Can be either number or biomass based –Can be limited age range or entire age range –Can be changed between models without impacting underlying truth

Growth Initial NAA distributed according to stdev1 Growth transfer matrices created for each age based on expected von B growth for age and stdev2 Fish not allowed to decrease in size Allows fishing to change distribution of length at age

Market Sampling Markets declared by user Sampling conducted per 100 mt of landings in each market each year

InputOutput

PopSim Limitations PopSim is not reality Annual Time Steps Does not contain spatial components Does not allow gender differences Does not allow density dependent effects No integrated management –Developing MSE wrapper to use PopSim, VPA, AgePro, and Control Rules

This Exercise Used utility to convert VPA run to PopSim –Gets Nyears, plus group age from VPA –Sets initial NAA and R from VPA –Sets annual Fmult from VPA –Estimates one logistic selectivity from VPA Length and biology stuff from user Market stuff from user Surveys and Indices defs from user Tuned markets, sampling, and surveys to represent actual assessments by lead

Farmed Out Assessments AIM – Rago ASPIC – Brooks SCALE – Nitschke VPA – Legault, O’Brien, Sosebee ASAP – Legault Used base case to get template settings reasonable Applied this base case to each of the test cases Some models did additional runs with modified templates to “fix” the problem

Results PopSim compares the distribution of 100 assessments with the known true values Exactly what is compared depends on model –E.g. VPA NAA & FAA, ASPIC B & F Many, many runs and scenarios –PopSim creates tables and graphs –R program to gather results and automatically create plots

Black Line True Circles and Grey Line Median Red dashed Lines 5 and 95%iles Started by looking at direct results

Decided Bias and CV Better

General Conclusions Given failure of all models tested (simple & complex), we suspect other models would also be vulnerable to “retrospective agents” Use of age-specific indices is robust to uncertainty in survey selectivity If ageing is uncertain, these simulations support using models w/o age or models which allow uncertainty in catch at age

General Conclusions (cont.) VPA and ASAP ‘failures’ were similar in pattern Magnitude of bias was less for ASAP Precision usually somewhat better for ASAP Given these similarities, we suggest that ASAP may offer some advantages to VPA (esp. in terms of flexibility)