1. Report Pilot Process – Refereed IPAC17 Proceedings
R, Assmann, IPAC17, May 2017
For the task force and helping colleagues: Ralph Assmann (DESY), Giovanni Bisoffi (INFN Legnaro), Marie-Emmanuelle Couprie (Soleil), Mike Seidel (PSI), Gianluigi Arduini (CERN), Juliana Pranke (ESS), Matthew Arena (FNAL), ...
Contributions and Input Acknowledged:
Martin Köhler (Head DESY Library, Germany)
P. Dumas (co-chair of SRI 2012 in Lyon)
Members of EPS-AG
11/9/2018
Refereed IPAC17 Proceedings

2. Special Acknowledgements
Giovanni Bisoffi and Ivan Andrian:
Implementation into the overall process
Dates and steps to be respected
Draft procedure
Mike Seidel:
Acceptance criteria
Matt Arena:
Upgrading SPMS refereeing module, new features, fixing bugs, follow-up on process
Ulrich Dorda et al:
Special DESY testing team (5 persons): checking by non-experts of SPMS process
The MC coordinators, acting as editors
Juliana Pranke:
Contact to IPAC17 team, forwarding s, IOP contract, ...
11/9/2018
EPS-AG Task Force

3. Refereed IPAC17 Proceedings
Discussed at IPAC16 in Pohang
Decided to go along with pilot process for IPAC17
Distributed to IPAC CC by Mike Seidel on 5 August 2016  for coordination and discussion with the regions
11/9/2018
Refereed IPAC17 Proceedings

4. IPAC2017: Goals Peer-Review
JACOW
PRAB
Non-peer reviewed
Conference publications
Open source
Peer reviewed
Journal publications
Open source
Peer reviewed
Conference preprints
Open source
82 %
15 %
3 %
By invitation only
Papers published before
Papers that will later be published in journals
Authors who do not care or have time for peer-review papers
Insufficient quality papers
Papers from authors interested in peer- review but not yet at level for high impact factor journals
Timely and well- organized submissions
Young colleagues: first peer-reviewed publication
Papers of very high quality aiming at high impact factor publication
Author choice First come, first serve
11/9/2018
Refereed IPAC17 Proceedings

5. Our Goals
Quality: Introduce quality control for a sub-set of IPAC proceeding papers which have been checked for correctness and scientific value.
Training: Authors experience the peer-review process and its quality assurance mechanisms.
Training: Volunteers perform peer reviewing, some of them their first reviewing experience.
Encouragement: Less stringent acceptance criteria provide many authors with a positive experience on peer reviewing.
Competitiveness: Fraction of conference papers should be countable in performance indicators of research organizations.
Note: This is not a new journal for accelerator physics but a stepping stone to high impact journals like our PRAB flagship.
11/9/2018
EPS-AG Task Force

6. 11/9/2018
EPS-AG Task Force

7. Refereed IPAC17 Proceedings
11/9/2018
Refereed IPAC17 Proceedings

8. Professional channel for publishing conference proceedings.
International – heavily used from all regions and many scientific fields.
Professional channel for publishing conference proceedings.
11/9/2018
Refereed IPAC17 Proceedings

9. IOP Conference Series Peer Review Policy
All conferences are requested to adhere to the following minimum standards:
Unbiased consideration is given to all manuscripts offered for publication in the proceedings regardless of race, gender, religious belief, ethnic origin, citizenship or political philosophy of the authors.
Conference papers should meet all the usual standards of quality for an IOP Publishing publication. However, referees should take into account the conference nature of the articles and so consider background papers more favourably than would be normal for a regular paper. These allowances should not go so far as to approve papers of low scientific standard or papers that have been published in written form elsewhere. Review papers are also welcomed and accepted.
Referees should give a clear statement of recommendation for each paper and supply comments to support their recommendation suitable for transmission to the author.
11/9/2018
EPS-AG Task Force

10. Demand at Abstract Submission
Status 12 December 2016
11/9/2018
Refereed IPAC17 Proceedings

11. Papers Received in Time (April 24)
Requested: papers
Available in time: 123 papers
Submitted at all: 245 papers
Assume that 2/3 of abstracts have submitted papers: 1,100 papers total
Fraction peer-reviewed estimated: %
11/9/2018
EPS-AG Task Force

12. Timeline over 16 days BEFORE IPAC17
April Matt Arena development work, final process, peer-review team tests, DESY team tests April 25 Papers selected and referees automatically assigned Approvers asked to check/optimize assignments 123 papers, 123 corresponding authors, 18 approvers April 27 end of business day: to referee – start refereeing 158 referees May 3 first round of reviews complete May 8 authors have included comments/feedback May 11 second round of review complete May 12 final decisions by approvers 121 papers decided, 2 open  all complete May 15
11/9/2018
EPS-AG Task Force

13. Could have handled 80% more papers!
158 Referees
Referees
Papers 81 1 71 2 3 4
Referee honor league
Boine-Frankenheim, Oliver
Boland, Mark
Casalbuoni, Sara
Kitegi, Charles
Koscielniak, Shane
Stem, William
Could have handled 80% more papers!
11/9/2018
EPS-AG Task Force

14. Refereeing results for 123 papers
110
Out of 123 papers handled:
Acceptance: %
Rejection: %
11/9/2018
EPS-AG Task Force

15. Impact for our field Quality: Peer review experience:
In many papers errors were identified during review and fixed by authors
13 “unacceptable” papers identified and authors informed  opportunity for these authors to learn and correct mistakes
Peer review experience:
Peer review experience for 123 authors and 158 referees
Positive peer review result for 110 authors  encouragement
Quantitative performance indicators accelerator science:
123 peer-reviewed conference papers: indexed by Scopus, Inspec, Conference Proceedings citation index (ISI Web of Science), Compendex and many others.
All papers fully citable: citations tracked via IOP publishing citing articles facility and Scopus
575 references in 42 papers in MC01-MC03. Many of them to journal articles: e.g. 36 citations to PRSTAB. Extrapolation: expect about 1500 references, e.g. about 100 PRAB citations generated
11/9/2018
EPS-AG Task Force

16. Peer Review Wish List - IPAC 2018
Check the name assignment in the log comment when referees are assigned
Don't allow a referee to be an approver in the same MC
Push referee assignment to the sub-classification level
Restrict approvers' dot assignment to a final status only (Green or Red)
Allow referees, approvers and authors to send messages to each other, keeping anonymity
Add referee removal to the abstract log
Allow administrators easier methods to send s to authors, referees and approvers
Add system parameters to disable features once executed (select papers, assign referees) to prevent accidental clicking
Document peer foreseen review date planning and process on IPAC web site, so required actions can be looked up by all actors
Use of same IOP and JaCOW templates?
11/9/2018
EPS-AG Task Force

17. Scalability Remember: 2 referees per paper. Finish before IPAC17.
With the volunteers we had for IPAC17 we could have done twice the papers, e.g. 200 – 250 papers feasible for IPAC17.
More papers in peer review process = more volunteers (authors also volunteer to do a peer review)
In theory: 1,350 IPAC delegates reviews per delegate, 2 referees per paper  maximum number of papers = 1,350
Must include non-availability / issues for referees (sickness, ...):  about 1,100 papers max but workload MC coordinators and editing team is an issue
My personal feeling: 400 papers max next time reasonably ambitious and realistic goal
11/9/2018
EPS-AG Task Force

18. https://conferenceseries.iop.org/content/authors
Next Steps
Ongoing: JaCOW editing process for all papers, also peer-reviewed IOP papers.
JaCOW team takes care that scientific content is not changed
Ongoing: Propose PRAB papers (see Mike Seidel)
Afterwards: Authors must reformat papers to IOP template, based on final JaCOW version.
In some future: Discuss usage of the same JaCOW and IOP (or other publisher) templates?
11/9/2018
EPS-AG Task Force

19. Conclusion
Pilot peer-review process for IPAC17 proceedings worked as planned. Successful. Thanks to help from many colleagues!
All papers ready in time were handled before IPAC17:
110 papers accepted
13 papers rejected
Many positive remarks, for example: “I found that the authors are very sincere and responsive to our requests. I wish the paper would be accepted for publication. That will be a very good (positive) lesson for them.” “Review was very good and paper afterwards much improved.”
Hick-ups solved on the way (many thanks to Matt Arena for implementing them fast): peer review wish list for IPAC18.
Step-wise approach was a good idea: Positive experience.
Next time: 400 papers (if demand exists) can be handled.
Important: Verify positive impact on journal publications!
11/9/2018
EPS-AG Task Force

20. Thank you for your attention
11/9/2018
EPS-AG Task Force

21. 11/9/2018
EPS-AG Task Force

22. 11/9/2018
EPS-AG Task Force