1. The Refereeing Module of the SPMS FEL2005: 22-26 August Heinz-Dieter Nuhn – Scientific Editor Beck Reitmeyer – Conference Editor Referee = Reviewer = Expert Refereeing = Reviewing

2. History Some JACoW conference series use a refereeing process, like ICALEPCS, but it was not included in the SPMS For FEL, there has always been a refereeing process for the proceedings Through 2003, published through Elsevier Joined JACoW in 2004; refereeing added to SPMS initially by Ivan, but it was a separate instance

3. Adding Refereeing Module to the SPMS: Proposed Workflow Workflow affected by SLAC Computer Security; worked out OK anyway for editing Referees for this conference: –Assigned 4-8 papers –Given 6 weeks to complete refereeing

4. How It Works: Before Step 1: System Parameter Setup Auto-assign referees Text for e-mail notice for paper available Text for e-mail notice for referee assigned Instructions Notify admin on decline (red) Notify admin on revisions (orange) Notify PC on decline (red) Notify PC on revisions (orange) Refereed conference Referees per paper

5. How It Works: Before Step 2: Processing Code Setup (May be different already: separate screen for referee and editing codes) Add refereeing codes in at the same place you add in Editing Codes ADDED LATER: A “no paper” code and an “in review” code UNCLEAR: Which takes precedence – editing or refereeing code?

6. How It Works: Before Step 3: Assigning Referees Location: Maintenance Tables > Classifications > Main Classifications > Experts (at the very end of the line) Each classification has a set of referees assigned to it That referee within each classification can be assigned a paper Reports > Referee > Unassigned identifies papers that were missed

7. How It Works: During Step 4: Editor Checks Theory: Just a quick check, make a PDF Reality: Full edit (OK – use the editors when you have them!) Kept future editing code and notes to ourselves and authors in the “Editor’s Notes” field, let author know that paper was being refereed in the “Comments to Author” field. Did not assign an Editing code during this first pass! When the editor uploaded a PDF, e-mail sent to the referee. PROBLEM: Author could still upload a paper; adding in “In Review” code would have stopped this, but again, which code takes precedence

8. How It Works: During Step 5: Refereeing Referees used three codes:  Referee Green: Good paper, accepted for the proceedings  Referee Orange: OK paper, needs some work before it can be accepted for the proceedings  Referee Red: Rejected for the proceedings based on scientific content NOTE: We expected mostly greens, got about half green, half oranges. This meant that a lot (>70) of the papers were going to be rewritten and resubmitted, most likely after the conference. PROBLEM (fixed): System wasn’t sending out notices of Referee Orange dot to authors; some found it on their own, some didn’t. A lot of people didn’t know what the Paper Dot Status Board was for.

9. How It Works: During Step 5: Refereeing (2) Regular login Extra “Referee” link available New screen listed all assigned papers If a paper was available for refereeing (signified by a PDF having been uploaded by an editor), then the Paper ID showed as a link

10. How It Works: During Step 5: Refereeing (3) Referees were asked to use the color codes and provide comments to the authors PROBLEM: Unclear what made the paper orange (content or style); needed clearer instructions 4000 characters was not enough for some

11. (Aside) Refereeing Timeline Most of the referees did not referee their papers during the conference Authors theoretically had 2 weeks to respond; almost all did, but not necessarily within 2 weeks ~98% of papers with an orange dot were resubmitted

12. How It Works: After Step 6: Reprocessing Resubmitted papers were processed as before Second review done by the Scientific Editor, not the original referee Second referee notice e-mail was just sent to the administrator e-mail (a mailing list)

13. How It Works: After Step 7: Finishing When a paper passed the refereeing process (that is, it got a Referee Green dot), it came back to the editors for final processing Basically, this meant copying the notes from the “Editor’s Notes” field to the “Comments to Author” field and selecting the correct editing dot

14. How It Works: After Step 7: Finishing (2) Because most of the papers marked Editing Yellow were notified as such after the conference, we stopped notifying authors since they either didn’t respond or responded with many corrections Papers were proofed against the author copy or carefully on its own Usually, this final proofing has occurred as part of the final QA

15. Actual Workflow 1.“JACoW editing” took place before refereeing; dotting took place after for Editing Green and Editing Yellow dots 2.After the first version of the paper, it went to the Scientific Editor, not the named referee 3.Editing Yellow papers after the conference were not sent back to the author, but checked by the editors 1 2 3

16. Next Time: Clearly Needed Add the additional codes (at least “In Review”) first Allow searching on the refereeing code Put refereeing statistics on main Statistics page or on the refereeing- only Master Status page? Definitely needs tallies somewhere. (may be done) Separate out the refereeing codes from the editing codes –Determine best place for “in review” dot – with editor or referee? –Determine which code takes precedence for author uploads? (done) Referees could see “owner” of the comments on their submissions

17. Next Time: Needed, but Confusing Modify the main Statistics report page to include refereeing – numbers of where papers stand is very hard to figure out Really hard to get a handle on what’s going on from an administrator perspective – papers come back to editors, and after the conference, they’re gone and not checking. This probably only happens with refereeing since there are steps that need to be taken after the paper comes back, and that mostly happens after the conference. Timing issue of editing vs. refereeing? Can/should this be a system parameter, or better handled “outside” the system like we did here? More process than programming. How does it affect precedence of referee vs. editing codes?

18. Suggested System Parameters for Refereeing Refereed conference (Y/N) Auto-assign referees (Y/N) Text for e-mail notice for paper available (Text) Text for e-mail notice for referee assigned (Text) Link for instructions (Link, not full text) Notify admin on decline (Y/N) Notify admin on revisions (Y/N) Notify admin on accepted (Y/N) Referees per paper (# -- incomplete; both at the same time?) Second review (who? Number of reviews?) Notify referee if referee dot is already green? (Y/N) Not needed if admin is cc’d?

19. Next Time: Would Be Nice Determine statistics for refereeing –How many papers were initially orange? –How many papers are orange now? –How many papers were resubmitted more than once? Increase the number of characters allowed for the referees to >4000 Put processing code in the subject line of the e- mails sent to authors/administrators Change “Edit Complete” to just “Editor”