Presentation is loading. Please wait.

Presentation is loading. Please wait.

Excellence in Research for Australia ERA at UWA Presentation to the UMG, 19 November 2009 Lorna Robertson, Manager Research Assessment Unit, Research Services.

Similar presentations


Presentation on theme: "Excellence in Research for Australia ERA at UWA Presentation to the UMG, 19 November 2009 Lorna Robertson, Manager Research Assessment Unit, Research Services."— Presentation transcript:

1 Excellence in Research for Australia ERA at UWA Presentation to the UMG, 19 November 2009 Lorna Robertson, Manager Research Assessment Unit, Research Services

2 Excellence in Research for Australia ERA is a research assessment exercise aimed at: Evaluating a range of indicators of quality at the discipline level Allowing for comparisons at an institutional level, national level, and international level Identifying excellence in research performance Identifying emerging research areas and opportunities

3 Units of Evaluation Research assessment is discipline based: –Units of Evaluation, defined by Field of Research (FoR) codes, eg 0205 Optical Physics, 1110 Nursing –Allows for comparison across institutions –Disaggregates interdisciplinary research into component disciplines The results of the ERA Trial will be publicly available at the national level The results of the full ERA will be publicly available at the institutional level

4 8 Clusters 1Physical, Chemical & Earth Sciences (PCE) –FoRs: 02xx, 03xx, 04xx 2Humanities & Creative Arts (HCA) –FoRs: 1201, 1203, 1205, 1299, 18xx, 19xx, 20xx, 21xx, 22xx 3Engineering & Environmental Sciences (EE) –FoRs: 05xx, 09xx, 1202, Social, Behavioural & Economic Sciences (SBE) –FoRs: 13xx, 14xx, 15xx, 16xx, 17xx 5Mathematical, Information & Computing Sciences (MIC) –FoRs: 01xx, 08xx, 1005, 1006, 1007, Biological & Biotechnological Sciences (BB) –FoRs: 06xx, 07xx, 1001, 1002, 1003, Biomedical & Clinical Health Sciences (BCH) –FoRs: 1101, 1102, 1103, 1105, 1107, 1108, 1109, 1112, 1113, 1114, 1115, Public & Allied Health Sciences (PAH) –FoRs: 1104, 1106, 1110, 1111, 1117, 1199

5 22 Divisions, 157 Disciplines 22 Divisions 157 Disciplines –0101 Pure Mathematics –0102 Applied Mathematics –0103 Numerical & Computational Mathematics –0104 Statistics –0105 Mathematical Physics –0199 Other Mathematical Sciences –… –01 Mathematical Sciences –02 Physical Sciences –03 Chemical Sciences –04 Earth Sciences –05 Environmental Sciences –06 Biological Sciences –07 Agricultural & Veterinary Sciences –08 Information & Computing Sciences –09 Engineering –10 Technology –11 Medical & Health Sciences –12 Built Environment & Design –13 Education –14 Economics –15 Commerce, Management, Tourism & Services –16 Studies in Human Society –17 Psychology & Cognitive Sciences –18 Law & Legal Studies –19 Studies in Creative Arts & Writing –20 Language, Communication & Culture –21 History & Archaeology –22 Philosophy & Religious Studies

6 Indicators Data collection and evaluation is based on four broad categories: Research quality, Research volume and activity, Research application, Research recognition These broad categories translate to specific Indicators for which we have to submit data: –Eligible researchers –Research outputs –Research income –Applied measures –Esteem measures

7 Eligible Researchers At census date (expected to be 31 March 2009), researchers must either: –Be employed at UWA; OR –Be a visiting, exchange, or seconded staff member employed elsewhere but working at UWA, AND show affiliation to UWA in their publication byline; OR –Be working for UWA in a non-salaried position (eg emeritus, honorary, adjunct, clinical, conjoint), AND show affiliation to UWA in their publication byline. Data to be submitted for each eligible researcher includes up to three 4-digit FoR codes describing person’s research focus Researcher and output data obligatory for all eligible T&R and RO staff Honorary/Adjunct staff may be eligible, full criteria to be announced Students are not eligible unless they meet the criteria (ie are also staff)

8 Research Outputs Outputs must meet all criteria for eligibility (be research, published during the reference period, Include at least one eligible researcher as an author) Each output must be assigned up to three 4-digit FoRs (one minimum) Eligible outputs: –Books (A1) and revisions/new editions (A4) meeting criteria –Book chapters (B1) –Journal articles (C1) – only those in ARC ranked journal lists –Conference publications, full paper refereed (E1) – may only be those in ARC ranked conference list –Creative works (4 non-HERDC categories – UWA H and J1-J8): Original Creative Works Live Performance of Creative Works Recorded/Rendered Creative Works Curated or Produced Substantial Public Exhibitions and Events –Other non-print and non-HERDC categories to be advised in final Guidelines

9 Citation and Volume Analysis of outputs All clusters will use volume analysis –Total number of research outputs against the FTE of eligible researchers by academic level and overall headcount (includes Honorary/Adjunct staff) Most clusters will use citation analysis –Relative citation impact assessed against world and national benchmarks –Distribution of publications compared with world centile thresholds and the rest of the field –Distribution of papers against relative citation rate bands All clusters will use ranked outlets –Ranked journals and/or ranked conferences

10 Peer Review of outputs Some disciplines will use a peer review process to evaluate research outputs instead of citation analysis –Those in HCA, SBE, and possibly some disciplines in EE and MIC Institutions identify a sample of outputs by 4-digit FoR to be peer reviewed Peer review outputs sourced by the Library and held in the UWA digital repository All creative works selected for peer review will be required to have a Research Statement –Addresses specific criteria, including an identification of the research component of the creative output

11 Income Research income to be reported in a very similar way to HERDC (institutional attribution) –Australian Competitive Research Grants (Cat 1) –Other Public Sector Research Income (Cat 2) –Industry and Other Research Income (Cat 3) Australian International peer reviewed International other –Cooperative Research Centre Research Income (Cat 4) Tagged with 4-digit FoR codes

12 Applied Indicators Applied measures: –Research commercialisation income –Patents –Registered Designs –Plant breeders’ rights –Possibly others Must have an identifiable link to the research that generated them Tagged with 4-digit FoR codes Some attributed to the institution (eg Research commercialisation income, patents), others attributed to eligible researchers (eg Plant breeders’ rights)

13 Esteem Esteem measures are being developed by ARC for ERA 2010 Likely to involve selection of esteem measures from predefined list, for example: –Editorial role of A*/A journal –Editor of prestigious work of reference –Contribution to a ranked conference –Membership of a learned academy –Recipient of a nationally-competitive research fellowship –Membership of a statutory committee (PAH only) –Recipient of an Australia Council Grant or Fellowship Esteem measures mostly get the relevant staff member’s FoRs Esteem measures were not assessed in the Trial

14 Background Statements Brief documents about research at UWA, providing context to the research data Required for each 2-digit FoR code Limited to 10,000 characters (approx 3-5 pages) The statements address the following sections: –Overview – brief outline –Capacity – significant staff or resource changes –Environment – eg support for ECR and HDR –Collaboration – interdisciplinary, inter-institutional, international –Type – pure basic research, strategic basic research, applied research, experimental development –Other – anything else that should be taken into consideration Only include information about research during the reference period, and not future directions

15 Evaluation Research Evaluation Committees (RECs) will be formed for each cluster. –Comprised of internationally-recognised members with expertise in research evaluation RECs will view a “dashboard” of indicator data reports by 4-digit FoR, including bibliometric data, (inter)national benchmarks, and research indicator data by FTE, year, level, etc Some disciplines will be subject to a peer review stage in lieu of citation analysis –Institutions nominate a sample of outputs, to be made available for peer reviewers Institutions will receive a rating from 1 (below average) to 5 (exceptional)

16 The ERA Trial: Two Clusters The ERA Trial assessed Cluster 1 and Cluster 2: Cluster 1 – Physical, Chemical and Earth Sciences (PCE) –02 Physical Sciences –03 Chemical Sciences –04 Earth Sciences Cluster 2 – Humanities and Creative Arts (HCA) –12 Built Environment and Design –18 Law and Legal Studies –19 Studies in Creative Arts and Writing –20 Language, Communication, and Culture –21 History and Archaeology –22 Philosophy and Religious Studies

17 ERA Trial Reference Period The staff eligibility census date was 31 March 2008 The reference period for outputs was 1 January 2002 – 31 December 2007 The reference period for income and applied indicators was 1 January 2005 – 31 December 2007

18 Data Sources and Socrates All data had come from difference sources, so it was necessary to bring the data together: –Outputs from UWA Publications Database, Research Services –Eligible researchers from HR –Staff FoRs from Planning Office –Income data from Research Finance Office –Applied data from Office of Innovation and Industry The ERA data was loaded into Socrates for checking and correcting Socrates underwent considerable development and testing to enable ERA data management and comparison Academic Stewards were asked to check the data and advise changes The final ERA data was loaded from Socrates directly into SEER (the online submission system)

19 ERA Trial Timeframe The final Submission Guidelines were released by the ARC in March 2009 The Evaluation Guidelines, outlining how the data would be assessed, were released by the ARC in April 2009 Submission period for PCE was 1 June – 3 July 2009 Submission period for HCA was 3 August – 21 August 2009 The results of the PCE and HCA evaluations are due to be released in November and December respectively

20 ERA10: the “Big Bang” The ARC has announced the submission period will commence in June 2010 –No information about closing date The final Guidelines and Technical Specification documents are due to be released in December This means a 6 month period to complete the submission: –Collect new data (eg ERA eligible staff outputs, new output categories, all esteem measures) –Check and correct data already held –Develop and test system functionality (developing and testing Socrates for esteem data collection, outputs, and other new indicators) –Select and source peer review items –Write research statements for creative works –Write background statements for all 2-digit FoRs

21 ERA10 ERA reference periods are likely to be moved forward one year: –Staff census date: 31 March 2009 –Outputs: 1 Jan 2003 – 31 Dec 2008 –Income, applied: 1 Jan 2006 – 31 Dec 2008 This has not been confirmed by the ARC –Greatly limits our ability to efficiently collect data PCE and HCA largely complete –2008 outputs, new indicators, reassess peer review sample There will be Academic Stewards, depending on the implications of the final Guidelines –Key staff with knowledge of the discipline, able to check data, identify gaps, draft background statements, and give advice for peer review selection where relevant The RAU has 3 FTE staff for this project –Annual HERDC and other tasks ongoing during the period as well

22 Further information ARC’s ERA website UWA’s ERA website Contact Lorna Robertson or Lizzy Przywolnik in the Research Assessment Unit, Research


Download ppt "Excellence in Research for Australia ERA at UWA Presentation to the UMG, 19 November 2009 Lorna Robertson, Manager Research Assessment Unit, Research Services."

Similar presentations


Ads by Google