Top 10 in 10 publications policy evaluation report

Executive summary

The National Health and Medical Research Council (NHMRC) is the Australian Government’s key entity for managing investment in, and the integrity of, health and medical research. NHMRC invests in the highest quality research and researchers, as determined through peer review, across the four pillars of health and medical research: basic science research, clinical medicine and science research, public health research and health services research.

NHMRC allocates funding to researchers through a grant system that relies on peer review providing rigorous assessment of applications for funding to ensure transparency, probity and fairness of process.

In 2021, with advice from its Research Committee, NHMRC undertook a targeted consultation with peak bodies in the Australian health and medical research sector on aligning NHMRC’s assessment of publications with other international agencies, to increase the focus on research quality and reduce peer reviewer burden.

Based on the results of this consultation, starting with the Investigator Grant scheme which opened for applications in January 2022, NHMRC began implementing a new policy relating to the assessment of publications for its track record-based schemes.

The new policy limited applicants to list no more than 10 of their top publications in the past 10 years (while also accounting for career disruptions). The full list of applicant publications from the past 10 years would no longer be provided to peer reviewers for their assessment.  

It was intended that this new ‘top 10 in 10’ policy (henceforth ‘the policy’) would:

• help drive sectoral change to value research quality rather than quantity of publications
• help make assessment of publications equitable for applicants across all career stages and research fields
• reduce peer reviewer burden
• align with the publication assessment practices of many international funding agencies.

This interim report presents Phase 1 of the evaluation work undertaken to review whether the change in policy, and its implementation, has helped achieve any of the above objectives in the first year of its implementation. The initial findings highlighted in this report indicate that the policy has been successful in increasing the emphasis on quality rather than quantity of publications and has not had any unintended consequences in the funding pattern (e.g., career stages and Broad Research Areas (BRAs)) compared to the year before its implementation.  

Further data analysis and the results of the 2022 Investigator Grant Peer Reviewer Survey show that the policy has also been able to reduce burden on peer reviewers.

However, the results of the survey have highlighted the need for further detailed guidance to applicants and peer reviewers on increasing their focus on the quality of the research in the publications, and consequently drawing the emphasis away from the total number of publications, journal impact factors and other quantitative metrics.

Key findings

1. Introduction of the policy did not appear to significantly change the application scoring pattern in 2022 in terms of Investigator level, gender, BRA, career disruption and career stage, compared to the year before its implementation.  
2. Most peer reviewers supported the implementation of the policy and agreed that it helped emphasise the quality, instead of quantity, of publications.
3. The policy has reduced the emphasis on the total number of publications, journal impact factors and other similar quantitative metrics.
4. Most peer reviewers considered that implementation of the policy led to a reduced burden, however 23% did not and some noted that in-depth assessment required careful consideration and time.

Opportunities

1. Improve guidance to applicants on the ‘quality and contribution’ section of the application.
2. Improve guidance to peer reviewers on the assessment of ‘quality and contribution’ of the selected publications.

Recommendations

1. Continue to monitor the implementation of the policy and its impact on the Investigator Grant scheme for the next two years.
2. Improve guidance to both applicants and peer reviewers on drafting and assessing the quality and contribution of research presented in publications.
3. Conduct Phase 2 of this evaluation work once the policy has been implemented for three consecutive years.