Application-centric peer review is the matching of the most suitable peer reviewers to each application. In this way, each application can have a unique best-fit set of reviewers. It is used by NHMRC’s Investigator, Ideas and Synergy Grant schemes.
Why application-centric peer review?
Placing grant applications at the heart of the peer review process, the shift to application-centric peer review has contributed to NHMRC improving the independence and matching of expertise for peer review while retaining a balance between the rigour of review and burden on reviewers.
An infographic shows how application-centric peer review has benefited the Investigator Grants Scheme (available in the Download section).
The application-centric peer review process
Application matching:
Research classification data from the peer reviewer’s Sapphire profile, such as fields of research, peer review areas and research keywords, are cross-referenced with the same data submitted by applicants for their research proposal, to generate a bespoke group of applications that best match a peer reviewer’s expertise, for the purpose of seeking conflicts of interest (CoI) and suitability declarations from a larger group of potential peer reviewers.
CoI and suitability declarations:
Potential reviewers self-assess their suitability to review an application and declare any CoIs based on comprehensive guidance. NHMRC examines declarations where peer reviewers seek policy clarification and where acceptable suitability is identified.
Application allocation for peer review:
Declarations inform the allocation of applications to the most suitable and non-conflicted peer reviewers. The allocation process aims to minimise workload while providing peer reviewers with an opportunity to benchmark their own approach to scoring to support consistency of assessments.
Application assessment:
NHMRC draws on the full pool of reviewers to allocate suitable reviewers for each application. Peer reviewers are instructed to undertake independent assessments and give constructive feedback that is provided verbatim to the applicant. Reviewers are supported by NHMRC secretariat staff, extensive documentation, a live briefing with Q&A and access to experienced peer reviewers who are available to provide guidance throughout the peer review period.
Improving the peer review experience
Substantial improvements in matching of peer reviewers to applications have been made over the last few years. However, the application-centric process relies heavily on the quality and completeness of fields of research data entered in an application by the applicant, and peer review information in a peer reviewer’s Sapphire profile. Furthermore, accurate declarations of suitability and conflicts by peer reviewers will ensure application matching is optimised to the expertise of the peer reviewer.
Application-centric peer review has proved effective and efficient for NHMRC’s largest schemes. However, a panel-based approach will continue to be used for the peer review of NHMRC’s smaller and targeted grant opportunities where good matching of groups of applications to panels can be achieved.
We recognise that many peer reviewers appreciated the opportunities that panel meetings in our larger schemes provided for in-person mentoring and professional development. We continue to refine and expand training and guidance opportunities for those involved in application-centric peer review and welcome peer reviewer feedback on how to best support peer reviewers.
NHMRC acknowledges the substantial time and effort researchers commit to applying for and reviewing grants and the support their administering institutions provide. We will continue to explore ways to improve and strengthen NHMRC’s peer review processes, based on advice from peer reviewers, applicants and our advisory committees, taking into account the experiences of other national and international funding agencies. This includes increasing the quality and efficiency of NHMRC’s internal processes using advanced technologies such as machine learning, while retaining the human-centric decision-making element of peer review.
Improving transparency and accountability
Several steps have been introduced to improve transparency and accountability of application-centric peer review.
- Investigator, Ideas and Synergy Grant schemes now require that peer reviewers provide feedback to applicants about the strengths and weaknesses of the application which will be useful for future applications.
- Peer reviewer feedback comments are shared anonymously between reviewers of the same application (for Investigator and Ideas Grants) to increase transparency and accountability of the peer review process and allow peer reviewers to compare their assessments with those of other reviewers. Transcription errors may be identified through this process.
- NHMRC undertakes outlier screening to identify reviews where scores differ substantially from those of other reviewers, or where it is apparent comments do not align with scores. This may involve NHMRC contacting peer reviewers to clarify their assessments, noting that it is common and expected that peer reviewers will vary in their scoring and justifications for their conclusions. Further errors may be identified through this process.
Development of application-centric peer review
In the past, our larger schemes relied on using ‘superpanels’ to group peer reviewers with groups of applications based on related research topics, then splitting them into multiple smaller grant review panels whose members met in person to assess applications.
In 2017–18, we undertook an extensive national consultation to inform the peer review processes for our then-new Investigator, Synergy, Ideas and Clinical Trials and Cohort Studies Grant schemes. Several themes emerged from this consultation including:
- preference for a larger number of independent assessments (minimum 4 or 5 per application)
- concerns that some panels did not have appropriate expertise and may have been overly reliant on an individual reviewer’s assessment
- concerns that applications were commonly reviewed by fewer than the target number of reviewers due to conflicts of interest
- the importance of feedback to applicants.
Following this public consultation, NHMRC has continued to consider ways to streamline and improve the quality of its peer review processes while striking the appropriate balance between rigour and burden. Feedback from the sector and the advice of NHMRC’s Research Committee and other advisory groups are instrumental in improving NHMRC peer review processes.
The COVID-19 pandemic necessitated innovative approaches to peer review, as lockdowns, illness, increased caring responsibilities and changed professional roles (such as front-line clinical and public health activities) limited the availability of peer reviewers to meet for panel discussions.
Analysis of scoring changes and outcomes of panel discussions from the 2019 Ideas Grant round, combined with data from published studies, showed that panel discussion does not significantly alter outcomes compared to independent evaluations. This prompted NHMRC to trial a hybrid application-centric peer review process for the 2020 Ideas Grant round, allowing the Ideas Grant scheme to save months in the process and deliver outcomes before the end of 2020.
Positive reviewer feedback from the trial and NHMRC analysis led to it being implemented for the 2021 rounds of Investigator, Synergy and Ideas Grants. In doing so, application-centric peer review has:
- made participating in peer review more flexible and inclusive by removing the need for peer reviewers to meet, allowing researchers to participate where they have caring/family commitments or other constraints on their ability to travel or spend large amounts of time in panel meetings
- reduced the average number of conflicts of interest and suitability declarations required of potential peer reviewers by pre-matching applications to a peer reviewer’s expertise
- reduced the average number of applications allocated to peer reviewers for full assessment, while ensuring a minimum number is allocated to improve the quality and consistency of assessments
- provided more flexibility in matching peer reviewers to applications to maximise the proportion of applications that receive the target of 5 assessments
- improved matching of applications to peer reviewer expertise, as evidenced by peer review survey results
- enhanced our ability to test and apply advanced technologies such as machine learning to improve the matching of applications to peer reviewers.
