How to assess publications for peer review video

This video was recorded in September 2024.

Video transcript

Speaker 

• Dr Julie Glover – Executive Director of the Research Foundations Branch, NHMRC 

Welcome – (0:06)
Hello everyone and welcome to this short video which will provide peer reviewers with some insights on assessing applicants' publication track record. I'm Julie Glover, the Executive Director of the Research Foundations branch, which is the branch responsible for delivering the Investigator Grant scheme.

I'd like to begin by acknowledging the Ngunnawal people, the traditional owners of the lands on which I'm joining you from here in Canberra, and to acknowledge all of the traditional owners on the lands that you are joining from. I also pay my respects to Elders past, present and emerging and acknowledge any Aboriginal and Torres Strait Islander people who maybe viewing this video.

Thank you for your contributions to NHMRC’s peer review process. This video is one in a series which aims to support peer reviewers to undertake their important task of peer review.

Investigator Grants – assessment criteria – (1:04)
Investigator Grants is largely a track record-based scheme, meaning that 70% of the overall score is determined by an applicant's past research performance.

The publications criterion contributes half of the track record assessment, contributing 35% to the overall score. As such, NHMRC expects peer reviewers to expend a proportionate effort in reviewing this criterion.

Publications assessment – (1:34)
Applicants have been asked to nominate up to 10 of their best publications within their 10-year assessment time frame.

Noting that the 10-year window may be extended for valid career disruptions. NHMRC is keen to focus on quality rather than quantity.

Applicants are asked to nominate up to 10 publications for the past 10 years, and we're asking you as reviewers to focus on an applicant's publication quality and contribution to science, rather than the quantity of the publications generated throughout their whole career.

Peer reviewers are asked to assess the nominated publications, including accompanying explanations, to form a judgement on their overall quality and contribution to science, including the applicant's contribution to each of the publications.

If you identify a publication that you believe is outside of the 10-year window, please contact your secretariat for further advice.

Please refer to the publication score descriptors to appropriately score each application and take into account any adjustments to the scoring for relative to opportunity considerations.

NHMRC’s expectations – (2:52)

When considering a publication's quality, we ask you to consider characteristics such as the creativity and innovation of ideas, the rigour of experimental design, appropriate use of statistical methods, reproducibility of the results, analytical strength of interpretations, and significance of outcomes. All of which serve as surrogates for measuring research quality of a publication, irrespective of the field of research and rather than the number of publications or the standing of the journal in which those articles are published.

We don't expect peer reviewers to replicate the review undertaken when manuscripts are submitted for publication. However, we do expect that you will use your expertise, your familiarity with the field of research, and the applicant's explanation for each publication to help you decide whether you need to read a nominated publication, either in full or in part, to verify the applicant’s claims.

Reviewers should ignore any additional track record information that the applicant might provide in the publication explanation field, where they're not satisfied that it's directly linked to the nominated publication, or where it is outside of the assessment of publications criteria. For example, career publication metrics.

Further information can be found in the Investigator Grants Peer Review Guidelines.

Use of impact factors and other metrics – (4:27)
NHMRC is a signatory to the San Francisco Declaration on Research Assessment, or DORA, and adheres to the recommendations outlined in DORA for its peer review processes, which makes recommendations for improving the evaluation of research assessment.

It is not appropriate to use publication metrics such as journal impact factors. Journal based metrics, if included by an applicant, should not be taken into consideration in the assessment of publications.

NHMRC also encourages the use of research quality guidelines such as the Hong Kong Principles for assessing researchers, which recommends focusing on responsible research practices, transparent reporting, open science, diversity of research, and recognition of all contributions to research as hallmarks of publication quality.

Score descriptors – (5:25)
When assessing an applicant's publications, please always refer to the score descriptors at Appendix B of the Peer Review Guidelines. According to feedback from Investigator Grant peer reviewers, applicants who scored well for the publications criteria showed an upwards career trajectory, clearly described and substantiated their role in the described work or nominated publications, and justified the quality, significance and impact of their nominated publications.

You may also find it helpful to hear from experienced peer reviewers about what they consider in assessing publications. We have a peer review mentor video which is available on the peer review resources tab of the Investigator Grants website. You can also make use of the Peer Review Mentor drop-in sessions throughout the assessment period.

Thank you – (6:23)
Thank you again for your contributions to NHMRC peer review. Please remember to make contact with your secretariat using the GRP inboxes should you have any questions or concerns.

End of transcript.