Tips and tricks for peer review video

This video was recorded in September 2024.

Video transcript

Speaker

• Dr Julie Glover, Executive Director Research Foundations, NHMRC

Welcome – (0:01)

Hello everyone, and welcome to the tips and tricks for the Investigator Grants peer review video. I'm Julie Glover, the Executive Director of the Research Foundations Branch that is responsible for delivering NHMRC's larger schemes, including the Investigator Grant scheme.

I'd like to begin by acknowledging the traditional owners of the lands on which I'm joining you here.

I'm in Canberra, which is Ngunnawal lands, and to acknowledge all of the other traditional owners on the lands throughout the country that you're viewing this video from. I pay my respects to Elder’s past, present and emerging, and acknowledge any Aboriginal and Torres Strait Islander people who may be viewing this video.

Thank you for your important contributions to this year's peer review processes. This video will provide some tips and tricks to help you undertake peer review.

Planning your reviews – (00:55)

One of the most important tips is to start your assessments early. We're asking you to assess between 10 and 18 applications in the 4-week assessment period. And peer reviewers, we know they spend a fair amount of time on each application, and this is a considerable commitment.

We suggest planning your time carefully to ensure that you can give each of your assigned applications a robust and fair assessment. Some of our peer reviewers recommend grouping your assigned applications by Level.

For example, you might look at all the L1 applications together before reviewing the L2 and L3 applications.

Other reviewers also recommend reviewing each criterion of all applications together before moving on to the next criterion. This approach may help you to benchmark and is mentioned in our peer review mentor video.

Another good piece of advice is to try and do your assessment in blocks, so maybe assign 2 or 3 blocks of time, and make sure you check back on your previous scores throughout the process to ensure you're being consistent and fair.

And as always, please keep in touch with your secretariat if you're having any issues, particularly around completing your assessments on time.

Working offline – (2:13)

We prefer that you enter your scores and applicant feedback directly into Sapphire because the system saves continually, and it reduces the chance of transcription errors. However, we do know that many peer reviewers like to work offline before entering their assessments into Sapphire.

We ask that if you do this, you please let us know as soon as possible. And if you're working offline, we will give you access to a template that can assist you with your assessments, where you can enter in your scores, your applicant feedback and general notes about the application into one document.

However, you still will have to enter those scores and applicant feedback manually into Sapphire.

Undertaking assessments – (2:56)

While the Peer Review Guidelines are your definitive source of guidance, here are a few key things to remember when undertaking your assessments. Peer reviewers must assess track record relative to opportunity, and where applicable, career disruption considerations.

When you're assessing applications, reviewers are also asked to consider, where relevant, the years spent since completing a PhD in their assessment of applicant track record relative to opportunity. And while not all applicants will have career disruptions, every applicant is required to provide a broad overview of the circumstances that have impacted their engagement in research within their 10-year assessment time frame.

Reviewers are to ignore any additional track record information that are provided in the applicant’s career context field. For example, career publication counts, total funding received, scientific summaries and of projects and outputs.

Applicants must justify in their applications their selected Category and Level of Investigator Grant. It is important that you consider this when reviewing your applications. Consider the applicant's track record relative to opportunity by referring to the Statements of Expectations at Appendix F of the Peer Review Guidelines.

We will provide a separate video about how to assess applications where you consider that the applicant has applied at an inappropriate level.

Where relevant, applications that have a focus on improving Aboriginal and Torres Strait Islander health outcomes, please consider in your assessment the report of the assessment against the Indigenous Research Excellence Criteria [IREC], which has been provided by a reviewer with expertise in Aboriginal and Torres Strait Islander health.

To do this, in your ‘My Assessments’ page, check to see if an application has been flagged in the IREC determination column and then review the assessment onscreen within Sapphire.

When you're ready to review the assessment criteria, please have the score descriptor tables on hand, which are available at Appendix C of the Peer Review Guidelines, this is your guidance to how to score the applications.

Assessment criteria considerations – (5:18)

Appendix C of the Peer Review Guidelines contains detailed information on the Investigator Grants assessment criteria and the associated score descriptors.

But here are a few points to remember.

When you're reviewing publications, you need to form a judgement on the overall quality and contribution to science of each publication, including the applicant’s contribution to those publications.

Publication quality refers to characteristics such as the rigour of experimental design, the appropriate use of statistical methods, reproducibility of results, as well as the analytical strength of interpretations and significance of the outcomes of those publications.

It's not about the number of publications or the standing of the journals in which they are published. We do not, however expect peer reviewers to replicate the review undertaken when manuscripts are submitted for publication.

Your expertise, your familiarity with the field and the applicant's explanations for each publication will help you decide whether you need to go more deeply into the publication to verify the applicant’s claims.

Please also check if any publications are outside of the applicant's 10-year assessment time frame, and if so, please let your secretariate know immediately.

When reviewing research impact, the applicants now have one field to respond to each of the 3 elements of research impact.

However, you still need to provide a score for each of the 3 separate elements. There is a separate field for applicants to provide corroborating evidence. A poorly or non-corroborated ‘Research Impact’ should receive a score of 1. For the assessment of reach and significance, it's important to note that the score descriptor tables are divided for Emerging Leadership and Leadership applicants to recognise that early and mid-career researchers may have had less time to accumulate research impact.

NHMRC recognises that a broad range of leadership contributions are necessary to create an environment that enables research excellence and stewardship, and this is reflected in the score descriptor table.

When reviewing leadership, peer reviewers should ignore leadership track record information that falls outside of the allowable 10-year assessment time frame. When reviewing ‘Knowledge Gain’, focus more on the scientific quality and potential for impact of the proposed new research and focus less on whether the existing or ongoing research has funding. Research that is not funded by the Investigator Grant can be included in the research proposal to help provide context for the proposed new research.

However, your assessment is of the proposed new research.

PRM Q&A drop-in sessions (8:13)

We encourage new and experienced peer reviewers to attend one of the 6 peer review mentor sessions which are available throughout the 4-week assessment period.

Peer review mentors are experienced reviewers, and their role is to assist with the advising and mentoring of peer reviewers on our processes. The mentors do not assess applications or provide specific advice on the scientific or other merits of individual applications.

We recommend that you send your questions through to your secretariat before the [Peer review] mentor session as this allows for the mentor to prepare a response. However, you can also just attend the sessions and ask a question without sending your question through beforehand.

Please remember that if you have any integrity of eligibility or eligibility concerns to please raise them with your secretariat [privately], not the peer review mentor.

Thank you (9:12)

Thank you for making the time to review this video and a reminder to please make contact with your secretariat using the GRP inboxes should you have any questions or if you have any concerns regarding your assessments.

Thank you again on behalf of all of us at NHMRC and the applicants to this round for your commitment to independent peer review. We look forward to working with you.

End of transcript.