The following data has been obtained from the 2019-2022 Postgraduate Scholarships (PGS) post-implementation review (PIR) surveys of peer reviewers. This document summarises key trends in these findings and details areas for improvement to better shape PGS peer review practices in the future.
Publication Data
Principles of peer review
Peer review is the impartial and independent assessment of research by others working in the same or a related field.
NHMRC’s Principles of peer review are high-level guiding statements that underpin NHMRC’s peer review processes and apply to all NHMRC funding schemes.
These principles include:
- peer review processes are fair and seen to be fair by all
- transparency applies to all stages of peer review
- peer reviewers provide independent advice
- peer review processes should achieve the highest-quality decision making.
Figure 1 shows, for each of the years from 2019 to 2022, the percentage of surveyed PGS peer reviewers who Agreed or Strongly Agreed that PGS peer review was consistent with these principles.
On average across all years, 83% of respondents Agreed or Strongly Agreed that PGS peer review processes were fair, transparent, independent and high quality.
Though, in 2022, some respondents felt that lack of visibility of other peer reviewer scores affected the transparency of the process.
Peer reviewer gender
NHMRC’s Guiding principles for peer reviewer nominations and appointments include balanced representation of gender as a goal. Figure 2 shows numbers of peer reviewers for the 2019 to 2022 PGS rounds by selected gender. On average across all rounds, 49.25% of peer reviewers selected female and 50.75% selected male.*
*Only these two gender selection options were available for these rounds. NHMRC expanded its gender selection options from 2023.
Time spent reviewing applications
A PGS application includes a Report (which contains track record and application specific information), a Supervisor Report and an academic transcript.
Across the period 2019–2022, PGS peer reviewers spent an average of 1-2 hours assessing each application (refer to Figure 3).
Assessment process
In 2022, an average of 85% of Peer reviewers Agreed or Strongly Agreed that they were comfortable with key elements of the PGS assessment process (refer to Figure 4).
Areas identified for improvement
The following areas for improvement were identified in the 2022 PGS round.
Stronger advice on Journal Impact Factors (JIFs)
Peer reviewers noted that some applicants included JIFs in the publication section of their application.
NHMRC is a signatory of the San Francisco Declaration on Research Assessment (DORA) which notes that it is not appropriate to use journal-based metrics such as JIFs as a surrogate measure of the quality of individual research articles when assessing an individual scientist’s contributions. The scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.
For 2023 and future PGS rounds, wording in the Grant Opportunity Guidelines will be updated to strengthen advice against inclusion of JIFs within applications.
Continued Sapphire improvements
Feedback from peer reviewers suggests that further improvements in accessing the Supervisor Reports in Sapphire will strengthen the PGS peer review experience.
Further information
- For further information about this data, please contact the PGS team on scholarships@nhmrc.gov.au
- You can find more information about NHMRC evaluation activities.
- NHMRC funding statistics and data are available.
Note
For the 4 years covered by this report, respondent numbers to the PGS PIR survey as compared with the total number of survey recipients were:
- 22 of 42 (2019)
- 19 of 42 (2020)
- 19 of 35 (2021)
- 16 of 35 (2022).
Figure descriptions
Figure 1
| Principle | 2019 | 2020 | 2021 | 2022 |
|---|---|---|---|---|
| Fair | 82% | 90% | 68% | 81.3% |
| Transparent | 91% | 74% | 74% | 62.5% |
| Independent | 95.5% | 95% | 100% | 100% |
| High quality | 81.8% | 85% | 79% | 68.8% |
Figure 2
| Round | Female | Male |
|---|---|---|
| 2019 | 50 | 50 |
| 2020 | 46 | 54 |
| 2021 | 46 | 54 |
| 2022 | 55 | 45 |
Figure 3
| Round | <1 hour | 1–2 hours | >2 hours |
|---|---|---|---|
| 2019 | 36.4% | 50% | 13.6% |
| 2020 | 36.9% | 52.6% | 10.4% |
| 2021 | 36.9% | 42.1% | 20.9% |
| 2022 | 25% | 62.5% | 12.6% |
Figure 4
| Assessment process element | 2021 | 2022 |
|---|---|---|
| I had sufficient time to adequately complete my conflict of interest declarations | 100.0% | 87.5% |
| Overall, I had sufficient time to review the applications I was assigned | 94.7% | 75.0% |
| The information supplied in applications was sufficient to perform an assessment | 89.5% | 93.8% |
| The Category Descriptors were an effective tool in helping me complete my assessments | 100.0% | 93.8% |
| The workload involved in performing peer review for the PGS scheme was reasonable | 68.4% | 75.0% |
