As part of our ongoing quality assurance processes for peer review, we recently conducted an analysis of our 2 largest grant schemes to investigate whether assessors have bias for applications originating from their own state or territory.
Publication Data
Table of contents
The outcome of this investigation, as detailed below, is that NHMRC did not observe evidence that suggests the presence of systematic bias by assessors favouring applicants from their own jurisdiction.
Analysis dataset
We examined the scoring behaviour of assessors who participated in the 2021 to 2024 Investigator Grant and Ideas Grant funding rounds, reviewing overall scores as well as individual criteria scoring patterns.
Applicants (specifically, Chief Investigator As (CIA) in these funding schemes) were classified into jurisdictions based on the location of the head office of the Administering Institution (AI) of their application. This was necessary as CIAs may work across a variety of sites, locations or organisations and Ideas Grant applications often have other Chief Investigators located in different jurisdictions.
Local assessors were compared to interstate assessors. This could only be undertaken in instances where applications were reviewed by both local and interstate assessors. Although NHMRC’s goals for peer review membership for each round are to have assessors from across states and territories, the primary factors for allocating assessors to each application are to maximise expertise suitability and minimise conflicts of interest.
Outcomes and results
1. Quantifying the proportion of local and interstate assessors by scheme
First, we set out to establish the proportion of applications reviewed by interstate and local assessors.
As illustrated in Figure 1 and Figure 2, most Ideas Grant applications (greater than 62%) and Investigator Grant applications (greater than 64%) were assessed by a combination of local and interstate assessors, with the proportions consistent across years. Slightly more than a third of applications (34% to 37% of Ideas Grants and 33% to 35% of Investigator Grants) were exclusively assessed by interstate assessors. Very few applicants in either scheme were reviewed entirely by local assessors.
Further dissecting the location of assessors by the location of an applicant’s Administering Institution within each scheme (see Figure 3 and Figure 4) shows that across the pooled 2021 to 2024 data, applications from NSW and VIC based Administering Institutions were predominately reviewed by a mix of local and interstate assessors. Just over half of all Ideas Grant and Investigator Grant applications from QLD were reviewed exclusively by interstate assessors. More than 78% of applications from SA and more than 79% of WA applications were exclusively reviewed by interstate assessors. Most applications from ACT and TAS (both greater than 97%) and all applications from NT were reviewed exclusively by interstate assessors across both schemes.
The raw data used to calculate these figures is provided in Table 1 and Table 2 for completeness.
2. Examining score differences between local and interstate assessors
We also examined whether local assessors provided more favourable scores compared to their interstate peers on the same applications. To undertake this analysis only applications where there were both local and interstate assessors (see the dark shaded populations in Figure 3 and Figure 4) were examined. Accordingly, no applications from NT and very few from ACT or TAS could be evaluated.
This comparison is visualised in Figure 5 to Figure 8 which shows the difference in scoring by local compared to interstate assessors for Ideas Grant and Investigator Grant applications from NSW, QLD, SA, VIC and WA. In these density plots and boxplots, each data point refers to the difference between the average local assessor score and the average interstate assessor score for an application. Positive values indicate that local assessors on average scored higher than interstate assessors for that application, and vice versa.
The score difference distributions for Ideas Grants (Figure 5) were centred around zero for each jurisdiction and appeared fairly symmetric (bell-shaped), indicating no systematic scoring bias by either local or interstate assessors and suggesting that the observed differences largely reflect normal variation in assessor scoring.
A closer examination of the score differences between local and interstate assessors by criterion and Administering Institution jurisdiction for Ideas Grant applications across the 2021–2024 rounds is visualised in Figure 6. For each jurisdiction and criterion shown, the median (black line) and mean (white dot) score differences are close to zero and the interquartile ranges (coloured boxes) are mostly symmetric, indicating no systematic tendency for either local or interstate assessors to score applications more favourably.
The below density plots (Figure 7) show that the score distribution of Investigator Grant applications from NSW, QLD and VIC were approximately symmetric and concentrated around zero, again indicting that differences are likely random variation rather than systematic bias.
The distribution of scores for Investigator Grant applications from SA appeared slightly bimodal. On average, local SA assessors scored slightly higher than interstate assessors resulting in a mean difference of 0.125. However, this difference was not statistically significant when a t-test was conducted using a 5% significance level.
Inversely, the mean difference between local and interstate assessor scores for applications from WA was minus (−)0.191 with statistical testing showing a significant difference. This indicates that local WA assessors tended to score Investigator Grant applications from their own state somewhat lower on average.
These findings are descriptive and should be interpreted in light of several analytical limitations, including small sample sizes, the need to aggregate across Investigator Grant levels, and variation in the balance of local to interstate assessors across jurisdictions.
The profile of Investigator Grants score differences between local and interstate assessors by criterion and Administering Institution jurisdiction across the 2021–2024 rounds is illustrated in Figure 8. Applications from NSW, QLD and VIC showed a median (black line) and mean (white dot) score differences that approached zero and most interquartile ranges (coloured boxes) were generally symmetric, indicating no systematic tendency for either local or interstate assessors to score more favourably.
The boxplots for applications from SA show greater spread across several criteria, particularly in Knowledge Gain, with medians slightly above zero, suggesting that local assessors more often scored applications marginally higher than interstate assessors. However, this pattern is accompanied by increased variability consistent with small sample sizes.
By contrast, WA applications show a consistent shift below zero across most criteria, with negative medians and distributions skewed towards lower scores from local assessors. The interquartile range of WA applications was slightly stretched downwards on the Knowledge Gain criterion which is consistent with the observed −0.191 mean score difference between local and interstate assessor scores for these applications. This indicates that local WA assessors tended to score applications of their own state slightly lower than their interstate peers.
Conclusion
This analysis revealed scoring distributions and assessor behaviour which were considered within the bounds of normal variation. There was no evidence to conclude that assessors showed systematic bias or favouritism for applicants from their own jurisdiction.
NHMRC remains committed to ensuring fair and equitable peer review of all applications. We will continue monitor and evaluate our peer review processes ensuring our principles of peer review are upheld.