The purpose of this report is to provide a summary of feedback received as part of the national consultation to review NHMRC’s Indigenous Research Excellence Criteria (IREC).
Publication Data
Executive Summary
NHMRC’s Indigenous Research Excellence Criteria (IREC) are designed to ensure that research into Aboriginal and Torres Strait Islander health is of the highest scientific merit and is beneficial and acceptable to Aboriginal and Torres Strait Islander peoples and communities.
A national consultation on the IREC was conducted to consider how the criteria are working in practice and whether improvements are needed. A discussion paper to inform national consultation was developed which included consideration of research application approaches used internationally, as well as by other research bodies in Australia. Stakeholders were asked to address four thematic areas on; whether the current IREC are still appropriate, how the related threshold should be applied, how we can ensure a rigorous peer review process using the IREC, and what other national or international ideas and approaches might be considered.
National consultation took place between June and September 2023 and included focus workshops, online submissions and an online webinar.
The main findings are summarised below and discussed in more detail in the body of this report.
The majority of respondents supported the continued use of IREC to assess applications involving Aboriginal and/or Torres Strait Islander health, however they also added that the criteria needed to be refreshed to reflect modern community expectations. The criteria would benefit from community-identified priorities (self-determination), be Indigenous-led, co-designed with communities, involve community-led governance (including data and intellectual property), respect Indigenous knowledges and research methods, value respectful relationships and result in research that is impactful and accountable with reciprocal benefits (for community and researchers), leading to appropriate knowledge translation and two-way capability strengthening.
Feedback was received that the current 20% threshold to qualify applications as addressing Aboriginal and/or Torres Strait Islander health (and thereby require IREC assessment) was confusing and difficult to measure. There was no consensus on an alternative approach, with ideas ranging from application rounds for Indigenous-led research to all NHMRC grants addressing a question relating to how their research might benefit Aboriginal and/or Torres Strait Islander people.
Ideas for strengthening of NHMRC’s peer review system included:
- linking IREC assessment to application scoring
- strengthening the assessment of Aboriginal and Torres Strait Islander health focused applications including through training and appropriate resourcing
- providing training for Indigenous and non-Indigenous assessors.
Other ideas received during the consultation included:
- all NHMRC applications to address how their research supports Aboriginal and Torres Strait Islander health (similar to New Zealand’s Health Research Council Māori Health Advancement approach)
- increasing the percentage of NHMRC’s Medical Research Endowment Fund (MREA) spending for applications that address IREC-related principles
- increasing institutional and NHMRC support for Aboriginal and Torres Strait Islander researcher workforce development (including pipeline support for students and community based researchers).
Introduction
The purpose of this report is to provide a summary of feedback received as part of the national consultation to review NHMRC’s Indigenous Research Excellence Criteria (IREC). The report was developed by the National Health and Medical Research Council (NHMRC) in consultation with NHMRC’s Principal Committee Indigenous Caucus (PCIC). More information on the history and current use of IREC can be found at Appendix A.
Consultation approach
A national consultation was undertaken between June and September 2023 to seek stakeholder views on the current IREC. The consultation was led by PCIC, with detailed development by a PCIC working group.
Consultation activities
The consultation process was supported by a discussion paper incorporating:
- An examination of other national and international principles and practices relevant to Indigenous health research (including from the Canadian Institutes of Health Research, Health Research Council of New Zealand, United States National Institute of Health - Native American Research Centres for Health, Australian Research Council, Central Australian Aboriginal Congress, Australian Institute of Aboriginal and Torres Strait Islander Studies [AIATSIS] and the Lowitja Institute).
- Learnings from an internal review of current IREC processes used by NHMRC staff and anecdotal feedback received from IREC assessors and peer reviewers.
- Questions developed by PCIC in response to identified challenges associated with IREC.
The consultation sought stakeholder views through a range of national public and targeted consultation activities, including:
- Initial workshops held at major conferences in Perth and Cairns (AIATSIS Summit 2023 and Lowitja Institute 3rd International Indigenous Health and Wellbeing Conference 2023). This provided an iterative process to refine the questions for the national and online consultations.
- Face-to-face workshops held in 10 key locations around Australia engaging relevant key stakeholders (Indigenous researcher workforce and related community groups). These were hosted and led/facilitated by PCIC members (both past and present).
- A webinar that provided background and key information, as well as an opportunity to ask questions.
- An online submission portal, to enable stakeholders to submit official responses to questions raised in the discussion paper.
More detail on the consultation coverage and analysis can be found in Appendices B and C.
What we heard
The following summary of outcomes draws on feedback captured at our workshops, webinar and online submissions. The methodology associated with the quantitative (thematic) analysis is discussed in more detail in Appendix C.
The common themes raised during the consultation process visually, represented in the word cloud below, show a diversity of ideas as well as related phrases.
(Phrases: Indigenous health research in Australia, Health and wellbeing, Peer review process, Ways of knowing, Indigenous knowledges, Indigenous communities, Sustainability and transferability, Research teams, Decision making, Aboriginal health, Indigenous leadership, Indigenous led, Peer review, Aboriginal people, Indigenous researchers, Indigenous data, Research team, Capacity building, Health researchers, IREC Criteria, Research effort, Community members, Community engagement, IREC Assessment, Aboriginal community, Research projects, Indigenous Peoples, Public health, Review process, Indigenous research, Indigenous knowledge, Building capability, Track record, Data Sovereignty, Indigenous People, Peoples and communities, Knowledge translation, Research excellence, Community controlled, Indigenous governance, Aboriginal Peoples, Proposed research, Capability building, Indigenous reviewers, Indigenous person.)
A snapshot of the key findings from the IREC Review consultation process is illustrated below.
Figure 2. Snapshot of key IREC Review consultation findings
Criteria are still relevant, but need to be refreshed with current terminology
Time to raise expectations: for example, community leadership/governance vs community engagement
Concepts that could be included: Indigenous knowledges, data sovereignty, translation, impact
Accountability – did people do what they said in their application?
The 20% threshold is confusing, and concern about potential to game it – but it’s not clear what’s better
Resources to train and support IREC assessors and panel members
Seeking greater consistency in how IREC report is applied in scoring
A question for every funding application: how will this project benefit Aboriginal and Torres Strait Islander health?
A more detailed summary of consultation findings, structured under the four questions outlined in the discussion paper, can be found below.
Question 1 - Are all of these 4 criteria still appropriate?
Question 1: Are all of these 4 criteria still appropriate? If not, why not and what should be used instead?
We invited participants to give us feedback on the relevance of the current IREC (Community Engagement, Benefit, Sustainability and Transferability, and Building Capability) and the criteria descriptors.
We heard that the criteria are still needed, however there are important elements that should be incorporated to better reflect Aboriginal and Torres Strait Islander peoples’ ways of knowing, being and doing (also see Figure 4 in Appendix C):
- Community-identified priorities / self-determination
- Indigenous-led research
- Shared co-design
- Shared governance (including data management)
- Respectful relationships
- Indigenous knowledges, pathways and research methods
- Two-way benefits / reciprocity
- Two-way capability strengthening
- Knowledge translation
- Impactful and accountable research
Consultation feedback recommended strengthening the language of the criteria to recognise the importance of respectful two-way ‘reciprocal’ partnerships with Aboriginal and Torres Strait Islander communities (and researchers) in the conception, co-design and outcomes of the research in line with current policy and related principles.
A high level summary of participant views in relation to each of the four IREC are described below.
Community engagement
The proposal demonstrates how the research and potential outcomes are a priority for Aboriginal and Torres Strait Islander communities with relevant community engagement by individuals, communities and/or organisations in conceptualisation, development and approval, data collection and management, analysis, report writing and dissemination of results.
Participants advocated that Community Engagement is important and Aboriginal and Torres Strait Islander communities must be engaged from the conception of the projects as equal partners and have an active role in the governance of research (including data and intellectual property). Research teams are expected to build trusted relationships and demonstrate respect for Indigenous knowledges and research methods. Many participants noted that research priorities should be determined by the community rather than by research teams.
'Requiring high standards for community engagement and relevance of needs as well as proposed research outcomes and benefits for Aboriginal and Torres Strait Islander peoples helps when resources need to be prioritised. We consider community engagement as the most important pillar on which the other principles depend.' (Submission 9)
Benefit
The potential health benefit of the project is demonstrated by addressing an important public health issue for Aboriginal and Torres Strait Islander peoples. This benefit can have a single focus or affect several areas, such as knowledge, finance and policy or quality of life. The benefit may be direct and immediate, or it can be indirect, gradual and considered.
Participants stated that Benefit arising from the funded research should bring a tangible health benefit or ‘impact’ to the involved community beyond ‘public health issues’, rather than ‘potential’ health benefits that are ‘indirect, gradual and considered’. The research team should be accountable for meeting the research aims as set out in the application and as assessed.
'The benefit criteria should be narrowed to direct and immediate benefit to research participants and their communities, and should encompass the concept of community capability building that is currently part of the ‘building capability’ criteria.' (Submission 16)
Sustainability and transferability
The proposal demonstrates how the results of the project have the potential to lead to achievable and effective contributions to health gain for Aboriginal and Torres Strait Islander peoples, beyond the life of the project. This may be through sustainability in the project setting and/or transferability to other settings such as evidence-based practice and/or policy. In considering this issue the proposal should address the relationship between costs and benefits.
There were a range of views in relation to Sustainability and Transferability including, building these concepts in from the start of a research project and funding appropriately, uncoupling the concept of transferability from sustainability, valuing knowledge translation, and acknowledging that not all research conducted at a local level will be transferable (for instance, recognising that Aboriginal and Torres Strait Islander culture is not homogenous).
'The criteria should not relate to sustainability or transferability but "knowledge translation". The focus should not be on the capacity to replicate research, but its transformative possibilities in bringing about change for those defining both the need for the research and the knowledge upon which that research is formulated.' (Submission 7)
Building capability
The proposal demonstrates how Aboriginal and Torres Strait Islander peoples, communities and researchers will develop relevant capabilities through partnerships and participation in the project.
With regard to Building Capability, participants noted that two-way capability is required to build successful mutually and culturally respectful relationships and that the focus should be on ‘strengthening’ capability that recognises existing capability (that is, moving away from deficit-based language). Building capability of non-Indigenous researchers and expert assessors of these criteria was also seen as important. Two-way capability building was also noted as both an institutional and research team responsibility.
The importance of Aboriginal and Torres Strait Islander leadership of, and within, the research team (including mentoring junior researchers and research workers on the ground) was advocated in multiple criteria.
'We can no longer accept Indigenous knowledge and conceptual framing as being a "nice to have" or a peripheral aspect of research grant applications. Indigenous knowledge and conceptual frameworks must be the focus and therefore interwoven in all aspects of research actions (framing, design, Indigenous methodologies, analysis, and so on). And yes, that requires us to privilege Aboriginal and Torres Strait Islander peoples as those best placed to lead Indigenous health research.' (Submission 10)
Some of the ideas raised at workshops and in submissions cut across multiple IREC criteria, especially self-determination, leadership, respect and value of Indigenous knowledge and research pathways, research prioritisation, knowledge translation and impact, reciprocity, data sovereignty, capability.
Question 2 - Is the 20% threshold still appropriate and relevant?
Question 2: Is the 20% threshold still appropriate and relevant?
- Is a ‘percentage’ qualifier an appropriate/relevant measure? What would be better?
- Is the focus on research effort and/or capacity building appropriate?
- How should we measure ‘capacity/capability building’? Whose capacity/capability should be built?
When asked about the appropriateness of the 20% threshold the key feedback received included:
- Support for changing the current threshold (including metric and non-metric criteria).
- A lack of understanding about the purpose of a threshold.
- Difficulty in quantitatively measuring a threshold.
- A full spectrum of ideas about the threshold level, from 0% (apply to all applications, per the New Zealand Māori Health Advancement approach) through to 100% (applications should be Indigenous-led and Indigenous-health focused).
- Ideas relating to a non-metric criteria including Indigenous leadership/team levels, geographic, population and checklist approaches.
- Mixed feelings about whether to measure research effort, capacity building (capability strengthening), or both.
- That NHMRC critically evaluate how various thresholds have impacted the outcomes of previously funded research, and what the magnitude would need to be, in order to determine the appropriate threshold/s going forward.
Responses about the level and appropriateness of a qualifying threshold varied. Suggestions for a threshold level ranged from 0% to 100%. Some suggestions about a threshold were based on matching relative population parity, whether there would be any Aboriginal and/or Torres Strait Islander participants or impact on Aboriginal and Torres Strait Islander peoples, or whether the research was Indigenous led.
There was a lot of support for the New Zealand Māori Health Advancement approach where all NHMRC applications need to address criteria but some commentary about needing to grow current capacity in the system to be able to assess all applications against IREC-like criteria.
An interim suggestion was to at least require all NHMRC applications to justify why (or why not) their research addresses Aboriginal and/or Torres Strait Islander health and wellbeing / Closing the Gap priority reforms.
This diversity of responses is reflected in the thematic analysis of workshop comments (see in Figure 5 in Appendix C).
Responses to whether a focus should be on research and/or capacity building favoured an approach that prioritised development of Indigenous leadership.
Responses to whose capability should be built prioritised capability building of Indigenous researchers, community members and community controlled organisations, including ongoing and stable employment opportunities. There were fewer comments in relation to capability development of non-Indigenous researchers.
'If it does not meet an essential criteria of being community driven, Aboriginal and Torres Strait Islander led and involves Aboriginal and Torres Strait Islander (people) as part of the research team and has a tangible and realistic opportunity to improve Aboriginal and Torres Strait Islander health then it would not qualify for a research grant' (Submission 5)
'…the focus on research effort and/or capacity building should be on the building (or strengthening) of emerging Indigenous researchers and the next generation of the Indigenous research workforce, as well as their well-demonstrated research partners, collaborators, and allies who clearly have a track-record of exercising professional and cultural humility in Indigenous research contexts.' (Submission 11)
Question 3 - How can we ensure a rigorous peer review process using the IREC?
Question 3: How can we ensure a rigorous peer review process using the IREC?
- For example, should consideration of the four IREC criteria be aligned to scoring of application assessment criteria?
Participant views were invited to see how the peer review process could be enhanced to ensure that it is rigorous and appropriate.
Figure 6 (in Appendix C) shows that a variety of ideas were received with strongest support for:
- Linking IREC assessment to application scoring.
- Strengthening the assessment of Aboriginal and Torres Strait Islander health focused applications, including through training and appropriate resourcing.
- Providing training for Indigenous and non-Indigenous assessors.
- Introducing alternative models for providing IREC assessment (for example, verbally).
The majority opinion was that the assessment of how well the applicants address the IREC should be tied to application scoring, including introduction of a minimum standard for funding. This approach ensures that the applicant’s ability to demonstrate success against the IREC, and an independent IREC assessor evaluation of those claims, are counted toward the success (or not) of the application. There was support for change to NHMRC budget rules (including Personnel Support Package or PSP levels) to recognise the true costs associated with this nature of research.
There was also strong support for involving IREC assessors as full scoring members of peer review panels (where capacity allowed), an iterative application approach (where promising applications receive feedback), broadening the pool of IREC assessors to recognise non-academic expertise, strengthening the pipeline of IREC assessors and Indigenous peer assessors through additional support and/or training, ensuring Indigenous assessors are appropriately compensated, strengthening training of non-Indigenous peer reviewers in relation to understanding the principles underpinning IREC assessment, and better accountability in the system to pick up applications that should have completed an IREC review but didn’t and ensuring grant holders deliver on their promises.
'IUIH recommends that attaching a score to the IREC might increase emphasis on these factors (contextualised understanding of Indigenous research) which are more aligned with Indigenous Ways of Knowing, Being and Doing than the existing scored criteria' (Submission 11)
Consider disqualifying an application if it does not meet minimum IREC scores. (Adelaide workshop)
Question 4 - Is there anything else you’d like to tell us?
Question 4: Is there anything else you’d like to tell us? For example, are there other models that you strongly favour?
Participants were invited to provide other IREC Review-related feedback, including what could be learnt from other national and international assessment processes.
Figure 7 (in Appendix C) shows there was strong support for:
- All applications needing to address how their research supports Aboriginal and Torres Strait Islander health (similar to the NZ Māori Health Assessment approach).
- Increasing % of NHMRC’s Medical Research Endowment Fund (MREA) spending for applications that address IREC-related principles.
- Increasing Institutional and NHMRC support for Aboriginal and Torres Strait Islander researcher workforce development (including pipeline support for students and community based researchers).
There was also strong support for NHMRC adoption of an approach where all NHMRC applications needed to answer a question or provide an impact statement about whether their research would provide benefits to the health of Aboriginal and/or Torres Strait Islander people, in line with Closing the Gap priority reforms and NHMRC health priority policy.
Participants shared ideas of how the NHMRC could use MREA funding to better support research prioritisation, related networks, effective knowledge translation and workforce development in addition to supporting research funding related to Aboriginal and Torres Strait Islander health.
'Lowitja Institute strongly recommends that the national research priorities in Aboriginal and Torres Strait Islander health be reviewed to give effect to the IREC…. We recommend that consultations regarding our peoples’ research priorities should engage only Aboriginal and Torres Strait Islander peoples' (Submission 17)
Next steps
The NHMRC would like to thank participants for their time and contribution to workshops, at the webinar and to online submissions.
The NHMRC will consider the findings of the IREC Review consultation process, drawing on the advice of PCIC and other relevant NHMRC Committees to implement changes to IREC and associated peer review processes.
This is the first national review of the IREC, with only a rapid review taking place in recent years. Consideration will also be given to where IREC (or its future equivalent) is reviewed on a regular basis to ensure that it remains fit for purpose into the future.
Appendices
Appendix A – Background to the Indigenous Research Excellence Criteria and review
In 1998, the NHMRC Aboriginal and Torres Strait Islander Research Agenda Working Group (RAWG) established certain requirements and processes (previous known as the ‘Darwin Criteria’) designed to ensure that research into Aboriginal and Torres Strait Islander health is of the highest scientific merit and is beneficial and acceptable to Aboriginal and Torres Strait Islander peoples and communities. This is part of NHMRC’s commitment to improving the health outcomes of Aboriginal and Torres Strait Islander peoples.
The following advice is provided to applicants of NHMRC grant schemes that wish to qualify as Aboriginal and/or Torres Strait Islander health research.2
Qualifying applications must address the NHMRC Indigenous Research Excellence Criteria as follows:
- Community engagement - the proposal demonstrates how the research and potential outcomes are a priority for Aboriginal and Torres Strait Islander communities with relevant community engagement by individuals, communities and/or organisations in conceptualisation, development and approval, data collection and management, analysis, report writing and dissemination of results.
- Benefit - the potential health benefit of the project is demonstrated by addressing an important public health issue for Aboriginal and Torres Strait Islander peoples. This benefit can have a single focus or affect several areas, such as knowledge, finance and policy or quality of life. The benefit may be direct and immediate, or it can be indirect, gradual and considered.
- Sustainability and transferability - the proposal demonstrates how the results of the project have the potential to lead to achievable and effective contributions to health gain for Aboriginal and Torres Strait Islander peoples, beyond the life of the project. This may be through sustainability in the project setting and/or transferability to other settings such as evidence-based practice and/or policy. In considering this issue the proposal should address the relationship between costs and benefits.
- Building capability - the proposal demonstrates how Aboriginal and Torres Strait Islander peoples, communities and researchers will develop relevant capabilities through partnerships and participation in the project.
Panels will consider these in their overall assessment of the application, together with the scheme-specific assessment criteria (refer to the scheme-specific funding rules).3
In 2002 NHMRC endorsed Road Map: A Strategic framework for improving the health of Aboriginal and Torres Strait Islander people through research (now in its third iteration), which set a target of 5% of the Medical Research Endowment Account (MREA) funding being expended on research that will provide better outcomes for Aboriginal and Torres Strait Islander people. This target was reached in 2008 and was 8% MREA in 2022.
In 2013 a 20% threshold requirement for applications involving Aboriginal and/or Torres Strait Islander health was added. In 2015, following the guidance from a review of NHMRC’s first Road Map and the advice of the PCIC, the six original criteria were consolidated into four criteria by including ‘Priority’ within ‘Community Engagement’ and Significance’ within ‘Benefit’ with the goal to increase the clarity of purpose of the Criteria.
A review of the IREC was originally planned for the 2018–2021 triennium. This was to be accompanied by a broader review of the peer review system for Aboriginal and Torres Strait Islander health research applications.
However, while extensive work was undertaken, ultimately PCIC decided to pause both reviews to provide time for New Zealand to fully implement the Māori Health Advancement Guidelines and to observe and review the results. COVID-19 also affected the timeline for the reviews.
Road Map 3 – 2021-2024 Triennium implementation: Action 94
'Review the NHMRC Indigenous Research Excellence Criteria and their use, and consider the other improvements that could be made to peer review of applications about Aboriginal and Torres Strait Islander health, with the advice of the Principal Committee Indigenous Caucus'.
The PCIC endorsed the review of the IREC Review at the 11 May 2022 meeting. PCIC Members advised best practice engagement with communities involves including consultation, ownership, and consideration of sustainability and transferability. The most important factor is responding to genuine community priorities.
Appendix B – Consultation locations and participants
Stakeholder Engagement
Our national consultation targeted the Australian medical and health research community, particularly Aboriginal and Torres Strait Islander researchers, communities, and advocacy groups.
| Workshop participants were drawn from across Australia’s health and medical sector (including but not limited to) | Other groups who were invited |
|---|---|
|
|
Workshop locations
The following table summarises details of the IREC Review national consultation workshops, involving 192 participants.
| Date | Location | No.5 | PCIC lead(s) | Notes |
|---|---|---|---|---|
| 5 June | Perth | 12 | Ms Samantha Faulkner (NHMRC Director of Aboriginal and Torres Strait Islander Health Advice) | This workshop was held at AIATSIS National Summit |
| 6 June | Perth | 10 | Professor Daniel McAullay (former PCIC member) | Hosted by Edith Cowan University |
| 15 June | Cairns | 7 | Professor Yvonne Cadet-James (PCIC Chair and Member-in-common with NHMRC Council) | This was a workshop as part of the 3rd International Indigenous Health & Wellbeing Conference 20236 hosted by the Lowitja Institute |
| 28 July | Canberra | 7 | Professor Yvonne Cadet-James (PCIC Chair and Member-in-common with NHMRC Council) | Hosted at NHMRC office |
| 8 August | Sydney | 6 | Professor Maree Toombs (PCIC member) | Hosted by the University of Sydney |
| 11 August | Brisbane | 17 | Professor Gail Garvey (PCIC member and Member-in-Common with the Health Research Impact Committee) | Hosted by Queensland Aboriginal and Islander Health Council (QAIHC) |
| 16 August | Adelaide | 21 | Associate Professor Alwin Chong (PCIC member and Member-in-Common with the Australian Health Ethics Committee) Associate Professor Odette Pearson (former PCIC member) |
Hosted by South Australian Health and Medical Research Institute (SAHMRI) |
| 17 August | Melbourne | 7 | Professor Catherine Chamberlain (PCIC Member and Member-in-Common with Women in Health Science Committee) | Hosted by the University of Melbourne |
| 22 August | Alice Springs | 7 | Dr Sean Taylor (PCIC Member and Member-in-common with Consumer and Community Advisory Group) | Hosted by Flinders University |
| 23 August | Darwin | 34 | Associate Professor Kalinda Griffiths (PCIC Member) Dr Sean Taylor (PCIC Member and Member-in-common with Consumer and Community Advisory Group) |
This included two workshops hosted by Menzies School of Health Research |
| 25 August | Tasmania | 10 | Professor Catherine Chamberlain (PCIC Member and Member-in-Common with Women in Health Science Committee) | Hosted by University of Tasmania |
| 18 August | Cairns (Zoom) | 16 | Professor Yvonne Cadet-James (PCIC Chair and Member-in-common with NHMRC Council) | Hosted by James Cook University |
| 30 August | Cairns (Zoom) | 38 | Professor Yvonne Cadet-James (PCIC Chair and Member-in-common with NHMRC Council) | Hosted by Central Queensland University |
| 192 |
Who we heard from
Workshops were hosted by PCIC members in all States and Territories, as shown in Figure 3 below. An interactive webinar allowed other interested stakeholders to participate. Online submissions to the consultation were received from 17 individuals or organisations.
(Canberra had 7 participants, Sydney 6, Brisbane 17, Cairns 7 participants in a face-to-face meeting and 54 online, Darwin had 34 participants, Perth 22, Adelaide 21 and Tasmania 10. Also included is a diagram showing that a total of 192 people attended the workshops across Australia, of which 17 people provided feedback via the online submissions portal and 33 people joined the webinar.)
Appendix C – Consultation report analysis
Consultation participation was promoted through widespread publication of the workshops, online submission and a webinar (for example, via the NHMRC Tracker, targeted emails, social media call outs, having PCIC members and other Aboriginal and Torres Strait Islander leaders promote the review and workshops to their networks).
Feedback was captured in a range of formats, including:
- notes from workshops, mostly provided by stakeholders
- written submissions via NHMRC’s Consultation Portal and/or via email
- webinar questions and feedback received.
This information was used to develop the quantitative and qualitative analyses contained in this report.
Qualitative analysis
All notes from workshops, the webinar and online consultations were analysed and summarised in the 'What we heard' section.
Quantitative analysis
A word-cloud was generated from the most frequently occurring phrases within the written submissions and workshop notes and is shown in Figure 1. The most frequently occurring phrases - Torres Strait, Torres Strait Islander, Aboriginal and Torres Strait Islander, Health research, and Indigenous health research – were omitted from the word-cloud as they occurred so frequently that their presence obscured other terms.
Notes from workshops were manually analysed for the frequency of commonly raised thematic issues (i.e., thematic analysis) in response to discussion paper questions 1-47. Common themes were not quantified from online submissions due to the small number of submissions received.
Each note taken was regarded as an individual comment and each common theme quantified from these comments. Not all comments could be assigned to a common theme: some contained information that was not clearly relevant to the discussion paper questions or else commented on a theme that was not mentioned more than once (a unique or ‘orphan’ theme).
For the purposes of this analysis, responses from participants at consultations in Perth and Cairns in June 2023 were grouped under the four overarching questions described in the IREC Review Discussion Paper.
Quantification was complicated by a number of factors including differences in note taking styles and the number of different and diverse themes discussed in response to questions. In particular, questions 3 and 4 led to a large number of responses containing orphan themes, and relatively few mentions of common themes (see Table 3 below).
| Question | Number of recorded comments8 | Number of common themes defined | Number of mentions of defined common themes9 | Comments assigned to common theme |
|---|---|---|---|---|
| 1 | 316 | 21 | 187 | 47.5% |
| 2 | 160 | 27 | 110 | 60% |
| 3 | 99 | 18 | 62 | 55.6% |
| 4 | 97 | 11 | 36 | 36.1% |
These results were collated against each of the four consultation questions and are shown in Figures 4 to 7 below.
Figure 4. Number of mentions of common themes at workshops in response to 'Q1: Are all of these four criteria still appropriate? If not, why not and what should be used instead'? (Figure description available at Table 5)
Figure 5. Number of mentions of common themes at workshops in response to 'Q2: Is the 20% threshold still appropriate and relevant'? (Figure description available at Table 6)
Figure 6. Number of mentions of common themes at workshops in response to 'Q3: How can we ensure a rigorous peer review process using the IREC'? (Figure description available at Table 7)
Figure 7. Number of mentions of common themes at workshops in response to Q4: Is there anything else you’d like to tell us? For example, are there other models that you strongly favour? (Figure description available at Table 8)
Appendix D – List of online submissions
Seventeen (17) online submissions were received from interested stakeholders.
Where permission to publish was received, individual research backgrounds or organisation names are included. Where permission to publish was not received, the submissions are listed as “Anonymous”.
| Submission number | Organisation |
|---|---|
| 1 | Individual: Researcher – Health/clinical/biomedical |
| 2 | Anonymous |
| 3 | Anonymous |
| 4 | Monash University |
| 5 | Individual: Researcher – Health/clinical/biomedical |
| 6 | Anonymous |
| 7 | Carumba Institute, Queensland University of Technology |
| 8 | Anonymous |
| 9 | Individual: Health care professional |
| 10 | Darak, The Australian Stroke Alliance |
| 11 | Institute for Urban Indigenous Health |
| 12 | Victorian Aboriginal Child Care Agency |
| 13 | Menzies School of Health Research |
| 14 | National Aboriginal and Torres Strait Islander Higher Education Consortium |
| 15 | Central Australian Aboriginal Congress |
| 16 | Anonymous |
| 17 | Lowitja Institute |
Figure descriptions
Figure 4
| AIATSIS Summit workshop - 5 June | Alice Springs Workshop - 22 August 2023 | Brisbane Workshop - 11 August 2023 | Canberra Workshop - 28 July 2023 | CQU - 30 August 2023 | Darwin 1 Workshop - 23 August 2023 | Darwin 2 Workshop - 23 August 2023 | ECU Workshop - 6 June | JCU - 18 August 2023 | LOWITJA conference (CAIRNS) - 15 June | Melbourne Workshop - 17 August 2023 | NHMRC IREC Session with STRIDE network | Sydney Workshop - 8 August 2023 | Tasmania Workshop - 28 August 2023 | Grand Total | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Culturally safe research | 1 | 1 | 2 | ||||||||||||
| Open access concerns | 1 | 1 | 2 | ||||||||||||
| Holistic health focus | 1 | 1 | 2 | ||||||||||||
| Community involvement in research team | 2 | 2 | |||||||||||||
| Training | 1 | 1 | 1 | 3 | |||||||||||
| Sustainability and transferability | 2 | 1 | 1 | 4 | |||||||||||
| Criteria need to be weighted/marked | 1 | 2 | 1 | 1 | 1 | 6 | |||||||||
| Strengthening capability | 1 | 2 | 3 | 6 | |||||||||||
| Community determined priorities/self determination | 1 | 2 | 1 | 1 | 1 | 1 | 7 | ||||||||
| Research impact | 2 | 1 | 2 | 1 | 1 | 7 | |||||||||
| More examples and guidance needed for review | 1 | 1 | 5 | 7 | |||||||||||
| Accountability | 1 | 2 | 2 | 2 | 7 | ||||||||||
| Two way benefits/reciprocity | 1 | 1 | 3 | 1 | 1 | 1 | 8 | ||||||||
| Capacity building | 1 | 1 | 2 | 2 | 1 | 2 | 9 | ||||||||
| Adjust IREC language/definitions | 2 | 3 | 2 | 3 | 1 | 1 | 1 | 13 | |||||||
| Valuing indigenous research methods/knowledge | 2 | 4 | 4 | 2 | 1 | 1 | 14 | ||||||||
| Data sovereignty and gov | 1 | 3 | 2 | 1 | 3 | 1 | 1 | 1 | 1 | 14 | |||||
| Community engagement | 1 | 2 | 5 | 2 | 1 | 1 | 1 | 1 | 14 | ||||||
| Translation/ Knowledge translation | 6 | 3 | 1 | 1 | 3 | 1 | 1 | 16 | |||||||
| Codesign/Shared governance | 1 | 2 | 5 | 5 | 1 | 1 | 3 | 1 | 2 | 21 | |||||
| Indigenous led research, respectful partnerships | 1 | 1 | 4 | 7 | 1 | 1 | 3 | 1 | 3 | 1 | 23 |
Figure 5
| AIATSIS Summit workshop - 5 June | Alice Springs Workshop - 22 August 2023 | Brisbane Workshop - 11 August 2023 | Canberra Workshop - 28 July 2023 | CQU - 30 August 2023 | Darwin 1 Workshop - 23 August 2023 | Darwin 2 Workshop - 23 August 2023 | ECU Workshop - 6 June | JCU - 18 August 2023 | LOWITJA conference (CAIRNS) - 15 June | Melbourne Workshop - 17 August 2023 | NHMRC IREC Session with STRIDE network | Sydney Workshop - 8 August 2023 | Tasmania Workshop - 28 August 2023 | Grand Total | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Staged removal/lowering of threshold | 2 | 2 | |||||||||||||
| Threshold should also relate to CI team | 1 | 1 | 2 | ||||||||||||
| IREC as opt out, no threshold | 1 | 1 | 2 | ||||||||||||
| Should be culturally appropriate/safe for all pops | 2 | 2 | |||||||||||||
| Current questions could lead to tokenism | 2 | 2 | |||||||||||||
| Experts/community as AI | 1 | 1 | 2 | ||||||||||||
| Value of involvement in project design | 1 | 1 | 2 | ||||||||||||
| Change question wording | 1 | 1 | 2 | ||||||||||||
| If 20%+ should be funded differently/different streams | 1 | 1 | 2 | ||||||||||||
| Yes still relevant | 1 | 1 | 1 | 3 | |||||||||||
| Shouldn't have a threshold | 1 | 1 | 1 | 3 | |||||||||||
| advantages/disadvantages of doing IREC criteria | 2 | 1 | 3 | ||||||||||||
| 20% not appropriate | 1 | 2 | 3 | ||||||||||||
| Allow opt in, even if not meeting 20% | 3 | 3 | |||||||||||||
| Checklist for IREC | 3 | 3 | |||||||||||||
| What happens when someone selects NOT IREC for IREC app | 3 | 3 | |||||||||||||
| Indigenous leadership | 1 | 2 | 3 | ||||||||||||
| Asking all could prompt engagement/access/involvement | 2 | 1 | 3 | ||||||||||||
| Proof of genuine IREC application | 3 | 3 | |||||||||||||
| Transparency/accountability | 1 | 1 | 1 | 3 | |||||||||||
| Using percentages not appropriate | 1 | 1 | 1 | 1 | 4 | ||||||||||
| Different thresholds for different (geographic) regions | 1 | 3 | 4 | ||||||||||||
| Workforce building to assess IREC | 4 | 1 | 5 | ||||||||||||
| Increase threshold (overall/for aspects) | 2 | 1 | 3 | 1 | 1 | 8 | |||||||||
| NZ (HRC) model | 3 | 1 | 1 | 2 | 1 | 1 | 9 | ||||||||
| Hard to measure/confusing what 20% is measuring | 2 | 2 | 2 | 2 | 2 | 10 | |||||||||
| All should address IREC | 2 | 8 | 2 | 1 | 3 | 1 | 1 | 1 | 19 |
Figure 6
| Row Labels | Alice Springs Workshop - 22 August 2023 | Brisbane Workshop - 11 August 2023 | Canberra Workshop - 28 July 2023 | CQU - 30 August 2023 | Darwin 1 Workshop - 23 August 2023 | Darwin 2 Workshop - 23 August 2023 | ECU Workshop - 6 June | JCU - 18 August 2023 | LOWITJA conference (CAIRNS) - 15 June | Melbourne Workshop - 17 August 2023 | NHMRC IREC Session with STRIDE network | Sydney Workshop - 8 August 2023 | Tasmania Workshop - 28 August 2023 | Grand Total |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IREC acts as a (good) framework for assessment | 1 | 1 | 2 | |||||||||||
| How to ensure IREC valued/impacts scoring | 1 | 1 | 2 | |||||||||||
| IREC review through workforce org | 2 | 2 | ||||||||||||
| More than 1 IREC assessor | 2 | 2 | ||||||||||||
| Challenges for incorporating community as investigators | 1 | 1 | 2 | |||||||||||
| IREC feedback to applicants | 2 | 2 | ||||||||||||
| IREC accountabilty/audit | 1 | 1 | 2 | |||||||||||
| 2 stage process | 2 | 2 | ||||||||||||
| Align with closing the gap | 1 | 1 | 2 | |||||||||||
| Aborignal and Torres Strait Islander person involved in peer review/IREC | 1 | 1 | 2 | |||||||||||
| Consumer reps on all panels | 3 | 3 | ||||||||||||
| IREC min score to go through to assessment | 2 | 1 | 3 | |||||||||||
| IREC apps need mid point review/grant monitoring | 2 | 1 | 3 | |||||||||||
| IREC needs to be scored and morerated panel score | 1 | 3 | 4 | |||||||||||
| Verbal IREC reports/presentation | 2 | 1 | 2 | 1 | 6 | |||||||||
| Training for indigneous and non indigenous | 1 | 1 | 1 | 3 | 6 | |||||||||
| Capacity/resourcing for IREC assessment/applications | 1 | 2 | 1 | 2 | 6 | |||||||||
| Introduce quantitative scoring (with rubric and advice) | 3 | 1 | 1 | 1 | 2 | 1 | 1 | 1 | 11 |
Figure 7
| Alice Springs Workshop - 22 August 2023 | Brisbane Workshop - 11 August 2023 | Canberra Workshop - 28 July 2023 | CQU - 30 August 2023 | Darwin 1 Workshop - 23 August 2023 | Darwin 2 Workshop - 23 August 2023 | ECU Workshop - 6 June | JCU - 18 August 2023 | LOWITJA conference (CAIRNS) - 15 June | Melbourne Workshop - 17 August 2023 | NHMRC IREC Session with STRIDE network | Sydney Workshop - 8 August 2023 | Tasmania Workshop - 28 August 2023 | Grand Total | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Specific schemes (DAATSIA/IREC only) | 1 | 1 | 2 | |||||||||||
| Public reporting | 2 | 2 | ||||||||||||
| IREC as a scoring element | 1 | 1 | 2 | |||||||||||
| PSP must value cultural excellence | 2 | 2 | ||||||||||||
| Divert funding from Go8/deinstitutionalising research | 2 | 1 | 3 | |||||||||||
| IREC grant audits/accountability | 2 | 1 | 3 | |||||||||||
| Workforce support (incl pipeline students) | 2 | 1 | 3 | |||||||||||
| Issues with clinical trial access/knowing baseline participation | 3 | 3 | ||||||||||||
| Two step process/iterative review | 1 | 1 | 1 | 1 | 4 | |||||||||
| Increase IREC target MREA | 4 | 4 | ||||||||||||
| All should address IREC/NZ model/AH impact statement | 1 | 1 | 2 | 1 | 3 | 8 |
Footnotes
1 The most frequently occurring phrases - Torres Strait, Torres Strait Islander, Aboriginal and Torres Strait Islander, Health research, and Indigenous health research – were omitted from the word-cloud as they occurred so frequently that their presence obscured other terms.
2 NHMRC Corporate Plan (2022-2023)
3 Funding rules involving Aboriginal and Torres Strait Islander peoples
4 2021-2024 Triennium implementation
5 Number of participants in attendance and workshops.
6 3rd International Indigenous Health & Wellbeing Conference 2023
7 For the purposes of this analysis, responses from participants at consultations in Perth and Cairns in June 2023 were grouped under the four overarching questions described in the IREC Review Discussion Paper.
