Future of peer review: Sustaining quality, fairness and trust in application assessment

In August 2026, NHMRC hosted a webinar exploring the future of peer review and its role in sustaining quality, fairness and trust in health and medical research funding. The webinar brought together leaders from across the research sector to examine the challenges and opportunities facing research assessment, discuss emerging reform directions, and consider what should be preserved, improved or redesigned to maintain confidence in peer review.

The session provided insight into NHMRC's peer review processes and ongoing work to strengthen quality, transparency and accountability across its funding schemes.

Facilitated by then NHMRC CEO, Professor Steve Wesselingh, the webinar featured a distinguished panel bringing together perspectives from health and medical research, consumer and community involvement, Aboriginal and Torres Strait Islander health and research leadership.

Meet the panel:

• Dr Doug Hilton AO (CEO, CSIRO)
• Professor Ginny Barbour (Editor-In-Chief, Medical Journal of Australia)
• Ms Debra Langridge (Consumer and Community Involvement Program Lead, Western Australian Health Translation Network)
• Associate Professor Nadeem Kaakoush (Scientia Associate Professor, UNSW)
• Professor Michelle Kennedy (Professor, University of Newcastle and Strategic Research Advisor, The Lowitja Institute.

Video transcript

0:02 Professor Steve Wesselingh
So I think we'll start and welcome you to this NHMRC webinar on the future of peer review. This is, I think, a really important webinar and one that I've been looking forward to for a while. I even had a haircut especially for the webinar. It's important because peer review sits at the absolute centre of our funding schemes. Not only that, I think at a time when we're seeing some loss of trust in science and in institutions around the world, I think particularly more overseas than in Australia, but we are seeing a bit of a loss of trust. I think we need to highlight the importance of peer review, and we also need to understand the impact of changes that are happening around us, like AI. So really important webinar. It's great that so many people have joined us, and I hope this webinar really starts to engage everyone in the discussion and hopefully answer some questions. We are certainly planning to have further webinars as well, by the way, to continue this discussion with the sector. Importantly, I'd like to acknowledge the Traditional Owners of the lands that everyone is on across the country. I'd also like to acknowledge Aboriginal and Torres Strait Islander peoples who are meeting with us today online. So, just going to the next slide. As I mentioned right at the beginning, peer review is evolving, the world is changing, and we really do need to understand how do we sustain quality, fairness and trust in the process. I think those are the three key words. What we want to do in this webinar is, firstly, I'll be presenting a number of slides that will outline NHMRC's models of peer review, what improvements we think we've brought to peer review, what the emerging challenges and issues are, and particularly, what the global and international view is on these emerging challenges. Then we have this terrific panel, who will discuss not only some of the questions that I'll give them, but also the questions that you have already given us and also questions that you will be adding online for us to give to the panel. So, I might at this point actually just go to the panel and get them to introduce themselves. I'll do that in the order that is on the screen, on the slides. So, despite the fact that it says Kimberly Glass, I'll go to Doug Hilton to introduce himself.

3:06 Dr Doug Hilton
Thanks, Steve. Doug Hilton, Chief Executive of CSIRO, and before that, medical researcher for 40-plus years.

3:16 Professor Steve Wesselingh
Next, I'll go to Ginny.

3:21 Professor Ginny Barbour
Hi everybody. Yeah, Ginny Barbour. I'm the Editor-in-Chief of the Medical Journal of Australia (MJA). I'm also the Co-Chair of DORA, the Declaration on Research Assessment and a background in open access and publishing for a number of years, and also publication ethics. Thanks.

3:37 Professor Steve Wesselingh
Thanks Ginny, Deb.

3:39 Ms Debra Langridge
Hi, Deb Langridge. I have the privilege of leading the Consumer and Community Involvement Program with the WA Health Translation Network, and I'm also a member of the NHMRC Research Committee and the NHMRC MRFF Consumer Advisory Group.

3:56 Professor Steve Wesselingh
Thanks Deb, Nadeem.

3:58 Associate Professor Nadeem Kaakoush
Good afternoon everyone. My name is Nadeem Kaakoush. I'm a Scientia Associate Professor in the School of Biomedical Sciences at UNSW.

My experience comes from the reviewing system, so the Ideas and Investigator schemes. I've done that for over four years for Ideas and three years for Investigator, as well as being a peer review mentor for the Ideas scheme for three years.

4:20 Professor Steve Wesselingh
Thanks, Nadeem. I just want to thank you for all of the mentoring that you've done in our schemes and the knowledge that you bring because of that.

And finally, Michelle.

4:32 Professor Michelle Kennedy
Hi everyone. Michelle Kennedy. I'm a Wiradjuri woman, Research Fellow at the University of Newcastle, and also the Strategic Research Advisor for the Lowitja Institute.

4:44 Professor Steve Wesselingh
Thanks very much. Such a terrific panel, and we'll obviously be going back to them after I've made a bit of a presentation. So I'll go to the next slide. I just wanted to highlight that we have had a lot of questions already submitted. Thank you very much to everyone who submitted those questions. You can submit new questions at any point through the Q&A, and the questions will be moderated and grouped into themes and topics. We've already had a lot of questions, but we are really hoping that you'll give us more. If we don't get a chance to address them today, we will obviously be answering those questions following the webinar. So don't hesitate to pile on the questions. Next slide.

Some of this, I think, will be self-evident to you, but you can see here the process that NHMRC uses to fund research, and central to that is peer review. You can see the process that we go through there at the top of the slide. Through that process, we fund excellent discovery research and priority-driven translation and innovation. Obviously, the ultimate goal is to develop new knowledge that will improve health outcomes. Why does peer review matter? I think, again, for this audience, it is self-evident. Importantly, I think peer review supports excellence and identifies the highest-quality health and medical research. It also helps ensure impartiality, minimises bias, upholds integrity and strengthens trust through independent expert assessment. Again, I think it underpins the way we do science, both through the funding of research and through the publication of our research via peer review publication processes. It also leverages diverse expertise. It brings together expertise within disciplines, consumer interests and industry interests, and brings them together in looking at what should and shouldn't be funded. Importantly, we believe it promotes transparency. We have very clear conflict management processes, clear published guidelines, observers to ensure those published guidelines are being adhered to, and we use clearly documented processes. Next slide.

A lot of the questions that have already come in relate to the peer review model and, particularly, one of the dichotomies that has occurred in the Australian system is the dichotomy between panel-based and application-centric peer review. Firstly, I just want to make it clear that we utilise both panel-based and application-centric peer review. You can see on this slide where we utilise panel-based peer review: CREs, Partnerships, Targeted Calls, Clinical Trials and Cohorts. Then there's application-centric peer review for Ideas, Investigators, Postgraduate Scholarships and Synergy. I will highlight this on another slide, but importantly, in both Ideas and Investigators, we have added shared reviewer identities, scores and comments to those processes. We also do the peer review for the majority of MRFF grant opportunities and that, in the main, utilises panel-based peer review. International collaborations are a bit of a mix between panel-based and application-centric peer review. Next slide, please.

If we look at panel-based peer review, I do understand that a lot of you see this as a very strong form of peer review. We do see that it supports in-depth dialogue, collaboration and shared probity responsibilities. We can conduct it either face-to-face or virtually. I think a lot of us think back to the time when everyone flew into Canberra and we chatted, had a great time, had a number of dinners, learnt, engaged, and brought younger researchers along. We acknowledge that in the past this has been a very suitable process. A panel can also, particularly around multidisciplinary research, bring in together lots of different people such as economists, biostatisticians, engineers and others. Their perspectives can come together to inform a decision. There are problems with panels, though, and we have found it's very difficult for us to optimally match each individual application to the best expert assessors. What we have found, and what we have measured, is that often there aren't people on the panel who have the particular expertise required for that particular grant, or if they are on the panel, they are conflicted and need to leave the room. So the matching, by our analysis, is about 30%, as compared as I’ll mention later with about 90% for application-centric peer review. Our data has also shown that spokesperson scores often have a significant impact on the panel, and rarely does the panel change scores above or below the spokesperson scores. Obviously, there are logistic issues with panels, particularly face-to-face panels and scale schemes. Then there are meeting dynamics, and I think we've all experienced situations where there's a very loud voice in the room. That is difficult, and the discussions are not always balanced, and chairing can be challenging. As I mentioned, getting everyone together, either on screen or in person, can be difficult and costly. Next slide.

With app-centric peer review, I think we have data showing that we actually get really good matching between a person's expertise and the grant. We get that matching above 90%. Obviously, for scale schemes with high application volumes, it is easier to use an application-centric peer review process. And obviously the flexibility for reviewers, who don't have to sit on a panel, and scheduling issues makes it easier for us to get peer reviewers. Obviously, that is one of the constraints of the system. It does mitigate some of the group dynamic biases and probably gets us to a more independent assessment process than perhaps panels do. Logistics are obviously easier. There are, however, risks and constraints. There's no discussion or conversation between reviewers. Recruitment logistics is difficult, but, as I mentioned, that's difficult across the board. There is less learning through the precise peer review process. Although, as we'll indicate later, we put a lot of effort into training, webinars and mentoring etc. to bring up the skills of our peer reviewers. Participation by reviewers with different perspectives can be a little harder to implement. We're thinking here, how do we bring consumers into our scale schemes with application-centric peer review, and that is an issue for us to think about. It’s probably harder and less opportunities for recognition, professional connections, particularly between senior and junior researchers, and the social capital and networking that may occur. Next slide.

As I mentioned, we have been working on our peer review processes and trying to improve transparency and accountability. We have feedback sharing, the five peer reviewers with app-centric peer review are shared with all the reviewers on the same application. Reviewers can identify errors and talk to NHMRC about it. Reviewers can view individual scores and also the identity of the four other reviewers through the application-centric process. We do think this means the process is no longer totally anonymous. The five people know who they are, they know what the comments are, and they know what the scores are. They can adjust their scores and feedback prior to the finalisation of the assessment, and they can also talk to the NHMRC about that. So we do think this has been a process of continuing improvement. Next slide.

We have also, in terms of improving our processes, started to broaden participation and the benefits of peer review, particularly in consumers of peer review. We've had consumer and community members involved in scoring of targeted calls for research applications since 2023, and the Clinical Trials and Cohort Studies scheme is implementing this approach with mentoring workshops and experienced panel chairs in the current rounds. For both of those schemes, consumer involvement criteria are weighted at about 20%. In addition, in Aboriginal and Torres Strait Islander health, we've had IREC, obviously, and I won't go into detail about that process, but it is a process that we recently reviewed and that is working very well. We've also found strong support for asking the questions: how does your research benefit Aboriginal and Torres Strait Islander health? It's such an important issue in Australia. Again, we've tested that in the CREs and we're going to roll that out across the board to get people to think about that question. Next slide.

If we look here at consumers in peer review in the TCRs and CTCSs, you can see how the process goes across the top and then along the bottom. We have our initial assessment, which is based on scientific criteria. 20 to 30% are shortlisted and the others are removed. We then have consumer scoring of the shortlisted applications. And then all of that then comes to the panel, where there is rescoring of both the scientific and consumer criteria, and the top-ranked applications are recommended for funding. So it does show how consumer engagement and involvement can really support our peer review. Next slide.

Obviously, through this process, we are building capacity and capability, with mentors and other scientific panel members working with NHMRC staff to make sure they are ready for the process and the panel meetings. Next slide.

So that leads nicely on to the other things that we do in the implementation of our processes to ensure the quality of what we are doing. Training and capability building are really important. We have webinars, interactive modules, mentors like Nadeem, and live Q&A drop-in sessions supported by senior peer reviewers. We've also been reporting everything we do. So we report the outcomes of our grants by grant opportunity, chief investigator, fields of research, institution, organisation type and by jurisdiction. That is all published regularly on our website. We have recently done some analysis to look at geographic bias. We analysed our two larger schemes, Ideas and Investigators, and looked carefully to see whether there were score differences between local and interstate assessors. Again, didn't observe any evidence suggesting systematic geographic bias. We do continually go out for expert feedback. We've had expert committees look at our peer review process both from statistical points of view, bias point of view and other points of view. We get feedback from applicants and peer reviewers, and we also, and I'll talk about this later, talk a lot to our international funders. We also work closely with people both in Australia and around the world who are doing research on research, or meta-science. We're seeing a lot of really important information come from groups such as RORI, and also Australian people working in that area. And we also look at the experiments that have been conducted around the world in peer review and consider the outcomes of those experiments. Next slide.

So, talking about the international environment, we signed up to DORA in 2013, and the mission was to advance practical and robust approaches to research assessment globally and across all scholarly disciplines. More recently, we joined CoARA, the Coalition for Advancing Research Assessment. As part of that process, we developed an Action Plan for Responsible Research Assessment, and that action plan is publicly available. It goes beyond impact factors and publication counts, looking at broader research contributions and impacts. Importantly, peer review remains the cornerstone of responsible research assessment. But again, DORA and CoARA also give us the opportunity to work internationally with other countries around the world in Canada, the EU, the UK, Asia and elsewhere to ensure we are moving forward in sync with our international funding counterparts. Next slide.

At a recent meeting that I was at in Paris called HIROs, Heads of International Research Organisations, where really we had all the major international research organisations coming into the same room to discuss issues that were of concern. We had NIH there, the European Research Council, the MRC and NIHR, as well as organisations from the Netherlands, Norway, Japan and Singapore. One thing that everyone was talking about was this significant increase in applications, which people think is driven by AI enabling people to write better higher quality grants more quicker and more easily. Some agencies were seeing 100% rise in grant numbers, and this is becoming, I think, an existential threat to peer review. A number of agencies are under strain because of this potential to overwhelm peer review. RORI has written a number of articles about this issue, and many people are questioning whether the current model is truly sustainable.

When we talk about peer review, it is important to look back at what we did and how we did it. But we absolutely have to look forward because the world is changing and AI absolutely is changing the world and changing how people write grants and how they review grants, and that has to be part of our thinking in terms of peer review. Next slide.

So if we look at this slide, this slide shows what some agencies are experiencing. Our increase in applications of around 20% which is still a lot year on year, but it is not as high as what organisations such as the Welcome Trust or the European Research Council are seeing. The Welcome Trust reported a 100% increase, the ERC 54%, Canada 48%, and the ARC 44%. These are the struggles and pressures other agencies are experiencing.
Next slide.

Going a little more in depth into these peer review pressures, they do impact on peer review in terms of peer review capacity that is the number of peer reviewers but also the quality is squeezing together. But also the quality is squeezing together, we are not seeing many applications that are more of low quality we are seeing more applications that are being squashed together which makes peer review more difficult and harder to differentiate between high and low qualities. We get into concerns regarding fairness, transparency, confidentiality and accountability, because of these processes with AI. One of the questions we got earlier this week asked if whether we used AI for peer review it would get rid of biases and so I think there is a lot to think about there.

If we look at the scale of NHMRC Ideas and Investigator Grants, around 20,000 assessments are required to be completed in a four-week period for each scheme. For the 2026 Ideas Grant round, around 3,839 peer reviewers were invited to assess approximately 2,400 applications, while for the 2027 Investigator Grant round, around 1,500 peer reviewers were invited for over 2,000 applications. This is all happening at scale, and that has to be taken into account. I was just thinking that one of our meetings a couple of weeks ago was with one of the Chinese funding schemes, and they had 500,000 applications. So, I guess we might count ourselves lucky. Next slide.

So, how are funders responding? They're responding as you might expect. They're putting caps on the number of grants people can hold, quotas on the number of applications from investigators and institutions, they’re reducing the number or funding calls, or the sequence of funding calls for example moving from yearly to every two years, or from six-monthly to yearly. They're putting limits on resubmissions, asking people not to resubmit if they weren't successful for two years as an example. They're introducing tiered assessment processes, and a number are looking at AI-assisted processes and triaging and shortlisting. They're also looking and experimenting at training AI systems using peer review data from past grants. You can see a number of articles in Nature and elsewhere on this issue. As I said, it is an issue affecting all funders around the world. Next slide.

So that's what I wanted to tell you. I think the top of this slide summarises the expectations of peer review continue to expand; the scale and complexity continue to grow. How do we ensure the system remains rigorous, fair, trusted and sustainable? We have to share this challenge with everyone on this webinar, with our researchers, reviewers. We also have to share it with our institutions and consumers. I think this is both a really exciting area but a really difficult area, and one that I'm looking forward to discussing with the panel and in answering your questions. Again, I'd emphasise that we need to be looking forward and working out what we can put in place that will withstand these growing pressures, as well as looking back and considering what worked, what we can learn from that and what we can take forward. But, acknowledging, as George Burns said, we're going to live the rest of our lives in the future. So we need to think about how we deal with these future challenges.

I think that's my last slide. Is that correct? Just flip to the next slide so I can make sure I haven't missed one. Yes, that's the last slide. So maybe we'll take the slides down and go to the panel. There are a couple of questions I'd like to start off with the panel, and then we'll move to some of the questions from the audience that we received both during the week and today. Maybe I'll go to Doug first.

What's surprising about how peer review has changed Doug? You've been in the system for I think you mentioned 60 years and you've probably seen it all. What are your thoughts?

30:05 Dr Doug Hilton
So I think my first peer review experience was on a panel that was called RGIC. That's where every state had a separate race for Project Grants back then. A team would go to Sydney and review all the grants for Sydney, and then another team would go to Melbourne. My sense of it is that there's a lot in common in the way we still do peer review. So I'm always bemused about the nostalgia. I don't think that, whether you've got a grant panel or whether you're reviewing a grant in your lounge room in isolation, you can take away the temptation for poor behaviour that comes to a small number of individuals in any scheme, in any group of people. I saw, in the old days, people who played games, people who voted low, and people who were charismatic who swayed the panel. Today, what do we see and what are the complaints that we get about application-centric peer review? We have people who mark hard. We have people who mark low. People say things are not fair. So I think what I would do is really echo what Steve said, which is that we need to chart a new course. The old ways of peer review, 10, 20, 30 or 40 years ago, were not perfect. They were flawed and not really fit for purpose. Now, probably the current path, if we continue on it, is not fit for purpose either. So what new course are we going to chart? I think the things that have changed enormously are the size of the system and the complexity of our sector. It's grown enormously over 30 years, with two doublings of NHMRC, the creation of the MRFF, and then, of course, AI. Probably the hardest thing for any institution, especially within government, is to adapt agilely to a changing environment. So that's probably the biggest challenge.

32:12 Professor Steve Wesselingh
Thanks, Doug. That's great. Thank you very much. I might go to Ginny now and ask the same question. In some ways, your experience has been with publication, but also with DORA.

So, Ginny, what are your thoughts about the evolution of peer review?

32:33 Professor Ginny Barbour
Yeah, thanks. Thanks very much for that, Steve. Look, I think the main thing is that we've allowed ourselves to be driven by external forces when we're thinking about peer review. One of my frustrations, I guess, is that we've drifted away from thinking about what the importance of peer review is and what its actual purpose is. I'm surprised that, as this really smart community, we've allowed ourselves to do this. It's one of the few areas where we don't think strategically about what the purpose of the system is. We say we're surprised by all these changes, but actually I don't think we should be. I think one of my biggest frustrations, and possibly surprises, is that we haven't yet done a good job of incorporating new tools into peer review. We're still doing things pretty much the same way we've done them previously. I would really like us to be thinking about what our opportunities are for incorporating some of these fantastic tools that are out there into the peer review system. Because, as you showed really clearly, Steve, we're going to have to think at a large scale about how we address the challenges in peer review now. We just can't carry on the way we've been going.

33:45 Professor Steve Wesselingh
Thanks, Ginny. That's great. Thank you.

I might go to Michelle now. Michelle, I know you've got some particular concerns about peer review, and perhaps you might want to air them right at the beginning.

33:55 Professor Michelle Kennedy
It's such a good opportunity to have this webinar coincide with a paper that we published. It literally came out last week. We had no idea that this would align so well, but we published a paper called Indigenous Excellence: Confronting Racism and Complicity in Peer Review. It was published last week in Health Promotion International. It's an all-Indigenous authorship team and it's generated from years and years of us having discussions about our experiences of racism in peer review, both in Category 1 funding but also in peer review for publications. The paper acknowledges that peer review functions as a critical upstream determinant of what the evidence base is, and that inevitably shapes policy, practice and resource allocation. It also critically calls out that racism in peer review occurs, and it's not even a singular act. It's upheld by really complex systems of editorial or commissioning governance and peer review practices of individuals, but also by forms of knowledge evaluation that has occurred throughout the academy, which excluded Indigenous peoples for so long. The paper identifies that what we need to do is more than just procedural reform. It requires educational, cultural and whole transformations across the research ecosystem. We make some specific recommendations for structural reform to address racism, and we encourage academics, journals and funding bodies to read this paper and funding bodies to think about the role that they play. I thought it was quite interesting talking about AI and the future of AI because this paper highlights that it could actually make experiences worse for Indigenous peoples. Because it’s going to generate and will be driven by the fact that Indigenous scholarship has been excluded, disadvantaged and devalued for so long. So it could actually be incredibly problematic, but it is possible to address this. It's not going to happen overnight, but it is something that we're working on. It's lovely to be on the panel with Ginny because we've worked closely together on this. I've worked really closely with the MJA over the past three years on their Indigenous health special issue, in really starting to look at the journal and looking at how publishing can be safer. I think, I'm probably biased, but I think we've done that pretty successfully over three issues. So, yeah, I'll stop there, but I'd love to have more conversations.

36:47 Professor Steve Wesselingh
Thanks, Michelle. That was terrific. Nadeem, I'll go to you and ask the same question. How have you seen peer review change? Particularly because of your work in mentoring, how do you think younger researchers and early career researchers are dealing with these changes?

37:08 Associate Professor Nadeem Kaakoush
Great comments from the rest of the panel members. I guess one of the things that I'll add in terms of the changes is this need to adapt to the changing situation, and how the criteria of what we're reviewing has slowly changed. For example, with the Ideas scheme, we've moved from focusing on track record to capability, and really focusing on that. As well as the Investigator scheme looks at impact rather than other aspects. So the criteria of what we're looking at is slowly adapting. In terms of the journals, for example, there's also been a push towards transparency, such as publishing peer reviewer comments and things like that to really show what the whole process is about. I think that's something that has really changed over the last decade or so. From the early career perspective, I guess, some people mention that panels provide this educational experience for early career peer reviewers. But perhaps the educational aspects of panels can be incidental, moreso, than structured. So I think focusing more on mentorship, workshops, modules and things like that provides more structured educational resources that peer reviewers can learn from. So I think if we focus more on that route and provide more structured education, that will help early career researchers a lot more.

38:37 Professor Steve Wesselingh
Thanks for that. That was really helpful. Actually, I want to get back to that question. I'm going to give Deb a chance first, but I'd like the panel members to think about this sense that our peer reviewers are anonymous and whether we should have annual lists of peer reviewers. How do we deal with that question? I know Ginny deals with it at the MJA. Nadeem, maybe I can get you to think about that because I'll come back to you on the question of greater transparency around peer reviewers and their involvement. But Deb, to be fair, I need to let everyone have their say. I'm sure from your perspective you've seen some pretty dramatic changes in peer review, so why don't you talk about that?

39:27 Ms Debra Langridge
I think, yes, Steve, absolutely there's been significant change. It follows on from what Nadeem was saying. The changes we've seen in peer review have coincided with changes with consumer and community involvement being recognised, but also therefore in governance of grants included in that criteria, in that process. So I think that's been a significant change. It's now a reflection of the National Health and Medical Research Strategy and the new Consumer Statement, which is fantastic. I also think one of the most significant changes has been the way where the criteria have changed and now there is 20% and some grants being allocated to consumer and community involvement. We've seen, this was piloted in the recent Clinical Trials and Cohorts Scheme 2025 round of consumer and community reviewers having the opportunity to be mentored so that there is consistency and an understanding of what the scoring process looks like. But also, panel members being involved in discussions where not only did consumer and community reviewers learn, but also science reviewers learn as well as part of that panel process. Theres been an adoption process which has been great, but I think it also sets the direction for the future in terms of what does consumer and community involvement in reviewing might look like for future grants where it currently isn't included but will be rolled out. A tailored process is important. It isn't one-size-fits-all. So, what involvement looks like in Clinical Trials and Cohorts may have differences in the process for Ideas, Investigator or Synergy Grants across the board. It's exciting to see more examples and more capacity and capability and training. Overarching all of this, though, the integrity of the process is really important. It's been at the top of the list for both MRFF and NHMRC in the process to ensure that the integrity of involving consumer and community review is comparable to the rest of the reviewers throughout the process.

41:46 Professor Steve Wesselingh
Thanks for that, Deb. Nadeem and Ginny, I'm just going to go to you now on the confidentiality issue. So, Nadeem, what are your thoughts about confidentiality and reviewers? Reviewers have to make some clear statements about what is good and what is bad about a particular grant. It's not always comfortable to know that your name will be associated with that. But obviously there's a transparency issue here as well. Do you have thoughts about that?

42:20 Associate Professor Nadeem Kaakoush
I think we're all adults. People have to put their name to some of the comments that they put out. I've been part of the ARS system for the Investigator scheme, where there was a revealing of the five reviewers for the grants and the comments, as well as the opportunity to change our scores. I think it operated pretty well, in my opinion. I didn't feel any sort of pressure from the perspective of feeling that my name was attached to my comments. I think transparency is part of the process of fixing some of the issues that people point toward in the peer review system. Beyond that, I guess transparency to the applicants rather than the reviewers is important. A roll call of who has reviewed grants over the year is perfectly fine and doesn't attach a name to a particular grant, but it does acknowledge the reviewers who have made the effort to review grants. So I think transparency is a really important aspect of the peer review process.

43:24 Professor Steve Wesselingh
Thanks for that. And Ginny?

43:28 Professor Ginny Barbour
Yeah, we've given a lot of thought to this. I must admit I've been involved at journals where we have had fully open peer review. At the MJA, though, the process we have is double-anonymous peer review. I think there are two things here. One is the transparency of the process. I'm absolutely in favour of publishing peer review comments, and we are looking to do that at the journal in the future. We don't technically have the means to do it at the moment, but that's not impossible to solve. The issue of anonymity, I think, is really important on both sides. I'm from the UK and I've moved to Australia. Australia is a very small system when it comes to health research. The anonymity of the review process, where reviewers do not know the authors, gives a degree of protection against bias. We know very well that applications are less likely to receive positive scores if there is a female first author or female last author. That's incredibly well shown. So from that perspective, I am actually quite in favour of anonymous peer review. The problem, of course, is that it's a small system, so people often do guess who the authors are. But some work has shown that when people guess, they don't always get it right. The other side of it is whether people should sign their peer reviews. I personally do sign my peer reviews, but I'm a senior academic and I don't have to worry about retribution. I've spoken about this quite extensively, and I vividly remember a conversation with a mid-career female researcher who was almost in tears because she thought she'd almost destroyed her career by signing a particular peer review. So I think we have to think really carefully about what we're actually trying to achieve with these different levels of transparency and anonymity.

45:09 Professor Steve Wesselingh
Thanks for that. Just looking at another question on the screen here, and that goes to lotteries. I'm just warning Doug and Michelle that this question is coming, but I'll go to Doug first. We've been talking about lotteries for a while. We do understand that in most grant schemes you can identify the very best applications and probably remove the bottom percentage, whatever that percentage may be, perhaps 50% or so. But the ones in the middle are very difficult to distinguish. Even with peer review, there probably is a lottery component to that. A number of people, both in Australia and overseas, have advocated that if you applied a lottery to that, you would get essentially the same outcome while removing a whole lot of biases and other influences. So what do we think about lotteries? Personally, as CEO, I also have the challenge of if we did use the lottery, which I’m not against actually, how I explain that to the minister. If we used a lottery, which I'm not necessarily against, how do I say that we agreed to fund these grants and chose who to fund on the basis of a lottery? There is a political and public education issue there as well. I'd like Doug and Michelle to perhaps comment on that.

46:40 Dr Doug Hilton
Maybe two comments. First, like you, Steve, I have absolutely no problem embracing the idea of a lottery when there is no statistically significant difference between grants and no clear way to separate them. The challenge is maintaining social licence, having done that, is a challenge and that’s one you’ve just articulated. Part of our mystique as a sector is that we do peer review, we can judge excellence, and we fund the best grant. Stepping away from that and admitting that we don't necessarily have a system that lets us do that perfectly, that does require thought and communication. The other question is how we get greater distinction between grants. How do we encourage reviewers to use more of the scoring range? I think we've drunk our own Kool-Aid in a way over the last 20 years when we talk about the fact that the quality of grants increasing because there are many more in the fundable range. We're scientists. Come on. We need to distinguish between drifts in scoring behaviour from actual quality of grants. Just because we score something as a 5 in 2026 doesn't necessarily mean it equates to a 5 in 1996. I think we do have a tendency to drink our Kool-Aid. Using a greater proportion of the scoring range, perhaps thinking about ranking systems rather than absolute numbers, and other approaches like that are worth considering. As an individual referee, I think those sorts of things, plus a lottery, are all worth considering.

48:23 Professor Steve Wesselingh
Thanks, Doug.

48:26 Professor Michelle Kennedy
Well, I've never won a lottery. I think of all those early career researchers being told, "Please spend hundreds of hours developing your first grant and you might just win in the lottery." I don't know how that sits with me. I also think of all the times we've put in a grant and then not got that grant, and had to spend so much time trying to get that idea up. I guess my perspective is a little different because of the consumer-led work that is increasingly occurring. It can take me 12 to 18 months with community engagement to develop an idea for a grant. If I was then to go back to the community and say, "I'm going to do my very best, but we may or may not win based on the lottery," I don't know how many communities would want to do that really deep work of developing a grant application and all the letters of support. Oh my goodness, that would just break my heart. In my role at Lowitja Institute, I've led the commissioning of our major grant schemes, and I completely understand why we end up with so many applications that are difficult to distinguish between. I think it would be really nice for schemes, when we're designing them, if we're very transparent and clear about what the priorities would be. Again, in my context of Aboriginal and Torres Strait Islander health, if we know that only a small numbers of research being done in Victoria and there's a priority project from Victoria, maybe those are the ways we should make decisions. Rather than using a lottery, I'd rather go back to: what would be our priority funding, and how can we be transparent that if you’re in that cluster, these would be our priorities and not just about remoteness. It could be about the types of teams that would be the most ideal to fund or the types of research topics that are priorities. I think that would be a much nicer way to move forward, as a non-lottery winner.

50:41 Professor Steve Wesselingh
Thanks, Michelle. Great comments. People have used other terms for this, by the way, such as "randomisation" or "algorithm-driven selection". There are euphemisms, but I thought I'd get it out on the table. I think that's been a really good discussion. Looking at the questions on my screen, I might as well on the table put the elephant in the room and I addressed it in my talk as well: the competition, in a way, between Grant Review Panels, or GRPs, and application-centric review. I'm going to ask everyone to comment on this because, to be honest, that's what a lot of the questions that we’ve had. As I've emphasised throughout, going forward we absolutely need to consider the really good parts of what we've done in the past, whether that's GRPs, application-centric review, or some combination of the two. So I think it would be an interesting discussion to get everyone to comment. I'm going to start with Nadeem. So tell us your views on GRPs, application-centric review, or some combination.

52:11 Associate Professor Nadeem Kaakoush
I was part of the NHMRC Ideas 2019 GRP. When it first started, it was still a GRP before switching to the application-centric system. My issue with it, initially, I thought GRPs were the way to go. Then I did a bit of reading on the topic and realised that I'm actually more in favour of application-centric review. The reason is because there are research papers around the fact that that the order of speakers influences the outcome of grant discussions. Whoever speaks first can influence the outcome of the grant. It's not just the spokespeople themselves having influence, but the order in which they speak. So there's a lot of bias around that aspect. Another point, which you highlighted in your slides, is the matching of expertise. You're really relying on two people to influence a full panel discussion, whereas in an application-centric process you can have five different reviewers matched to an application based on expertise. For the big schemes, I'm more of a proponent of application-centric review for that reason. But, I do believe there is a way to combine both approaches. Which is what ARS is doing. There could potentially be more discussion around the applications that I guess need some discussion. We don't necessarily need to discuss the really high-scoring applications or the un-fundable, but we might need to discuss those in the middle, as we discussed earlier.

53:46 Professor Steve Wesselingh
And just importantly, by ARS, you mean the current process of sharing reviewer scores, identities and comments?

53:52 Associate Professor Nadeem Kaakoush
Yes, sharing the reviewer scores. Yes

53:58 Professor Steve Wesselingh
All right. No, thank you for that, Nadeem, Deb?

54:02 Ms Debra Langridge
Yeah, I'm a huge fan of GRPs. In terms of consumer and community reviewers, there's still a lot of learning across the research sector in that space. In both the Targeted Calls for Research and Clinical Trials and Cohorts schemes where panels has been used, the process leading up was app-centric to some degree in terms of expertise of science reviewing first, and then those that progress. So it was a two-part process, and I think it worked well. The expertise and perspective that consumers and community members bring is different. They're not there to become overnight scientists. They're there to bring a perspective of lived experience to questions such as: what should involvement look like and what should reviewers be looking for? I think there is a role for that to play panel discussion, particularly as more grant programs role out consumer and community involvement. Even if there is only a panel component at part of the process, it allows discussion about what meaningful and valuable involvement actually looks like versus what might look good on the page but is not resilient enough to be comparable and score appropriately. But I do understand the strengths of application-centric review in terms of matching expertise to the lens of the grant. However, nationally, we don't yet have an abundance of consumer and community reviewers as per yet. However, it's been great working with the NHMRC and MRFF in building, training and mentoring consumer and community reviewers. So it maybe further down the track, in an application-centric process, we can match lived experience and consumers to applications. But I think there are still some obstacles before we could move wholly to application-centric review, having some process in having a panel involved. Especially in this beginning process where we are at the moment where there’s great learnings and the relevance and the meaning of what true and meaningful involvement looks like is still being recognised.

56:38 Professor Steve Wesselingh
Thanks, Deb. Ginny?

57:45 Professor Ginny Barbour
Yeah, my experience has mostly been with medical journals, and I was giving some thought to how what we do relates to grant review panels versus application-centric approaches. Although reviewers at journals such as the MJA don't interact directly with one another, the editors certainly do. We have a process where reviews come back and then we have an meeting panel where we sit down and discuss them. My experience has absolutely been that the more diverse the views we have in that meeting, the better the decisions we make. The more collaborative we are, the better the decisions we make. For me, that's been particularly highlighted by the work of Michelle and her colleagues, who have been guest editors for our NAIDOC Week special issues. The degree of collaboration and thoughtful consideration is incredibly advanced compared with other panels I've been part of. For me It's really opened my eyes to the importance of having a collaborative space to discuss feedback. It produces much better results in my view.

Professor Steve Wesselingh
Thanks. Michelle?

57:47 Professor Michelle Kennedy
I agree with Deb and some of my first experiences of peer review for Category 1 grants was as the Aboriginal IREC reviewer. Even though in those, and I've always had very positive panel experiences with both NHMRC and MRFF and being able to bring a very different perspective, exactly what you said Deb. And it looked really good on paper, but let me just break it down for you. Have had panel reviewers change, you know, their scores and opinions based on my reviews. So I've had a really positive time. I do know prior to the years when I began participating in these, there had been pretty negative experiences. If you have, you know, only one Aboriginal and Torres Strait Islander person in the room and whether their voices were heard and accepted. Mine's always been very positive, but it hasn't been for everyone. But then, you know, last year I was also an assessor of the Investigator scheme and I think that was excellent as well because even as an assessor, when I got the sheets, I could even see bias as an Aboriginal assessor, it was wonderful for me to see there were a lot of Aboriginal and Torres Strait Islander people also assessing content that was relevant for our people and being able to see the other reviewers and their assessors and their scores and be able to change mine. I think that process was really good. It was wonderful, I guess, for me to see and be able to reassure people that are applying this year and into the future years that there is a considered approach, even if we don't have that panel and deep discussion, which we definitely need for some schemes, I could see that people took that very seriously and genuinely provided as much as possible. So, I think that it's useful for us to diversify what we're using and to draw on both.

59:52 Professor Steve Wesselingh
Thank you. That's terrific. Doug?

59:54 Dr Doug Hilton
I have a mixed view about this. As you know, Steve, my high-level comment would be that I don't see why NHMRC uses panels for some schemes and application-centric review for others. I don't buy the argument that some types of grants are more complicated than others and needing more diverse skill sets. Most grants are complicated. They require people with different discipline skills to come to assess them properly. So I would ask, first, for some level of policy consistency. I was a big fan of grant panels, and through listening to experts on peer review through what I think is eight or nine years on the Research Committee, I've been convinced of the merits of application-centric review from the viewpoint of getting independent thoughts that aren't coloured by strong personalities to assessments. My preference would be for application-centric review across the board, with some opportunity for moderation. Then, if we're thinking about AI, I'm going to say almost the opposite. Maybe we need to go back to something like a viva-type discussion around grants where you actually interview people in person, because I'm not sure how we understand the deep sense of ownership of a grant with a scientist if most grants end up being written or modified by AI. Maybe I'm very old-fashioned, but perhaps I'm getting to the point where I want to look people in the eye and gain an understanding of the depth of their creativity, humanity and their knowledge.

1:01:47 Professor Steve Wesselingh
Thanks, Doug, and thanks to all of you. That was a terrific range of comments on a really complex area. That was great. I'm getting lots of questions. I'm just going to mention a few areas so people can think about them: relative to opportunity, EOIs, incentives for peer reviewers, and early and mid-career researchers. I'll start with relative to opportunity, which is a bit of a vexed question, and I'm going to give this one to Nadeem and Doug. How do we do relative to opportunity better Nadeem?

1:02:31 Associate Professor Nadeem Kaakoush
That's a tricky one. I guess one thing that could potentially be done is potentially looking into people commenting about their assessment of relative to opportunity and then have some form of deliberation around whether that has been considered appropriately. It would be a sort of post-review process asking has relative to opportunity has been integrated appropriately into the assessor scores. Beyond that, it's a very tricky subject. I'm not too sure what else can be done to really ask people to look into relative to opportunity as deeply as required.

1:03:24 Professor Steve Wesselingh
Thanks. Doug?

1:03:27 Dr Doug Hilton
I have some pretty strong opinions on this. The first thing I would say is that the reason we have valued 70 plus years of peer review is because we get experts to do it. I would not want non-Indigenous people necessarily commenting on cultural issues on grants. I'd rather have people with the expertise do it. What I would say would be I don't think there are many scientists who are experts at judging relative to opportunity. Certainly, when I started as a thirty-something-year-old man, I had no idea how to judge the lived experience of a forty-something-year-old woman who might be juggling parents with end-of-life care issues and children. I was so not the expert. So I would ask that we go back to principles. How could we achieve a level of uniformity in assessing relative to opportunity, and how could we get a group people who are actually experts in the area, rather than assuming every scientist has some capacity to assess it? Then you'd have to think very, very carefully about which impacts on opportunity you wanted to consider, because it is a can of worms.

1:05:06 Professor Steve Wesselingh
Thanks, Doug. Thank you. I think that's really important. I did mention those areas, but I've got a question here specifically for Ginny. It says: Ginny mentioned the need to use new tools that are available for peer review. Can we hear something about what those tools are and how they might be applied to grant review?

1:05:30 Professor Ginny Barbour
Look, I'm absolutely not the expert in grant reviews, but I can tell you that we are thinking very hard about how we might apply these in journals. Papers are getting more complex. I think we need dissect out what are the parts of peer review that we need to happen. There are clearly objective components. Was the statistical analysis done appropriately? Is the data available? Is the data solid? All those kinds of issues. Those can be done by machines. I don't think we have got the tools to do it yet, but those can be done by machines. When we get to subjective elements, like is this grant the most appropriate for a community that Michelle is working with? Is it most appropriate for an approach Australia needs to have a focus on at the moment? That is absolutely subjective. I'm very keen to see new tools come forward. At the moment, for example, we use plagiarism screeners, but we know that's quite a rough tool. I strongly suspect we'll see more tools come out around methodological assessment, both for grants and for publications, and that's what I'm hoping to see soon.

1:06:46 Professor Steve Wesselingh
Thanks, Ginny. Moving on to incentives for peer reviewers. Maybe I'll go to Ginny first and then Deb. How do you incentivise peer review at the MJA? How do you get people to review?

1:07:06 Professor Ginny Barbour
We're very lucky because I recognise that a journal like the MJA finds it easier to get reviewers than other journals do. I hope people do it because they value being involved with the journal and also, we use their time wisely. We spend a lot of time at the MJA doing triage so that the papers we send out to peer reviewers are worth their time. You're absolutely right, though, that we don't yet have a formal system of recognition. That's part of the work we're doing through DORA. How do you reward the entirety of the work academics do, that can be from mentoring through to peer review and all the activities that happen? I'd really like it to be part of academics, what they’re counted for at their universities. It's not just a job for journals; it's a job for universities as well. A And I think as a system wide; we should be looking to see how we can consistently recognise it. At the moment, we do provide recognition at the end of the year. We don't require people to sign their reviews, we can do that. I'm actually very interested in turning the question around: what would academics like to see? How would they like to be rewarded for peer review? And please don't say money, because we know that money doesn't necessarily improve the quality of peer reviews. It's not a simple answer I’m afraid.

1:08:31 Professor Steve Wesselingh
Thanks, Ginny.

1:08:33 Ms Debra Langridge
I think, as I mentioned before, there will continue to be a growing demand for consumer and community member reviewers as we move through the process. And what has happened up until now, it's often been networks where there's been a call and an invitation for consumers or community members to put their hand up to potentially become a reviewer. What hasn't been part of that process until recently has been a strong commitment and an intentional commitment to train and build the capacity of those consumers and community members across the board. And so, whereas there are some opportunities for consumers and community members to get training and to learn what does it, what should involvement look like? The scoring process, I'm not supposed to mention this, but how to use Sapphire, you know, all of those things, all those bits and pieces which are foreign to people outside of the research sector. So there's that. That incentive will be really important moving forward to say not only do we recognise that people's lived experience is important in the assessment process, but also integrity and consistency is important. And so therefore training is, I think, something that will become a predeterminant to ensure that not only the people who are putting their hand up to become consumer and community reviewers have the understanding, but also that they're supported and it's a safe process as well. So I think that's one incentive. The other incentive that just comes naturally is that people either from a personal diagnosis or a diagnosis of a family friend or member, they want that lived experience to make a difference. And so we have the opportunity to really engage with passionate people, support them in that process and knowing at a national level now across the research sector that lived experience is a point of difference in terms of not only value of being, but research being valued, but also the outcome in terms of in turning, turning great things and findings into the translation into making a difference. That's a brilliant incentive that we don't need to really publicise for consumers and community members. But it's that understanding of the process and then ensuring that it's done in the right way, and training. But also importantly, that consumer reviewers and community member reviewers are remunerated for their time and expertise as well. That's a non-negotiable. It's in the Consumer Statement. It's agreed across the board that that is recognition of that expertise together with researchers and clinicians and others, but people absolutely don't do it for the money. They do it to make a difference because they're passionate about it.

1:11:21 Professor Steve Wesselingh
Thank you. I'm just looking at the screen. It's at some of the other questions. There's a couple of questions that relate to the ARC and how we might synergise with the ARC and how we might align with the ARC. So I'm going to answer that question because I just want to put on the table a wonderful meeting we had last week where the Council of NHMRC, AMRAB and the Board, yeah, the Council of NHMRC, AMRAB and the Board of ARC, met together for a morning, for three or four hours, and workshopped a number of different things, including peer review and trust in science.

And it was a really great meeting, and certainly Ute, the CEO of ARC, and myself have been working together for quite a while to try and see how the agencies can work better together. And we've had a workshop between our directors as well. And as an example, the ARC shared their findings on EOIs and what they found, and how they analysed that, and whether or not they would continue with it.

So I do think there's a lot of learnings between the two organisations. I do think that we should try and align more of what we do between the organisations. And someone else has said, what about aligning RTO with ROPE? And, you know, maybe we can improve ROPE and RTO at the same time and come up with a better system.

And maybe, in a future that I can imagine, we actually have the same system for people to put their grants into. It won't be called Sapphire; it'll be called something else. But it could be a system where people put in a grant either for ARC or for NHMRC or for MRFF, but into the same system with the same requirements. So I do think there's a lot of excitement and possibility in working together with the ARC.
We've had a number of questions about early- and mid-career researchers. And so I'm just going to ask Nadeem and Michelle whether there are aspects of our peer review that you think don't help, or help, early and mid-career researchers, or could we improve it for our early and mid-career researchers? I'll go to Nadeem first.

1:13:51 Associate Professor Nadeem Kaakoush
So I guess there's a lot of aspects around the grant review panels having helped early-career and mid-career researchers learn how to review grants. But I do believe there are other ways to help them. And so mentorship obviously is one of these areas. But just touching on one of the previous questions around what journals do that could be implemented into grant review, there's also this idea of co-review. So some journals operate with this co-review thing where a senior researcher reviews a paper and invites an early-career researcher to review it with them. And that's so that one-to-one learning and mentorship really helps the early-career researcher learn. And I think that's something that could potentially be integrated into the mentorship programme that the NHMRC does. Potentially there are other aspects also that can benefit early-career and mid-career researchers. I don't think GRPs, at least in my opinion for the Ideas and Investigator schemes, would be the ultimate way to help them through the process. I think structured education will probably be a much better option.

1:15:05 Professor Michelle Kennedy
Yeah, I'm going to agree with Nadeem. We have, at the Lowitja Journal, we’ve built a mentorship system within even the back end of how the editorial system works, where we've been able to identify, you know, "I'm mentoring an early-career researcher through the process of the peer review." And I think that's really useful. I mean, we can use all these advances in technology to do this without it being a face-to-face, sit-down-next-to-someone-and-walk-them-through-the-process situation. I also think I was very lucky. I had a lot of mentors that were able to help me navigate which schemes were appropriate for me when I was an early-career researcher. And I think that maybe our schemes could do a little better in saying, you know, "We want to have two seats on this, particularly for early-career researchers." And we're going to embed that structured support for you within that. I know that I've got a few early-career researchers in my team, and for them to be able to identify, well, where would their first go at this be, I think that's really the tricky bit. And once you get in and build your confidence, you know, you're fine. But being able to know where to start is really a tricky point. So any way that we can try and help identify that I think would be really positive.

1:16:24 Professor Steve Wesselingh
Excellent, thank you. Now moving on to, and I thought there were actually more questions on this, but there are a number, and I'll probably just read out this one question because it does summarise quite a few. How can peer review processes best leverage AI for efficiency while preserving expert human judgement and assessing originality, significance and impact? So if AI is going to test us, how are we going to use AI to respond? I'll just see if someone wants to put their hand up for that question. Nadeem, well done. And then Ginny. I love it. Fantastic.

1:17:05 Associate Professor Nadeem Kaakoush
So I guess, in my opinion, it wouldn't be using AI to review grants. I think that that will be the ultimate mistake. It will be about decreasing the workload of reviewers. So thinking about maybe not even screening, but potentially, I know the NHMRC does reviewer selection through machine learning, but that can potentially be expanded to use some of the more advanced AI techniques to improve that. I think maybe AI can be used to sort of retain reviewer memory. So grants that have already been reviewed, already have five reviewers, and if that's somehow retained by AI, then potentially that can be leveraged to decrease the workload of reviewers as well. The other aspect I can see AI working really well in is potentially crunching the numbers. So we've talked about the ARS system where reviewers look at all the scores, but if AI can be used to sort of crunch the numbers and provide individualised reports to peer reviewers saying, "Oh, you're doing this well, you're not doing this well. You're very different to people on this score," then that's really a good way to leverage AI to help and to be an educational sort of tool for people.

1:18:20 Professor Steve Wesselingh
Thank you. Ginny?

1:18:22 Professor Ginny Barbour
Look, I think in the first instance, one of the things that AI can do is screen out really problematic papers. So this is from the point of view of journals, but also problematic authors. We do know there is a challenge with kind of problematic papers coming into journals. And this is the sort of trust marker approach that RORI talks about in their report, their approach to peer review. So I think that there's some initial screening that AI can absolutely do very early on to look for patterns of problematic authors, patterns of problematic papers. As I said, I also think it has this opportunity to be able to screen the methodology. But for me, at this point, I would not yet trust using it to synthesise opinions. I think that, at the moment, we probably still want to be doing that with humans.

1:19:09 Professor Steve Wesselingh
Thank you. Anyone else want to comment on AI, Doug?

1:19:17 Dr Doug Hilton
Yeah, look, I think there are some really interesting tools being developed, some by prominent Australians, that allow the detection of AI-generated text and images. So I'm really interested in that. And, you know, from what I've seen in early tests, the Australian ones seem incredibly accurate. So I'm really interested in that. It has to be obviously aligned to the NHMRC submission policy, but I think that's interesting. I really like the idea that Ginny has spoken about of using AI maybe across grants as an NHMRC-driven pre-assessment on things that are objective, like statistical tests, like perhaps picking up data that is probably not statistically distributed and therefore likely to be manipulated. I think there's a whole lot of ways that NHMRC could deploy AI in a really selective way to raise flags, not necessarily to, Robodebt-style, make determinations, but to raise flags.

1:20:31 Professor Steve Wesselingh
Thanks for that. Deb, I saw you were going to unmute.

1:20:36 Ms Debra Langridge
I was happy to let Doug go first. But I think, in terms of consumer and community involvement, AI absolutely plays a role. And I think that'll be an ongoing process of discovery. But that people element, the human factor, I think, thankfully, we're still not at a point where AI can replicate it. If it does, it replicates a one-dimensional version. The people factor is a three-dimensional aspect. And writing grants and reviewing grants still requires, thankfully, that people element. So I think that's an ongoing process that we need, just as a checkpoint, to make sure that we don't lose that. Efficiency is important, but we don't lose the people factor across the board. Because you really can tell the difference, and that's the beauty of reading and reviewing great grants, where you can read and see the passion and the reason and the meaning across the board, which I don't think I've yet to see AI fully capture, thankfully. So I think, yeah, it absolutely plays a role, but definitely not the 100% hand it all over.

1:21:54 Professor Steve Wesselingh
Thank you for that. We're getting close to the last five minutes. There was one question there that I find quite interesting: how do we do peer review of multidisciplinary research? There is this convergence issue in science where we want physicists, engineers and medical researchers working together on some of these really difficult problems. But how do we then review those grants? We don't have a lot of time, but a quick comment on that.

1:22:34 Dr Doug Hilton
Maybe just stating the obvious is to have a range of disciplines reviewing the grants. But I also think there are areas where peer review and having lots of people score doesn't necessarily come up with the right answer. And I think for truly innovative breakthroughs, or truly innovative ideas that are going into grants, if you think about some of the big breakthroughs like mRNA technology or immunotherapy, the vast majority of the field viewed those as being science fiction before they were actually shown to be really useful. So I think for really brave science, the last thing you want to do is democratise and average. You need a few people who think they're fantastic ideas and are willing to go out on a limb. So I think peer review has its limits.

1:22:36 Professor Steve Wesselingh
Thanks, Doug. Nadeem?

1:25:18 Associate Professor Nadeem Kaakoush
I was just going to say that a lot of the time we think about peer review as an alignment of expertise, and it works very well for some schemes. But I guess, for multidisciplinary stuff, it's more around aligning perspectives, and it can be the global perspective, consumers and Indigenous voices. But at the granular level, it could be even things like a disease-specific perspective or a method-specific perspective. So if we align perspectives rather than expertise, maybe that's the way to approach multidisciplinary research.

1:25:50 Professor Steve Wesselingh
All right, thanks for that. So we're going to wind up soon, and we're almost certainly going to go overtime because I'm going to ask you all one question. I'll probably need you to keep your comments to 30 to 60 seconds if possible. What will peer review look like in 10 years' time? I'm going to start with Michelle.

Professor Michelle Kennedy
I'm not normally optimistic, but I am optimistic because I think the fact that we're talking about it now, and we're calling things out, and we're being very realistic, is a really positive thing. So I think that it can only continue to get better and utilise, hopefully, things like AI with a lot of caution, which I know we've already spoken about. I love Doug's idea about the interview and trying to think, you know, not lose our creativity in how we can advance things. I hope that's what the future looks like.

1:25:13 Professor Steve Wesselingh
Thanks, Michelle. Terrific. Nadeem?

1:25:18 Associate Professor Nadeem Kaakoush
I think there'll be a push towards more transparency, perhaps better incentivisation. AI definitely will be integrated, for better or for worse, into the different systems. I think perhaps we'll come to an appreciation of, I guess, what the limitations of peer review are in terms of deciding on, you know, some grants. Some grants, you know, they're exceptional, some grants are un-fundable, but perhaps what do we do with that middle ground? So I think we'll appreciate that more and find processes to fix that.

1:25:50 Professor Steve Wesselingh
Thank you. Deb?

1:25:53 Ms Debra Langridge
I think we, well, this is without saying, that consumer and community involvement will absolutely be a part of all grants, not just those that, at this point, that makes sense. But parallel to that will be intentional training and capacity-building for both researchers submitting grants, but also for the peer review process, so that there is integrity across the board in its recognition. And we won't lose the people factor. That'll be evident and at the foundation of whatever grant it is.

1:26:31 Professor Steve Wesselingh
Thanks, Deb. Doug?

1:26:31 Dr Doug Hilton
So, I definitely think it'll be AI-assisted. I'm hoping it will be a little more in person as well. So blending that humanity, that creativity, with the fast-developing tools. And I hope we're also funding much larger teams to work together so we can get away from researchers having to spend 50, 60 or 70% of their time applying for and reviewing grants. So I think if we could do that, then you could have much greater rigour of peer review for a far smaller number of applications for larger teams.

1:27:11 Professor Steve Wesselingh
Thank you, Doug. Ginny, last word.

1:27:15 Professor Ginny Barbour
So there's three things. One is I hope that we'll be rewarding reviewers. I think it'll be part of the kind of expectation that people are appreciated and rewarded for what they do. The second is I think that I really hope that we'll have peer review that's appropriate for the papers. So all the grants, that we get community involved where it's necessary, where we get consumer involvement where necessary. It's not the one-size-fits-all. But the other thing I've been thinking a lot about is, as we move to more of an AI future, we are going to focus less on the final output. And I think we're going to be thinking more about the processes that went into either the grant or the papers. And it's kind of like when we were kids. Do you remember we had to show our working, and you got more marks for showing your working than just the final outcome? I feel like that's where peer review is going to go.

1:28 00 Professor Steve Wesselingh
Thank you very much. So we're going to finish on that. I'd firstly like to thank the panel. Fantastic panel. Thank you for your time and effort in being involved, and for your answers and engagement. It went so well. I'd also like to thank the audience. We had a terrific audience, over 400 or 500 people. I saw a lot of people online, lots of questions before the webinar and lots of questions during the webinar. I just want to emphasise that we want you to continue to give us your questions. We want to hear your questions, and we want them to impact the way we think at NHMRC, and hopefully at the ARC and the MRFF, about the grants process. So please keep sending us your questions and comments. As I said at the beginning, our plan is to continue having these webinars so we can continue to engage with the sector, hear your concerns and hear your ideas. I'm optimistic, like Michelle. I think that in 10 years' time we're going to have a terrific peer review system. I think it will use a lot of AI, but I think that will enable us to concentrate on the really important parts. As Nadeem and others said, we need AI to do some of the harder grant-processing work, and then we can concentrate on the human aspect and the really important parts of peer review, to make sure that we fund the best grants and the grants that are most likely to improve human health. So thank you to the panel, thank you to the audience, and I also really want to thank the staff at NHMRC for setting this up, getting it all organised, and organising the questions and everything else. Thank you, everyone. Have a great rest of the day, and hopefully we'll see you again soon at another webinar. Thank you.

End of transcript.