Investigator Grants Peer Review Mentor (PRM) video

This video was published in April 2022.

Video transcript

Introduction by Doctor Julie Glover, NHMRC Executive Director, Research Foundations

Welcome to the Investigator Grant Peer Review Mentor Video. This video is part of the training and support available to peer reviewers that will help you prepare for the types of issues you'll need to consider during your independent assessments. Our peer review mentors are strong leaders in health and medical research from a range of different research fields who have experience with peer review and have agreed to share their thoughts and approaches.

We've asked our mentors a series of questions based around themes that come up commonly in the peer review of investigative grants, but also in NHMRC peer review more generally, you'll see from our mentor responses that there are different ways that reviewers approach their tasks, but you'll also see that there is a common thread of reviewers striving for quality, consistency, and fairness in their approaches. This video will address some of the key changes made to the peer review of investigative grants. These include a change in publications track record assessment which will shift the focus from the quantity of the publications to the quality of the research and the contribution to science. The other key change that will be discussed by our mentors is the clarification on how to assess those applications that you feel have been submitted at the wrong Level.

It is essential for the fairness of peer review that our assessments of application track records continue to be assessed relative to opportunity. Our relative to opportunity policy recognises that not all research careers are the same and that an applicant's contribution and research productivity should be assessed commensurate with the opportunities that have been made available to them. Please also remember to give full consideration to career disruptions, particularly where these can have an ongoing impact on the research career.

To help us identify important improvements for future realms, we really appreciate your participation in the peer review survey which will be available to you following the completion of your assessments. NHMRC is committed to improving our processes and the feedback that we receive through this process will be used to inform potential improvements to the scheme. We hope that this video addresses some of the key questions you may have and helps you in completing your assessments along with the category descriptors and other documentation you have been provided. Thank you for taking the time to view this video and for your important contribution to independent NHMRC peer review.

Peer Review Mentor introductions

Prof. Patsy Yates: I'm a registered nurse and have been working in clinical education and research aspects of nursing and health services with a particular focus on cancer and palliative care research. 

Prof. Yvette Roe: I'm a Njikena Jawaru woman from the West Kimberley. So I'm a very mid-career first nation researcher whose work is focused in health service redesign. 

Prof. Nicholas Talley: I'm a gastroenterologist. I'm a clinician researcher and I've been doing research into gastrointestinal disease particularly inflammatory diseases recently in the microbiome for quite a number of years. 

Prof. Jennifer Stow: I'm a molecular cell biologist. Been doing cell biology research for over 30 years. 

Prof. Sarah Russell: I'm a immunologist cell biologist.

Prof. Christopher Fairley: For the last 20 years, I've been director of Melbourne Sexual Health Centre and carrying out research in an area of sexual health and public health. 

What makes a good application?

Prof. Patsy Yates: So to me, an excellent research program in terms of knowledge gain is research that makes an extremely significant contribution to knowledge in a field that in terms of addressing a particular problem, it's not necessarily about research that is about the most prevalent problem in our community, but it's research that truly is adding significantly to our understanding of a particular health issue and is ultimately creating advances in how we can improve health of our community.

Prof. Yvette Roe: The characteristics of an excellent research program specifically to knowledge gain includes clarity of point that the researcher is writing about, that it's well supported, that the statements being made can be justified and verified, that I can coherently understand what the researcher is trying to do and how they're planning to do it, and also that there's a clear gap in evidence and the proposed research program will fit into that gap in knowledge. So clarity, acceptability, relevance, and also priority, especially in first nation health.

Prof. Nicholas Talley: I must admit, I am influenced by how easy to read the application is, how clear the hypotheses and aims are, how logically if the program follows a plan, how the importance and the relevance of the research is made very clear, how the methods are going to be robust. It doesn't have to be very detailed for this, but it has to be convincing.

Prof. Sarah Russell: I think it has to be exciting. If you don't walk away wanting to think more about it, then that's a bit of a problem, and unique and with a very clear focus on a big picture of relevance to health.

Prof. Christopher Fairley: So I think there's one overriding characteristic that no grant can get up without, and that is it has to be easy to read and clear to someone who is not an expert in that field. And if you can't do that, if you can't create a grant that someone can read and understand without laboriously going back and forth, you just won't score highly enough to get funded in my particular view.

Using the Category Descriptors

Prof. Yvette Roe: Category indicators are going to be your guide to assessing the document. Each research application deserves to be assessed individually. What distinguishes a research proposal from outstanding to excellent is often those fine tune things that there is clear program of design, there's solid scientific contribution, the team is leaders in their field and is going to make a significant impact, and those things are well justified by the author.

Prof. Nicholas Talley: It's very much a judgement call, and that's why it's so important to look at those descriptors very, very carefully before doing any of the grants. And again, refreshing your mind each time you look at it. Looking at each grant independently, not relating the grants to each other.

Prof. Jennifer Stow: I try to review each application on its own merits to begin with, and then later on, once I've done them all on their own merits, go back and try to compare them with each other and make sure they're all ranked appropriately. I think any truly exceptional applications stand out immediately. 

Track Record

Prof. Patsy Yates: Exceptional to me in publications is really around about that sort of the quality and the contribution that that publication is made to science. So here I look, for example, at how significant and original the publication was, how it added to new knowledge around a particular problem, how it's been used by other people, taken up or influenced other people, for example, or other researchers in the field. Metrics aren't necessarily the key to assessing always how significant publication is. So I look very carefully at how an applicant has described the significance and using the narrative and the explanation, you can often get a feel for how significant that publication has been.

Prof. Christopher Fairley: This is a relatively small amount of text to look at, but it's really important that assessing these publications is done thoughtfully and fairly. And you can't just look at them and say, "Gee, that's great. They’ve got two Lancets, two New England and a JAMA”. Seven out of seven or whatever the top score is. You’ve got to put some thought into it, and the key is what is this applicant's contribution to these papers?

Publications: shift to top 10 in 10

Prof. Patsy Yates: So how I see this is that I'm coming to do a peer review where I'm bringing my understanding of my disciplinary context. I'm bringing my understanding of publication practices. I'm bringing my understanding of the significance of a particular research finding or publication to the field, and that's what I'm taking into account when I'm looking at the person's description and their explanation of what that contribution has been to science. When I'm looking at the individual and how the individual has explained what their particular part of that particular publication is as well.

Prof. Sarah Russell: So remember, you are looking for the quality of research and the contribution to science. Well, to my mind, this is a very welcome shift in emphasis from quantity to quality, and it'll hopefully make it much easier to address the publication record. Exactly how you determine the quality and contribution of each paper is no different than it has been for a very long time, the only change is that there's no longer an opportunity or responsibility to factor publication numbers into the score. 

Assessing Research Impact 

Prof. Patsy Yates: And I think it really does reflect an important change in the way we think about the value of research to our communities. And what's great about the development of this in NHMRC peer review processes is that we've thought about impact in terms of not just knowledge gain, but also in terms of health impacts, economic impacts and social impacts.

Prof. Christopher Fairley: There are three areas for research impact, reach and significance, research programs contribution, and the applicant's contribution. So these three sections, I'm trying to get a feel for, okay so, what is it this person's claiming? What is their individual contribution to it? What is the contribution of a particular research program? So they're ahead of the entire program, it's relatively easy, but they might just be one part of an entire research program and I'm trying to get a feel for those three things, as I read through it. It's very important that it's clearly written, it makes sense, it's easy to read and the claims are reasonable and entirely justifiable.

Prof. Patsy Yates: So this is actually where, as reviewers, your expert knowledge of the context in which that particular impact is being undertaken becomes really important.

Prof. Yvette Roe: Try not to force things that aren't naturally forced because you'd like it to be that. Assess the document with what's in front of you.

Prof. Nicholas Talley: Research impact, again, can be quite difficult because of the varying groupings. Some are easier than others and I think that's one of the things, one of the traps for the assessors here. So it's really important to look at which areas have been picked for research impact and how they've been justified.

Prof. Jennifer Stow: The evidence, I think, is important always and that has to come from the quality of the outputs themselves, any measures or metrics that people can provide. And also, I think, in addition to those things, plain English statements from the applicant describing their impact are really, really powerful for reviewers.

Assessing Leadership

Prof. Yvette Roe: Track record on leadership should be considered in a very diverse way. Leadership might not be at the front of the pack leading a whole team, but it might be attributes that the author has described. Critical ability, able to bring a team together, able to do a collective effort, able to be an advocate or a specialist in a team. So be mindful that when you assess leadership, that you're looking at a very diverse definition of what that might look like.

Prof. Jennifer Stow: And you'd expect leadership to be commensurate with both career level and opportunity, and also the level of impact someone's having. So you like to see that people are not only having impact in their own, right, but exerting leadership in a way that brings other people along with them, provides opportunities for trainees and students and grows research capability around them as they progress through their careers.

Assessing overall Track Record

Prof. Sarah Russell: So rather than saying, for instance, that you can't have an excellent without at least three first author papers a year, I would say there are several reasons why you might score in that top category. And if you miss one, so you haven't got quite that many last author papers, you can still be in the running by, for example, articulating how you were a critical piece of papers on which others were first or last.

Similarly, for things like leadership, you can say you've built up an enormous team and won loads of grants and being a keynote speaker in lots of very important conferences. But then there are lots of reasons why you might not be a keynote speaker in lots of different conferences so they could be personal reasons that you just don't travel or other things. There's got to be many ways where you can achieve these scores and one example would be in leadership where I'll ask the question, "What have you done that goes beyond expanding your own success?"

So I look for a degree of altruism here in the leadership section, and there's many ways and many people do a lot of leadership that it goes beyond their own success, and I think it's very important to reward that. So whether it's that they've realised that Australia is missing a key network that connects up people with a particular skill or resource, or whether it's nurturing a particular cohort of people that needs a bit of help to get them over the line, there's a lot of things that are not just making sure that your postdoc wins a fellowship.

Assessing Track Record Relative to Opportunity

Prof. Sarah Russell: I think it's worth keeping in mind that to provide a score for track record, we are tasked with converting qualitative information on both achievements and opportunity into a single number. At one level, there's a temptation to look at this task as first, scoring achievements as a whole. Second, scoring opportunity as a whole, and then somehow modifying one number by the other. I don't find this a helpful approach. More valuable to me is to remember that our primary goal is to ensure value of the public's money by using past achievements as a reflection of a future potential. And in that light to consider each of the applicant's achievements in light of their opportunity.

Prof. Patsy Yates: So when I'm doing a relative to opportunity assessment, there's many things that I take into account and probably one of the most important points that I like to frame this assessment is in a holistic way because when you think about sort of a career interruption or something that's impacted on your opportunity to achieve research outputs, for example, they're often related. You can't just sort of take a three month gap or a six month gap and say that's going to have the same impact for everyone, because that might also have impacted subsequent research activity that a person may have had the opportunity to contribute to.

Prof. Sarah Russell: And again, if the applicant's been in and out of the workforce due to young children for a few years, I recognise that as well as the reduced publications over that time that's accommodated in the application, there'll also be missed opportunities to recruit students and submit grants and so on and that'll impact on their productivity for years to come. So again, I consider that as part of the decision as to how impressive was their achievement in light of that track record.

Prof. Yvette Roe: Not all of us have a very traditional pathway when we think of our research career, so relative to opportunity must consider not only their ability to be part of research team, to produce outputs, but also some of the personal considerations. So people come into research with a whole range of social commitments, be mindful of that. Research is not a linear trajectory, so be mindful that people will have their own journey and does the author, again, justify their opportunities to participate as a high performing researcher.

Prof. Nicholas Talley: You really need to be mapping out the particular applicant's course and trajectory from all of the material that's available to you in the application. And it's so important here to be as objective as possible, and obviously to look at the applicant circumstances. I think it's important that peer reviewers don't compare themselves or their colleagues or their mentees or mentors as they're thinking about the relative to opportunity scoring. I think it's really important to recognise every situation really is quite different, and you have to judge each career path and trajectory individually.

For example, if they're clinician researchers in their career overview statement or their teaching load if they're an academic and that needs to be taken into account. Sometimes that's a little bit difficult to judge particularly admin load and what that means, but again, hopefully the applicant has expressed this clearly. So you understand how to adjust your scoring depending on all of this.

And then you have to use your experience to understand the impacts of all of these factors, teaching clinical duties, moving labs, research years and the likely productivity issues as you judge it. I think this is one of the most difficult things to do in an application, but critically important.

Prof. Jennifer Stow: So I think the new career context section in the application is very helpful in putting relative to opportunity considerations in context of someone's whole career. So it makes it easier to see what impact career disruptions have had at different times in someone's career. And for careers that might be research focused or teaching in research or clinical, I think the new career trajectory, career statement helps you to judge relative to opportunity for all those different types of careers much more clearly.

And so I think for new reviewers having to assess this, it's a matter of looking at individual justifications and career trajectories and seeing overall how much opportunity, how much time, how much bandwidth people have had to spend on their research and what their productivity is relative to that, what their leadership is relative to that, what their impact is relative to that.

Assessing applications across different fields

Prof. Patsy Yates: You've been chosen because you're an expert reviewer. So you are bringing your understanding of what's relevant to that field, what's appropriate in that field. So my advice is, again, that you bring that understanding of what's relevant.

Prof. Yvette Roe: My work is in health service to research. Often, I'm looking at apples and oranges. So when I look at that, I'm trying to look at each application in its own merits. What I think I know about a field is really important, so I try not to make assumptions. Again, recognising my biases, what are those biases based on? So ensuring that what's in front of me, I read independently that I make assessment and I question why I make those assessments so the applicant, even if they're not in my field, has the best way of getting an independent score from me.

Prof. Nicholas Talley: This can be somewhat difficult, but I think, again, if you are clear in your mind before you start the process of peer review, what your expectation are according to the field or area, I think this can help you to judge these fairly across all of the applications that you are assessing. I mean, it wouldn't be reasonable to expect a basic scientist to have an impact on clinical practice directly. It does happen actually, it can happen, but that would not be expected and in fact will be unusual. So if it's basic science or research programs that you are looking at, that is a reasonable expectation to have in your mind before you start.

Prof. Jennifer Stow: So judging research opportunities and quality of impact and things like that across different career paths can be quite difficult. I do think the new career context statements make that easier. It's easier to see how much time clinicians have had to spend on research and compare them to people who are doing teaching in research or research only. I think given the whole package of somebody's application, that's what's important to take into account when you're comparing clinical researchers, public health researchers, basic researchers, and in all their different career positions. 

Benchmarking applications

Prof. Yvette Roe: Be mindful that the applicant has spent a significant amount of time to present a program of work, and they deserve your time and commitment to do a solid assessment of it. I will usually run through and do a first read of all the grants that I'm reviewing, and then I'll make time to go actually a deep dive and then review them independently in a lot of detail. This is a time consuming task, but we want to make sure we make solid informed decisions regarding these applications.

Prof. Jennifer Stow: Reviewing a lot of different applications can be quite daunting. I personally scan through them in the beginning and try to find one that's a really clearly very good one and well, that's a weak one. And I do those first to set the benchmarks at either end, and then that helps to position the others in between those outliers. Then I go back and sit down with them all and look at them all and see how I've ranked them according to each other, and if there's ones that are anomalous in that whole cohort, that I might adjust them.

Prof. Sarah Russell: I think there can be a concern with letting your view of a person's track record influence your judgement of the knowledge gain section, the proposal. For this reason, I try to score the knowledge gain section before the track record and sometimes I try to do all the knowledge gain sections, create a score for those, and then go back and review each applicant for the track record sections after that. And that way you can get your mind a little bit more focused on one particular aspect and achieve a bit more consistency that way.

Prof. Christopher Fairley: The first point I’d make is be under no illusion that this is hard work and requires a lot of time and a lot of commitment. What I do to try and make them as fair as possible is I tend to do the sections all together. So when I'm doing publications, I’ll have a quick read of the application, then I'll score the publications for that one. I'll go to the next one, quick read, score the publications. I'll score all the publications of all of them together, then I'll come back and I'll do the research impact all together. Then I'll come back and do the knowledge gain all together.

So I try and not do the application, if you would like, in a sort of linear way, but I'll go across the same section across all the applications and come back and do it that way. And I think for me, that was much more helpful.

Assessing applications at incorrect Level

Prof. Patsy Yates: What do we do if we sort of feel that it is not quite right? Well, I think that's where we need to take into account that we are doing an assessment on relative to opportunity. So in some ways, by taking that relative to opportunity assessment into account, we're really going to score people on where they're at in their career, not necessarily needing to worry about sort of benchmarking in its particular Level, it's really we are looking at their career and where they're at. 

Prof. Sarah Russell: Well, you are asked to take into account the Level they applied for, but the guidelines are very clear that it's up to you to take into account a range of considerations when scoring. So if the Level applied for doesn't feel right in the context of their descriptions of the category level justification and all the other relative to opportunity considerations in the application, I would probably give it a much lower weighting. And from that perspective, I think you should be very comfortable ranking them against the cohort that you feel they do belong to. 

Single Summary Statement/Feedback

Prof. Patsy Yates: We want to enable them to develop their research in ways that continues to improve. So if you put yourself in their shoes and you think about, well, why are you providing this feedback? I think that's always a really good starting point. You should be able to defend that feedback if you were providing that feedback face to face to someone, how you would be able to defend that.

Prof. Yvette Roe: We know that research can be better, it can be tighter, it can be more concise and we want it to make an impact. So be mindful when you're writing those comments, how are you going to provide constructive feedback to the applicant that allows them to enhance better their grant, whether it's successful or unsuccessful, we've got a community and we want to support better research. How will your comments contribute to that?

Prof. Jennifer Stow: We can in that space highlight strengths and weakness particularly if there's something we think they have taken the wrong approach to in the application or left out some very meaningful information in their justifications or their career trajectory statements.

Prof. Christopher Fairley: You should stop and think for a minute what was it that made me score this application the way I scored it? I think you've got to focus on why it was that you felt that this application wasn't very good and then provide some constructive evidence. The best way to do that is to start up the sentence with the application could have been improved if there was more. So you're not saying there was none or it was terrible or anything, just saying if there was more attention paid to explicitly describing the evidence that supported their claim, which I think was probably the single thing that's done the worst.

Unconscious bias

Prof. Patsy Yates: The most important way of addressing unconscious bias is actually through self-awareness, and that's so critical that you are aware of what biases you might bring to the assessment of any application. I think once you're aware of them, it does, again, help you when you are looking at your overall assessments to ask yourself that question, have any of my biases perhaps potentially influenced the way in which I've undertaken that assessment? I think unconscious biases is also something that one of the ways in which I have tried to address unconscious biases in the past is just simply by talking about those with other people as well.

Prof. Yvette Roe: Do not be frightened by your unconscious bias, we all have it. We have it when we walk down the street, we have it when we look at a TV show and we have it when we review applications. Be mindful about some of the things about your assumptions that you're making about either the discipline, how the problem might be solved, who's in that team. Address those biases, be honest with and ask yourself, "Can I do this, an independent review to the best of my ability?" So the thing would be to address them and also make the honest assessment, how does this impact on the score that you're giving them?

Prof. Christopher Fairley: I think if you're aware that you might have unconscious biases, so you make them conscious, then you can deal with them.

Overall approach

Prof. Patsy Yates: I think we take for granted how important it is that good peer review is what will enable us to undertake the best research in the community to achieve the best health outcomes for our community. So it's not something that's trivial, it's a really important part of the research process, and I think that means taking on board all of the guidance and advice and support and development opportunities that NHMRC is providing peer reviewers, because that means that that service that you're providing as a peer reviewer will have the best possible impact that it can.

Prof. Yvette Roe:The first time that you will do this, you'll be nervous. And you should be, it's a big responsibility. The thing is NHMRC is about ensuring that we're supporting the best research, but we're also encouraging the best researchers. As a grant reviewer, this is a really exciting opportunity for you to contribute your expertise, your understanding, and your vision for research in Australia. Ensure that you will be able to provide a really insightful, independent and terrific experience both for yourself and for the applicant.

Prof. Nicholas Talley: Our job as reviewers is to do the very best we can by each individual to avoid bias where we possibly can to consider each proposal fairly individually and as expertly as we can. 

Prof. Jennifer Stow: So it's important not to be overwhelmed by the process, take a quick scan through the applications when you get them, but then assign times to go through them in detail and a time for looking at them all at the end. 

Prof. Sarah Russell: I think it's a really healthy thing in the system that we've got at the moment, that there is an opportunity for different people to have different views. And there's no question that there are many ways to be a great medical researcher, and there are many ways to perceive what is a great medical researcher. 

Prof. Christopher Fairley: These applications matter. These are people's lives and livelihoods, and it is important that we do what we can to get them right. So these things matter and you shouldn't take this job on, unless you are prepared to do it properly. And to do it properly, you need time to think and do it carefully and be self critical of your reviewing along the way, and try to do it in a way that is as fair and reasonable as possible.