Dr Julie Glover, Executive Director, Research Foundations, National Health and Medical Research Council
Julie Glover: Welcome to the Ideas Grants Peer Review Mentor video. This video is part of a new process that we’re undertaking for NHMRC, where we’re looking at ways to draw on the knowledge of experienced reviewers. This will help to prepare reviewers for the types of issues they will need to consider during their independent assessment of Ideas Grants. Our peer review mentors are strong leaders in health and medical research from a range of different research fields who have deep experience with NHMRC peer review and have agreed to share their thoughts and experiences.
We’ve asked our peer review mentors a series of questions which were based around themes that came up frequently in peer review and are particularly relevant to Ideas Grants. You’ll see from the mentor responses that they have different ways of approaching their tasks but you’ll also see that quality consistency and fairness are all underpin the different ways they approached their work. Thank you for taking the time to view this video.
We hope that it addresses some of your key questions and helps you in completing your assessments along with the category descriptors and the other documentation that you have been provided. Thank you for your contributions to NHMRC peer review.
Peer Review Mentor Introductions
Professor James Bourne, Monash University
James Bourne: I'm a professor in neuroscience at Monash University. Research wise, my work is in basic discovery science, and animal base research.
Professor Rosalie Viney, University of Technology, Sydney
Rosalie Viney: By background I'm a health economist, so I work in health economics research and health services research. I am the Director for the Centre of Health Economics Research and Evaluation at University of Technology, Sydney.
Distinguished Laureate Professor Nick Talley, University of Newcastle
Nick Talley: I'm a clinician investigator, I'm a gastroenterologist, interested in neuro-gastrointestinal disorders.
Associate Professor Misty Jenkins, Walter and Eliza Hall Institute
Misty Jenkins: I'm a laboratory head at WEHI. I'm Gunditjmara.
Distinguished Professor Patsy Yates, Queensland University of Technology
Patsy Yates: I'm Executive Dean Faculty of Health at QUT. I'm a nurse by background.
What makes a good application?
Misty Jenkins: I think first of all a really good application is really clear storytelling. So a great application realises that it may not necessarily be reviewed by an expert directly in their field. It has to be super easy to read. The research questions have to be crystal clear.
It has to be an application that's underpinned by approaches that are outlined in a way that they can be really easily evaluated for feasibility. So when this is done really well, you can review a grant in almost any field.
James Bourne: So, for me personally, a great application is one that you feel that you have understood the aims, even if not directly in your field of research, but it's well justified, clearly articulated, and has the potential to be game changing. Further, the application is written in a way that allows you to easily interrogate those four review criteria. And in a way, it is like a book, it makes you want to read past the first page.
Rosalie Viney: First of all a good application has a really compelling idea, a really clear articulation of the problem that the research will solve, and why it's something that needs to be solved.
So that's, that's in the way that the problem is described.
It'll explain the importance of the health problem and the potential benefits and it will have really clear and compelling methods.
Applying the Category Descriptors
Patsy Yates: So I think the way that I go about this is I first start, you know, make sure that I do familiarise myself with the aims of the scheme and particularly this, the Ideas Grant scheme, which is new, and it's got some very unique aspects of it. So I, I try and understand, I read, I read in detail the guidelines for assessors and guidelines for applicants, and I really make sure that I'm very familiar with the category descriptors and how they apply for this particular grant scheme and not just make assumptions that I'm applying those in the same way as what I might've done for, you know, when I was doing project grant reviews, for example. So I do recommend that you take some time to, to familiarise yourself with those aspects.
But personally, what I try and do is I try and do applications, you know, in groups in a relatively close period of time together. So, and I keep notes on applications, particularly where I'm not as confident in where I'm sort of scoring. And so I might go back to those applications again, at the end of completing the grants that I've been allocated, just to be confident again, that I'm applying the category descriptors in consistent ways.
Rosalie Viney: So the first thing I want to say about category descriptors is they're really absolutely critical. And in my experience as a panel chair, I found myself a number of times, going back to saying, "Go back and look at the category descriptors.
When you're thinking about the scoring is what you're saying matching up with those category descriptors". And I think that's actually really good advice to take in, to thinking about the reviewing of a grant application as well.
So the first thing I'd say is it's really important to be familiar.
So read and familiarise yourself with the category descriptors before you approach the application and think about what the different categories under which you're assessing or the different criteria mean and how they differ. I think they provide a really useful way of calibrating across applications and across different problems, different researchers and different methods.
So what I would say is that it's useful to read the application through, make some notes on the key elements, what the application's trying to do, identify any questions that you might have, and then go to the category descriptors and try to write the comments against the scoring criteria. So the comments that you actually write about the grant should try to match against those scoring criteria and against the category descriptors, noting any issues or problems or anything that you think is really compelling, the significance of the problem, those sorts of things.
Assessing Multidisciplinary Grants
Patsy Yates: You often do get a range of applications. That, that's typical and it's the same in any grant scheme, and, and you can't be an expert in all of those areas. But that doesn't mean that you should necessarily say that it excludes you from a good, undertaking a good peer review.
You know, you’re reviewing aspects of the quality of the science. You're reviewing aspects of the capability of the team. You're, you're reviewing aspects of the innovation. And so, again, a well-written application should give you enough information to be able to judge those, even if you're not, you know, absolutely, that's directly, you know, your field of expertise.
James Bourne: So, the category descripts are essentially used as a guide, and they're general, and cover the full gamut of research activity sent through the NHMRC Ideas Grant Programme.
Therefore, as the particular expert, you have to evaluate the application accordingly. Some reviewers feel that they may not be expert in some aspects, but in these cases it should be clear that there is a very good capability, in terms of the team's expertise, and the resources and infrastructure available, to complete the experiments. Further, highlighting the potential risks and their management is important.
For me, the difference in the research quality is guided by a very well justified research plan, including the aims and hypotheses and the one that would be highly competitive internationally.
Identifying and Managing Risk
Misty Jenkins: Projects that skirt around risk, have not thought deeply enough about their project and they will not have mitigated them well enough and will be scored accordingly. And I'm talking about obvious things like a project might rely on collecting human samples. And then the risk is that you don't collect enough samples to reach power. And if a study can't reach power and then no conclusions can be drawn, then it's clearly a waste of investment.
So these sorts of, outlines of risk are absolutely essential for researchers and as reviewers we are always looking for, for, have the researchers really clearly outlined their risks, because every single project has risks. The worst, the worst proposals are the one that say, ones that say, "There are no risks. This is perfect". And of course it's not. So, we all have risks.
We have risk of technology failing, we have risks of personnel issues. We have risk of, you know, samples and all sorts of things and so you need to really provide evidence that you have the right infrastructure, and study design.
Patsy Yates: I think one of the principles underpinning the Ideas Grant scheme is recognising that perhaps some of our most important discoveries may not necessarily fit within that paradigm that it might be a little bit more risky.
And so I think that means, you know, in the research plan, we should sort of see really clear indication that the researchers have given attention to what risks there might be, and that they've given clear indication to how they’re monitoring, preventing those risks if possible, monitoring for those risks.
And that should sort of really be built into the research design. And that gives you confidence that, that the outcome is likely to be achieved.
James Bourne: There will inherently be aspects of the project which are more risky, and novel. These components should have been acknowledged by the applicant, and highlighted, as to how the risk is being managed, and that the risk is not so great, that the results underpin the project aims failing.
Assessing Research Quality
Rosalie Viney: I would want to see that it's absolutely possible to collect the data that are required to address the research question, that really strong consideration’s been given to issues such as sample size, the feasibility of collecting the data, the analysis methods and the appropriateness of the data to the question.
So I guess, it's not that you necessarily have to have done it but you’ll need to have shown that you’ve really thought about the fact that it is possible to do what you're saying that you're going to do in this research project, and you've made a convincing case, that it will be possible, and that you know what the steps are going to be.
Nick Talley: It's very much about the hypothesis and aims are clear and sensible and the methods that will be applied to address the hypothesis and aims are robust and will address, they will actually address them, and that's verifiable in the application from the information provided by the applicant, or applicants.
I also think it's about you know, looking at, for research flaws.
If there are serious design flaws for whatever reason which can't be really mitigated, that is an issue about research quality, and, and, and you can judge the application on this, and of course that can lower the score significantly, or substantially, if, if there are what you consider to be fairly fatal design flaws or major design flaws. And one of the, one of the obvious things is if you've got three aims and one of them is totally co-dependent on another aim and perhaps one won’t work, then that clearly could be a risk and that may also affect the scoring that you're, that you’re doing.
So, so I think it's about being convinced by the application in the absence of preliminary data, the work can be done, can succeed and can answer the questions.
Assessing Innovation and Creativity
Patsy Yates: So I think innovation can be seen in many different ways. And I think the definition that's being provided, it reminds us that it can actually be about a novel concepts or, or, interventions, it can be about new applications of those concepts, or it can also be about integrating other areas of knowledge to a new field. So all of that should be considered, I think. And probably when I'm assessing this, I think the one most important area should be that the applicant themselves should be letting you know, what's that point of difference. You know, how is it that this really is demonstrating that innovation.
James Bourne: Innovation is more about the ideas and scope of the project, rather than the tool chest being used to get there. For example, a project is not innovative just because it's using the latest whizz bang machine.
Misty Jenkins: I think to score well in innovation and creativity, you're really looking for a project that is not necessarily just making small and incremental advances, but really approaching a scientific problem from, from a new angle. It might be using new or cutting edge technology that hasn't been applied to a problem before. It might be just, cross-disciplinary merging of a variety of well-founded techniques that just haven't been integrated before in, in that dynamic. Science for the most part is incremental, you know. It relies on building of new ideas on foundational knowledge.
Nick Talley: I think it is field specific, basic science innovation differs from clinical innovation and health sciences innovation.
You can't expect the basic scientist to impact clinical practise directly, although it can happen.
And it would be unreasonable to expect a clinician to mechanistically dissect the biology of clinical, of a clinical practise issue but these pathways do overlap and I think you can look at these issues from an innovative perspective and come up with a score that's fair and reasonable.
Assessing Significance
James Bourne: So, the first thing to understand is that the application would be highly significant if, if it advances knowledge in the field, or have a clinical translational outcome.
Both of these have equal weighting, and the Ideas Grant Scheme is there to fund research at any stage of the discovery pipeline.
Isolating the significance of an application in Section D alone, should help in restricting the overlap with other aspects under review, such as the research quality innovation. From this section, it should be apparent that the application has demonstrated that it addresses an important issue, in relation to the field of research, or health. The outcomes will be significant, in terms of the knowledge gain, practise or policy, and need to output, such as publications, IP, conference presentations, et cetera. It should be clear to you as the reviewer, the importance of the advances to be made, and that you can, alone, as well, see their app impact.
Rosalie Viney: I guess the issue here is that scientific significance is something that will advance knowledge in a way that changes the science, that adds to the science and can help to solve public health problems or health problems but the translational significance is about how that would actually change what happens in the health system.
So whether it's the significance in terms of being a new discovery that might lead to a new medicine, or a new treatment, or a new way of doing things, or a new way of funding, whatever it is.
So something that, that has the capacity to change what happens in the health system, to people in the health system, and make a difference to their lives.So that's the translation element whereas I guess the, the scientific significance might be I guess the background material to that.
The things that, that change knowledge in a way that could then lead to those changes and both are really important and both are really clearly things that are important for NHMRC, and for the health system.
Assessing Capability
Rosalie Viney: I think the question about team capability and the criterion of team capability is, is this the right team to do this research? So do they have the skills, the experience and the capacity to conduct the research? Is there evidence that they've been able to undertake research in the past? So, it's not necessarily about having a really long track record and indeed I, I like the fact that it's not a track record based assessment but that doesn't mean that you don't want to see that the right people and the right mix of skills, the right group of people are identified in the project.
So for example, if the research involves consumers, you really want to see that consumer engagement. If you're using qualitative methods, you really want to see that there's somebody with experience in those qualitative methods.
You want to see that there's knowledge of the health problem that's being addressed and understanding of that health problem, so the right set of experience there. And indeed I think often we might find that the people who are the right people to conduct the research might not have really long track records but what they've got is on the ground experience and on the ground understanding of what's needed to address this health problem.
I guess the other things is are there appropriate mentors or associate investigators to provide the backup to the research team, and I guess is there evidence in the description of the team's capacity to conduct similar research or of having worked together and having been successful in conducting research. Or if they haven't worked together, that there's a reason why this group of people are together to do this research and that there's some evidence of their capacity to work together in the future.
James Bourne: So, the capability of the team should be evaluated from the statements in Section E.
So, while not necessarily publishing together, it should be clear that the CIA can lead the team, and achieve the aims of the project, in conjunction with other team members, including both the CIs, and AIs. There should also be evidence of them of having technical resources, infrastructure and facilities required to achieve the project. One important aspect is that there is a balance of integrated expertise. And this should encompass the stage of career of some members of the team, that it's not just weighted to senior people. Just because an individual is senior, and a world leader in the field, doesn't necessarily mean that they are more likely to achieve the aims of the project, compared to an upcoming leader in the field.
Patsy Yates: Capability is where we assess the skills of the team in being able to deliver on that grant. And, and that doesn't necessarily mean that they need to have had an extensive track record of publications.
It does mean, though, that what we're looking at, and when we're assessing is, is that we're confident that that team can have put things in place that they've got the skills to manage the project, to deliver the outcomes, to manage the risks that might be inherent in, in an Ideas Grant. An overall assessment and a holistic assessment of a range of different dimensions. Applicants should be demonstrating and putting in what their skills are, what they're bringing, what the resources are, that they're bringing to address this particular research question that they're answering.
Mitigating Unconscious Bias
Misty Jenkins: Reviewers really, absolutely must be aware of their own biases. Just as we would say in the leadership literature, you can't lead others until you can lead yourself.
We all need a bit of introspection here. Everyone has bias, everyone. It may be biased for a particular approach or field, it maybe gender, it may be a bias just to score highly for CIs that have a bag of nature papers. You may have a generational or a hierarchy bias to score highly for more senior well-established CIs.
So first of all, it's important to be aware of what kind of bias you do have when you are assessing track records, because you will have one. Only then by identifying this risk, I think you can mitigate it. So here you're looking for capability, you're looking for evidence the CI has a balanced and appropriate mix of expertise.
James Bourne: OK, so, unfortunately we all possess bias, especially in terms of people, including gender, stage of career, and academic achievement. So, it is important to try and be aware and remind yourself when reviewing the application of this.
I would suggest to try and look at the project as a whole, and not the individual parts. But evaluate the project before the team. Try not to necessarily rely on an investigator’s previous research, as a proxy for the quality of the proposal under review.
Indigenous Research Excellence Criteria
James Bourne: So, there are four specific categories in the indigenous excellence research criteria, community engagement, benefit, sustainability, and transferability in building capacity. It is important to understand the full gamut under these criteria, and not to be biased by more traditional types of research. For example, community members may be applicants on an Ideas Grant, because of their ability to assist with data collection, and achieving the aims of the project, but may not necessarily have the traditional academic qualifications that we're more accustomed to seeing. Further, building capacity is not necessarily the same either, as this could be leadership, or a cultural skillset. So, it's important to look at the broader picture, rather than the individual, traditional mechanisms of categorising an application.
Misty Jenkins: The indigenous research excellence criteria in my mind is… sits across all of the other four criteria in the Ideas Grant scheme. What I mean by that is that an indigenous lens is applied and that criteria is applied to all of the other rubrics and all of the other category descriptors.
For example, community engagement, you quite often see community engagement layering across the research quality category descriptor, because when this is done really well in projects, what it means is that indigenous people or Aboriginal and Torres Strait Islanders have been involved in a co-design way in the conceptualization development approval of the research proposal itself. It's a really different way of thinking to mainstream studies. It often involves a co-design component. Again, research and community engagement criteria also speaks to feasibility.
The research question might be founded on excellent hypothesis, you might have a really good study design and approach, but if that means going out into communities and taking samples or doing interviews or doing lung function tests or whatever it might.... whatever the study might be, and you haven't engaged with that... with any communities, then that, then you have, you have feasibility issues and so it will be scored accordingly. And so it's really important to get all of those elements right when working with Aboriginal people.
The next criteria around this rubric is the, the benefits, so the potential health benefits for this specific project must demonstrate that it will actually benefit Aboriginal and/or Torres Strait Islander people. Now this benefit maybe in health, but it also might be in policy. It might be in quality of life. It even could be financial, but the benefits might be short term and immediate, they might be long-term. Whatever they might be, but they... the, the application should really very clearly define and articulate what the benefit to Aboriginal people will be.
The next rubric is around, the criteria is around sustainability and transferability. So it really must outline how it will lead to sustainable change. This might mean in the community, it might mean from a policy perspective. Will the work continue beyond the life of the grant? Does it have the capability to result in real change? And the final and the most important criteria, and I think, you know, one that can be really difficult for non-indigenous assessors to review is capability.
The projects must absolutely demonstrate how Aboriginal and Torres Strait Islander peoples, communities, and researchers will develop the relevant capabilities through partnership of the project. And really excellent proposals have indigenous leadership and participation embedded within the proposal. They might be indigenous CIs, they might be projects that are working in particular communities that are employing healthcare workers or nurses or teacher or whatever the project might be.
It's a really essential element to embed the indigenous workforce and capability within the project design itself. And that can really also speak to feasibility. So for example, a project that might have a really exceptional study design but requires working with Aboriginal people and if they haven't been embedded within the proposal from the start it's less likely to lead to really sustainable outcomes.
The authors really need to provide evidence that the research will be conducted in a culturally safe way otherwise it can be scored down as well.
And on the flip side reviewers are really good at determining level of Aboriginal involvement being authentic and not tokenistic.
For example putting someone on, on as a CI on, on a grant, will be, will be, will be seen through, and be pretty, fairly transparent if that’s not an obvious person with the appropriate mix of skills to be lead... to be on that specific project.
It’s a really important criteria. It is something that reviewers struggle with to do well. Indigenous reviewers do this really well.
So I would encourage anyone if they're really struggle, if they're are reviewing grants, and they are struggling with this to, to reach out and to, to use those resources and, and talk to your Aboriginal colleagues to receive some advice on how best to apply the criteria.
General Comments
Rosalie Viney: As a peer reviewer, I think it’s really important to remember that you are also often a grant application and you want your grant to be reviewed in a way that’s fair, and I guess, so to bring that fairness to the way that you review other people’s grants, and to try and sort of put yourself in the, the shoes of the persons who is the applicant, who has put all that effort in and try to be as fair as possible to that application in a way that you’d want your own applications to be treated.
Nick Talley: To the assessors, it's an absolute privilege to be an assessor for the NHMRC. It really is a privilege it, it's, it's an important role, it's a role you learn a lot in so you actually get a lot out of doing this as well as giving a lot to the research community.
But it's so important to take it very, very seriously.
I, I know that's usually the case but it is so important as I, as people’s careers are affected by the decisions of NHMRC, and it's critical that the best research, the very best research does get funded.
So, that's the role of the assessors to sort out, and that's what is so critical for the whole sector.