Independent review

Independent review can improve the quality and enhance the legitimacy of your guidelines

Independent review

Independent review is a chance to receive feedback from individuals with expertise and perspectives that may not be represented in the development group. Independent review includes review by peers, experts (e.g. with clinical, methodological or technical expertise) and others (e.g. consumers, expected users and other stakeholders). It presents an opportunity for specialists in relevant fields, including non-experts, to comment on a full draft guideline, including recommendations, before it is published.

Independent review is mainly used in guideline development to assess:

the rationale applied in searching for and examining the body of evidence
the quality of the evidence on which the recommendations are based
the rigour of the development process
the usability and acceptability of the recommendations and the overall guideline
the applicability and consensus of international perspectives.

In addition to quality assurance, independent reviews can improve uptake of guidelines by increasing the legitimacy of recommendations, signifying to users that they are a trustworthy resource. It is recommended that at least two reviewers are engaged in the process— guidelines represent the synthesis of a large body of evidence and their quality can be difficult for a single reviewer to judge.

Independent review is a routine step in the guideline development processes of international bodies such as the National Institute for Health and Care Excellence (NICE), the Scottish Intercollegiate Guidelines Network (SIGN) and the World Health Organisation (WHO). NHMRC also regularly invites independent experts to review draft guidelines and information papers that it develops, as well as commissioning independent clinical and methodological experts to assess the quality of externally-developed clinical practice guidelines submitted through its approval program.

Given that one of the purposes of independent review is to invite new perspectives from individuals not otherwise involved in the guideline’s development, this module is closely related to Engaging stakeholders, Consumer involvement, and Public consultation. Table 1 outlines the principles that should underpin all aspects of independent review in guideline development.

**Table 1.** Principles of independent review in guideline development
Transparency	Independent review processes should be transparent and any specific questions or frameworks that are provided to reviewers should be made publicly available.
Appropriateness and balance	Reviewers should have relevant expertise and experience to inform the topic. There should be a broad representation of perspectives where possible to reduce the risk of bias.
Confidentiality	Confidentiality is important for the fairness and robustness of independent review. The reputations of reviewers, the guideline development group and stakeholders should be protected.
Independence and impartiality	Reviewers should provide objective, impartial advice. There should be appropriate processes in place to identify and manage any conflicts of interest.
Quality and excellence	Independent review should be undertaken with reference to international standards and established quality indicators used where possible.

What to do

1. Decide which aspects of the guideline are to be reviewed

Due to the length and complexity of many guidelines, it may be unrealistic to expect a single reviewer to undertake a thorough assessment of all aspects of the guideline and the development process. It is best to focus independent review on specific aspects of a guideline that are considered priority topics and on which the reviewer is knowledgeable. For example, a clinical expert may be asked to comment on recommendations involving treatment strategies, a consumer might provide comment on the relevance and acceptability of recommendations and an expert reviewer with a background in research methods would focus their efforts on critiquing the evidence appraisal and synthesis (Vintzileos, Carvajal et al. 2013; SIGN 2015).

It is important to remember that independent reviewers are responsible for appraising the draft guideline and cannot modify the scope or questions on which the recommendations were developed.

Table 2 provides a summary of the different aspects of a guideline that could be the focus of independent review (based on discussion in Shekelle, Woolf et al. 2012).

**Table 2.** Independent review of guidelines (based on discussion in Shekelle, Woolf et al. 2012)
Focus of review	Relevant reviewers	Examples
1. Comprehensiveness and balance of the scientific evidence	Guideline methodologists Experts in systematic reviews Experts in relevant discipline (e.g. clinician, researcher, policy maker)	Invite experts to assess the quality of the systematic review on which the guideline is based. They can also be asked if they know of existing guidelines or evidence that has not been cited. Reviewers might identify: omission of relevant terms and sources from search strategy mistakes made in describing studies or their results previously unaddressed risk of bias methodological (including statistical) errors (e.g. in meta-analysis) imbalance or asymmetry in presenting the evidence.
2. Certainty of evidence and rationale for the recommendations	Guideline methodologists Experts in relevant discipline (e.g. clinician, researcher, policy maker)	Invite experts to assess the way in which the evidence has been synthesised and the validity of the rationale used to formulate the recommendations. For example, reviewers might: comment on the appropriateness and specificity of the recommendations call attention to biases, political pressure or other factors that may affect the group’s judgement.
3. Relevance and feasibility of recommendations	Consumers, expected users (e.g. clinicians), and other stakeholders (e.g. program managers, health care administrators, policy makers) Experts in relevant discipline (e.g. clinician, researcher, policy maker)	Invite consumers, expected users and other stakeholders to comment on how useful the recommendations are in their current form for guiding their decisions. They might also provide feedback on the clarity of the recommendations, the supporting information and the accompanying resources (e.g. whether the intended users, settings and target populations are clearly defined). Reviewers might also be able to identify potential problems with the feasibility of recommendations, such as issues related to: lack of providers or technology to support them implications for reimbursement or medico-legal liability impracticalities created for information systems or performance review criteria upcoming legislation or policies that bear on the topic political resistance the recommendations could face.
4. Rigour and transparency of the development process	Guideline methodologists	Invite experts to assess the overall trustworthiness and legitimacy of the guideline development process. Reviewers might identify issues such as poor documentation of key decisions, inadequate management of conflicts of interest, or failure to engage with key stakeholder groups.

2. Choose specific questions and existing frameworks

In assessing the quality and trustworthiness of a guideline, it is intended that reviewers use their expertise and experience to provide feedback on particular aspects of the guideline according to the scope of the review. You can assist reviewers in this process by providing them with existing frameworks or by referring them to international standards for guideline development. For further information on potential frameworks and standards, see the Useful resources section of this module.

For example, if the purpose of the review is to assess the quality of the systematic review of the evidence on which recommendations are based, consider referring reviewers to the Cochrane handbook for systematic reviews of interventions, the AMSTAR checklist or the PRISMA checklist for reporting of systematic reviews and meta-analyses. Related advice in the GRADE handbook (including criteria in chapter 8 for determining whether the GRADE approach was used) and in modules in the Develop section might also assist reviewers assessing the rationale and process of forming recommendations based on the evidence.

For reviewers assessing the clarity and feasibility of recommendations, consider referring them to the Cochrane style guide for advice on clear language in scientific communication to assess guideline usability.

There are a number of frameworks that you can use to assess the quality of clinical practice guidelines (Siering, Eikermann et al. 2013). If you are seeking NHMRC approval for a clinical practice or public health guideline, use of the AGREE II tool is mandated. The NHMRC procedures and requirements (and the methodological review template based on the same standards) can also be referred to when assessing overall guideline quality.

It is helpful to provide the reviewer with specific questions in order to focus the review. For example, NHMRC asks clinical experts three questions when assessing the quality of clinical practice guidelines:

Has the appropriate evidence been identified and reviewed in line with the scope and clinical questions covered by this guideline?
Have the risks and potential harms of recommendations been fully considered in the context of clinical practice, including any medico-legal implications?
Are there relevant international or well referenced guidelines (recommendations) on the same topic that conflict with this guideline? If so, is this acknowledged and justified in this guideline?

Similarly, Kidney Health Australia’s CARI Guidelines are subject to independent review based on a series of specific questions such as whether the reviewer agrees with the recommendations and grading of evidence.

3. Identify suitable reviewers

The reviewers you select should have relevant expertise and experience and be independent of the guideline development process. They should have an understanding of the condition on which the guideline is based and context in which it will be implemented, from the perspective of an expert, consumer or other stakeholder (e.g. policy maker, health administrator).

Members of the guideline development and working groups will usually be able to suggest experts in the health discipline of interest (e.g. clinicians, researchers, policy makers). However, it can sometimes be difficult to find local experts who are not already involved in developing the guideline so it can be useful to look outside Australia for experts to bring an international perspective.

If consumer expertise is sought, consider contacting a relevant condition group or consumer organisation and ask them to nominate someone suitable to be an expert reviewer (see Consumer involvement module). In some cases, it may benefit the development process if the commissioning agency or steering committee independently recruits independent reviewers.

4. Contact reviewers

Consider contacting potential reviewers at least two months in advance of sending them the draft guideline so that they can plan for the work involved. This will also leave you enough time to seek alternative reviewers if selected individuals are unable or unwilling to contribute, or if they are unsuitable due to conflicts of interest.

It is important to provide sufficient information to individuals when requesting their involvement in the review process. This material should include specific information about the guideline and the development group’s expectations of the reviewer, including:

basic information about the guideline, including the name of the guideline, its aims, and the organisations funding and developing it
the format of the guideline, its expected length (e.g. number of pages) and any other information that will be provided with the guideline (e.g. administrative or technical reports)
the scope of the review, including any specific questions they will be asked to answer and any frameworks they will be asked to refer to
the date they should expect to receive the guideline and the length of time they will have to complete the review (e.g. four weeks)
information regarding the guideline development group’s declaration of interests policy, to which they will be expected to adhere (see Identifying and managing conflicts of interest)
how they will be acknowledged in the published guideline and whether they should expect to be compensated for their time
any plans to publish feedback received from reviewers (if so, asking the reviewer for permission), and whether or not it will be de-identified.

Independent reviewers should be subject to the same declaration of interests policy as members of the guideline development group. Once an individual has confirmed their availability to review the guideline, you should request that they complete a declaration of interests form in advance of confirming any arrangements.

If the guideline development group identifies any conflicts of interest, they are required to either exclude the individual from the review process or develop a strategy to manage the conflict (e.g. by excluding the individual from reviewing recommendations involving a subject for which he or she has a declared financial interest). Avoid publishing the conflicts of interest of individual reviewers; however, an aggregate statement regarding the conflicts of interest of independent reviewers would be appropriate.

5. Record and consider all comments received

Comments received by independent reviewers should be presented clearly for consideration by all members of the guideline development group. Consider using the same format used for comments received during public consultation (see Sections 5 and 6 of the Public consultation module). Keep a record of whether each comment resulted in a change to the guideline, what that change was, or that the comment was considered but did not result in a change, and the reasons why it did not.

It is not practical or appropriate to modify the guideline based on every comment received. Table 3 lists a variety of potential concerns relating to comments and ways to deal with them. In particular, evidence shows that reviewers are more likely to introduce bias into their review when the content contradicts either their own or mainstream thinking (Phillips 2011).

**Table 3:** Types of concerns and associated responses (adapted from Rosenfeld, Shiffman et al. 2013)
Type of concern	Suggestions for Responding
Concerns substantiated with research evidence	All concerns supported by literature citations must be taken seriously, potentially resulting in revision of the guideline.
Concerns based on experience, personal bias or expert opinion	Must be addressed, however can be difficult to deal with as they may not have a rational or evidence-based foundation. Mention in supporting text if deemed appropriate.
Concerns based on misinterpretation	If sections of a guideline are misinterpreted by a reviewer, they should be revised to provide more clarity.
Concerns based on misunderstanding of guideline development methods	Reviewers may consider themselves to be better qualified to have written the document than the development group and/or be concerned that the guideline was not developed according to best practice. Seeking clarity regarding the reviewer’s position and educating them on the transparent, evidence-based methods used to develop the guideline may help to alleviate these concerns.
Concerns about abuse, misuse and legal precedents	There may be fear that guidelines will be misused by insurers or result in restricted care. Basing the guideline on an objective, evidence-based protocol that addresses bias, and seeking input from a broad range of stakeholders, can alleviate these concerns.
Concerns about spelling, grammar, terminology and choice of words	These are the easiest comments to deal with and are useful in picking up oversights by the authors. Changes should be made as appropriate.

If you wish to publish the feedback provided by reviewers with or without an explanation of changes made to the guideline as a result, you should make sure you have permission from the reviewers before doing so.

NHMRC requirements

Guidelines approved by NHMRC must meet all requirements outlined in the Procedures and requirements for meeting the NHMRC standard. The following requirements apply to the Independent review module:

D.15 The guideline and recommendations have been assessed by at least two reviewers, independent of the guideline development process, using the AGREE II instrument

In addition, guidelines seeking approval from NHMRC will be subject to independent clinical and methodological review prior to being considered by Council.

NHMRC Standards

The following Standards apply to the Independent review module:

6. To be evidence informed guidelines will:: 6.1. Be informed by well conducted systematic reviews; 6.2 Consider the bodyof evidence for each outcome (including the quality of that evidence) and other factors that influence the process of making recommendations including benefits and harms, values and preferences, resource use and acceptability; 6.3 Be subjected to appropriate peer review.

Useful resources

PRISMA checklist for reporting of systematic reviews and meta-analyses

Chapter 8 of the GRADE handbook (on determining whether the GRADE approach was used)

Cochrane style guide for advice on clear language in scientific communication

CARI Guidelines reviewer form

AGREE II toolkit

References

Phillips, J. S. (2011). Expert bias in peer review. Current Medical Research and Opinion 27(12): 2229-2233.

Rosenfeld, R. M., Shiffman, R. N., et al. (2013). Clinical practice guideline development manual, Third Edition: a quality-driven approach for translating evidence into action. Otolaryngology - Head and Neck Surgery 148(1 Suppl): S1-S55.

Scottish Intercollegiate Guidelines Network (SIGN) (2015). SIGN 50: a guideline developer’s handbook.

Shekelle, P., Woolf, S., et al. (2012). Developing clinical practice guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the emerging issues of enhancing guideline implementability and accounting for comorbid conditions in guideline development. Implementation Science 7: 62.

Siering, U., Eikermann, M., et al. (2013). Appraisal Tools for Clinical Practice Guidelines: A Systematic Review. PloS one 8(12): e82915.

Vintzileos, A. M., Carvajal, J., et al. (2013). Challenges in the peer review of systematic reviews and meta-analyses. Journal of Maternal-Fetal and Neonatal Medicine 26(8): 768-771.

Acknowledgements

NHMRC would like to acknowledge and thank Professor Dianne O’Connell from Cancer Council NSW for her contribution to this module as editor.

Version 4.3. Last Updated 22/11/2018.

Suggested citation: NHMRC. Guidelines for Guidelines: Independent review. https://nhmrc.gov.au/guidelinesforguidelines/review/independent-review. Last updated 22/11/2018.