Guidelines for Guidelines

Public consultation is included in many international guideline standards and can improve a guideline’s quality, legitimacy and its acceptability to end users and the public.

NHMRC’s role in providing health advice includes a long standing policy of public consultation which is mandated in the legislation as set out in the National Health and Medical Research Council Act 1992 (NHMRC Act) and the National Health and Medical Research Council Regulation 1996 (NHMRC Regulation) (NHMRC 1992; NHMRC 2016). NHMRC also recognises the importance of using additional methods of stakeholder consultation early and often in the guideline development process. This is discussed further in the Engaging stakeholders module.

Public consultation methods can be either open, targeted or a combination of the two (GIN Public 2012). By inviting specific stakeholders to comment, targeted consultation methods enable feedback to be sought in a relatively controlled manner; however, this also runs the risk of important viewpoints being overlooked. Open consultation is more transparent and can capture views from individuals or groups that might not otherwise have been planning or expected to engage. It is recommended that you use a combination of both open and targeted consultation methods—although this may produce a large volume of feedback it ensures that all stakeholders have the chance to comment on the content of your guideline

What can public consultation achieve?

• Controversial issues can be identified and managed early.
• A wider range of views from target audiences, the public and patients can be incorporated into the guideline.
• Gaps in the evidence can be identified.
• Advance notice of the publication of new evidence can be given.
• The wording and presentation of the guideline can be improved.
• Users can provide advice about ways to effectively disseminate and implement the guideline.
• Feasibility and acceptability of recommendations can be improved.

The purpose of this module is to outline methods for conducting fair and effective public consultation on guidelines; however, specific requirements apply to third party developers of clinical practice guidelines seeking NHMRC approval (see NHMRC requirements).

What to do

1. Plan for public consultation

You should ensure that you have enough time, resources and expertise to conduct a rigorous public consultation process (see Organisation and budgeting). In particular, plan for how best to:

• communicate with the public
• manage queries during the consultation period
• consider and respond to submissions as they inform the final guideline.

This will ensure that the process runs as smoothly as possible and will reduce some of your administrative burden.

When you notify the community about public consultation you will need to make sure that they understand the guideline being released is a draft version. It is important that the public understands that the consultation process will contribute to the end product—this will avoid the perception that final decisions have already been made.

You will also need to allow enough time for key stakeholders to undertake their own consultation process. For example, medical colleges or special advisory committees might need to table the guideline for discussion at committee meetings. They might also need to distribute the draft guideline to their own network and collate responses before providing a final submission.

You can prepare a communications strategy in advance of the public consultation release date in order to:

• identify the key objectives and messages in the draft guideline
• prepare answers to questions the public may ask
• brief any agencies that might respond to submissions
• consider whether the release should be timed with an event (e.g. a conference)
• decide how to notify the public (e.g. email, social media, television, newspaper, event launch)
• identify topic areas or questions you would like specific feedback or advice on
• direct the community to topical or potentially controversial areas in the guideline through ‘trigger’ questions
• nominate a spokesperson from the development group.

You should clearly articulate the purpose and scope of both the guideline and the consultation process; if key messages in the draft guideline are misinterpreted it may be difficult for you to overcome these perceptions at the implementation stage.

2. Define parameters for the consultation

Clear parameters are needed to ensure a consistent and transparent process is in place for deciding which submissions are relevant and in scope and which are not. This will also help you to control the volume of feedback received so that you can give due regard to all submissions.

Approaches to developing the parameters of consultation

• Ask specific questions.
• Describe what type of feedback will and won’t be considered.
• Describe the type of responses that will be accepted (e.g. comments, journal articles, guidelines, policies).
• Request that submissions are concise and set a maximum word limit if necessary.
• Set out the information in a logical order including line items, page numbers or structured forms to help with the collation process.
• Set clear time frames and a firm deadline for submissions.
• Clearly state if extensions will be offered and how to apply.

For example, in preparing for public consultation on the Australian Dietary Guidelines, NHMRC outlined specific parameters for how additional studies provided by the public during the consultation would be considered. These were based on the same requirements for studies included in the supplied Evidence Report (i.e. with regard to the former NHMRC levels of evidence and grades for recommendations), including:

• that the study is a high quality systematic review, meta-analysis, randomised controlled trial, or intervention, cohort, or observational study, but not an editorial or opinion piece
• that the outcome of the study relates to health or chronic disease
• that the study results can be generalised to the Australian population
• that the study relates to foods or the total diet rather than nutrients.

NHMRC determined that if a relevant systematic review or meta-analysis was provided, the recommendations in question would be revisited to incorporate the new evidence. If relevant high quality studies of other types were provided (according to the parameters above) the new evidence would be incorporated in the surrounding text supporting the recommendations.

3. Notify the public

NHMRC uses a number of methods to communicate with the public, including an email subscription service, website notices, social media and directly emailing stakeholders on its stakeholder database. When you are developing a guideline it is important that you collate and maintain your own list of stakeholders at the beginning of the process and ensure they are notified about your public consultation.

Notices inviting public comment on a guideline should contain:

• the subject matter of the draft guideline
• where a copy of the draft recommendations or guideline can be obtained
• the manner in which submissions should be made
• the timeframe for receipt of submissions. 

Although NHMRC’s legislated minimum requirement for public consultation is 30 days, you should consider extending the submission period to accommodate the needs of your stakeholders as required.

4. Request information

To better understand the context for comments and the motivations of respondents, you should request the following information from them:

• their name and the city or state in which they are based
• their profession or the type of organisation in which they are employed
• whether they are responding as an individual or on behalf of an organisation
• any conflicts of interest they might have
• whether or not their submission can be published.

5. Prepare submissions for consideration

When the consultation process has ended you should carefully read all submissions. You can then apply the parameters of the consultation process (Section 2) to determine what is relevant to the scope. If submissions are excluded from the decision making process because they do not fall within the stated parameters, you should record your decision and the reason for excluding them. The guideline development group should still be provided with copies of all submissions, as well as the record of decisions to exclude material.

You might find that the volume of submissions you receive is overwhelming, particularly if your guideline is controversial. If there are a large number of submissions consider forming subgroups of the guideline development group or project teams to process different sets of submissions. Exactly how this is done may vary depending on:

You should also be aware that well-resourced stakeholders might use a public consultation period to submit multiple comments that promote their interests. For example, recent calls for evidence for public and environmental health guidelines have resulted in NHMRC receiving over 500 submissions with near-identical content. If this situation arises you may find it easier to respond to submissions if duplicate or near-identical comments are grouped together. This approach should still enable the unique content of the submissions to be considered in full (while noting the potential skewing of total submission numbers) while reducing your administrative burden. Subgroups of your development group can also be formed to respond to these kinds of submissions.

As with all public consultation processes it is important to be transparent about any decisions you have made to manage lobbying campaigns, should the situation arise. For example, you can document the approach taken and the number of duplicate submissions received in a given time period to justify that a change in approach was necessary.

In these cases it may also help if you (or a technical writer) prepare a document that summarises the submissions for discussion within the guideline development group. You must proceed with caution— brief summaries only provide a cursory examination of relevant material and may be subject to challenge.  Summaries and any associated cover papers must be sufficiently detailed so as to alert the group of the need to undertake a more thorough analysis if necessary. You should also provide the full submissions to members along with any summary documents in case they would like further details.

You (or a technical writer) can also help the guideline development group in their deliberations by extracting and tabulating all specific issues arising from the submissions. There is no designated standard for how comments should be categorised but it is important that the end product is clearly presented to assist the guideline development group’s decision making.

For example, comments and group responses can be tabulated by:

Editorial comments and suggestions are often suggested during public consultation. To make best use of a guideline development group’s time you can summarise editorial comments in a separate table for project staff to consider. Overtly biased comments and suggestions will require more careful consideration, particularly if the guideline is controversial; however, is up to the guideline development group to collectively decide on the best way to respond.

An example of a public consultation table with sample content is provided in Table 1. The final two columns are filled in after the development group has considered the comments (see also Section 6).

Table 1. Example of public consultation summary table

Name/ID	Section of guideline/issue	Comment	Guideline development group response	Action taken
ID39	Section 2.1 Lifestyle interventions	Include a, b and c in this section.	Agreed. While consideration of a, b and c is out of scope for the recommendations, these should be included in the supporting text of the guideline.	Guideline amended in Section 2.1 and 4.2 to include a, b and c
ID42	Section 5.2 Pharmaceutical interventions	Please note use of the word d in this section may be outdated.	Noted. The GDG has decided to retain current wording since it is correct in this context.	No change.
ID52	Section 6.1 and Section 6.3 Family history	Parameters e and f are inconsistent as listed in these two sections.	Noted and agreed.	Guideline amended as per suggestion.

 

Public consultation offers an opportunity for any stakeholder to submit comments on your guideline. The primary role of stakeholders in guideline development is to represent their own interests and their views should be understood in this context. Stakeholders are expected to promote the interests of their organisation and this can be straightforward to manage when their communication is open and honest.

It may be prudent to take steps to minimise the risk that a guideline will be biased by the interests of well-resourced stakeholders. For example, NICE considers tobacco companies separate to other stakeholders to differentiate them from their standard procedures (NICE 2014).

The guideline development group must adhere to the parameters specified at the beginning of the process (Section 2) irrespective of the volume or nature of comments received—they must not be selective in what they read. For example, if it is specified that all comments will be considered you must take care to consider submissions in all forms. This might include published literature, personal stories, or images.

This requirement has been challenged once in court by the tobacco industry which submitted a large volume of material in response to the public consultation of NHMRC’s passive smoking guidelines. The Federal Court found that NHMRC did not fulfil its obligation to give proper consideration to all materials received as the steering committee at the time elected to retrospectively consider only literature published in peer reviewed journals which meant that literature published by the tobacco industry was deemed out of scope (Jamrozik, Chapman et al. 1997).

If stakeholders identify new evidence you need to determine whether it meets the specific inclusion criteria. If the evidence was published outside the timeframe of the guideline search parameters the development group may still wish to consider its potential impact and decide how it should be referred to in the guideline. For example, developers of the Australian Dietary Guidelines determined that relevant high quality studies published after the search dates of their literature review would not be formally incorporated into the recommendations. Instead, this information was included in the surrounding text of the guidelines to support the recommendations and ensure currency.

The guideline development group is ultimately the decision maker for the content of a guideline. You and other members of the group have a responsibility to maintain the scope and to judge the evidence while taking into account the needs of the target audience. It is important that this judgement is applied at all times while reviewing submissions.

You should document all comments made by stakeholders, any decisions made as a result of the comments and the reasons for the decisions. This always needs to be recorded regardless of whether a change was made to your guideline or not. It is important that the consultation report provides a clear and accurate record of this decision making.

Once the guideline is finalised and ready to be released it is prudent to include a summary of the public consultation process and the changes made to your guideline as a result. The summary should capture the key issues and their corresponding responses (e.g. the public consultation report of the NHMRC Draft Information Paper: Evidence on Wind Farms and Human Health).

A public consultation report is usually provided as part of the process report of a guideline, either as an appendix or published separately.

People or organisations who respond to a public consultation will want to know how their comments were received and if your guideline changed in response to their comments. When providing feedback to respondents it is important to make reference to the parameters of the public consultation so it is clear how you determined whether a comment was in or out of scope.

If respondents have consented to having their individual submissions made publicly available you must make sure that you redact their direct contact details and any personal information of other people that may be contained in the submission. For example, submissions may contain personal stories and experiences including the names of patients or health professionals that may be unaware they have been referred to in the submission, or may be inferred (e.g. “I am making the submission as the mother of a suicidal teenage daughter”).

Finally, you must ensure that your guideline acknowledges those individuals and organisations that have made a submission in the final published guideline (provided they have given consent to do so). This can be included in the process report (as an appendix) or published separately to the guideline (e.g. the NHMRC public consultation portal publishes submissions online). As mentioned in Section 8, take care to consider your obligations under privacy legislation.

NHMRC requirements

The NHMRC Act states, inter alia, that when it is practicable to do so, NHMRC should adopt a policy of public consultation in relation to individual and public health matters.

Section 13 of the NHMRC Act details the requirements for consultation on guidelines developed by NHMRC internally, namely that:

    Before:

    (a) the Council provides guidelines (other than human research guidelines) to the CEO for the purposes of subsection 9(1); or

    (b) the Australian Health Ethics Committee provides human research guidelines to the Council for the purposes of subsection 10(2);

    the Council or Committee must:

    (c) prepare a draft of the guidelines; and

    (d) publish a notice, in the manner and form specified in the regulations:

       (i)containing a summary of the draft guidelines; and

       (ii)stating where copies of the draft guidelines can be obtained; and

       (iii)inviting persons or bodies to make submissions relating to the draft guidelines in accordance with the procedures, and within the period, specified in the notice; and

    (e) have regard to any submissions received as a result of the invitation referred to in subparagraph (d)(iii).

Section 14A of the NHMRC Act details the requirements for consultation on third party guidelines seeking approval by NHMRC, namely that:

    1) The CEO may, on the advice of the Council, approve guidelines prepared by a person or body from outside the NHMRC.

    2) The Council may only advise the CEO to approve the guidelines if the Council is satisfied that the person or body, before submitting the guidelines to the CEO for his or her approval:

       (a) prepared a draft of the guidelines that the person or body proposed to submit to the CEO; and

       (b) published a notice, in a manner and form acceptable to the Council:

          (i) containing a summary of the draft guidelines; and

          (ii) stating where copies of the draft guidelines could be obtained; and

          (iii) inviting persons or bodies to make submissions relating to the draft in accordance with the procedures, and within the period, specified in the notice; and

       (c) had regard to any submissions received pursuant to the invitation referred to in subparagraph (b)(iii).

Note: Subsection (2) does not apply if the guidelines are of minor significance: see section 14B

In addition, regulation 7 of the NHMRC Regulation details NHMRC’s requirements regarding the manner and form of notice of such consultation, namely that:

       (1) A notice under paragraph 13(d) of the Act must be published on the NHMRC website.

       (2) A notice under paragraph 13(d) of the Act must include the following:

          (a) the subject matter of the draft guidelines to which the notice relates;

          (b) the last day on which the Council or the Australian Health Ethics Committee will accept submissions relating to the draft guidelines, which must be a day that is at least A days after the notice is published under subsection (1) of this section.

Third party developers of clinical practice guidelines seeking NHMRC approval must meet all requirements outlined in the Procedures and requirements for meeting the NHMRC standard, including the following in relation to public consultation:

• F.1 The process for public consultation on the draft guideline complies with section 14A of the [NHMRC Act and the NHMRC Regulation].
• F.2 Details of submissions received during public consultation and the response of the guideline development working group to the submissions (including whether, why and how the guideline was altered) are provided as a separate document to the NHMRC.
• F.3 During the public consultation period, the developer has undertaken and documented consultation with: – the Director-General, Chief Executive or Secretary of each state, territory and Commonwealth health department – relevant authority/ies, when a guideline makes any recommendation/s specifying interventions that are not available or restricted in Australia (see Requirement D.10).
• F.4 The developer has identified and consulted with key professional organisations (such as specialty colleges) and consumer organisations that will be involved in, or affected by, the implementation of the clinical recommendations of the guideline.

NHMRC Standards

The following Standard applies to the Public consultation module:

1. To be relevant and useful for decision making guidelines will:

1.3 Be informed by public consultation

 

Useful resources

European Commission Code of Good Practice for consultation of stakeholders

European Commission Guidelines on stakeholder consultation

GIN Public Toolkit 2015

NHMRC public consultation portal

NHMRC public consultation submission guidelines

References

Guidelines International Network (GIN) Public Working Group (2012). GIN Public Toolkit. Chapter 1: How to conduct public and targeted consultation.

Jamrozik, K., S. Chapman, et al. (1997). How the NHMRC got its fingers burnt. The Medical Journal of Australia 167(7): 372-374.

National Health and Medical Research Council (NHMRC) (1992). National Health and Medical Research Council Act 1992, Australian Government.

National Health and Medical Research Council (NHMRC) (2016). National Health and Medical Research Council Regulation 2016, Australian Government.

National Institute for Health and Care Excellence (NICE) (2014). Developing NICE guidelines: the manual.

 

Version 5.1. Last updated 22/11/2018.

Suggested citation: NHMRC. Guidelines for Guidelines: Public consultation. https://nhmrc.gov.au/guidelinesforguidelines/review/public-consultation. Last published 22/11/2018.