Forming the questions
This draft module is currently open for public consultation. You are encouraged to make a submission through the online public consultation portal available at consultations.nhmrc.gov.au. Consultation closes on Wednesday 6 February 2019 at 5:00pm AEDT.
Forming the questions
The scope of the guideline must be translated into a series of specific research questions - this is a critical step which requires careful attention.
The questions that are formed will be used to map the planned guideline against existing guidelines or systematic reviews, determine the type of evidence needed (e.g. qualitative or quantitative, observational or trial-based), design a search strategy to find relevant evidence, structure the analysis and synthesis of evidence and structure the guideline’s recommendations (O’Connor, Green et al. 2011; NICE 2014).
If the right questions are not clearly defined at the beginning of the process then your evidence search will be impacted—you won’t have the information you need when the guideline development group comes together to consider its recommendations. The compiled evidence may have the wrong focus, or there may be too much untargeted information. Repeated searches or retrospective changes to the evidence synthesis may be needed, increasing both the risk of working with incomplete evidence and the cost in time and resources. There is also an increased risk that the recommendations in the guideline will not effectively address the purpose of the guideline, the uncertainties faced in the work of health professionals and policy makers or the choices made by consumers. Even if the topic of the guideline is of critical importance, it may be misdirected and little used if these issues are not addressed.
Guideline development groups will usually need to consider several types of questions (WHO 2014):
- Definition/background questions. For example, what is this condition? What is the burden of disease? Is this a priority condition? What are the current practices or policies? What are the contextual factors or relevant values and concerns of stakeholders? What is the epidemiology of the condition and risk factors? This information may be available without requiring detailed systematic reviews, but may inform the background to the guideline and the context in which decisions are made.
- Facts/foreground questions. For example, what are the effects of this intervention? What is the accuracy of this test? These questions will form the main structure of the evidence informing the guideline and are more typically the subject of systematic reviews.
- Recommendation/decision questions. For example, should we recommend this course of action? These questions are usually considered later in the process (see the Evidence to decision module).
This module provides practical advice on how you form specific research questions to be addressed by your guideline. Other modules that you might find useful at this stage include Equity, Considering purpose and context and Consumer involvement. Guidance on determining your scope is outlined in the Scoping the guideline module. The process of seeking evidence to answer your questions is outlined in the Deciding what evidence to include and Identifying the evidence modules.
What to do
1. List questions within the scope of the guideline
At this stage in the development process, the overarching purpose and scope of the guideline should be well documented (see Considering purpose and context and Scoping the guideline modules). Taking these as a starting point, the next step is to list individual questions within the scope that can be answered by the guideline.
Ground the selection of questions in the priorities and choices faced by guideline users, including health professionals, consumers, policy makers and others in the Australian context:
- Who will use this guideline? Who will it apply to?
- What are the choices or uncertainties that arise in this area in practice, in the community, or in policy?
- What actions or changes are you hoping to influence?
- Will a guideline add value?
Addressing these issues will make the guideline useful and ensure that the time and effort expended to develop the guideline are worthwhile (Eccles, Grimshaw et al. 2012; GRADE Working Group 2013).
The list of possible questions should be informed by consultation with consumers and other stakeholders. This input may be found through:
- your own consultations as part of the guideline development process (see Scoping the guideline, Consumer involvement, Engaging stakeholders, Implementability and Equity)
- published literature on the priorities and outcomes important to these groups
- gaps identified in previous reviews or guidelines
- data on the burden of disease or variations in practice (such as those identified in reports from the
- reports from organisations that regularly conduct consultation and priority-setting activities to inform research, such as the
The guideline development group is also likely to have relevant expertise and may contribute additional questions.
As an example, a relatively specific guideline on alcohol use in pregnancy might be intended for use by general practitioners, midwives and obstetricians working with pregnant women and by public health and health promotion staff. Some of the key questions to resolve uncertainty and inform action might include:
- What level of alcohol use is safe in pregnancy?
- Is any level of alcohol use safe in pregnancy?
- In what stages of pregnancy does alcohol convey higher risk?
- What are the most effective ways to identify health problems arising from alcohol consumption during pregnancy?
- What are the most effective care interventions for people who may have consumed risky levels of alcohol during pregnancy?
- Why do women use alcohol in pregnancy and what are their experiences?
- Are there differences in alcohol use between population groups?
Each of these questions may break down into additional, more specific questions—for example, about individual intervention options or populations at higher risk.
Depending on the purpose of the guideline, related questions may be placed outside the scope, such as:
- What are effective health promotion interventions aimed at changing alcohol-related behaviour?
- What are the effects of alcohol consumption unrelated to pregnancy, such as violence, liver disease, cardiovascular risk, neurological development in teenagers, etc.?
- What are effective care interventions for children born with fetal alcohol spectrum disorders?
You may have some idea about the evidence base during the scoping phase, but you will not know exactly what the evidence is until a systematic search is undertaken (see Identifying the evidence). Do not exclude important questions because you anticipate that few studies will be found.
If a logic model or similar framework has been developed (see Considering purpose and context), this can provide a useful structure to identify the different component areas of the guideline. Dividing the guideline into component areas (such as stages of care, sites of delivery, specific diseases within a vaccine program or specific populations at risk) allows you to consider answerable questions within a manageable limit without trying to tackle all components of the guideline.
Some guidelines may need to break questions down into separate elements of the logical pathway from exposure or intervention to outcome. For environmental guidelines, a seemingly straightforward question such as ‘what level of this chemical is safe in drinking water?’ may need to be broken down further into questions such as:
- Is this chemical associated with toxicity in laboratory settings?
- If this level of the chemical is present in water, how does that translate to the level of exposure in humans and over how long?
- Is this level of exposure associated with important health outcomes in the community?
- What interventions are effective in changing the level of exposure?
For interventions likely to be used at a community level, you may be interested in both the laboratory-tested efficacy of interventions such as measured dietary intake of specific nutrients or supervised exercise for obesity interventions, and the effects of pragmatic community-based interventions such as interventions to modify dietary habits or physical activity.
Updating guidelines: Before embarking on an updated systematic review, first consider the questions posed in the existing guideline. Are they still relevant? Do they reflect current practice or policy decisions? Some parts of the existing guideline may not need to be included in the new version and new areas of evidence may need to be considered for the first time. Cochrane has published some useful guidance on considering updates of systematic reviews that may be helpful (Garner, Hopewell et al. 2016).
Adopting or adapting guidelines: You should still conduct a process to form the questions of interest before considering adopting or adapting an existing guideline. This will enable you to map how well the existing guideline answers your questions and identify gaps or differences that will need to be addressed. For more information see Adopt, adapt or start from scratch.
2. Translate elements into structured questions
Using a structured and specific approach allows each question to be directly addressed by mapping existing guidance, targeted literature searches, synthesis of the available evidence and clear recommendations.
An answerable question has several essential components, depending on the nature of the guideline and the questions asked, for example, intervention, exposure, diagnosis and prognosis. The most common structure used to articulate intervention questions is ‘PI/ECO’ (Population/participants, Intervention/Exposure, Comparison, Outcome). The PI/ECO structure can be readily amended for different question types (Guyatt, Oxman et al. 2011; NICE 2014). There are also options available for framing qualitative/mixed methods questions such as the SPIDER tool (Sample, Phenomenon of Interest, Design, Evaluation, Research type) although these are less widely used (Cooke, Smith et al. 2012).
The first three components of the PI/ECO structure are:
- P: Population/participants—the people who will receive the proposed intervention or test, or who are experiencing the exposure or condition of interest. For prevention questions, this may be anyone in the national population. For other questions, you may need to define health conditions or stages of disease. In addition to the overall definition of the population, consider whether there are groups of people who may experience different outcomes and for whom the answers to the research questions might be different. Consider characteristics of the people (such as age, gender, comorbidities, Aboriginal and Torres Strait Islander status, culturally and linguistically diverse groups, risk profiles), the setting (hospital, community) and the context (such as geographic locations, resources and social factors such as culture, historical context and equity).
When considering the definitions of disease, you should be cautious about expanding current definitions to encompass new populations previously defined as healthy or with less serious conditions (e.g. by shifting thresholds of blood pressure for hypertension or fasting glucose for pre-diabetes). A proposed
- I: Intervention (or E: exposure)—the interventions, exposures, behaviours or variations of care that may be within the scope of the guideline. Timing, delivery, setting and resources may be important and consider whether alternative or complementary therapies, consumer self-management options, or interventions currently not available or funded in Australia are within the scope of the guideline. For multi-component interventions such as packages of care (e.g. rehabilitation) or community-level interventions (e.g. promoting exercise), you may need to define the key intervention components or groupings and how these will be differentiated from each other and from options outside the guideline’s scope.
Special considerations should also be made for the complex nature of exposures. For example, do multiple exposures need to be considered? Are there different methods or routes of exposure and can these be measured differently (such as dermal versus inhaled arsenic)? Does the exposure have different effects on the stages of human development (such as fetal exposure versus adult exposure to per- and polyfluoroalkyl substances [PFAS])? These issues should all be flagged so they can be included in the research questions.
For diagnostic questions, two different components should be defined: firstly, the diagnostic tests, thresholds or signs to be tested and secondly the ‘target condition’ that these tests aim to diagnose. For prognostic questions, define the prognostic factors or models to be assessed, such as attributes of the patient or features of the condition.
There are tools available that may assist you in thinking through your interventions or behaviours. For interventions, the Template for Intervention Description and Replication (
- C: Comparison—for questions comparing two or more options, comparisons of interest may include alternative options, no intervention/exposure or varied levels of exposure. Other questions may not need defined comparisons, such as questions of prevalence. Any comparisons should be described as clearly as each intervention or exposure group, including placebos or ‘usual care’. For diagnostic reviews, the ‘reference test’ (or gold standard for defining the diagnosis) can be defined here, which may be different from diagnostic tests used in current practice (such as confirmatory tests performed late in disease progression). Ensuring the comparators are clear is essential to inform the recommendations, especially when the available evidence does not directly compare the choices of interest (for example, comparing interventions to an older practice that is no longer recommended, or to no intervention) (Guyatt, Oxman et al. 2011).
The O: Outcomes in PI/ECO will be discussed in the next section. More detailed discussions of different question types are available elsewhere (NICE 2014; WHO 2014; Munn, Stern et al. 2018).
Depending on the purpose of the review, your questions may need to be very specific (what dose or level of exposure or dose is optimal for older people with specific comorbidities?) or very general (is drug intervention beneficial?). Questions that are too specific for your purpose may find too little relevant evidence, be repetitive in their findings, or provide guidance that is less useful for health professionals and consumers. Questions that are too broad for your purpose will be difficult to answer comprehensively, may require literature reviews that are infeasible within the time and resources available and may obscure important differences in the response among different populations (such as groups with higher and lower baseline risk levels), or variations on the intervention.
A research question should not always outline the ‘ideal’ version of a population, intervention, exposure or test, but rather the minimum components that define it and make it identifiable. Communities, current practice or policy and the available research are likely to include many variations, large and small and so it is important to keep in mind the boundaries within which the population, intervention, exposure or test is similar enough to be applicable to your context (e.g. studies conducted in low-income settings may or may not be applicable in Australia, depending on the question) and similar enough to be grouped together for the purposes of your question (such as whether different variations of cognitive behavioural therapy can be considered together).
Remember that your consideration of the evidence may be conducted at more than one level. Within your primary questions of interest, you may begin by considering the overarching evidence for broader populations or categories of intervention. You may then wish to define important subgroups that may modify the effects of interest (such as high and low risk populations, or different intervention components or levels of exposure) and explore these within the overall synthesis using subgroup analysis or other techniques (see Synthesising evidence).
If the questions have been drafted by the guideline development group to express uncertainties raised by consumers and stakeholder groups, it is prudent to re-check the framing of these questions with the same stakeholders to ensure that the meaning has been accurately captured.
3. Prioritise and select the final question list
It is likely that more questions will be suggested than can possibly be addressed within your guideline. Consider the time and resources available for the completion of the guideline and establish a realistic limit on the number of questions that will be feasible to consider. Ensure that these address the areas of greatest uncertainty, impact or variation in current practice or policy. For example, in the alcohol and pregnancy example presented above, the question of ‘what are the health impacts of excess alcohol consumption in pregnancy’ may be considered a lower priority question, perhaps a background question, as it is not a key area of uncertainty and would not inform specific actions to improve policy or practice in the same way that other questions might.
Guideline development groups should conduct a transparent process of prioritisation to select questions. This can be a direct discussion in a meeting of the group, or can be conducted using more formal, Delphi-style techniques, either in person or online. The use of logic models can also help inform this selection. The GRADE Working Group recommends a process that includes:
- an initial suggestion-gathering phase
- a review of the list of suggestions, with members able to rank the different questions by priority and add new questions that arise
- a review of a ranked list to reach consensus (Guyatt, Oxman et al. 2011).
Online tools can be used to support this kind of process, including simple survey tools like
4. Select and prioritise outcomes critical to answering the questions
Once the questions have been selected, the fourth component of the PI/ECO (or related) structure can be addressed, which is to select the outcomes that will be measured in order to answer the questions.
- O: Outcomes—a limited list of the outcomes that are most critical to inform decision making. Outcomes can be of many kinds, including health outcomes, prevalence, diagnostic test accuracy, social outcomes, economic outcomes, perceptions or behaviours. The effects of an intervention, exposure or condition on these outcomes will form the basis for the guideline’s recommendations. Always consider both possible benefits and adverse effects. Consider outcomes of relevance to consumers, health professionals and other decision makers and at the level of individual patients, communities or health systems (Guyatt, Oxman et al. 2011; GRADE Working Group 2013).
Like questions, there are likely to be many more measurable outcomes than would be feasible to consider in the guideline. Guideline development groups must be pragmatic in selecting the most important outcomes for decision making. Including too many outcomes can substantially increase the resources required to find and synthesise the evidence, for instance requiring a large number of separate systematic reviews, and may produce a complex body of evidence that is overwhelming for guideline development groups to meaningfully interpret. For example, stroke can impact on a broad range of outcomes from immediate mortality to long-term independence and include a long list of specific physical and psychological outcomes, care services and social impacts. However, not all of these outcomes should have the same level of weight in guiding decision-making throughout the care pathway, nor is it necessary to provide exhaustive evidence on all of them to inform a meaningful decision (see Case study below). On the contrary, attempting to balance the positive and negative effects and relative uncertainty across all the possible outcomes would be entirely infeasible (see Evidence to decision).
Identifying the outcomes of most importance to consumers and other stakeholders should be based on consultation, whether reported in the published literature or conducted as part of a larger consumer and community engagement strategy for the guideline (Guyatt, Oxman et al. 2011; Armstrong, Mullins et al. 2018) (see Section 1 and Consumer involvement). For some fields of research, core outcome sets have been established to inform researchers, often based on usefulness in decision making and importance to patients and care providers (see the
Guideline development groups should undertake a transparent process to list and then prioritise the outcomes to be measured. This can be similar to the process conducted to prioritise questions, but should be conducted as a separate step. Groups may consider a single set of outcomes to be applied to all studies, or different sets of outcomes for different types of questions where appropriate. For example, the expected benefits and harms relevant to pharmacological interventions may be different to those expected from surgical or health systems organisation interventions. The GRADE Working Group outlines a process of numerical rating and consensus to select a limited number of the most important outcomes (Guyatt, Oxman et al. 2011; GRADE Working Group 2013). Again, online tools like
Do not exclude important outcomes because you anticipate that few studies will be found. If the outcome is important for decision making by consumers, health professionals or policy makers, then you should consider ways to address that need. You may find that:
- more evidence may be available than you anticipate, which may be found through a comprehensive search
- different sources of evidence can be explored to address the outcome, beyond the preferred study design for the question type (see Deciding what evidence to include), such as routinely collected data sets, observational studies or qualitative research
- an appropriate surrogate measure may be used, although caution should always be used in selecting surrogates as they may not be reliable predictors of the outcome of interest.
Remember that the most important outcomes for stakeholders may not be those most frequently reported in the research literature. An oft-cited example is that people with arthritis may prioritise reduction in fatigue over reduction in pain. In some cases, deeply held community concerns may present a barrier to the implementation of a guideline if they are not clearly addressed, including cases where the published evidence may rarely identify the effects of concern (such as community fears about possible harmful effects of water fluoridation or proximity to wind farms).
The use of any less rigorous study designs (see Assessing risk of bias) or surrogates may lead to recommendations being considered less certain (see Assessing certainty of evidence) (Guyatt, Oxman et al. 2011; GRADE Working Group 2013). Note that the selection of appropriate study designs to answer the questions is discussed in a separate module (Deciding what evidence to include).
Be specific about acceptable measures of those outcomes, especially where many different measures are likely to be encountered in the evidence (such as multiple scales used to measure depression, variable definitions of ‘treatment failure’ or ‘sufficient physical activity’). It may not always be possible to obtain the ideal measures consistently, so a hierarchy of preferred measures may be useful. Consider also the time points at which it may be meaningful to measure outcomes.
5. Use questions to structure the guideline
As discussed above, the questions and outcomes identified can be used throughout the process—mapping against existing guidelines and reviews, designing search strategies, synthesising results and structuring the final recommendations.
It is possible that new questions or outcomes of importance will emerge through the process of reviewing the available evidence. If this occurs, changes can be made, but should be transparently considered to ensure that they meet the criteria of questions and outcomes of importance for decision making and not ‘drift’ in the project based on retrospective consideration of study findings.
In many cases it will be useful to report the full list of questions in:
- The technical report—demonstrating that your guideline answered clear, pre-specified questions and describing the methods you used to form and prioritise the questions, will increase the credibility of your guideline and may also increase the re-use of the guideline in future updates or for adaptation by other guideline developers.
- The guideline itself—where readers can explicitly make the connection between the question and the resulting recommendation/s.
During development of its Clinical Guidelines for Stroke Management the Stroke Foundation conducted a formal process to select and prioritise questions of interest. This was based on consultation with working groups for different areas of the guideline (broken down by stage of care) and considering areas of possible overlap with related guidelines. The Technical Report includes a list of the resulting 89 clinical questions, structured in the PICO format.
A very short list of critical outcomes is listed for each question and these vary according to the question. For example:
These questions then formed the structure for the recommendations in the guideline itself (Stroke Foundation 2017).
Guidelines approved by NHMRC must meet all requirements outlined in the
- A.3 A multidisciplinary group that includes end-users, relevant disciplines and clinical experts is convened to develop the purposes, scope and content of the guideline, and the process and criteria for selecting member are described.
- B.1 The purpose of the guideline is stated, including the clinical questions, issue or problems the guideline addresses.
- B.2 The health care setting to which the recommendations apply is described, including the health system level (e.g. primary care, acute care) and clinical stage (e.g. whether the guideline covers prevention, screening, assessment, treatment, rehabilitation or monitoring).
- B.3. The intended end users of the guideline are clearly defined, and any relevant exceptions are identified.
- B.4 The population to which the guideline recommendations will apply is defined (e.g. children, adolescents, adults or older adults) and population subgroups for which specific information is required are identified and described.
- B.5 Issues relevant to Aboriginal and Torres Strait Islander peoples (such as particular risks, treatment considerations or sociocultural considerations) are identified and described.
- B.5.1 (desirable) Issues relevant to special-needs groups such as culturally and linguistically diverse communities or groups with low socioeconomic status (e.g. particular risks, treatment considerations or sociocultural considerations) are identified and described.
- C.1 Clinical questions addressed by the guideline are stated in a structured and consistent format to define the boundaries of the topic, i.e. by specifying the relevant population, intervention/s (e.g. treatment/s or diagnostic test/s), comparator/s and outcomes measured.
- D.10 Where a guideline makes any recommendation/s specifying intervention/s that are not available or restricted in Australia, the text clearly indicates this, and the developer has consulted the relevant authority/ies.
- D.12 The harms (risks or side effects) and benefits of each recommended intervention and its alternatives are described in the guideline text and the rationale for the recommendation is explained.
- D.9.2 (desirable) The resource implications and cost effectiveness of any recommended practice, compared with current or established practice, are explicitly stated in the guideline text.
- D.16 If evidence for complementary and alternative medicine options is identified, the risks and benefits of these are stated in the guideline text and appropriate recommendations included.
- D.18 Recommendations that consider consumer self-management options are included, where relevant.
The following NHMRC Standards apply to the Forming the questions module:
1. To be relevant and useful for decision making guidelines will:
1.1. Address a health issue of importance
1.2. Clearly state the purpose of the guideline and the context in which it will be applied
1.3. Be informed by public consultation
3. The guideline development group will:
3.1. Be composed of an appropriate mix of expertise and experience, including relevant end users
5. To be focused on health and related outcomes guidelines will:
5.1. Be developed around explicitly defined clinical or public health questions
5.2. Address outcomes that are relevant to the guideline's expected end users
5.3. Clearly define the outcomes considered to be important to the person/s who will be affected by the decision, and prioritise these outcomes.
Eccles, M. P., Grimshaw, J. M., et al. (2012). Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implementation science : IS 7: 60-60.
GRADE Working Group (2013). GRADE Handbook. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. H. Schünemann, J. Brożek, G. Guyatt and A. Oxman.
Lewin, S., Hendry, M., et al. (2017). Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Medical Research Methodology 17(1): 76.
Munn, Z., Stern, C., et al. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC medical research methodology 18(1): 5.
O’Connor, D., Green, S., et al. (2011). Chapter 5: Defining the review question and developing criteria for including studies. In: Cochrane Handbook for Systematic Reviews of Intervention. . J. Higgins and S. Green (editors), Version 5.1.0 (updated March 2011). The Cochrane Collaboration. Available from http://handbook.cochrane.org.
NHMRC would like to acknowledge and thank Miranda Cumpston (author) and Adjunct Professor Davina Ghersi (editor) from NHMRC for their contributions to this module.
Version 3.0. Last updated 22/11/2018.
Suggested citation: NHMRC. Guidelines for Guidelines: Forming the questions. https://nhmrc.gov.au/guidelinesforguidelines/develop/forming-questions. Last published 22/11/2018.