Identifying the evidence

This draft module is currently open for public consultation. You are encouraged to make a submission through the online public consultation portal available at consultations.nhmrc.gov.au. Consultation closes on Wednesday 6 February 2019 at 5:00pm AEDT.

Identifying the evidence

Recommendations made in guidelines should be informed by well-conducted systematic reviews.

The quality of systematic reviews will depend on the ability of the reviewer to identify as much relevant evidence as possible.

Conducting broad searches will capture all relevant evidence; however, this might also identify many studies that are not relevant. You should aim to find a balance between the sensitivity (the proportion of relevant articles retrieved) and specificity (the proportion of irrelevant articles not retrieved) of your search (Montori, Wilczynski et al. 2005).

There is an ever increasing body of evidence that can be used to inform guideline recommendations. However, before you consider reviewing this evidence you should determine whether this has already been done. Existing reviews and guidelines can save a lot of time and resources as they may address most or all of the questions relevant to your review. These can be found by searching databases such as the Cochrane Database of Systematic Reviews (CDSR) and the Guidelines International Network (G-I-N) library.  

Bibliographic databases of peer-reviewed literature are the primary source of information on which to base a systematic review. There is now a recognised need to include unpublished studies and other non-commercial evidence as its inclusion can also provide a more comprehensive view of the topic, and excluding these data can introduce significant bias (Adams, Smart et al. 2017). For example, there is a high incidence of unpublished data for recommendations involving drugs as results from clinical drug trials are sometimes not reported or published (Brænd, Straand et al. 2016). Non-commercial and non-peer-reviewed literature is often referred to as ‘grey literature’ (e.g. government reports, public health monitoring or surveillance data, data from clinical trials registries). Unpublished evidence, which may not even be in the public domain (e.g. unpublished industry data, cultural knowledge), can be challenging to identify and may include actively seeking information from individuals or organisations through interviews or written correspondence.

You should also be aware that in very specific circumstances, such as if you need to conduct a narrative synthesis as part of your evidence review, other types of evidence relevant for your guideline might not fit within the boundaries of knowledge as defined above. These forms of evidence exist and are equally valid for consideration in an evidence evaluation, although they can’t be subjected to meta-analysis in a formal systematic review process (Popay, Roberts et al. 2006). These forms of evidence might include traditional knowledge or community knowledge and might only be obtained through unconventional search strategies involving discussions with traditional knowledge holders during the guideline development process. These kinds of strategies are not discussed in this module.

Systematic review search methods are currently evolving. Automation is providing a transition from conventional ‘pull’ methods (searching databases to identify studies and then ‘pulling’ them out) to ‘push’ methods. Particularly useful for living systematic reviews, these automated systems search databases using the stored search criteria terms. They then alert the reviewer when a citation that matches the search criteria is found (Thomas, Noel-Storr et al. 2017). Two automated systems that simplify the search process are the Epistemonikos and Turning Research Into Practice (TRIP) databases. In addition, the ‘My NCBI’ functionality within PubMed allows the reviewer to save searches and set up alerts, thereby speeding up the search process.

Whichever search method you use, the key is to produce a comprehensive search strategy that balances sensitivity and specificity. You need to ensure that your search includes a variety of sources and is accurately recorded for transparency and reproducibility.

This module focuses on developing search strategies, database navigation, citation management and how best to document the process. Methods for developing inclusion and exclusion criteria or determining what type of studies you should include are described elsewhere (see Deciding what evidence to include). Other modules you might find useful include Forming the questions and Selecting studies.

What to do

1. Plan your search

Before you start planning your evidence search, you need to know what you are looking for. This can be done by first defining the questions (see Forming the questions) and determining what types of evidence you would like to include (see Deciding what evidence to include).

Your initial search should focus on identifying existing guidelines that may be suitable for adoption or adaptation (see Adopt, adapt or start from scratch). Important sources of clinical guidelines include the NHMRC Clinical Practice Guidelines Portal and the Guidelines International Network (G-I-N) database (membership required). Alternatively, the National Institute for Health and Care Excellence (NICE) Public Health Guidance and advice list and Australian Department of Health Series of National Guidelines (SoNGs) on communicable diseases are important sources for public health guidelines.

Identified guidelines may address some or all of the questions that will be covered in your guideline. Where some of the questions are not addressed, you should search for existing systematic reviews that may help to fill the gaps. Important sources of systematic reviews include PROSPERO, the Cochrane Database of Systematic Reviews (CDSR), the Trip Database and the Joanna Briggs Institute Database of Systematic Reviews and Implementation Reports.

Where existing guidelines and systematic reviews fail to address all of the questions relevant to your guideline, you will need to conduct an additional search to identify this evidence. The search strategy will consist of two components: a search for evidence relating to questions that have not been addressed; and a date-limited search of evidence to update the relevant guideline or systematic review from the time of the most recent search on which it was based.

Conducting a comprehensive search of multiple evidence sources is a complex task and you should avoid developing your own search strategies if you lack experience. Talk to someone who knows what they are doing—for example, Cochrane review authors are prompted to use (or seek advice from) a trials search coordinator (Cochrane Handbook section 6.3.1). Consider consulting information specialists who are experienced in developing search strategies such as health librarians. If you do not have a systematic reviewer or methodologist within your guideline development group and have sufficient resources, it may be beneficial to commission a systematic review. Consider approaching organisations such as Cochrane Australia, the Joanna Briggs Institute or other health technology assessment (HTA) agencies.

Some key questions to consider when planning a literature search include:

  • What key words will you use in your search?
  • Where will you search? (e.g. which bibliographic databases, grey literature sources)
  • Will you use any filters? (e.g. language, date limits)
  • How will you manage identified references?
  • What bibliographic software will you use for managing your references?

In addition to searching for published data and grey literature, you should devise a plan for identifying unpublished data (Wolfe, Gotzsche et al. 2013). For example, you might need to identify companies, organisations or individual research groups that have conducted research in the topic area and might not have published the results. You will also need to determine the most appropriate method of contacting them, what information you would like to request and how you might acknowledge this work in your review and final guideline.

2. Determine sources of information

There are a wide range of published and unpublished information sources that can inform a systematic review. The primary published sources are bibliographic databases of peer-reviewed journal articles such as MEDLINE, CINAHL, PsycINFO, EMBASE, Cochrane Central Register of Controlled Trials (CENTRAL) and Global Health (via Ovid) (Table 1).

Grey literature is an increasingly important source of evidence. These are defined as any literature or information that is not commercially published or searchable within standard databases. Key sources of grey literature include clinical trials databases, government reports, theses and dissertations. A 2008 Cochrane review demonstrated that results from grey literature can significantly affect the outcome of a review (Hopewell, McDonald et al. 2007). This is due to their increased reporting of negative or inconclusive findings, which generally aren’t included in published studies. In addition, systematic reviews that rely on published evidence miss an average of 64% of data on adverse effects (Golder, Loke et al. 2016).

In addition to bibliographic databases and grey literature, a search for unpublished data should be conducted. This evidence can take many forms, such as clinical trial results, pharmaceutical company data (e.g. after market adverse event data) and cultural knowledge (see Table 1). Primary sources of cultural knowledge might include letters, diaries, taped interviews, photographs and personal journals.

The sources of information that you search will be informed by your topic. For example, if you were conducting a review on treatment options for prostate cancer it would be little use searching TOXNET (a toxicology database) for evidence. However, this database would be integral to any review assessing the environmental health impact of polycyclic aromatic hydrocarbons. Be sure to tailor your search strategy to your topic of interest.

Table 1: Potential sources of evidence to inform systematic reviews

Source of evidence

Description/Examples

Bibliographic databases

  • Allied and Complementary Medicine Database (AMED)
  • Campbell Collaboration Library of Systematic Reviews
  • Cochrane Central Register of Controlled Trials (CENTRAL)
  • Cochrane Database of Systematic Reviews (CDSR)*
  • Cumulative Index to Nursing and Allied Health Literature (CINAHL)
  • Education Resources Information Center (ERIC)
  • EMBASE
  • Guidelines International Network (GIN) Library
  • Health Technology Assessment (HTA) Database
  • Joanna Briggs Institute Database of Systematic Reviews and Implementation Reports
  • MEDLINE
  • Physiotherapy Evidence Database (PEDro)
  • PROSPERO
  • PsycINFO
  • TOXNET (Toxicology Data Network)

Grey literature

  • Conference proceedings
  • Government reports
  • Policy statements
  • Environmental health monitoring or surveillance reports
  • Clinical study reports
  • Newsletters and bulletins
  • Theses, dissertations and essays
  • Research reports
  • Clinical trials registry information
  • Government and non-commercial databases (e.g. Trove)
  • Newspaper articles
  • Essays
  • Company reports
  • Industry data reports

Unpublished evidence

  • Cultural knowledge (e.g. passed on through oral traditions)
  • Confidential industrial data (e.g. pharmaceutical industry)
  • Unpublished research data

* The Australian Government funds free access to the Cochrane Library for all Australian residents.

3. Test your search strategy

Developing a comprehensive search strategy is usually an iterative process. Once you have determined where evidence might be available, you can conduct an initial search to expand the key terms. Your research question(s) will outline the population of interest, the target intervention/exposure, the comparator and one or more outcomes (see Forming the questions). For example, your objective may be ‘to determine the difference in overall survival and sexual function between active surveillance and active treatment for men with low risk prostate cancer’. The goal is to use these basic terms to identify synonyms and related terms to maximise the retrieval of relevant evidence (Table 2).

Search one or two core databases (e.g. MEDLINE, EMBASE) using key terms to identify studies related to your topic of interest. Check that the search has identified some relevant articles. These can be reviewed and additional relevant keywords from within the title, abstract or index. For example, searching ‘active surveillance’ may identify additional terms such as ‘watchful waiting’, ‘active monitoring’ and ‘observation’. You should also search for Medical subject headings (MeSH) that are high level categories used for indexing journal articles within MEDLINE (Table 2). Assessment of the quality of your search strategy is described in the next section.

Your review will generally focus on several outcomes related to your topic of interest. For example, a systematic review may assess safety and effectiveness of chemotherapy for treating liver cancer. This review will include some primary outcomes of interest such as overall survival, response rate and toxicity. However, numerous other secondary outcomes may be reported such as quality of life, incidence of hand-foot syndrome and infection. It is difficult and time consuming to include all possible outcomes in your search strategy. It has also been demonstrated that including adverse events in search terms can fail to identify 4–8% of relevant studies (Golder, Wright et al. 2014). Leaving the outcome field blank ensures that studies presenting all relevant outcomes are identified.

Table 2: Example of basic versus expanded search terms

PI/ECO element

Review question (basic terms)

Expanded search terms

Population

Men with low risk prostate cancer

Prostate cancer, prostate carcinoma, prostatic neoplasms [MeSH], prostate/pathology [MeSH]

Intervention/Exposure

Active surveillance

Active surveillance, watchful waiting, active monitoring, observation

Comparator

Active treatment

Active treatment, radical prostatectomy, brachytherapy, radiotherapy, chemotherapy, cryotherapy, TURP, transurethral resection of prostate, high intensity focused ultrasound, external beam radiotherapy, androgen deprivation therapy

Outcome(s)

Overall survival, sexual function

[avoid using outcomes within your search strategy]*

* The inclusion of outcomes in search strings can reduce the number of relevant studies identified (Golder, Wright et al. 2014)

4. Conduct your search

Searching for evidence can be a complex and time consuming process. For instance, many public health guidelines involve a complex topic that often will have numerous scientific questions to seek answers for (e.g. guidelines for reducing risk from drinking alcohol, dietary guidelines). If your review focuses on a broad or complex topic which needs a complex search strategy it may be more practical to employ an expert to develop the strategy for you. If you develop your own search strategy, consider asking an information specialist to review your work before beginning the search.

Published literature

Once you have developed a comprehensive list of search terms, the next step is to map them into your target databases. Each database is different. For example, a basic search for articles with prostate cancer and radiotherapy in the title or abstract is structured differently depending on the database searched (Table 3). Following a web-based survey of experts, the Cochrane Collaboration has published a Peer Review of Electronic Search Strategies (PRESS) evidence-based checklist (McGowan, Sampson et al. 2016). This validated checklist is used to evaluate the quality and completeness of an electronic search strategy, and criteria fall into six categories:

  • translation of the research question
  • Boolean and proximity operators (these will vary based on the search service)
  • subject headings (database specific)
  • text word searching (free text)
  • spelling, syntax and line numbers
  • limits and filters.

Numerous guides have been developed instructing the user how to design comprehensive search strategies using popular databases (e.g. PubMed tutorial, University of Adelaide EMBASE guide, Monash University Database operators). To assist in the development of search strings, Bond University’s Centre for Research in Evidence-Based Practice (CREBP) has developed a Polyglot Search tool. This program converts search strings specific to one database into a format applicable to other databases. You should also consider reviewing previous systematic reviews for examples of appropriate search strategies.

Table 3: Examples of variations in search strings between databases

Database

Search string

Search terms

prostate cancer and radiotherapy in title or abstract

MEDLINE

‘prostate cancer’ [TIAB] AND radiotherapy [TIAB]

EMBASE

‘prostate cancer’ .ti,ab. AND radiotherapy .ti,ab.

CINAHL

(TI ‘prostate cancer’ OR AB ‘prostate cancer’) AND (TI radiotherapy OR AB radiotherapy)

You should consider applying filters to your search such as language, date or gender. For example, if you were conducting a review on robotic prostatectomy for prostate cancer you could limit your search to studies after the year 2000 when the first robotic prostatectomy was described (Shuford 2007). Although applying filters can improve the specificity of your search, care must be taken to avoid excluding relevant studies. This most frequently affects studies published before 1990 due to non-standardised reporting and indexing (Relevo 2012).

You should also consider hand-searching relevant journals. A Cochrane review demonstrated that 35% of randomised controlled trials identified by hand-searching were not indexed in MEDLINE. The majority of these were reported in meeting abstracts and supplementary articles published prior to 1991 (Hopewell, Clarke et al. 2007).

Grey literature

Unlike bibliographic databases, there are no single strategies for grey literature searching. Due to the large range of potential sources, searching grey literature can be highly subjective. You need to take great care to ensure that the most important sources of information are checked. The Canadian Agency for Drugs and Technologies in Health has developed a source guide of grey literature where it lists a variety of government and HTA organisations through which grey literature can be searched (Grey Matters). Other popular sources of grey literature include clinical trials databases (Australian New Zealand Clinical Trials Registry, ClinicalTrials.gov), dissertations and theses databases (e.g. Proquest, EThOS) and the New York Academy of Medicine’s Grey Literature Report.

In addition, guideline development groups and expert committees are also a source of relevant grey literature, with members having a prior history of working on issues that resulted in grey literature reports, or knowing colleagues who have done so.

5. Manage references

When you have conducted your searches and identified references you will need a flexible system to store and sort these references. Citation management software provides a platform where you can record the number of citations identified and where they were identified. This is both important for record keeping and also helps in the screening of articles, which is the next step in developing your review (see Selecting studies).

There are several programs available to manage citations, such as Endnote, Refworks, RevMan (Cochrane), Covidence (Cochrane), Mendeley, Zotero and Rayyan. These all differ in cost, functionality and the ability to import and export multiple file types. Many offer a free trial period and you should consider testing a few to determine which one works best for you. When determining which citation management software is right for you consider the following key attributes:

  • its ability to remove duplicate references
  • its ability to add full text and store PDFs
  • its ability to sort and organise references into subfolders
  • whether it contains a cite-while-you-write feature, enabling easy integration of references into word processing programs
  • its ability to export the references in multiple formats.

Once your search is completed, export all of the identified references to your citation management software. You should then separate citations into subfolders based on information source (e.g. one folder for MEDLINE, one for CINAHL). When this process has been completed you are ready to remove duplicates and screen individual references (see Selecting studies). Numerous tutorials have been developed to assist users (e.g. ‘The Little Endnote How-To Book’, the Zotero ‘quick start guide’, ‘Getting started with Mendeley Desktop’). Several library guides specific to citation management for literature reviews are also available. Cochrane provides extensive instruction on the use of Covidence and REVMAN.

6. Document the process

Thorough documentation of the search process is needed to demonstrate transparency and reproducibility. You should record the following:

  • the databases searched
  • the exact search strategy employed in each database
  • any filters used during the search
  • the exact date(s) of the searches
  • the number of articles identified within each database.

This information makes it both easier to update the review in the future and also provides the basis for the development of a study selection flow diagram. This diagram demonstrates the evidence identification and study selection process and is an important part of reporting for a systematic review. The most widely used method is the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram (Figure 1). This method has been refined and now contains several extensions focused on individual patient data, network meta-analysis and equity (see Equity). In this figure the red circle represents the first step and lists how many references were identified and where they were identified. This template is used when selecting studies for your evidence synthesis (see Selecting studies).

Figure 1: A version of the PRISMA flow diagram demonstrating the process of systematic review study identification and selection

* Number of studies may differ to the number of articles, as multiple articles may present data relating to an individual study

Your review should also document the exact strategies used during the search process. Be sure to present the exact search strings used, the search date, the filters that were used and any other information that improves transparency. The information you provide should facilitate exact replication of your search strategy. These can either be incorporated into the final guideline or be included in a technical document that accompanies the guideline.

7. Select your articles/studies

After running your searches, the next step is to screen the identified articles to determine whether they meet your inclusion criteria (see Decide what evidence to include). You will likely find that many of the articles will be irrelevant as you sort through them. The correct use of citation management software is an important means of tracking the process. You should develop subfolders for different categories such as duplicate references (the same article identified in two or more databases), articles excluded following title/abstract screening and articles excluded following full text screening. For more information on the study selection process see Selecting studies.

NHMRC requirements

Guidelines approved by NHMRC must meet all requirements outlined in the Procedures and requirements for meeting the NHMRC standard. The following requirements are specific to the Identifying the evidence module:

  • C.2. Systematic searches for evidence are undertaken and the search strategy is documented, including the search terms and databases searched.
  • C.3. The population groups specified in the search strategy include Aboriginal and Torres Strait Islander peoples and any population subgroups that have been identified (see Requirement B.4 and B.5)
    • B.4. The population to which the guideline recommendations will apply is defined (e.g. children, adolescents, adults or older adults) and population subgroups for which specific information is required are identified and described.
    • B.5. Issues relevant to Aboriginal and Torres Strait Islander peoples (such as particular risks, treatment considerations or sociocultural considerations) are identified and described
  • C.4. The publication period covered by the searches is stated, and the latest date is within 12 months of the first day of public consultation and within 20 months of submission of the final draft guideline to NHMRC for approval.
  • C.3.1. (desirable) The population group specified in the search strategy include groups such as culturally and linguistically diverse communities or other group for whom specific sociocultural factors (including ethnicity, gender, age, disability, socioeconomic status and location) in treatment or prevention outcomes should be considered.
  • C.3.2. (desirable) Search strategies include search terms to identify evidence related to consumers’ perceptions and experiences.
  • C.3.3. (desirable) Dependent on the guideline scope, the search strategy is designed to identify evidence for all relevant alternatives for screening, prevention, diagnosis or treatment of the condition addressed by the guideline, including relevant complementary and alternative medicine approaches.
  • C.3.4. (desirable) Search strategies include search terms to identify evidence related to cost effectiveness and resource implications of practice.

NHMRC Standards

The following Standards apply to the Identifying the evidence module:

2. To be transparent guidelines will make publicly available:

  2.1. The details of the processes and procedures used to develop the guideline

  2.2. The source of evidence.

6. To be evidence informed guidelines will:

  6.1. Be informed by well conducted systematic reviews

  6.2. Consider the body of evidence for each outcome (including the quality of that evidence) and other factors that influence the process of making recommendations including benefits and harms, values and preferences, resource use and acceptability.

8. To be up-to-date guidelines will:

  8.1. Ensure that the recommendation is based on an up-to-date body of evidence.

Useful resources

Cochrane Handbook, Chapter 6 (Searching for studies)

Canadian Agency for Drugs and Technologies in Health (CADTH): Peer Review Electronic Search Strategy (PRESS) explanation and elaboration (including checklist and assessment form)

The International Centre for Allied Health Evidence – A short course in searching the literature

The InterTASC Information Specialists’ Sub-Group Search Filter Resource (a collection of search filters by study design or focus)

The Joanna Briggs Institute Reviewers Manual: p. 137-164 ‘An introduction to searching’

Automated search databases Epistemonikos and the National Institute of Health and Care Excellence Health Database Advanced Search (HDAS)

Canadian Agency for Drugs and Technologies in Health (CADTH): Grey Matters: a practical tool for searching health-related grey literature

References

Adams, R. J., Smart, P., et al. (2017). Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies. International Journal of Management Reviews 19(4): 432-454.

Brænd, A. M., Straand, J., et al. (2016). Publication and non-publication of drug trial results: a 10-year cohort of trials in Norwegian general practice. BMJ open 6(4).

Golder, S., Loke, Y. K., et al. (2016). Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Medicine 13(9): e1002127.

Golder, S., Wright, K., et al. (2014). Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review. Systematic Reviews 3: 113-113.

Hopewell, S., Clarke, M. J., et al. (2007). Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reviews(2).

Hopewell, S., McDonald, S., et al. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews(2).

McGowan, J., Sampson, M., et al. (2016). PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of clinical epidemiology 75: 40-46.

Montori, V. M., Wilczynski, N. L., et al. (2005). Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 330(7482): 68.

Popay, J., Roberts, H., et al. (2006). Guidance on the conduct of narrative synthesis in systematic reviews: a product of the ESRC methods programme (Version I). Lancaster, UK, University of Lancaster.

Relevo, R. (2012). Effective Search Strategies for Systematic Reviews of Medical Tests. Methods Guide for Medical Test Reviews [Internet]. C. S.M., M. D.B., S. G.W. and e. al. Rockville (MD), Agency for Healthcare Research and Quality (US). Chapter 4.

Shuford, M. D. (2007). Robotically assisted laparoscopic radical prostatectomy: a brief review of outcomes. Proceedings (Baylor University. Medical Center) 20(4): 354-356.

Thomas, J., Noel-Storr, A., et al. (2017). Living systematic reviews: 2. Combining human and machine effort. Journal of clinical epidemiology 91: 31-37.

Wolfe, N., Gotzsche, P. C., et al. (2013). Strategies for obtaining unpublished drug trial data: a qualitative interview study. Syst Rev 2: 31.

Acknowledgements

NHMRC would like to acknowledge and thank Professor Dianne O’Connell from Cancer Council NSW and Associate Professor Zachary Munn from the Joanna Briggs Institute for their contributions to this module as editors.

 

Version 3.0. Last updated 23/11/2018.

Suggested citation: NHMRC. Guidelines for Guidelines: Identifying the evidence. https://nhmrc.gov.au/guidelinesforguidelines/develop/identifying-evidence. Last published 23/11/2018.

ISBN: 978-1-86496-024-2