Baudard, M., Yavchitz, A., Ravaud, P., Perrodeau, E., & Boutron, I. (2017). Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ, 356.
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015). Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews, 4(1).
Haddaway, N. R., Collins, A. M., Coughlin, D., & Kirk, S. (2015). The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PLoS ONE, 10(9).
Hartling, L., Featherstone, R., Nuspl, M., Shave, K., Dryden, D. M., & Vandermeer, B. (2017). Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Medical Research Methodology, 17(64), 1–11.
Higgins, J.P.T. & Green, S. (Eds.). (2011). 6.2.3 Unpublished and ongoing studies In Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration. Available from www.handbook.cochrane.org.
Hopewell, S., Mcdonald, S., Mj, C., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions (Review). Cochrane Database of Systematic Rev, (2. Art. No.:MR000010.).
Mahood, Q., Van Eerd, D., & Irvin, E. (2014). Searching for grey literature for systematic reviews: challenges and benefits. Research Synthesis Methods, 5(3).
McAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000).Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? The Lancet, 356(9237), 1228–1231.
Saleh, A. A., Ratajeski, M. A., & Bertolet, M. (2014). Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evidence Based Library and Information Practice, 9(3), 28–50.
Schmucker, C. M., Blümle, A., Schell, L. K., Schwarzer, G., Oeller, P., Cabrera, L., … Consortium, on behalf of the O. (2017). Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLOS ONE, 12(4),
Stansfield, C., Dickson, K., & Bangpan, M. (2016). Exploring issues in the conduct of website searching and other online sources for systematic reviews: how can we be systematic? Systematic Reviews, 5(1).
Publication bias
Papers with ‘interesting’ results are more likely to be
- Submitted for publication
- Accepted for publication
- Published in a major journal
- Published in the English language
Goldacre, B. (2012) What doctors don't know about the drugs they prescribe, Available [Accessed 2nd June 2014].
Publication bias refers to the fact that studies with more ‘positive’ results – i.e. which show a definite effect for a particular treatment – are more likely (three times more likely in fact) to be published than ones which show little or no positive effect for a treatment.
Researchers themselves often think that studies which show no or little effect for a treatment aren’t worth publishing so these studies are less likely to be submitted for publication – the so-called ‘file-drawer’ problem. Once submitted they are less likely to be accepted by a journal, less likely to be published in a ‘high-impact’ journal and less likely to be published in English making them all the more difficult to find.
Less scrupulous researchers are sometimes known to selectively report the results of trials, reporting results that show treatments in a good light while glossing over ones which show that it is ineffective in a process called HARKing – Hypothesizing After the Results are Known.
Looking for grey literature can unearth the unpublished trials which show interventions in a less-than-glowing light and can significantly affect the outcome of a systematic review. A good example of this is this study into the anti-depressant Agomelatine (see the box on this page).
Grey Literature resources/Databases
Global Health (Ovid) King’s username and password for access
PsycEXTRA (APA) Senate House Library membership required - click on more for full details
Other internet resources
Best Practice (BMJ) (King's) This link opens in a new window King's username and password for access off campus
UpToDate This link opens in a new windowKing's username and password required for access off campus. Initial registration required - see More for details
TRIP This link opens in a new window
Agomelatine: an example of publication bias
Howland, R.H. (2011), 'Publication bias and outcome reporting bias: agomelatine as a case example', Journal of Psychosocial Nursing & Mental Health Services, 49(9): 11-4. DOI: 10.3928/02793695-20110809-01
Publication bias and outcome reporting bias contribute to distorted perceptions of drug efficacy and the underreporting of adverse events. To demonstrate these biases, this article describes how the clinical profile of the antidepressant agent agomelatine (Valdoxan) has been presented in the literature. Agomelatine has been systematically assessed in 10 short-term placebo-controlled studies and three long-term placebo controlled relapse prevention studies. Five published trials demonstrated clinically modest but statistically significant benefits over placebo. Five unpublished trials did not find agomelatine more effective than placebo, but in two of these studies the active comparison drug fluoxetine or paroxetine was found to be more effective than placebo.
Agomelatine was more effective than placebo in only one of three relapse prevention studies, but only the positive study was published. Based on what is evident in the entire published and unpublished dataset, agomelatine does not have a tremendously superior sleep and sexual effects profile. The risk of liver toxicity is also not prominently highlighted in the published literature.
Data repositories
A data repository is an archive that manages the long-term storage and preservation of digital resources and provides a catalogue for discovery and access. There are a large number of data repositories some with a general coverage and many that are subject specific. Depending on the subject of your systematic review you may find it useful to consider searching for data. The links below are just a small number of the data repositories that are available. Some funders and journal publishers specify specific data repositories so check the guidelines for researchers in your field to find which ones they recommend.
The King's Research Data Management System is a a research data repository service providing long term storage and public access for datasets that support published research and/or have long term value. Search strategies fall into this category. View the Libraries & Collections Research Support webpage (Preserve > Deposit with King's tab) for more information on how to submit. Please note that this is normally limited to researcher and PhD level.
Other data repositories are also linked from the Libraries & Collections Research Support webpage (Preserve > Deposit your data tab) and some key ones are listed below:
What is Grey Literature?
"The term ‘grey literature’ is often used to refer to reports published outside of traditional commercial publishing. Review authors should generally search sources such as dissertations and conference abstracts" Cochrane Handbook v.6.3 - Part 2, section 4.3.5 Other sources.
Formats include:
- Registered Controlled Trial Registers
- Technical or research reports from government agencies
- Reports from scientific research groups
- Working papers from research groups or committees
- Doctoral (PhD) dissertations
- Some conference proceedings and official publications
- Preprints (journal articles not yet peer-reviewed and/or published)
PhD Theses and Dissertations
One of the best ways of searching for dissertations and theses is the WorldCat database produced by OCLC which is available via the university's database page. Follow the links through to WorldCat in the same way as you access other databases. When you get there you will need to select King’s College and log in with your King's user name and password.
EThOS This link opens in a new windowRegistration may be required - see More for details
WorldCat Dissertations and Theses (OCLC) This link opens in a new windowKing’s username and password for access. Once logged in, select 'List all Databases'
Open Access Theses and Dissertations This link opens in a new windowFree access
Controlled Trial Registers
Trial Registers are a useful source of unpublished and ongoing trials:
US ClinicalTrials.gov
The US ClinicalTrials.gov registry of clinical trials also links to articles and other publications relating to a trial where known.
WHO International clinical trials registry platform search portal
The Clinical Trials Search Portal provides access to a central database containing the trial registration data sets provided by various country registries. It also provides links to the full original records.
ISRCTN (International Standard Randomised Controlled Trial Number) Registry
A primary clinical trial registry recognised by WHO and ICMJE that accepts all clinical research studies (whether proposed, ongoing or completed), providing content validation and curation and the unique identification number necessary for publication. All study records in the database are freely accessible and searchable.
NIHR Be Part of Research
Search the NHS National Institute for Health Research's (NIHR) Be Part of Research website to find clinical trials open to UK participants. The data is retrieved from 2 databases ISRCTN and ClinicalTrials.gov. Note: the search initially retrieves only trials which are actively recruiting but this setting can be changed in the filters.
Conference Proceedings
You can search for conference proceedings using a number of the databases available via the Library Services’ web page. On Medline you can search for them by combining a search for Congresses with a subject search for whatever topic you are looking for. You can also restrict your search to conference proceedings when using the Web of Science database.
SCOPUS (Elsevier) This link opens in a new window King’s username and password for access off campus
Web of Science Core Collection This link opens in a new windowKing’s username and password for access off campus
Hand-searching and scanning reference lists
Hand-searching involves looking through the contents pages of journals, conference proceedings and abstracts page by page. This process can identify articles (and other items, e.g. letters) which have not yet been included in electronic databases, and those which are not indexed or have been indexed incorrectly. Deciding which journals to search through in this way can be done by analysing the results of your database searches to see which journals contain the largest number of relevant studies.
You can also find other studies by looking through the references of papers you have found by searching electronically.
Citation indexes are another extremely useful tool to help you locate papers and other relevant research which may have been missed in your more formal database searches. Consider using a database like Web of Science or Scopus, or undertake a search on Google Scholar, Microsoft Academic, Semantic Scholar or ResearchGate to look at 'Cited by' information for papers you know are of interest and seeing who has cited them since they were published. Some search engines, and journal websites also offer 'recommended for you' links which can also surface research of interest.
Contacting authors, experts, and other organisations
You may be able to get hold of further information about unpublished or ongoing research by making contact with study authors, and organisations, e.g. drugs companies and research centres. Experts can also be contacted and shown the list of evidence which you have found to see if they know about any articles you have missed out. If you have a supervisor for your systematic review then remember to check with them to see whether they can spot any key research not included in your results.
Why is Grey Literature important?
Searching Grey Literature to include in your systematic review:
- gives currency to your review as upcoming research can be located, for example in conference proceedings, preprints or trial registers, which show research which is ongoing but
hasn't yet been formally published.
- PhD theses can be either at the cutting edge of research and/or in a niche or esoteric area where there might not be much (if any) published research.
- Conference proceedings – either a paper, presentation or just a poster – can often be the first place where new developments or breakthroughs are announced
- helps avoid positive results publication bias (negative results are less likely to be published in a formal journal but may have been reported at a conference).
- acts as a double check – it may allow you to find important published research which has been missed in your database searches (allowing you to edit your search strategy to improve results if there is time or simply report as one located during the grey lit search).
- One example was where a search in Medline for RCTs on dietary interventions during pregnancy failed to return a relevant result because the abstract mentioned “lifestyle interventions” only so the keyword search failed and the MeSH headings also did not pick up the specific ‘diet’ element in the indexing. However, the diet intervention was highlighted on the clinicaltrials.gov website and so when this was searched it flagged up that results had been published and gave a link to the abstract.
There are a variety of ways you can search grey literature but you need to look in the right places for the right resources. Some databases specialise in technical reports, unpublished research and the work of government departments while others specialise in clinical trials, conference proceedings and theses. Depending on what kind of material you are looking for, what topic you are researching and how much time you have available you might decide to search only one of these databases, a few of them or, if you want to be really thorough, search them all.
Grey Literature Guides
Grey Matters
The Canadian Agency for Drugs and Technologies in Health have produced this very useful guide to international sources of grey literature – including those in the UK – available online.
Exercise on Grey Literature
Exercise on Grey Literature
Exercises on grey literature resources including Open Grey. ClinicalTrials.gov, Web of Science Conference Proceeedings and WorldCat Dissertations and Theses.
Finding out more details - Conference Proceedings
If you have located the abstract of a conference paper published in a journal and then indexed in a database then you may wish to follow this up in order to get more information.
Two common further actions you may wish to take:
- See if there is a website for the conference in question (search the internet for the name of the conference) as sometimes you find archived information over and above what is included in abstracts which appear in journals.
- Search for the author(s)/their institution (this information is normally included under author affiliation in the abstract) to locate a list of their publications.
- Some institutions have an institutional repository where conference posters or fuller information than an abstract provides may have been stored.
- Many institutions have profiles for
academics where they list their publications (or link to a full list on a site like ORCID)
- This can be useful to see whether the conference proceeding has been written up later as a full article (just in case you have not located this in your database searches).
- You could also consider contacting the author directly to see if they can send you some further information/are happy to discuss their research with you with regard to the specific findings you are interested in.
- Sometimes authors have moved institutions so it becomes a bit of a mission to track them down – LinkedIn, ResearchGate or Author ID systems where academics create profiles (including ResearcherID on Web of Science and ORCID) are all sites which can bring together all publications for an author and hopefully a link to the current institution.
Preprint servers
A preprint is a version of a scholarly or scientific paper that often precedes formal publication. Free preprints may be available before and/or after a paper is published in a
journal. There is no guarantee when reading a preprint that the article has undergone formal peer review (or even publication) in a peer-reviewed scholarly or scientific journal, without undertaking further checking for publication details.
A preprint will therefore allow free access to an article by all interested parties regardless of whether they subscribe to a particular journal or not. On some occasions the preprint may not yet have been formally accepted to a journal or
undergone peer review but the author may have chosen to make a preprint available in order to quickly circulate some new information or ask for informal feedback from their peers prior to publication.
Preprints are hosted in many places:
- Institutional repositories e.g. King's Research Portal
- Preprint databases or archives or servers e.g. arXiv.org; (Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, and Economics); biorxiv (Biology); SSRN (multi-disciplinary including Medicine, Biology and Chemistry); OSF Preprints (covers wide subject area across arts and humanities, social sciences, life sciences and medicine).
- Open access publishing platform e.g. Wellcome Open Research; F1000 Research
Many preprint servers and open access publishing platforms do undertake a basic screening of content uploaded and may check for plagiarism but whilst some offer a post publication peer review option (community mediated peer review) others are simply repositories.