The Evidence Based Pyramid: Laser-focused Search Technology

 Abe Lederman, Deep Web Technologies and Dr Sam Keim

Imagine front-line medical personnel under time pressure to diagnose and treat patients with a wide range of conditions. These brave souls have only their skills, their experience, and their resources (human, paper, and electronic) to rely on. A new breed of technology is evolving to help medical staff to quickly identify the treatments most likely to be effective.

The intersection of best medical practices and leading edge computerized clinical decision support systems (CDSS) is fluid and fast-moving. Clinicians are likely amazed by the wide array of information sources that are available. Unfortunately, most are naïve to the quality of the information source and have little time or experience to do fact-checking. Many products are also difficult to navigate without additional training. For these reasons, clinicians have been relatively poor adopters of CDSS.

Emerging computerized search technology, however, might allow more clinicians access to state of the art medical information. This technology can increase accuracy of information acquisition and reduce frustrating time delays. One such technology already helps emergency clinicians and other medical staff quickly assess medical answers and identify the best treatments faster. This technology has two foundational goals: first, to provide the clinician with information pre-graded by quality; and second, to deliver accurate information extremely fast. Quality ranking is accomplished by utilizing the scientific strength underlying each clinical publication. These concepts are central to the now widely promoted discipline of Evidence Based Medicine (EBM).

EBM is the mindful use of modern technology to bring the best relevant medical information to bedside clinical decisions. Incorporating EBM greatly benefits patients who gain access to the best and latest scientific advancements in the world. EBM simultaneously benefits the provider by facilitating powerful uptake of new knowledge. In short, EBM is the solution to the chasm that exists between proven medical discoveries and clinical practice. The alternative to practicing EBM for clinicians is to “just use your personal experience and gut-feelings.” This is increasingly recognized as irresponsible and unacceptable to patients, providers and healthcare policy makers.

Federated search, also known as distributed search, is a technology that searches huge parts of the web that Google and the other crawlers can’t get to. This “deep web” is where a large amount of the highest quality academic information, scientific reports, legal documents, and medical research and clinical study information lives. It’s the fact that much of the medical information that a clinician needs is in the deep web that makes diving beneath the surface so critical to the success of EBM in providing bedside care. Not only is Google not good enough for practising EBM, but neither are medical textbooks or individual online databases. The tools supporting the practice need to be much focused and comprehensive.

URNation provides a great introduction to EBM in the form of slides along a timeline. The presentation begins with the first clinical trial in 1747 and covers nearly twenty milestones leading up to the state of EBM today.

EBM is built on a foundation of evidence that is organized into hierarchical levels. Higher levels of the pyramid include evidence that is more scientifically rigorous and therefore higher quality. The evidence is also likely more problem specific. Evidence at the lower levels includes evidence that has resulted from less scientifically-rigorous methods and is typically more general in nature. Search technology that delivers results filtered according to the EBM pyramid allows for powerful advantages to the user in both time efficiency and quality impact.

Traditional federated search meets the needs of scientists, researchers and academics who need to find quality articles that are out of Google’s reach. Santa Fe-based Deep Web Technologies (DWT) has extended the traditional offering to add a laser focus component in developing its flagship medical product, Explorit Everywhere! EBM. DWT collaborated with Samuel Keim, MD, MSc, Professor, and Chair of the Department of Emergency Medicine at the University of Arizona to develop the system.

It takes three major elements to deliver a focused search tool informed by EBM best practices:

  1. Start with the information sources backed by the best evidence
  2. Provide an intuitive and extremely friendly user interface
  3. Organize results within the framework of the evidence pyramid

Below we highlight the steps a researcher or clinician takes to search with Explorit Everywhere! EBM.

Select the speciality

The user selects from one of 28 specialties.

Selecting a specialty determines which targeted (curated) resources are used for the search and narrows the focus vs. a shotgun approach of searching a large pile of medical resources. Our focus is on selecting fewer sources for each specialty, in particular the ones medical experts select that provide the strongest clinical evidence. Contrast this with the Google approach of providing a ton of results, many of questionable value. We eliminate the effort of sifting through irrelevant results and of wondering whether the sources are vetted.

We select the best sources using a combination of Doody’s review service (for textbooks) and the input of experts within the specialties. Doody’s is highly regarded in the health sciences community for its expert reviews of specialty titles.

Beyond specialty selection, we improve search efficiency in various ways:

  1. We search sources simultaneously and present result summaries in a single page. This includes sources requiring authentication. So, there’s no need to perform multiple searches and compare result screens.
  2. We can integrate a library’s subscribed content with public content into the evidence pyramid, and select sources to include by specialty. The aim is to provide users with exposure to a broad range of content. And, we balance this focus on breadth with the ability to highlight favourite sources such as DynaMed Plus and UpToDate in their own tabs in the result page.
  3. For PubMed, we limit searches to “core clinical” and specialty high impact factor journals. We further limit searches to the last ten years of human studies in English-only articles.
  4. We search only three to five of the best textbooks per specialty in the textbook tab. Customers can select online textbook sources that they have subscriptions for.
  5. Other resources include all of ClinicalKey, STAT!Ref and Psychiatry Online. Like the textbooks, customers can customize these. 

Perform quick or advanced search

The optional advanced search allows for further focusing based on full record, title, author, date, and other fields. And, in setting up the form for the advanced search, the user can target individual layers of the pyramid. For example, you can choose to search only the resources at the top layer of the pyramid — Systematic Reviews.

Regardless of whether or not the user customizes the search fields, we optimize the search results in various ways to further maintain user focus.

Review search results

  1. We organize results into sequential and colour-coded tabs, following the levels of the evidence pyramid with the Cochrane and systematic reviews (coral coloured) results on the left.

  1. We provide highlighted abstracts and snippets directly on the results page, saving the effort of clicking through to an article that’s not what the user is looking for.

  1. Similar to the point above, we provide inline author conclusions (for some sources).

  1. BrowZine integration (for some sources) allows users to browse other articles in the journal in which a particular result appears.

5. We customize ranking to favour a mix of relevant and recent articles. In particular, articles that are three years old or newer are displayed early in the search results if they are relevant to the search terms.

  1. We recommend medical terms (using the UMLS service) to replace common terms. UMLS – the Unified Medical Language System – per Wikipedia, “is a compendium of many controlled vocabularies in the biomedical sciences.” The UMLS feature suggests, for example, somnambulism, as a replacement search term for sleepwalking. Medical resources are more likely to find better matches using the medical term. We also catch typos and suggest the correct spelling.

A demo with public sources is available at

The future of EBM

EBM is a model for selecting a patient treatment that is supported with the strongest evidence. The future of EBM is to structure patient data so that it is easy to analyze with machine learning techniques to determine the best treatment given historical data for millions of patients. Such a system, of course, will learn continuously as artificial techniques evolve and as the gathered data grows larger and more comprehensive. While the technology may change, the focus won’t — we learn from what has worked before.

Biographical note

Dr. Samuel Keim has dedicated thirty years of research and teaching focused on EBM. Dr. Keim’s pioneering work with federated search and EBM is described in an article he co-authored, Evidence-based Medicine Search: a customizable federated search engine.

Abe Lederman is the founder and CEO of Deep Web Technologies

K & IM Refer 33 (3), Winter 2017


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s