Criteria for Evaluating a Search Strategy

A search strategy is evaluated by examining whether it retrieves the right information completely, accurately, efficiently, and reproducibly. In information retrieval and evidence synthesis, no single measure is sufficient: a strong evaluation combines quantitative retrieval measures, methodological checks, user-centered outcomes, and quality assurance.

The central evaluation problem is balancing Recall and Precision. Systematic reviews generally prioritize high recall or sensitivity, while operational and web searches may place greater emphasis on precision and ranking quality.2

A useful evaluation model is:

Footnotes

1. Evaluation measures (information retrieval) - Definitions of precision, recall, F-measure, and test-collection evaluation. ↩

2. Sensitivity vs. Precision - Explains the balance between sensitivity and precision in systematic-review searching. ↩

1. Coverage and Completeness

Coverage is the first criterion. A search strategy should retrieve the important records needed to answer the research question, including difficult-to-find, older, poorly indexed, or differently worded sources.

Indicators of coverage

• Retrieval of all or nearly all known-item records.
• Representation of every major concept in the research question.
• Inclusion of synonyms, spelling variants, acronyms, historical terminology, and related expressions.
• Coverage across appropriate databases, catalogs, websites, repositories, and grey-literature sources.
• Retrieval of seminal studies, recent studies, and records from different geographic or disciplinary contexts.
• Successful identification of records found through citation searching, expert recommendations, or reference lists.

In systematic reviews, sensitivity is commonly expressed as:

$$\text{Recall} = \frac{\text{Relevant records retrieved}}{\text{All relevant records available}}$$

The denominator is often difficult to know because the complete set of relevant records is rarely observable. Therefore, evaluators use a benchmark set of known relevant records, reference checking, citation searching, and supplementary databases.2

Known-item testing

A practical test is to compile a set of relevant articles and determine whether the strategy retrieves them. If a strategy misses known relevant studies, investigate whether the cause is:

• Missing synonyms or subject headings.
• Incorrect Boolean logic.
• Inappropriate database limits.
• Inadequate truncation or phrase searching.
• Indexing differences between databases.
• An overly restrictive publication or language filter.

Footnotes

1. Evaluation measures (information retrieval) - Definitions of precision, recall, F-measure, and test-collection evaluation. ↩

2. Write a Search Strategy - Discusses synonyms, subject terms, databases, and methods for improving sensitivity and precision. ↩

2. Precision and Relevance

Precision measures the exactness of the search. A search with high precision produces fewer irrelevant records and reduces screening burden.

$$\text{Precision} = \frac{\text{Relevant records retrieved}}{\text{Total records retrieved}}$$

A search may achieve high recall by using many synonyms and broad terms, but this can lower precision. Conversely, a highly restrictive search may improve precision while excluding relevant material. This trade-off is especially important in systematic reviews, where high sensitivity is usually preferred even at the cost of screening more irrelevant records.2

Questions for evaluating precision

• Are most retrieved records related to the research question?
• Does the search retrieve records because of meaningful conceptual matches rather than incidental words?
• Are ambiguous terms producing large quantities of irrelevant material?
• Are field restrictions, phrase searching, proximity operators, or subject headings being used appropriately?
• Are unnecessary databases or search blocks producing duplicate or irrelevant results?

A useful combined measure is the F-measure:

$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision}+\text{Recall}}$$

The $F_1$ score gives equal weight to precision and recall. When recall is more important than precision, a weighted measure such as $F_\beta$ may be used.

Footnotes

1. Sensitivity vs. Precision - Explains the balance between sensitivity and precision in systematic-review searching. ↩

2. How to write a search strategy for your systematic review - Describes the preference for sensitivity in comprehensive evidence searches and the resulting precision trade-off. ↩

3. Evaluation measures (information retrieval) - Definitions of precision, recall, F-measure, and test-collection evaluation. ↩

3. Relevance to the Information Need

A strategy should be judged against a clearly defined Information need rather than against a vague topic description.

Evaluation should determine whether the strategy reflects:

• The population, setting, or context.
• The intervention, phenomenon, technology, or subject.
• The comparator, if relevant.
• The outcome or information type.
• The study design or document type, when appropriate.
• The date, geographic, language, or publication requirements.
• The intended audience and level of detail.

A search can be technically correct yet conceptually weak. For example, a strategy may use valid Boolean syntax but omit a central population synonym or represent an outcome too narrowly. Conceptual alignment is therefore evaluated by mapping each search block to the research question or eligibility criteria.

Concept-block evaluation

A well-designed strategy generally:

1. Identifies the main concepts.
2. Connects synonyms within each concept with OR.
3. Connects distinct concepts with AND.
4. Uses exclusion operators such as NOT cautiously.
5. Applies filters only when their effects are understood.

For example:

$$(\text{Concept A}_1 \ OR \ \text{Concept A}_2) \ AND \ (\text{Concept B}_1 \ OR \ \text{Concept B}_2)$$

4. Logical and Technical Correctness

Technical correctness concerns whether the search platform interprets the strategy as intended. Evaluation should include:

Controlled vocabulary can improve consistency when records use different natural-language terms. However, controlled vocabulary alone may miss newly published or poorly indexed records, so it should normally be combined with free-text searching.

Footnotes

1. Write a Search Strategy - Discusses synonyms, subject terms, databases, and methods for improving sensitivity and precision. ↩

5. Efficiency and Workload

Efficiency evaluates whether the strategy uses resources responsibly.

Relevant indicators include:

• Number of records retrieved.
• Number and percentage of duplicates.
• Number of relevant records per 100 or 1,000 records screened.
• Screening time per relevant record.
• Search development and translation time.
• Cost of database access, expert time, and document retrieval.
• Number of databases searched relative to additional unique records found.
• Contribution of each search block or supplementary method.

A strategy with lower precision is not automatically poor. In a systematic review, retrieving many irrelevant records may be acceptable if it prevents missing an important study. The correct judgment depends on the consequences of false negatives and false positives.

Yield by source

The evaluator should compare the unique contribution of:

• Each database.
• Citation chasing.
• Reference-list checking.
• Grey-literature searching.
• Expert consultation.
• Handsearching.
• Search alerts or update searches.

A source that produces many duplicates but no unique relevant records may be less valuable than a smaller source that identifies unique evidence.

6. Ranking and Result-Order Quality

When a system ranks results, evaluation should examine whether relevant records appear early enough to be useful.

Common ranking measures include:

• Precision@k.
• Recall@k.
• Mean reciprocal rank.
• Mean average precision.
• nDCG.

For a top-$k$ result set:

$$P@k = \frac{\text{Relevant records among the first } k}{k}$$

Ranking evaluation is particularly important when users inspect only the first page or first few results. A strategy may have acceptable overall recall but poor practical usefulness if relevant records are buried deep in the result list.

Footnotes

1. A practical guide to search relevance metrics and evaluation - Reviews precision@k, recall, F1, MAP, MRR, and nDCG. ↩

7. Reproducibility and Transparency

Reproducibility is a central criterion for academic and systematic searching. A reproducible search records enough information for another person to understand and repeat the process.

At minimum, document:

• Database or information source name.
• Platform or interface used.
• Complete search string.
• Database-specific subject headings and field codes.
• Search date and time, when relevant.
• Date coverage.
• Language and publication limits.
• Study-design or document-type filters.
• Number of records retrieved.
• Deduplication method.
• Search updates and amendments.
• Citation-searching and supplementary methods.
• Names or roles of search developers and reviewers.

PRISMA 2020 and PRISMA-S emphasize reporting the full search strategy and search details so that the process can be assessed, replicated, and updated.2

A reproducible strategy should also be version-controlled. If the search is modified, preserve the original version, explain the change, and report how the modification affected retrieval.

Footnotes

1. PRISMA 2020 Statement - Reporting guidance for systematic reviews, including transparent reporting of search methods. ↩

2. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches - Detailed reporting guidance for literature searches. ↩

8. Peer Review and Quality Assurance

Peer review evaluates both the technical construction and conceptual adequacy of a search strategy. The PRESS guideline is designed to identify errors in electronic search strategies, including problems with:

• Translation of the research question.
• Boolean and proximity operators.
• Subject headings.
• Text-word searches.
• Spelling, syntax, and line combinations.
• Limits and filters.
• Missing concepts or inappropriate terms.

Cochrane guidance strongly encourages peer review by an experienced searcher, information specialist, or librarian and recommends use of the PRESS checklist. Peer review should occur before the search is executed whenever possible, because early correction prevents downstream screening and reporting problems.

A quality review should answer:

1. Does the strategy represent every important concept?
2. Are the terms appropriate for the target database?
3. Are subject headings and free-text terms combined appropriately?
4. Are limits justified and safe?
5. Does the strategy retrieve benchmark records?
6. Are the syntax and line references correct?
7. Can another researcher reproduce the search?

Footnotes

1. Cochrane Handbook, Chapter 4: Searching for and selecting studies - Recommends peer review of search strategies and use of the PRESS checklist. ↩

9. User-Centered Evaluation

Technical metrics do not fully capture search quality. A strategy should also be evaluated by its users or searchers.

Important user-centered criteria include:

• Task success: Did the search support the required decision or research task?
• Satisfaction: Did users judge the results useful and understandable?
• Effort: How much time and cognitive effort were required?
• Confidence: Do users trust that important evidence was not missed?
• Learnability: Can users understand and reuse the search process?
• Accessibility: Can users interact with the search system effectively?
• Explainability: Can users understand why records were retrieved or ranked?

For professional searching, user satisfaction should not replace recall and precision testing. Instead, it complements those measures by showing whether the retrieved evidence is usable in practice.

10. Stability, Robustness, and Updating

A robust strategy continues to perform when conditions change. Evaluate whether it remains effective across:

• Different databases and search interfaces.
• New terminology and emerging technologies.
• Changes in indexing policies.
• Newly published records.
• Different date ranges.
• Different document types.
• Minor spelling or wording variations.
• Search updates conducted by different researchers.

Robustness can be assessed by rerunning the strategy at different times, translating it across platforms, and testing it against newly identified relevant records.

An update search should be checked for:

• New subject headings.
• Newly adopted terminology.
• Changes in database coverage.
• Records added since the original search.
• Changes in eligibility criteria.
• New citation networks or influential studies.

11. A Practical Evaluation Scorecard