Estimating the agreement and diagnostic accuracy of two diagnostic tests when one test is conducted on only a subsample of specimens

Hormuzd A. Katki, Yan Li, David W. Edelstein, Philip E. Castle

Research output: Contribution to journalArticlepeer-review

19 Scopus citations


We focus on the efficient usage of specimen repositories for the evaluation of new diagnostic tests and for comparing new tests with existing tests. Typically, all pre-existing diagnostic tests will already have been conducted on all specimens. However, we propose retesting only a judicious subsample of the specimens by the new diagnostic test. Subsampling minimizes study costs and specimen consumption, yet estimates of agreement or diagnostic accuracy potentially retain adequate statistical efficiency. We introduce methods to estimate agreement statistics and conduct symmetry tests when the second test is conducted on only a subsample and no gold standard exists. The methods treat the subsample as a stratified two-phase sample and use inverse-probability weighting. Strata can be any information available on all specimens and can be used to oversample the most informative specimens. The verification bias framework applies if the test conducted on only the subsample is a gold standard. We also present inverse-probability-weighting-based estimators of diagnostic accuracy that take advantage of stratification. We present three examples demonstrating that adequate statistical efficiency can be achieved under subsampling while greatly reducing the number of specimens requiring retesting. Naively using standard estimators that ignore subsampling can lead to drastically misleading estimates. Through simulation, we assess the finite-sample properties of our estimators and consider other possible sampling designs for our examples that could have further improved statistical efficiency. To help promote subsampling designs, our R package CompareTests computes all of our agreement and diagnostic accuracy statistics.

Original languageEnglish (US)
Pages (from-to)436-448
Number of pages13
JournalStatistics in Medicine
Issue number5
StatePublished - Feb 28 2012
Externally publishedYes


  • Gold standard
  • HPV
  • Kappa
  • Sensitivity
  • Specificity
  • Symmetry test
  • Two-phase design
  • Verification bias

ASJC Scopus subject areas

  • Epidemiology
  • Statistics and Probability


Dive into the research topics of 'Estimating the agreement and diagnostic accuracy of two diagnostic tests when one test is conducted on only a subsample of specimens'. Together they form a unique fingerprint.

Cite this