This data book contains the Directed Design of Experiments for Validating Probability of
Detection (POD) Capability of NDE Systems (DOEPOD) analyses of the nondestructive
inspection data presented in the NTIAC, Nondestructive Evaluation (NDE) Capabilities Data
Book [1]. DOEPOD is designed as a decision support system to validate inspection system,
personnel, and protocol demonstrating 0.90 POD with 95% confidence at critical flaw sizes,
a90/95. Although 0.90 POD with 95% confidence at critical flaw sizes is often stated as an
inspection requirement in inspection documents, including NASA Standards [2], NASA critical
aerospace applications have historically only accepted 0.978 POD or better with a 95% one-sided
lower confidence bound exceeding 0.90 at critical flaw sizes, a90/95. (see Figure 11 of [3]).
The test methodology used in DOEPOD is based on the field of statistical sequential analysis
founded by Abraham Wald,
“Sequential analysis is a method of statistical inference whose characteristic feature is that
the number of observations required by the procedure is not determined in advance of the
experiment. The decision to terminate the experiment depends, at each stage, on the results of
the observations previously made. A merit of the sequential method, as applied to testing
statistical hypotheses, is that test procedures can be constructed which require, on average, a
substantially smaller number of observations than equally reliable test procedures based on a
predetermined number of observations.” A. Wald [4]
Details of the analysis methods used in DOEPOD are fully described in the DOEPOD [5]
manual, and “Directed Design of Experiments for Validating Probability of Detection Capability
of a Testing System” US Patent Serial Number: US 8,108,178. Additional details are available
on the operation [6] [7] and proof property validation [7] of DOEPOD.
The critical importance of validating methodologies used for establishing POD have been
highlighted [3] and this data book provides the DOEPOD validation of POD capabilities for
NDE systems, materials, structures, and flaw types presented in the NTIAC, Nondestructive
Evaluation (NDE) Capabilities Data Book [1].
The maximum likelihood estimation (MLE) method used in DOEPOD to estimate the probability
of detection using a two parameter logit model (MLE-Logit) are identical to that used in NTIAC
[1]. This MLE method was chosen as a verification of data integrity so that the MLE POD plots
in NTIAC [1] and this data book are identical except where this data book provides a correction
to NTIAC [1] analysis. Corrections to NTIAC [1] are indicated in the Errata listed at the end of
this document. Other MLE-Logit methods may be used, and a simple grid search for
maximizing parameters has been demonstrated [3] to be effective. The POD analysis methods of
NTIAC [1] and a military handbook [8] use a predetermined number of observations.
Detection (POD) Capability of NDE Systems (DOEPOD) analyses of the nondestructive
inspection data presented in the NTIAC, Nondestructive Evaluation (NDE) Capabilities Data
Book [1]. DOEPOD is designed as a decision support system to validate inspection system,
personnel, and protocol demonstrating 0.90 POD with 95% confidence at critical flaw sizes,
a90/95. Although 0.90 POD with 95% confidence at critical flaw sizes is often stated as an
inspection requirement in inspection documents, including NASA Standards [2], NASA critical
aerospace applications have historically only accepted 0.978 POD or better with a 95% one-sided
lower confidence bound exceeding 0.90 at critical flaw sizes, a90/95. (see Figure 11 of [3]).
The test methodology used in DOEPOD is based on the field of statistical sequential analysis
founded by Abraham Wald,
“Sequential analysis is a method of statistical inference whose characteristic feature is that
the number of observations required by the procedure is not determined in advance of the
experiment. The decision to terminate the experiment depends, at each stage, on the results of
the observations previously made. A merit of the sequential method, as applied to testing
statistical hypotheses, is that test procedures can be constructed which require, on average, a
substantially smaller number of observations than equally reliable test procedures based on a
predetermined number of observations.” A. Wald [4]
Details of the analysis methods used in DOEPOD are fully described in the DOEPOD [5]
manual, and “Directed Design of Experiments for Validating Probability of Detection Capability
of a Testing System” US Patent Serial Number: US 8,108,178. Additional details are available
on the operation [6] [7] and proof property validation [7] of DOEPOD.
The critical importance of validating methodologies used for establishing POD have been
highlighted [3] and this data book provides the DOEPOD validation of POD capabilities for
NDE systems, materials, structures, and flaw types presented in the NTIAC, Nondestructive
Evaluation (NDE) Capabilities Data Book [1].
The maximum likelihood estimation (MLE) method used in DOEPOD to estimate the probability
of detection using a two parameter logit model (MLE-Logit) are identical to that used in NTIAC
[1]. This MLE method was chosen as a verification of data integrity so that the MLE POD plots
in NTIAC [1] and this data book are identical except where this data book provides a correction
to NTIAC [1] analysis. Corrections to NTIAC [1] are indicated in the Errata listed at the end of
this document. Other MLE-Logit methods may be used, and a simple grid search for
maximizing parameters has been demonstrated [3] to be effective. The POD analysis methods of
NTIAC [1] and a military handbook [8] use a predetermined number of observations.
