Compliance

Safety and performance information

In the following, performance and safety information is provided for the entire device, including its variants P001-01and P001-02.

The confusion matrix of statistical classification is shown below with the abbreviations used in this document. 

		Comparator method		Total
		Positive	Negative
Test	Positive	TP	FP	TP+FP
	Negative	FN	TN	FN+TN
	No calls or invalid calls	E	F	E+F
	Total	TP+FN+E	FP+TN+F	N

Please note that not all entries in the confusion matrix can be determined for all genomic variant types [1]. For example, theoretically, there are an unlimited number of Indel variants, which means that TN cannot be determined for SNV/Indel discovery. The performance characteristics that were determined for this device are: 

Performance characteristic	Abbr.	Formula	Variant type	Alias
Positive Percent Agreement	PPA	TP/(TP+FN)	SNV/Indel, CNV	Sensitivity
Negative Percent Agreement	NPA	TN/(TN + FP)	CNV	Specificity
Technical Positive Predictive Value	TPPV	TP/(TP + FP)	SNV/Indel, CNV	Precision
Limit of Detection	LoD	-	SNV/Indel, somatic	-
Limit of Blank	LoB	-	SNV/Indel, somatic	-
Reproducibility and Repeatability	-	-	SNV/Indel	-

 

Detection of SNV/Indels in NGS data (germline) 

The analytical performance parameters for SNV and Indel discovery in germline samples are listed below (in %) in conjunction with the confidence intervals. 

Parameter	Variant type	WGS	WES	NGS panel
PPA	SNV	99.8	99.4	99.1
PPA 95% CI	SNV	99.8..99.8	99.4..99.4	98.9..99.3
TPPV	SNV	99.8	99.5	99.8
TPPV 95% CI	SNV	99.8..99.8	99.5..99.6	99.7..99.9
PPA	Indel	97.1	86.9	71.9
PPA 95% CI	Indel	97.1..97.2	86.7..87.1	69.3..74.4
TPPV	Indel	95.9	72.4	56.9
TPPV 95% CI	Indel	95.9..96.0	72.1..72.7	54.4..59.4
PPA	all	99.5	98.3	96.7
PPA 95% CI	all	99.4..99.5	98.3..98.3	96.4..97.0
TPPV	all	99.2	96.9	95.0
TPPV 95% CI	all	99.2..99.2	96.8..96.9	94.6..95.4

 

Confidence intervals were calculated using the Wilson score method [2]. 

The performance was achieved with data sets of the following quality: 

Analyte	Depth of coverage	% bases covered at 20x	Insert size	On-target rate
WGS	> 90	n/a	> 300	n/a
WES	> 80	> 96%	> 180	> 75%
NGS panel	> 320	> 99%	> 200	> 75%

 

Note that the analytical performance of NGS-based tests strongly depends on quality parameters like the depth of coverage and the selected target region [3]. 

It is therefore recommended to determine the performance characteristics for the particular NGS assay and laboratory process according to applicable guidelines [4]–[7]. 

Reproducibility and repeatability 

Repeatedly processing the same data set with the device yields identical results (100% repeatability). Reproducibility was assessed using replicates of the same DNA sample thereby keeping conditions like the type of sequencing instrument, the day of measurement, the operating conditions remain the same. This means that only the sequencing process of the DNA is repeated which represents a sampling of different NGS reads. 

This assessment of reproducibility therefore encompasses 

• Variation in the preparation of DNA replicates (e. g. amount of DNA), 

• Variation due to errors in the sequencing process, 

• Variation due to different local optimization of alignment in each sampling. 

Under these conditions the PPA varied by 0.2% and the TPPV varied by 0.3%. 

Limitations 

The following limitations should be noted: 

• Genome-in-a-bottle samples have been characterized using NGS technologies, i. e. that the reference data set is biased towards variants that can be easily discovered by NGS.  

• In addition, the performance evaluation is restricted to the trusted regions of the GiaB data sets. Typically, these trusted regions represent those genomic regions that can easily be assessed using NGS.  

• The analytical performance outside trusted regions should be expected to be lower than reported here [3]. 

• The performance evaluation was performed within the manufacturer’s target regions plus 50bp around these regions. This is the region that is typically evaluated in practice. The performance reported here may be lower than when reporting limited to the manufacturer’s target regions. 

Detection of SNV/Indels in NGS data (somatic) 

The performance characteristics of the somatic analysis are provided in % in the table below. 

Parameter	Variant type	WGS
Genome build		hg38
Limit of blank		10.0
Limit of detection		15.0
PPA	SNV	97.7
PPA 95% CI	SNV	97.7..97.8
TPPV	SNV	99.3
TPPV 95% CI	SNV	99.2..99.3
PPA	Indel	79.0
PPA 95% CI	Indel	78.6..79.5
TPPV	Indel	93.8
TPPV 95% CI	Indel	93.5..94.0
PPA	all	95.6
PPA 95% CI	all	95.5..95.7
TPPV	all	98.7
TPPV 95% CI	all	98.7..98.8

 

These performance characteristics include both germline and somatic variants. The WGS data set, a mixture of two GiaB samples, had an average depth of coverage of 110. 

Detection of CNV in NGS data 

General 

The analytical performance does not depend on the genome build. hg38 or GRCh37 provide equivalent performance.  

The CNV discovery of this device presumes that CNVs and pathogenic CNVs in particular are rare. CNVs that are present in 1 out of a pool of 5 samples or in 4 out of a pool of 10 can be detected with high sensitivity. This must be considered when analyzing data from families. CNVs with high prevalence or polymorphisms cannot reliably be detected with the device. 

Since the performance characteristics of a specific NGS assay depend on the configuration of the assay and the laboratory process (e. g. sequencing capacity, depth of coverage, uniformity across samples), it is recommended to perform an assay-specific validation according to applicable guidelines [4]–[7]. 

Performance evaluation using public patient data 

A performance assessment using the public data set ICR96 [8] comprising 96 patient samples is provided here. The data set is publicly available and may serve as a benchmark between different devices.  

Note that the ICR96 data set has been characterized as “less homogeneous” than other data sets by the reviewers of the publication. It should be expected that a results with a more modern laboratory set up can generate data of greater uniformity and therefore better analytical performance. The results reported here therefore represent a lower limit. 

The quality thresholds are: 

Parameter	Description	Threshold
Reference spread (RefSpread)	Quality value assigned to a single target region	< 0.2
Bivariance (bivar)	Quality value assigned to a data set as a whole	< 0.2

 

The following table summarizes the performance characteristics from the ICR96 data set: 

Name	Loss	Gain	All
PPA	98.8%	97.5%	98.4%
PPA 95% CI	98.82..98.82%	97.53..97.53%	98.40..98.40%
NPA	99.9%	100.0%	99.8%
NPA 95% CI	99.86..99.86%	99.96..99.96%	99.82..99.82%

The following table indicates how many samples and how many individual target regions were excluded from the analysis (no-calls) after application of the quality thresholds: 

Parameter	Result	Fraction
Total number of samples	96	100.0%
Samples failing thresholds	10	10.4%
Total number of targets	31203	100.0%
Targets failing thresholds	417	1.3%

 

The choice of reference genome (hg38 vs GRCh37) has no significant impact on the overall analytical performance regarding the data set. It should be noted, though, that the analysis using hg38 enhances the mapability of NGS reads and leads to a larger number of samples being acceptable for analysis. Partial deletions or mosaic variations were not assessed. 

References 

[1] P. Krusche et al., “Best practices for benchmarking germline small-variant calls in human genomes,” Nat Biotechnol, vol. 37, no. 5, Art. no. 5, May 2019, doi: 10.1038/s41587-019-0054-x. 

[2] R. G. Newcombe, “Improved confidence intervals for the difference between binomial proportions based on paired data,” Statistics in Medicine, vol. 17, no. 22, pp. 2635–2650, Nov. 1998, doi: 10.1002/(SICI)1097-0258(19981130)17:22<2635::AID-SIM954>3.0.CO;2-C. 

[3] M. H. Cleveland, J. M. Zook, M. Salit, and P. M. Vallone, “Determining Performance Metrics for Targeted Next-Generation Sequencing Panels Using Reference Materials,” The Journal of Molecular Diagnostics, vol. 20, no. 5, pp. 583–590, Sep. 2018, doi: https://doi.org/10.1016/j.jmoldx.2018.04.005. 

[4] C. Rehder et al., “Next-generation sequencing for constitutional variants in the clinical laboratory, 2021 revision: a technical standard of the American College of Medical Genetics and Genomics (ACMG),” Genetics in Medicine, pp. 1–17, Apr. 2021, doi: 10.1038/s41436-021-01139-4. 

[5] E. Souche et al., “Recommendations for whole genome sequencing in diagnostics for rare diseases,” Eur J Hum Genet, pp. 1–5, May 2022, doi: 10.1038/s41431-022-01113-x. 

[6] G. Matthijs et al., “Guidelines for diagnostic next-generation sequencing,” Eur J Hum Genet, Oktober 2015, doi: 10.1038/ejhg.2015.226. 

[7] P. Bauer, “S1 Leitlinie: Molekulargenetische Diagnostik mit Hochdurchsatz-Verfahren der Keimbahn, beispielsweise mit Next-Generation Sequencing,” medgen, vol. 30, no. 2, pp. 278–292, Jun. 2018, doi: 10.1007/s11825-018-0189-z. 

[8] S. Mahamdallie et al., “The ICR96 exon CNV validation series: a resource for orthogonal assessment of exon CNV calling in NGS data,” Wellcome Open Res, vol. 2, p. 35, May 2017, doi: 10.12688/wellcomeopenres.11689.1.