Q: Once upon a time, I added up some results from you from ClinVar how many variants with no known pathogenicity can we update this query?

The ClinVar dataset comes as gzipped xml from:
ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/ClinVarFullRelease_00-latest.xml.gz
and we happen to have a fresh copy in dipper/raw/clinvarxml_alpha/
If we needed freah copy we would issue



In [1]:

    
cd ../../raw/clinvarxml_alpha
# wget --timestamping ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/xml/ClinVarFullRelease_00-latest.xml.gz

There are of course a number of ways to approach the question,

First what counts as variant?

a submitters assertion (SCV)?
ClinVars interpertation of a bundle of SCVs (RCV)?
a "variant" ID which may ocuur in multiple RCVs?

"no known pathogenicity" is conceptually simple but the answer is supplied as freetext

First isolate the bits containing the answer.

pathogenic call associated with a SCV
RCV assocaiated with SCV
variant ID associated with an RCV



In [7]:

    
# path shorteners 
CVSET=/ReleaseSet/ClinVarSet
RCV=${CVSET}/ReferenceClinVarAssertion
SCV=${CVSET}/ClinVarAssertion



In [8]:

    
# do not try this at home

#xmlstarlet sel -t  \
# -m ${SCV} -v ClinVarAccession/@Acc -o '|' -v ClinicalSignificance/Description -o '|' -n \
#ClinVarFullRelease_00-latest.xml.gz   | sort > FULL_SCV_ClinicalSignificance.unl

#xmlstarlet sel -t \
#    -m ${SCV} -v ClinVarAccession/@Acc -o "|" \
#    -v ancestor::ClinVarSet/ReferenceClinVarAssertion/ClinVarAccession/@Acc -o "|" -n  \
#ClinVarFullRelease_00-latest.xml.gz | sort > FULL_SCV_RCV.unl

#xmlstarlet sel -t 
#    -m ${RCV} -v ClinVarAccession/@Acc -o "|" 
#    -v 'MeasureSet/@ID' -o "|" -n  \
#ClinVarFullRelease_00-latest.xml.gz | sort > FULL_RCV_VARID.unl



In [16]:

    
ls -l FULL_[RS]CV_*









    



-rw-r--r-- 1 tomc tomc  7589528 Mar 24 16:13 FULL_RCV_VARID.unl
-rw-r--r-- 1 tomc tomc 12160305 Mar 26 16:33 FULL_SCV_ClinicalSignificance.unl
-rw-r--r-- 1 tomc tomc 10967778 Mar 24 16:13 FULL_SCV_RCV.unl

see what is in the supplied freetext



In [18]:

    
cut -f2 -d\| FULL_SCV_ClinicalSignificance.unl | sort | uniq -c | sort -nr









    



 154368 Uncertain significance
  73853 Likely benign
  72911 Pathogenic
  56946 Benign
  19980 not provided
  14175 Likely pathogenic
   2867 pathologic
   1191 other
   1161 probable-pathogenic
   1024 cancer
   1005 risk factor
    921 pathogenic
    747 unknown
    566 drug response
    544 untested
    481 likely pathogenic
    407 Unknown
    384 benign
    351 Likely Benign
    293 variant of unknown significance
    263 no known pathogenicity
    239 Uncertain Significance
    207 uncertain
    190 association
    156 likely benign
    151 Affects
    117 Benign/Likely benign
     98 conflicting data from submitters
     96 protective
     90 Moderate
     85 probable-non-pathogenic
     84 non-pathogenic
     73 Uncertain
     28 probably not pathogenic
     28 Associated with leiomyomas
     25 MUT
     20 POLY
     17 Pathologic
     14 likely pathogenic - adrenal pheochromocytoma
     12 drug-response
      9 VOUS
      6 Pathogenic/Likely pathogenic
      5 Unknown significance
      5 likely pathogenic - adrenal bilateral pheochromocy
      3 VLM
      3 unknown significance
      3 pathogenic variant for Bardet-Biedl Syndrome
      3 
      2 VSB
      2 Suspected Pathogenic
      2 probably pathogenic
      1 Suspected Benign
      1 HSCR variant
      1 confers sensitivity

We have a mapping of most of these in Clinvar translation table and should add the new ones
I am isolating them for this query.



In [24]:

    
cat patho_label.map









    



Benign								benign
conflicting data from submitters	uncertain significance
Likely benign						likely benign
Likely Benign						likely benign
likely pathogenic					likely pathogenic
Likely pathogenic					likely pathogenic
no known pathogenicity				uncertain significance
non-pathogenic						uncertain significance
pathogenic							pathogenic
Pathogenic							pathogenic
Pathogenic/Likely pathogenic		pathogenic
pathologic							pathogenic
probable-non-pathogenic				likely benign
probable-pathogenic					likely pathogenic
probably pathogenic					likely pathogenic
probably not pathogenic				likely benign
Suspected Benign					likely benign
Suspected Pathogenic				likely pathogenic
uncertain							uncertain significance
Uncertain							uncertain significance
Uncertain significance				uncertain significance
Uncertain Significance				uncertain significance
unknown								uncertain significance
Unknown								uncertain significance
variant of unknown significance		uncertain significance
MUT									uncertain significance
Unknown significance				uncertain significance
likely pathogenic - adrenal pheochromocytoma	likely pathogenic
pathogenic variant for Bardet-Biedl Syndrome	pathogenic
untested							uncertain significance
other								uncertain significance
not provided						uncertain significance
cancer								pathogenic
risk factor							likely pathogenic
drug response						uncertain significance
drug-response						uncertain significance
association							uncertain significance
Affects								uncertain significance
Moderate							uncertain significance
protective							likely benign
unknown significance				uncertain significance
likely pathogenic - adrenal bilateral pheochromocy	likely pathogenic
Associated with leiomyomas			likely benign
benign								benign
Benign/Likely benign				benign
likely benign						likely benign
##################					##########################
confers sensitivity					uncertain significance
Pathologic							uncertain significance
HSCR variant						uncertain significance
POLY								uncertain significance
VLM									uncertain significance
VOUS								uncertain significance
VSB									uncertain significance

The last half dozen or so are SWAG,
maybe someone has better info on there classification.

Convert the freetext pathogenic call to its classification:



In [2]:

    
# head patho_label.map FULL_SCV_ClinicalSignificance.unl

tr -s '\t' < patho_label.map > patho_label.txt
tr '|' '\t' < FULL_SCV_ClinicalSignificance.unl > SCV_ClinicalSignificance.txt

head patho_label.txt  SCV_ClinicalSignificance.txt









    



==> patho_label.txt <==
Benign	benign
conflicting data from submitters	uncertain significance
Likely benign	likely benign
Likely Benign	likely benign
likely pathogenic	likely pathogenic
Likely pathogenic	likely pathogenic
no known pathogenicity	uncertain significance
non-pathogenic	uncertain significance
pathogenic	pathogenic
Pathogenic	pathogenic

==> SCV_ClinicalSignificance.txt <==
SCV000020145	Pathogenic	
SCV000020146	Pathogenic	
SCV000020147	Pathogenic	
SCV000020148	Benign	
SCV000020149	Pathogenic	
SCV000020150	protective	
SCV000020151	association	
SCV000020152	Pathogenic	
SCV000020153	Pathogenic	
SCV000020154	risk factor



In [40]:

    
awk -F'\t' 'NR==FNR{p[$1]=$2}NR!=FNR{print $1,p[$2]}' \
 patho_label.txt SCV_ClinicalSignificance.txt > SCV_class.txt



In [42]:

    
head SCV_class.txt









    



SCV000020145 pathogenic
SCV000020146 pathogenic
SCV000020147 pathogenic
SCV000020148 benign
SCV000020149 pathogenic
SCV000020150 likely benign
SCV000020151 uncertain significance
SCV000020152 pathogenic
SCV000020153 pathogenic
SCV000020154 likely pathogenic



In [44]:

    
wc -l SCV_class.txt
grep -cv  pathogenic SCV_class.txt









    



406214 SCV_class.txt
311637

~75% of the records submitted to Clinvar (SCV) we would not classify as pathogenic.



In [47]:

    
tr '|' '\t' < FULL_SCV_RCV.unl > SCV_RCV.txt



In [26]:

    
join -j1 -o 2.2,1.2 1.3 SCV_class.txt SCV_RCV.txt | sort> RCV_class.txt



In [27]:

    
head RCV_class.txt









    



RCV000000002 pathogenic 
RCV000000003 pathogenic 
RCV000000004 pathogenic 
RCV000000005 benign 
RCV000000006 pathogenic 
RCV000000007 likely benign
RCV000000008 uncertain significance
RCV000000009 pathogenic 
RCV000000010 pathogenic 
RCV000000011 likely pathogenic



In [28]:

    
#  How many RCV-call combinations
wc -l < RCV_class.txt



In [29]:

    
#  How many distinct RCV-call combinations
sort -u RCV_class.txt | wc -l



In [46]:

    
echo "scale=3;(377430 / 406214* 100)" | bc



In [ ]:

    
About 7% of the calls Clinvar bundles together are perfectly redundant, or faithful reproductions.



In [2]:

    
cut -f1 -d ' ' RCV_class.txt | sort -u | wc -l



In [2]:

    
echo "scale=3;(377430 - 366567) | bc



In [3]:

    
echo "scale=3;(10863/377430 * 100)" | bc

This is saying that 11k or 2.8% of RCV have some variation in the pathogenic calls.
Note well this is any variation in our classification and not just signigificant disagreement.



In [49]:

    
grep -c pathogenic RCV_class.txt



In [81]:

    
grep pathogenic RCV_class.txt |cut -f1 | sort -u | wc -l



In [78]:

    
echo "scale=3;(94577 / 406214 * 100)" | bc 
echo "scale=3;(82798 / 377430 * 100)" | bc 
echo "scale=3;(82798 / 94577 * 100)" | bc

86.8% of all RCV have no pathogenic classification 88.1% of the unique RCV have no pathogenic classification

About 12.5% of the pathogenic call have some replication.



In [64]:

    
awk  '$NF!="pathogenic"{p++}{c++}END{print p, c, p/c*100}' RCV_class.txt









    



311637 406214 76.7174

(Without collapsing RCV this just recapiulates the SCV counts)



In [12]:

    
tr '|' '\t' <  FULL_RCV_VARID.unl | sort > RCV_VARID.txt



In [69]:

    
join -j1 -o 1.2,2.2,2.3 RCV_VARID.txt RCV_class.txt |sort -n > VID_class.txt



In [71]:

    
head VID_class.txt









    



2 pathogenic 
3 pathogenic 
4 uncertain significance
5 pathogenic 
6 pathogenic 
7 pathogenic 
7 pathogenic 
8 pathogenic 
9 benign 
9 likely pathogenic



In [73]:

    
wc -l < VID_class.txt



In [77]:

    
cut -f1 -d ' ' VID_class.txt | sort -u | wc -l



In [79]:

    
grep -v pathogenic VID_class.txt | cut -f1 -d ' ' | sort -u | wc -l



In [80]:

    
echo "scale=3;(206300 / 268916 * 100)" | bc



In [82]:

    
sort -u VID_class.txt | awk '$NF=="pathogenic"{a[$1]=or(1,a[$1])}$NF=="benign"{a[$1]=or(2,a[$1])}END{for(v in a){if(a[v]>2)print v}}' | wc -l

Note these conflicts do not involve calls of uncertain significance

So far we have

406,214 unique SCV no conflicting calls 76.7% non pathogenic
377,430 unique RCV no conflicting calls 88.1% non pathogenic
268,916 unique VID 653 conflicting calls 76.7% non pathogenic

( note the two 76.7% are via completly different pairs numbers more than 100k apart )

And here is where we are glad we did it this way.

I need to be counting "Variant of Uncertain Significance" (VUS) not the known pathogenic.



In [87]:

    
grep -vc significance SCV_class.txt
grep -v significance SCV_class.txt | cut -f1 |sort -u | wc -l
grep -v significance RCV_class.txt | cut -f1 |sort -u | wc -l
grep -v significance VID_class.txt | cut -f1 |sort -u | wc -l

406,214 unique SCV no conflicting calls 226,625 or 55.8% VUS
377,430 unique RCV no conflicting calls 202,837 or 53.7% VUS
268,916 unique VID 653 conflicting calls 159.850 or 59.4% VUS