PharmGKB introduces scoring system for variant and clinical annotations and updated Levels of Evidence

Since their introduction in 2010, clinical annotations have become one of the most popular features of PharmGKB. Each clinical annotation summarizes a phenotypic association between a drug and genetic variant, shows relevant findings from the curated literature as variant annotations and is assigned a level of evidence to indicate the strength of support for that association in the literature.
As the number of variant and clinical annotations in PharmGKB has increased, it became a challenge to maintain consistency when assessing all the available evidence and assigning a level of evidence to clinical annotations. To address this, PharmGKB began a project at the end of 2019 to improve standardization across clinical annotations by establishing new curator tools and protocols. We are pleased to be able to release the first phase of this project to users today.Central to this work has been the development of a scoring system which assigns scores to both variant and clinical annotations as a numerical summary of the evidence underlying each annotation. Variant annotations are scored in a five-step process which assesses various attributes found in both the main annotation and in the study parameters. This scoring of variant annotations is not a judgement of study quality. It is a metric used by PharmGKB curators when comparing variant annotations against each other as part of the process of creating and updating clinical annotations.

Annotation Scoring

The scoring process can be described using the following formula, with a list of attributes scored in each step given below. A more detailed description can be found on our Variant Annotation Scoring page.

(Step 1 + Step 2 + Step 3 + Step 4) x (Step 5a x Step 5b)

Step 1 – Phenotype category (toxicity, efficacy, etc.)
Step 2 – Reported p-value
Step 3 – Cohort size
Step 4 – Effect size
Step 5a – Weighting by study type
Step 5b – Weighting by reported association and significanceWhen a variant annotation is attached to a clinical annotation, the variant annotation’s score contributes to the score for the clinical annotation. Curators can mark any variant annotation which reports an association in the opposing direction to the assertion made in the clinical annotation as a conflicting variant annotation. For example, if a clinical annotation reports that the G allele is associated with decreased response to a drug, a variant annotation which reports that the G allele is associated with increased response would be considered to be conflicting.Variant annotations which are marked as conflicting are assigned a negative score and receive a tag which can be seen on a clinical annotation’s page. It is important to realize that a variant annotation is only considered to be conflicting within the context of a specific clinical annotation and that the score of the variant annotation only changes in that clinical annotation.
Supporting CPIC and DPWG clinical guidelines or FDA-approved drug labels can also be added to a clinical annotation by curators and can contribute to that annotation’s score.Assigning a score to guidelines and labels has helped us to better define our Level 1A clinical annotations. Now, any annotation with support from a qualifying CPIC or DPWG guideline or an FDA label is assigned as 1A. More information about clinical annotation scoring, including how a clinical guideline or drug label qualify for addition to a clinical annotation can be found here.In this first release, only DPWG guidelines which mention specific alleles in the recommendation text have been used to support 1A clinical annotations. We are aware that DWPG provide additional mapping in supporting documents and plan to look at these in more detail in the near future. This will allow more DPWG guidelines to be used to support 1A clinical annotations.A clinical annotation’s score is used to assign a suitable level of evidence for the annotation. A table with scoring ranges and detailed descriptions of each level can be found on our level of evidence page. We have introduced a separate scoring range for rare variants to account for the fact that these tend to only be reported in small studies, even though there is an underlying pharmacogenomic association. Information on how PharmGKB defines a rare variant can be found here. A clinical annotation’s score is only used to determine the level of evidence and is not intended to be used to rank or compare clinical annotations within a given level of evidence.A clinical annotation’s score is now displayed on clinical annotation pages. This includes a score breakdown to indicate how different types of evidence are contributing to the annotation’s score. In rare cases, the team may feel that a clinical annotation’s score is not reflective of the underlying evidence and can, after group discussion and consensus, choose to override the scoring system. When an annotation’s level has been overridden, it is displayed on the annotation page along with a written justification for the override.This scoring system minimizes subjectivity in the assessment of clinical annotations and makes the assignment of levels of evidence more consistent, reproducible and transparent to users. To complement the scoring system, PharmGKB clinical annotations are now being written to new standards as well as displaying additional information for users.

New Clinical Annotation Features

Clinical annotations will now begin to use a new template and set of standardized sentences to highlight caveats and other considerations to users. They are also now written only on a single drug, drug combination or drug class and a single phenotype category (e.g. dosage, efficacy, etc.). Additionally, we have introduced extra checks to ensure that all level 1 and 2 clinical annotations are supported by two independent pieces of evidence.PharmGKB now offers two distinct types of clinical annotation: gene-level and variant-level. Variant-level annotations provide genotype-based summaries for a specific rsID (example), while gene-level clinical annotations display summaries for one of more star alleles of a gene (example). These formats have always been part of our clinical annotations, but have now been formalized with templates to standardize annotation writing.The score and level of evidence assigned to gene-level clinical annotations represents the strength of evidence underlying the association at the level of the gene rather than at the level of the individual variant. A ‘Limited Evidence’ tag is used to highlight alleles which are supported by substantially less evidence than the overall level indicated by the level of evidence. Where possible, we also now display allele function as assigned by CPIC.These changes mean that clinical annotations are easier to compare against each other and that the level of evidence is more representative of the evidence supporting a phenotypic association for the variant-drug pair. We acknowledge that this has reduced the number of clinical annotations at levels 1B, 2A and 2B however, these annotations are now more consistent and based on quantitative criteria.This project has entailed detailed review of over 350 clinical annotations by our curation team and, as part of this release, all level 1 and 2 clinical annotations have been rewritten to our new standards and reviewed by at least two curators. Updating clinical annotations at levels 3 and 4 to our new standards will continue as part of our regular curation activities. Users can check the history section of each annotation to see if there has been a recent update.
We are excited to bring the first phase of this project to PharmGKB users and welcome user comments or suggestions sent to feedback@pharmgkb.org.