The below text is a transcript from the webinar. Because it is a transcript, there may be oddities that arise from the process of translating speech into text. We recommend accessing the recording, above, to gain full context.
Download the slides used in the webinar here
Overview
Optimism for next generation sequencing (NGS) technologies to be used in a clinical setting quickly turns to numerous conversations about the challenges involved. These challenges range from implementing and validating tests, finding people with expertise, and efficiently and accurately interpreting data from the tests. In this webinar, we discuss the CAP (College of American Pathologists) Distributive Model and how your organization can leverage it to rapidly and economically conduct or scale clinical NGS testing.
So why the distributive model? Deploying an NGS test in the LDT environment can take anywhere from 18 to 36 months. Where we go from initial planing to establish a laboratory, select an assay get institutional approval procure the equipment set up the actual laboratory. Optimize and train based on the assay or assays that you're planning to deploy. Validate that test in an LDT environment. Prepare to go live engage with your customers to order assays. Ensure that your IT systems are set up to run. Report out on the test and push that end in the medical record. And then go live.
Barriers to Building an NGS Test Menu
Barriers are that there's a scarcity of informatics expertise. Obviously in the LDT environment and in the environment there's a validation of clinical testing protocols. There's a large amount of data to curate in order to be able to report out, on NGS assays efficiently and effectively. The technology's changed very rapidly. So the moment you have validated an assay and deployed it a better assay comes along. The expense of implementation is not to be underestimated. Each essay validation can cost tens of thousands of potentially hundreds of thousands of dollars of personnel and reagent costs. And if this is your very first NGS assay the difficulty of that first application deployed shouldn't be underestimated. A more rapid and economical approach wherein the distributive model may be leveraged in order to deploy an assay within your own environment practice precision medicine report out on NGS assays is possible in the distributive model. And one that both the CAP and PierianDx have embraced especially in a transitional model as laboratories learn how to actually in source an assay. And with that I'd like to turn it over to Karl who will again describe the distributive model itself and the CAP requirements. Karl?
Elements of a Next Generation Sequencing Test
The CAP accreditation requirements are structured to accommodate the distributive testing model. And to begin, I'd like to just review the elements of a next generation sequencing test as viewed by the CAP. So essentially we have a total analytical laboratory process comprised of what we refer to as a wet lab and dry lab activities, followed by a professional activity. And within the wet lab, activity is essentially the sample handling through library preparation and sequence handling with generation of typically a fastQ file followed by alignment and variant calling. And various aspects of annotation and prioritization as performed by algorithms and software. This data has been utilized for variant evaluation and assertion and report generation. And that portion of it we refer to as medical practice.
One of the aspects to be aware of in terms of a distributive model is that the prioritization of variants and their evaluation is and can be at times an overlap zone into otherwise referred to as medical practice. Next slide. In sort of a parallel fashion of how we defined the elements of next generation sequence of test I thought it would be of interest to the audience to also review the recent elements of an NGS test as articulated by the FDA in its final guidance in April of this year on essentially analytical validation of NGS based test. And you can see the following steps that they illustrate here from A to I. And it's essentially quite parallel to the elements of the test that we talked about in terms of CAP, viewpoint. You ca see that the FDA recognizes that there is an overlap zone essentially with variant evaluation and assertion. And you can see here that in the quotes it say says "Interpretation of the clinical significance of a variant, performed by health care providers is not considered part of the test." Essentially they do think of that as medical practice.But interestingly they say that "Certain standard operating procedures, including but not limited to" and then in blue "Protocols for variant evaluation and some software products may be considered test elements."
How Many Labs are Performing Molecular Pathology Testing?
So I also want to point out to the audience if we look from a CAP accreditation laboratory statistics as of 2017, we have over 800 laboratories worldwide performing molecular pathology testing in general, that are CAP accredited. And then specifically we have about 280 laboratories that are accredited performing NGS based molecular pathology testing. So the question becomes how are these laboratories actually performing their testing in terms of, operations.
What we know so far is that the majority of laboratory performing NGS testing are performing it essentially in one primary physical laboratory location, that would have a single overarching CLIA license. So the wet bench dry bench and interpretation of reporting are under one roof. But we have also observed, if we go to the next slide, that interesting new developments are an increasing movement of laboratories to essentially upload data in to several different cloud based services where they're either utilizing those services for data storage, perhaps exclusively for data storage. Or they're also essentially running algorithms on cloud based servers. And this brings of course into the picture the importance the transfer of data the integrity its security and privacy. In this model it's still considered under one essential primary physical laboratory.
CAP Distributive Model for NGS Testing
However we are seeing some laboratories that are outsourcing process steps and this really relates back to the comments by, Rakesh regarding how laboratories look at their own operational set up and what is more efficient for them to do. So one model is shown here where the primary lab receives an actual patient sample, they do the sample handling and sequencing, and then they outsource. So essentially typically what they're doing is they're uploading their fastQ files in to some type of computational server system that is developed and maintained by what we refer to under CAP as a reference laboratory. And therein the bioinformatics processes are executed and then the data is sent back to primary laboratory. So typically the process of bioinformatics would send back to the laboratory annotated variants. And they may be in some degree filtered in or prioritized. This is what we refer to as a distributed testing model and the key aspect of this particular model is what is being transferred between the primary and the reference laboratory are patient data files at different levels of evolution and maturity through the total analytical process.
Another model of a distribution is that the primary lab receives the patient sample and then accessions it catalogs it and then sends it, a physical sample. Or perhaps they've taken a physical sample and extracted DNA and they're sending that to a reference laboratory that performs the sequencing and then sends the sequencing data back to the primary laboratory for bioinformatics, interpretation and reporting. So in this case what's important to consider is not only is there transfer of patient physical sample but then also what's being returned back this actual typically a fastQ file. If we go to the next slide ... So how do we accommodate this diversity of distributive testing in our current CAP requirements?
CAP Accreditation Requirements for Test Component Distribution
The first theory I'd like to highlight is accreditation requirements that pertain to the issue of a primary and referring laboratory and the policy that a laboratory needs to have in effect to meet CAP accreditation. Next slide. I will walk through several of these, actual accreditation requirements to give you the highlights of this. This particular requirement really focuses on referral laboratory selection. And as you can see in the red dashed box, in the text of this requirement "The referral may include the total analytical process or portions-"for example as we spoke about the wet bench or for bioinformatics. And importantly from the CAP perspective if it is a laboratory based in the US or the US provinces in terms of US regulations being pertinent, any referral of the process or portions of the process must be made to a CLIA certified laboratory or a laboratory equivalent or more stringent requirements, as determined by CAP or CMS. And this requirement also assigns responsibility to the primary lab director for selection and evaluation of referral laboratories.
There's two important requirements which have to do with the provenance and tracking with samples. So essentially this is an aspect where laboratories that are referring samples to reference laboratory and conversely when reference laboratory sends data back to a primary laboratory for example that both parties are responsible for tracking specimens between the respective laboratories. And also the transfer of such data needs to be deployed in a manner that the transfer of the data is, intact, that there's confidentiality and security associated with it. And of course each of these will be-one we'll try to address this but it's also very important that in addressing this that you meet any regulatory requirements that may be local state and or other aspects of federal requirements.
NGS Test Validation in the CAP Distributive Model
Let's highlight and talk about the validation process in a distributive model. And if we go to the next slide, back to our total analytical process and coupled with medical practice. What's important here is that it is recognized that the validation the overall validation of a next generation sequencing test is a highly interdependent process between the generation of the sequence data and then the subsequent bioinformatics. So during the optimization and development of a next generation sequencing test laboratories typically undergo an iterative process that results in a refinement of the test leading to an optimized performance based on the characteristics that one desires in terms of sensitivity specificity. As you can imagine this is a process that operationally become more straightforward if everyone is done under one roof in terms of having all of the appropriate personnel with the appropriate expertise. But logistically for laboratories that need to explore and or deploy the distributive testing model, it's very very critical that that interdependency be maintained and that can require numerous conference calls site visits between the two respective parties the primary lab and the referral laboratory. So critical to this is to select a referral laboratory partner with whom one has a close working relationship and is open to working through that type of iterative process.
Therefore in our checklist requirements we require that laboratories have an integrated validation, between the wet bench process and the bioinformatics. And as you can see here in our dashed red box we highlight the fact that the validation must be acceptable as a beginning to end test performance activity and must be integrated, with the bioinformatics. And then conversely on the next slide ... To compliment this we also have a requirement that during the NGS analytical bioinformatics process that laboratories must integrate this with the wet bench process validation. And these requirements pertain to either a model in which all activities are occurring under the CLIA license or are occurring in a distributive testing model.
In terms of choice of specimens the requirements of CAP have evolved over the last couple of years. The core principle that we go buy is a methods based validation where the sample contain a representative spectrum, of variant types that the test is designed to detect. And also essentially a hybrid approach where if you have specific kinds of testing for gene panels for example, that you have specific common mutations for example recurrent hot spots in a malignancy. Or specific variants in a cardiomyopathy panel. That it's very critical during the validation to supplement one's methods based validation with examples of such specimens for common mutations in an [inaudible 00:20:46] based validation.
Role of the Primary Lab Director in the CAP Distributive Model
Essentially one of the key things in the distributive test model is the role of the primary laboratory director. So their role is really three fold. One is the initial selection of the referral laboratory, in which they must-in our model under CAP accreditation the referral laboratory must have a certification status either CLIA CAP or an equivalent that's recognized by CMS. The referral laboratory needs to be able to perform the required testing elements that the primary laboratory needs. And the referral laboratory must be able to meet the turnaround times. And then the primary laboratory's director's ultimately needs to ensure an integrated validation as we spoke about before. Requiring typically a very close working relationship with referral laboratory. Review of all validation documents between their laboratory and the referral laboratory. And to ensure that performance metrics are achieved. And then once the test goes live critically the primary laboratory director needs to make certain that there's a continuous quality monitoring of this complex distributive process. Including how to assess and address upgrades to the various processes. And also to be able to say "We've made a change in the overall process that requires a revalidation." And then one circles back to the iterative process of what degree a validation would need to be done, to assess and determine performance metrics.
Case Study: How PierianDx Implemented the CAP Distributive Model
So the very first step that we took was we defined our test features. So with respect to the PierianDx test we selected 179 gene test. We defined the genes that would be a part of this test and the sub tumors from solid tumors to metabolic disorders breast tumors CNS tumors, genital urinary tumors gynocologenic tumors head and neck tumors melanoma, and thoracic tumors for this test. The chemistry that we chose for the test was hybridization captured targeted chemistry.
And we also defined the variant types that we would be reporting via this test. So we chose single nucleotides variants insertions and deletions up to 21 base pair, three IDDS and structure variants. With respect to structure variants we defined that we were reporting fusions with respect to seven specific genes that include [ALC, MLL, ROS-1, RED and TRKFGFR and FGR3 00:24:33] since these were clinically relevant for many of the tumors that I just mentioned about in that sub panel section. With respect to the sample type we decided that we would be performing this testing on hematological specimens that is bone marrow blood and bone marrow aspirates for blood, and on the solid tumor tissue which is embedded in tissue. We also defined the intended use of this test where the molecular pathology results would aid and diagnose as prognosis 10 therapy selection of hematological and solid tumors. And the estimated turn around time for this large panel we accept would be 21 days.
Defining Test Components
So in the step two what we did was we identified the beginning to end components of our NGS test. And defined the sites for execution that adverse or primary or the referring laboratory referral laboratory activities. So I'll take a moment to orient you to this work flow and then kind go where the portions that we outsource versus decided to retain in house. So in this NGS work flow the first section of the test is where the sample in accession. Then it goes through what we call the wet lab [inaudible 00:26:13] tumor nuclei evaluation happens. Nucleic acid extraction and library preparation happens. And then sequencing happens. And this is essentially what we call as wet lab. Then the dry lab or the bioinformatics compliment including base calling sequence alignment or mapping data visualization and the sequencing review, variant annotation variant prioritization. And then the last component of this NGS workflow is the professional component which includes the variant evaluation and assertion and generation of the test report and sign off. And in this workflow for our PierianDx purposes we outsource the sample receipt and the wet lab components to our referral laboratory. And we decided to keep or retain in house the dry lab and the professional component.
CAP Checklist Items
So then what we did is we looked up and defined the CAP checklist items that we believe we applicable to ensure a beginning to end test performance So between these four checklists approximately we found 250 checklist items that applied to us to ensure a beginning to end checklist performance. And to kind of navigate through these and help us develop the procedures and policies we consolidated them under the following big categories I would say. First being assay validation. Second procedure manuals for various components of the test that is the wet bench dry bench and reporting including requisitions and the results reporting. Next section was the specimen which included specimen collection requisition handling tracking et cetera. Then proficiency testing or alternate of assessment as applicable, personnel laboratory computer services. Physical facilities and safety. And quality management program.
Evaluating the Referral Laboratory
Then the next step that we performed was we evaluated the referral laboratory for NGS referral testing. What we did for our test was we interacted with the referral laboratory director and of course we chose a referral laboratory where we felt that we could have a very close working relationship. Also another point to consider this referral laboratory was to consider was the test that we wanted to bring up was already validated in operation in this referral laboratory. So with that in the background we interacted with our referral laboratory director. We reviewed our the CLIA and CAP certificates assay validation for the test that we were interested in and the laboratory policies and procedures the were relevant for this test. We also assessed the overall quality of the performance for the referral laboratory in accordance with the CAP checklist requirements. And in this interaction with our referral laboratory director, we agreed upon the assay validation and the procedures.
Laboratory Personnel in CAP Distributive Model
Once our referral laboratory was decided what we did was we identified the personnel in our primary referring laboratory and including their role and how they will contribute to the development of the policies and procedures to the PierianDx laboratory. So again by reminding you the workflow that I just went to work from sample to accession and wet bench dry bench and the professional sign off. The components for which we had to development procedures and policies for our laboratory included sampling receiving and accessioning dry bench and the professional compliment. So for this we identified the different roles of the personnel in our laboratory. So first was the laboratory director who would provide complete oversight and review of all documents and procedures prior to finalization in accordance with the CAP and the CLIA requirement. The second role and the personnel that we identified was a laboratory supervisor who of course would satisfy the CAP requirements for the qualifications and experience and training. And they played the role in developing the physical facilities and safeties policies and procedures. Personnel policies and procedures. Proficiency testing or alternative assessment as applicable to our laboratory and our quality management program.
The third personnel role that we identified and Karl had also mentioned in his section was the personnel who would do the sample tracking accessioning and show that requisitions report delivery over all appropriate for the test. And also provide help test support to work lines. So this person helped us in developing the specimen collection requisition handling and tracking. And as a procedure manual procures. The other roles that we identified for our laboratories was our bioinfomaticians the clinical bioinfomaticians. And these play a big role with the laboratory director in execution with the dry and the professional component of the test. This person also helped us in development with the assay validation procedures procedure manuals for various components of the test of the applicable including the wet bench dry bench reporting. And including requisitions and the results reporting. And laboratory computer services.
So in the step five once we had identified our personnel we had identified our roles how we would develop generate the policies and procedures we validated our test. And ensuring keeping in mind the CAP requirement of [inaudible 00:33:24] integrated validation and ensuring a beginning to end to performance. And I just highlight these checklist items and Karl also had gone over these in his section. Simultaneously when we were validating our test we also applied further CLIA certifications and CAP accreditation. And both of these aspects require separate applications that need to be filled. So CLIA certification requires first we have to apply to the State Health Department for CLIA certification or actually just call certificate of registration. And once you receive a certificate of registration one gets 11 months to get CAP accredited and then receive a certificate of accreditation. So after we applied the CLIA certificate of registration we also completed our application for CAP inspection and CAP accreditation. So PierianDx integrated the both wet bench and the dry bench portions of the test and showed an acceptable beginning to end test performance. And satisfied all the CAP and CLIA requirements for a laboratory test validation.
Gaining CAP Accreditation
And once one receives a full CAP accreditation they get a full CLIA certification for two years. That is again they're renewable at the end of two year cycle with the successful CAP inspection. Next slide. So in the last step once we had validated our test, then we developed all of our procedures established a quality management program. Launched the test. Performed a mock inspection before going for an actual CAP inspection. And then full through our full CAP inspection according to our applications and the timelines. One recommendation that I would like to share with everybody here is that during this process if we had any question about sort of CAP checklist requirement or associated evidence of compliance we actively reached out to CAP for guidance. Because we did not want to imagine or speculate things. And that was very beneficial. CAP really guided us and helped us a lot and we immensely benefited from this approach. So I urge everybody who is going through a CAP accreditation that they should take advantage of CAP and reach out to them. The personnel are very helpful and they guide you through the process and help you with any aspect of your application and your accreditation requirement.
Distributive Models in Place at PierianDx
So now I would like to share with you the two distributive models that are currently operational at the PierianDx laboratory. So again looking at that workflow where we have a component of the sample receiving in accessioning, wet bench dry bench and professional sign out. So in the first distributive model that is our operation for our NGS test is that the sample receiving and accessing tumor nuclei evaluation nucleic acid extraction. Library preparation and sequencing are happening [inaudible 00:36:59] numbers in the same institution. And these are all called or actually our referral laboratories. And then the dry bench component even here the base calling the sequence alignment is mapping is happening at the referral laboratory and I'm identifying it as CLIA lab one because of their CLIA license. The data visualization sequencing [QC metric 00:37:26] review which is basically happens once the fastQ files are generated also happens in our referral laboratory or the CLIA lab number one with that particular CLIA license. And then at that point once the medical director or the laboratory director over there is satisfied with the sample evaluation.
And then the level one QC metricsthey push the fast Q files to our bioinformatics pipeline that sit under our CLIA license which I'm identifying as CLIA lab number four. And at that point we do it in calling in notation and calling and prioritization in the primary laboratory or the PeridianDx clinical laboratory. Again the variant evaluation and assertion and the professional sign off on the report also happens in the primary laboratory or the PierianDx laboratory that I've identified as CLIA lab number four with the respect to the CLIA license.
The next model that is operational in our laboratory is where five labs are actually participating. So again similar to the first model the sample receiving accessioning and the different components of the wet bench are happening under three different CLIA licenses of our laboratory which happens to sit under one single academic institution. Again some components of the dry bench which include the base calling sequence alignment and mapping on data visualization and sequencing QC metric for the fast Q files also happens in the referral laboratory. Once all of that is satisfactory the fast Q files get pushed to our CLIA instance that I have identified as CLIA number four here or the PierianDx laboratory. And once we have performed the variant calling annotation and prioritization we push the VCFs to our customer CLIA laboratory that I have identified here as CLIA laboratory number five as a separate CLIA license. And they and this model serve as the primary laboratory and they are performing the variant evaluation and assertion and generation of the test report and sign off.
And in both of these models the beginning to end test performances and showed and the three laboratory directors between the participating laboratories work together and hand off from one step to another to ensure a beginning to end test performance. And the validation also was conducted in an integrated manner to keep all of these components of the test integrated.
Quality Management in the CAP Distributive Model
I wanted to give you a briefly an idea of our quality management program because this is such an important component for CAP accreditation and maintaining CAP CLIA accreditation and also ensuring patient safety in your laboratory. So what we did was to develop our quality management program we kept a couple of things in mind. First in guidance as CAP guides it and CLIA requires it we ensured that it covers all the sections of four laboratory. To develop our [QN 00:40:55] program we utilized CAP guidelines and CLSI guidelines to develop the program. And we developed two things. One is our quality management manual and second is the quality management report template. We ensured when developing this manual and report template that all applicable CAP checklist items for the QN program are covered and addressed properly. We also documented all the evidence of-we also ensured that this [skilled 00:41:24] report template that we had developed our quality controlled report would be utilized to document all the evidence of compliance during our monthly quality meetings.
So just to give you an overview of what our quality manual looks like I thought it would be useful to share my index. So my quality manual for my laboratory is divided into 13 sections in accordance with the guidance from CLSI and CAP. The first section is organization. The second section is customer focus. Third section is facilities and safety. Fourth section is personnel. Fifth is purchasing and inventory. Sixth is equipment. Seventh is process management. Eight is document and records. Ninth is information management. Tenth nonconforming event management. Eleventh is assessments. Twelfth is continual importune. And thirteenth is we also listed our all the CAP checklist items that are not applicable to PierianDx clinical laboratory. And we thought it would be useful to list this out because during the CAP inspections when we are asked "Why certain CAP checklist items applicable or not applicable?" It doesn't look like we forgot about it so we listed it out so that we can actually show that we thought about it and for X Y Z reason it's not applicable to our testing model or to the test that we are operating.
Under these different sections of our quality manual briefly the CAP checklist items are basically the written quality management program and its implementation its extent of coverage and its effectiveness. CLIA significant type federal state regulations organizational chart. Employee and patient quality communication. Terms of accreditation. Customer sanctification. CAP sign. Notification from vendors. Quality management indicators of quality. Monitoring analytical performance. Monthly QC review. Test results statistics. Correction of laboratory records. Error detection and correction. Quality management patient care services. Turn around time. Document control. Record specimen retention. Record retention. Interim self inspection. And laboratory director responsibility for interim self inspection.
Quality Control Reporting
I though it would be nice to also share with you what our quality control report looks like this. So this is basically the template we developed based on our quality manual. And this quality control report is designed to cover again all sections of the laboratory and all the checklist items that our laboratory needs to track, and maintain evidence of compliance. We also kind of designed it in such a way that we don't have to keep going back to CAP checklist to remember what all we need to cover and ensure to have an effective QM program. So what we did we divided this in to four sections. Things that we have to cover monthly we call that section monthly update. Items that we have to track quarterly we call them quarterly update. Items that we have to cover semi-annually we call them semi-annual updates. And the items that we have to cover annually per CAP and CLIA guidance we call them annual updates.
So we review all of these sections so that we don't forget and whichever section is applicable on our monthly quality meetings. We complete them and we add the evidence of compliance for them and continue to implement an effective QM program. So briefly under our monthly update section we cover pre analytic section of the laboratory analytic section of the laboratory post-analytic section of the laboratory. Physical facilities and safety. Personnel training and competency continuing education. RND updates. Employee and patient quality communication. Key quality indicators. Policy procedures exchanges and updates. CAP notification to satisfy the CAP terms of accreditation. And referral laboratory meeting minutes and updates.
In our quarterly updates we cover our vendor policy and procedures updates and changes and key quality indicator tracking. Under our semi annual updates we cover personnel training competency and continuing education. And under our annual updates we cover again personnel training competency continuing education. Technical incident and corrective action annual review. Physical facilities and safety. Laboratory policy and procedure review. HIPA audit. Customer satisfaction survey. Annual review of key quality indicators. Annual review of test result statistics. Referral laboratory policy. And self inspection.
