External Quality Assessment Automated Laboratory
Joseph Murray
What is External Quality Assessment? EQA – “is a program in which multiple samples are periodically sent to members of a group of laboratories for analysis; whereby each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratories and others.”
Some key points... • Analyte value unknown to participants prior to analysis • EQA specimens should be analysed using exactly the same process as patient specimens • Allows for inter-laboratory comparison of reagents, methods, instruments • The data is kept confidentially from other participants ensuring anonymity •Unlike Internal Quality Control (IQC) which provides an indication of how an assay is performing in real-time the results of EQA are retrospective •EQA schemes are available for most of the tests available across all disciplines in pathology •Provided by a number of different organisations •In Clinical Chemistry mainly focus on the analytical process and method performance –rather than pre- and post-analytical phases •Participation in EQA schemes is a requirement for UKAS accreditation
An overview of the EQA Process... External Quality Assessment Scheme PROVIDER 1. Clinical material dispatched to the USER laboratory
5. RESULTS from all participating USER laboratories analyzed and a report indicating the performance of an individual laboratory's performance in relation the performance of all participating laboratories
6. Report indicating the performance of an individual USER laboratory's performance in relation to the performance of all participating laboratories
External Quality Assessment Scheme USER 2. Clinical Material received by the USER laboratory
3. Clinical material examined by the USER laboratory and the results recorded
4. Examination results returned to the External Quality Assessment scheme PROVIDER 7. USER laboratory reviews its performance in relation the to the performance of all participating laboratories and takes action to remedy any problems
Why do it? QUALITY ASSURANCE!!!!!! EQA is one facet of QUALITY ASSURACE. In addition to IQC, health & safety, clinical governance , training & competency and audit it helps us as a service to achieve our QUALITY goals and form part of the laboratory’s Quality Management System (QMS).
Further info can be found in the laboratory ‘Quality Manual’ - PROC-G-QualMan - on iPassport!
How does EQA fit into the overall Quality framework in UK Pathology?
EQA schemes should be accredited by UKAS to ISO standard 17043
National Quality Assurance Advisory Panels (NQAAPs) (discipline specific experts) oversee EQA schemes and set performance criteria. Joint Working Group for Quality Assessment (JWGQA) (multidisciplinary group within RCPath), ultimately responsible for EQA schemes
So what happens when performance is poor? If issue fails to get resolved passed to JWGQA
JWGQA (no mandatory powers) contact relevant hospital CEOs and reports to Care Quality Commission (CQC) and UKAS!!!
Lab reported to relevant NQAAP if fails to engage with EQA provider/improve performance
When performance poor EQA provider works with lab to improve
Performance criteria differs between EQA providers – performance standards not harmonised
Automation EQA Schemes... Provider
NEQAS
NEQAS – Cardiac Marker NEQAS -IMMQAS
WEQAS
Scheme
Analyte
Distribution Frequency
AFP, CEA & hcG
AFP, CEA, HCG
Monthly
Antibiotic Assays
Gentamicin, Vancomycin, Amikacin, Tobramycin
Monthly
Clinical Chemistry
Albumin, ALK, ALT, Amylase, AST, Bicarbonate, Calcium, CK, Cl, Creatinine, GGT, Glucose, Iron, K, LDH, Lithium, Mg, Na, Osmolality, Phosphate, Tbil, TP, Urate, Urea
Bi-weekly
CRP
CRP
Monthly
Fructosamine
Fructosamine
Monthly
Peptide Hormones
FSH, LH, Prolactin
Monthly
Pregnancy Testing
HCG-Urine
PTH, ACTH, hCT
PTH
Bi-Monthly
Salicylate, Paracetamol, Ethanol
Carbamazepine, Digoxin, Ethanol, Lithium, Paracetamol, Phenobarbitone, Phenytoin, Salicylate, Theophylline, Valporate
Monthly
Specific Proteins
A1 antitrypsin, Caeruloplasmin, Haptoglobin, IGA, IGG, IGM, Orosomucoid, Transferin, Cortisol, DHAS, Female Testosterone, Male Testosterone, Oestradiol, Prgesterone, SHBG
Monthly
Thyroid Hormones
FT3, FT4, TSH
Monthly
Urine Chemistries
ACR, Albumin, Amylase, Calcium, Cl, Creatinine, Glucose, K, Mg, Na, Osmolality, PCR, Phospate, TP, Urate, Urea
Monthly
eFGR
Creatinine, eGFR
Monthly
Cardiac Marker
Troponin T
Monthly
CSF Biocehmistry
Glucose, Lactate, TP
Bi—Monthly
PSA
PSA
Monthly
Tumour Markers
CA125, CA153, CA199
Bi-Monthly
Ammonia
Ammonia
Monthly
Bile Acids
Bile Acids
Monthly
Bilirubin
Dbil, Tbil
Monthly
Lipids
Cholesterol, HDL, LDL, Trigs, LPA
Monthly
NT PRO BNP
BNP
Bi-Monthly
Serum ACE
ACE
Bi-Montlhy
The Process – Booking To Submission
Booking In • EQA specimen labelled (primary/secondary tube and paperwork) and booked in by trained member of staff in specimen reception • Specimen treated like external laboratory requests – ‘EQA’ entered as ‘Requesting Source’ • Each scheme allocated unique patient registration number with demographics providing additional info. E.g.
• Under ‘Specimen Comments’ distribution number and Sample number entered •A number of test super-sets have been created for each scheme
Procedure outlined in SOP ‘Booking in External Quality Assurance Samples’ - PROC-CB-H5.3/Booking - on iPassport!
Sample Processing • Once labelled and booked Specimen Reception pass specimens (with paperwork) to Automated laboratory • Important to analyse EQA samples on day of receipt – if not possible should be stored appropriately and delay in analysis recorded on paperwork • Any preparation instructions provided should be adhered to ‘exactly.’ Especially important when reconstituting lyophilised samples e.g.
TROPONIN...
Reconstitution instructions for Troponin EQA
Sample Processing • Once labelled and booked Specimen Reception pass specimens (with secondary tubes and paperwork) to Automated laboratory • Important to analyse EQA samples on day of receipt – if not possible should be stored appropriately and delay in analysis recorded on paperwork • Any preparation instructions provided should be adhered to ‘exactly.’ Especially important when reconstituting lyophilised samples e.g. Troponin • Once prepared samples transferred to labelled (bar-coded) secondary tube for analysis – transfer just enough for analysis so that remainder in primary tube can be stored • Ensure specimens placed on correct analyser – special consideration should be made when analysing NEQAS General Chemistry distributions...
CITY General Chemistry EQA:
CITY General Chemistry EQA: Two sets of specimens received for NEQAS General Chemistry Distributions per site Need to examine test requests on each sheet and decide most appropriate analyser for analysis!!!! Ideally each set should be analysed on SAME analyser every distribution – this allows performance issues to be correctly attributed to specific analysers Any discrepancies (i.e. moving samples between analysers to complete requests) should be raised with Section Senior. Set repertoire can be updated with NEQAS for subsequent distributions Best practice to record on distribution sheet which analyser(s) was (were) used for analysis
Which Analyser Would You Use To Analyse These Specimens?
Post Analysis • Check the specimen in Telepath. Ensure all requests have been completed... • Check the EQA distribution sheet against what has been booked in on Telepath. Ensure no requests have been missed!!!! Better to resolve problems now, rather than having to locate specimens later!
Transcription of Results • Once analysed results manually transcribed onto distribution sheet • Transcribe all results from Telepath system to sheet!!! Complete ALL relevant fields e.g. Date of receipt/analysis, storage conditions, kit/calibrator lot number
Transcription of Results • DON’T transcribe directly from analyser screen - units and decimal places may be different!!!
Retention of Specimens & Paperwork • Completed samples stored in ‘Complete EQA Draw’ on bottom shelf of ‘QC Fridge’ at Sandwell and ‘EQA draw’ in Freezer in old Haemoglobinopathy lab at City •Ensure when placing specimen into specimen bags that lids are tightly in place to prevent spillage!!! • Once reports have been returned from Scheme Provider EQA specimens at Sandwell are moved to Freezer in Manuals lab. At City EQA specimens are moved from Freezer in old Immunology Automation lab on a monthly basis and retained in appropriate rack for period of three months • The paperwork is placed in the ‘EQA tray’ mounted on the wall at Sandwell and in the tray on the cabinet bellow the Communication Screen at City ready for submission •Paperwork should NOT be transported across site so as not to go astray!!!
Result Submission • The Results for Automation EQA schemes are submitted manually via the appropriate scheme Providers website – details of which are provided in the table below... EQA SCHEME PROVIDER
WEB ADDRESS
NEQAS
www.birminghamquality.org.uk
WEQAS
http://www.weqas.com/
IMMQAS
www.immqas.org.uk
NEQAS - Cardiac Markers
www.ukneqas-cm.org.uk
Preparation, Analysis, Transcription and Result Submission covered in SOP ‘External Quality Assurance Preparation and Reporting’ PROC-CB-H5.3/Reporting - on iPassport!
EQA Providers – Performance Criteria & Reports
Performance & Reports • Performance criteria differs between EQA providers e.g. UK NEQAS & WEQAS • Performance criteria also differs between different divisions of same provider e.g. UK NEQAS ‘Birmingham Quality Clinical Chemistry’, ‘Edinburgh Peptide Hormones’ and ‘Immunology, Immunochemistry & Allergy’ • Report format very different between providers • Although different reports are designed to be easily and quickly interpreted whilst performance scoring systems have similar underlying principles • NQAAP is attempting to harmonise performance criteria between providers using Minimum Analytical Performance Standards (MAPS)
UK NEQAS
– UK National External Quality Assessment Scheme Birmingham & Edinburgh
Performance of each analyte measured based on an A, B, C scoring system... • A is for Accuracy (total error) • B is for Bias • C is for Consistency of bias Each score is calculate using all specimens within a rolling time window – usually six distributions. As new data is added the oldest is lost Calculated as follows.... • B score is the trimmed mean of % bias for each specimen within time window % bias = ((result-target)/target)*100 • C score is the trimmed SD of the B Score • A score is the trimmed mean of the Specimen Accuracy Indices -Specimen Accuracy Index calculated for each specimen using transformed bias -Transformed bias = specimen % bias/degree of difficulty -Degree of difficult is assigned to each specimen by NEQAS based on the ‘difficulty’ in obtaining the target value -Modulus of the transformed bias gives the Specimen Accuracy Index
REPORT - Feedback Page • Provides any information NEQAS wants to convey to users
• This includes any comments made by the lab at the time of result submission (e.g. analytical issues) or any performance issues NEQAS wants highlighted
Performance Summary Icons • Gives and indication of how each analyte is performing at a glance – ideal for quickly reviewing reports
• Penalty box plots based on NEQAS scoring system used to illustrate labs performance in relation to peers – a traffic light system helps distinguish between GOOD, ACCEPTABLE & POOR performance. • Based on data from last six distributions!!!
Distribution Summary • Summarises the date from the current distribution for each analyte
Distribution Summary • Summarises the date from the current distribution for each analyte
Method Summary • List the methods (including underlying principle) used by the laboratory for each analyte
Method Summary • List the methods (including underlying principle) used by the laboratory for each analyte • The labs A score for each analyte tabulated alongside the median A scores for ‘Method’ and ‘All Lab’. Allows comparison with peers
• Data represented as box & whisker plots
Analyte Specific Pages
• ‘Specimen level’ statistics - detailed breakdown of how each analyte performed for each specimen in current distribution
• ‘Rolling time window level statistics’ for each analyte
‘Specimen Level’ Stats Comm0n terms: • ALTM – All Laboratory Trimmed Mean • GLTM – Group Laboratory Trimmed Mean • MLTM – Method Laboratory Trimmed Mean • % Bias Vs. Transformed Bias Analytes A, B and C scores with limits – rolling time window data, not just current distribution Labs method info – key relates to histograms
Specimen composition – usefully for identifying possible matrix effects
‘Specimen Level’ Stats Histogram displays graphically data from table Table shows mean, SD & CV for All, Group & Method
Current distribution Box Whisker Plots of B & C scores for All, Group & Method
‘Rolling Time Window’ Stats
‘Rolling Time Window’ Stats
a.
b.
c.
d. a. b. c. d.
Specimen Accuracy Index A score Specimen % Bias B score
Plotted over a number of distributions – allows observation of Trends
‘Rolling Time Window’ Stats
a.
c.
b.
d. a. Difference (result-target) Vs. Concentration b. C Score c. Specimen % Bias Vs. Concentration d. Penalty Box Plot of B Score against C Score a. and c. allows observation of concentration dependent bias. b. Provides trend data for C score. d. allows between lab comparison – are we in agreement?
‘Rolling Time Window’ Stats
a.
b. a. Specimen Accuracy Index Vs. Concentration – enables observation of concentration dependent inaccuracy b. Between-laboratory agreement by concentration
UK NEQAS For Immunology, Immunochemistry & Allergy (IMMQAS) Performance of each analyte measured based on a Variance Index Scoring System (VIS). Performance expressed in terms of... • MRVIS - Mean Running Variance Index Score • MRBIS - Mean Running Bias Index Score • SDBIS - Standard Deviation of Bias Index Score MRVIS gives an indication of the degree of imprecision. MRBIS gives an indication of any consistent tendency (positive or negative) SDBIS gives an indication of the variability of the bias index (hence imprecision) Data is collected within a rolling time window and ‘trimmed’ or ‘smoothed’ to remove erratic outliers An arbitrary scaling factor, Chosen Coefficient of Variation (CCV), is applied to each analyte to reflect current ‘state of the art’ and provide a common means of comparison between analytes
IMMQAS Report Very similar to other UK NEQAS reports in terms of layout which uses...
IMMQAS Report Very similar to other UK NEQAS report in terms of layout which uses...
A Table of mean, SD & CV for All, Group & Method
IMMQAS Report Very similar to other UK NEQAS report in terms of layout which uses...
A Histogram to display graphically data from table
IMMQAS Report Very similar to other UK NEQAS report in terms of layout which uses...
Graphs to evaluated trend
IMMQAS Report Very similar to other UK NEQAS report in terms of layout which uses...
Scatter or Penalty box plot to displays inter-lab consensus
IMMQAS Report Very similar to other UK NEQAS report in terms of layout!!!!
Exception being the scoring system!!!
WEQAS -
Wales External Quality Assessment Scheme
Performance of each analyte measured based on an Standard Deviation Index (SDI) scoring system used in conjunction with a WEQAS SD SDI is a Total Error score and is calculated as: SDI = (Participant result – target value) / WEQAS SD WEQAS SD is fixed for a given analyte and reflects current ‘state of the art’ and provides a common means of comparison between analytes
WEQAS Report – Performance Summary
WEQAS Report – Performance Summary
Table referring to current distribution Performance... -Overall Lab SDI is score for all specimens (inc. multi-analyte) ran by lab on current distribution - Median All Laboratory is median SDI score for all labs. - 97.5th percentile is SDI score within which 97.5% of participants fell
WEQAS Report – Performance Summary
Performance criteria
Graph shows Overall Lab SDI score over a number of distributions – allows observation of trend
WEQAS Report – Performance Summary
Table of SDI scores including individual analyte SDIs for current distribution
Analyte Specific Pages
Analyte Specific Pages
Table shows mean, SD & uncertainty for lab, Method & Instrument SDI given for each specimen as well as average analyte SDI
Analyte Specific Pages
Graph shows analyte SDI scores over a number of distributions
Analyte Specific Pages
X-axis target value, y axis target +/-3SD Bias Plots (Bland-Altman): - Left hand plot represents Current distribution – includes lab, method and instrument performance - Right hand plot shows Previous distribution results, including current distribution
Analyte Specific Pages
Precision and Accuracy indicators
Review of EQA Reports • EQA reports reviewed by Pervaz - EQA Officer for Clinical Chemistry • Each section has a responsibility to monitor their own performance • An EQA Evaluation Sheet is completed by the EQA Officer when performance issues arise •Performance issues are brought to the attention of Senior staff in the relevant section and urgent action initiated to investigate/resolve the issue •Biochemistry hold monthly EQA meetings where different sections report their performance to the group • EQA reports and evaluation sheets are stored in the EQA folder on the ClinChem drive Interpretation and review of EQA reports is covered in SOP ‘Review and Interpretation of External Quality Assurance results’ - PROC-CB-H5.3/Interpretation - on iPassport!
EQA –
Providing Quality Assurance
Consistent Poor Performance
EQA Provider Intervention
Improvement (Lab changed reagent supplier)
In-House Report Summaries
Review of EQA reports and monthly summarisation of data – to be displayed on EQA board to keep staff informed!!!
END
