As the holiday season rapidly approaches, I find myself sitting on an airplane pondering about the voluminous amount of prose, Dr. D has entered into his trusty laptop, in an effort to enlighten and guide readers through the often-ambiguous world of medical device regulations. In an adventure that began in early spring, the doctor has been traversing through the Quality System Regulation (QSR), providing banausic (look-it up) insights and salient points for compliance, while injecting humor and some sarcasm from the perspective of a long-time quality professional. Dr. D’s writing is motivated by the need to ensure all device manufacturers enjoy the success that can be eventuated from sustained compliance to the QSR.

That said, 21 CFR, Part 820 – Subpart O (Statistical Techniques) is the proverbial “Final Act” in regards to the QSR. This edition of DG will bring to a close a series of articles that I hope the readers have thoroughly enjoyed while gleaning some practical information.

Warning letter violations
Sometimes the doctor just feels like screaming – AAAARRRRGGGGHHHH! For this week’s edition of DG, the three lucky recipients of FDA’s “You Suck” award (a.k.a. warning letter) each received an observation that commences with the infamous and often used introduction of “Failure to Establish and Maintain Procedures.”

Week after week, FDA routinely inspects the effectiveness of device manufacturers’ quality systems; and week after week, the agency routinely issues Form 483s that often translate into warning letters. Dr D sincerely hopes that the readers will learn from this edition and all of the previous editions of DG that procedures are everything in the medical device industry. The expectation of FDA is that the procedures device manufacturers create encompass the entire QSR and not just the ones the device manufacturers believe they need to comply. People, I have been there and want to reinforce the belief: “Warning Letters are no Fun!”

For device manufacturer’s wishing to exacerbate their predicament with the agency, go ahead and provide a response to an observation without sufficient evidence supporting the issues associated with an observation have been resolved. If there are doubts about what the agency is looking for in regards to corrective action and closure, pick up the phone and ask the agency for clarification, just do not shoot from the hip.

Two of FDA’s “You Suck” Award recipients provided FDA with responses that were deemed insufficient. Do you know what happens to device manufacturers that provide poorly positioned responses to Form 483s? The Form 483s morph into warning letters. Do you know what happens to device manufacturers that provide poorly positioned responses to warning letters or fail to respond to a warning letter within 15-days of receipt? They enter FDA’s house of pain and take the first giant leap toward Consent Decree!

Warning letter One (June 2010): Observation 8 of 14 – Failure to establish and maintain adequate procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed as required by 21 CFR 820.250(b). For example, your firm’s procedure (b)(4) Sampling of Finished Products, Revision (b)(4) describes the requirements for collecting samples of the finished product. However, the procedure does not discuss sampling for additional testing to overcome a failure of the finished product, other than to say that an investigation shall be conducted. As a result, the (b)(4) which is integrity testing, was used to release lots of prefilled syringes when defects affecting the container closure integrity were found after or during manufacturing. Sampling for (b)(4) consisted of (b)(4) regardless of the size of the prefilled syringe lot. In some cases this size was as low as (b)(4) of the lot. Your firm released the following lots of I.V.

Warning letter Two (May 2010): Observation 6 of 6 – Failure to establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capabilities and product characteristics, as required by 21 CFR 820.250(a). For example, there was no recognized sampling plan methodology incorporating a valid statistical technique to verify the acceptability of the process and take appropriate action when nonconforming components or products are identified that do not meet the acceptable quality limit (AQL).

FDA Response to Observation 6 of 6 – We have reviewed your response dated December 17, 2009, and have concluded that it is inadequate because it does not include documentation demonstrating that the sampling plan methodology issues have been appropriately addressed.

Warning letter Three (March 2010): Observation 5 of 5 – Failure to establish and maintain procedures for identifying valid statistical techniques required for establishing, controlling and verifying the acceptability of process capability and product characteristics, as required by 21 CFR § 820.250(a). For example, your firm does not have adequate statistical rationale to support the techniques used in trending of customer complaints per your film’s procedure SP-14122, “Complaint Trending and Escalation Process,” revision F, issued October 02, 2008

FDA Response to Observation 5 of 5 – Your firm’s response dated October 16, 2009, is not adequate because your firm has not provided adequate statistical rationale for your film’s Complaint Trending and Escalation Process procedure. Please contact the Center for Devices and Radiological Health’s (CDRH) Office of Compliance (OC) with further responses.

Quality System Regulation – 21 CFR, Part 820
QSR – Subpart O Section 820.250 Statistical Techniques

(a) Where appropriate, each manufacturer shall establish and maintain procedures foridentifying valid statistical techniques required for establishing, controlling, and verifying the acceptability of process capability and product characteristics.

(b) Sampling plans, when used, shall be written and based on a valid statistical rationale. Each manufacturer shall establish and maintain procedures to ensure that sampling methods are adequate for their intended use and to ensure that when changes occur the sampling plans are reviewed. These activities shall be documented.

Statistical techniques
Before I dive into statistical techniques, Dr. D would like to refresh the readers with a definition of statistics. According to Merriam-Webster’s On-Line Dictionary, “statistics” is a branch of mathematics dealing with the collection, analysis, interpretation, and presentation of masses of numerical data. Device manufacturers need to understand the purpose and value of applied statistics in supporting decisions associated with initial design, process development, and ongoing compliance to a published and approved product specification. When I visit suppliers, and I ask about applied statistical methodologies, and get the answer, “we 100 percent inspect all characteristics,” I must inform the suppliers this is not an acceptable statistical technique. In fact, it is not even an effective approach to inspection. My belief is that process capability studies, with the results interpreted in CpK and PpK, is the only true path to establishing effective statistical techniques and the rationale required by the regulation.

When discussing the employment of statistical methodologies, Dr. D always like to point engineers in the direction of Juran’s Quality Handbook, which some of us in the industry refer to as the Quality Engineer’s Bible. Dr. D also recommends visiting Dr. Wayne Taylor’s website and reading his work on effective sampling plans. Finally, with so many statistical tools available, such as Minitab™ (no – Dr. D is not a paid spokesperson for Minitab) there should never be an excuse for device manufacturers not having a robust and documented approach to statistical techniques..

Procedure(s)
It is now time for another Dr D broken-record time. As the doctor has opined on multiple occasions; DG Rule # 6 is not an option; “All procedures, work instructions, drawings, specifications, etc. must be written, well-documented, and controlled within a defined document control system.” Nestled into every single requirement of the QSR is the phrase, “Shall establish and maintain procedures.” It seems pretty clear to Dr. D, so I struggle to understand why device manufacturers continue to misinterpret the need for procedures.

In regards to statistical techniques, the agency is really looking for device manufacturers to collect data and then interpret the results of the data through the application of statistics. For example, if Acme Device Corporation (fictional) has a critical requirement for device length, the agency expects Acme to prove that the device meets this critical requirement on continuous basis. Can you say process capability studies? In this example, Acme should be collecting dimensional data and using statistics, e.g., to ascertain if ongoing processes are capable and remain in statistical process control – English translation “the device length is within specification and we have the data to prove it – done!” If Acme validated the process as being Six Sigma capable and the data is now trending below a PpK of 1.0, there now appears to be a problem.

How a device manufacturer identifies the problem, though the employment of statistics, and how the problem is corrected is one of the fundamental foundations of this requirement. If device manufacturers are collecting data just for the sake of having the data, well – what is the point?

In summarizing 820.250(a), device manufacturers shall have a procedure that delineates robust statistical methodologies for driving process control; and supporting ongoing inspection activities in determining that measured characteristics are acceptable and within their specification limits.

Sampling plans
As I stated earlier, Dr. D strongly suggest reading Dr. Taylor’s work on statistics and sampling plans as a whole. Additionally, the American Society for Quality maintains two of the most recognizable standards for determining appropriate sample sizes and creating effective sample plans, ANSI/ASQ Z1.4-2003 for the inspection of attributes and ANSI/ASQ Z1.9-2008 for the inspection of variables. Adherence to these mainstays of acceptable approaches to sampling methods will result in a solid foundation for device manufacturers to support compliance to 820.250.

Over the years, I cannot count the times that Dr. D has been asked to evaluate the statistical significance when the sample size employed was a N=3 or an N=5. Do you have any idea how difficult it is to defend that type of sample rationale to any regulatory body, especially FDA? It is the doctor’s opinion; there is no way to defend the approach, if it is documented by procedure. The approach is irresponsible, reprehensible, lackadaisical, nonsensical, impractical, laughable, and indefensible, when sitting across from the agency during one of their friendly visits.

To summarize, the doctor strongly suggests that the documented approach to sampling be premised on recognized standards. Not only does the sampling methodology need to be delineated with the procedure, so does the ability to adjust sampling plans premised on the results. The doctor strongly suggests that sampling plans are routinely reviewed and the results of the review documented. Additionally, the need to review sampling plans needs to be depicted in the actual procedure.

Sampling plans need to be linked back to risk and risk indices. When determining an appropriate level of sampling for a component or device, it is imperative that the PFMEA and DFMEA be evaluated, because the failure mode effects analysis will lead you to applying the appropriate sample size premised on the actual risk of failure. Finally, ensure your supplier base is capable of understanding and employing industry recognized statistical concepts. Suppliers that routinely employ effective statistical approaches to process control will be in position to deliver product that meet specification.

Takeaways
There is a plethora of data, standards, and websites that can provide useful information needed to create robust procedure(s) for establishing effective statistical control. Additionally, this same information is available for establishing effective sampling plans. As captured in the warning letter extractions, FDA will evaluate a device manufacturer’s approach to statistics and sampling during one of their friendly visits. Establishing robust procedures, in advance, will mitigate the potential receipt of a Form 483.

In closing, thank you again for joining Dr. D and I hope you find value in the guidance provided. Until the next installment of DG, when Dr. D will provide some guidance for managing a visit from the FDA  – cheers from Dr. D. and best wishes for continued professional success.

References:

1. Aczel, D., & Sounderpandian, J. (2006). Complete business statistics (6th ed.). Boston: McGraw-Hill Irwin.
2. Code of Federal Regulation. (2010, April). Title 21 Part 820: Quality system regulation.  Washington, D.C.: U. S. Government Printing Office.
3. Devine. C. (2009, July). Exploring the effectiveness of defensive-receiving inspection for medical device manufacturers: a mixed method study. Published doctoral dissertation Northcentral University. Prescott Valley, AZ. 
4. FDA – U.S. Food and Drug Administration Website. (2010). Warning letters. Retrieved November 12, 2010, from http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/
5. Juran, J. & Godfrey, A. (1998). Juran’s quality handbook (5th ed.). New York, New York: McGraw-Hill.
6. Merriam-Webster’s On-Line Dictionary. (2010). Statistics definition. Retrieved November 25, 2010, from http://www.merriam-webster.com/dictionary/statistics
7. Sampling procedures and tables for inspection by attributes. (2003). American Society for Quality ANSI/ASQ Z1.4-2003. Milwaukee, WI.
8. Sampling procedures and tables for inspection by variables. (2008). American Society for Quality ANSI/ASQ Z1.9-2008. Milwaukee, WI.
9. Taylor, W. (1993, November). Classifying defects and selecting AQLs. FDC Control, Food Drug & Cosmetic Division ASQC, 103. Retrieved March 23, 2007, from http://www.variation.com/techlib/as-1.html
10. Taylor, W. (1996). Selecting statistically valid sampling plans. Quality Engineering 10(2). Retrieved March 5, 2007, from http://www.variation.com/techlib/as-7.html