The FAA has released a new draft Advisory Circular that is intended to describe acceptable statistical methods for developing substantiating data for comparative test and analysis compliance findings to support FAA approval of turbine engine and APU parts produced under a PMA, TC, STC, repair or alteration. The guidance is intended to help generate substantiating data to support compliance with the airworthiness requirements of 14 C.F.R. 21.303 as well as parts 33, 43 and TSO C77.
Several members have brought this AC to our attention and expressed their concerns about the details of this guidance. We welcome specific comments from members about problems and/or ways to improve this document. Based on member comments we have received so far, the proposed guidance appears likely to discourage applications to produce replacement parts. One reason for this is the large suggested sample size, and associated high costs, necessary to satisfy the proposed statistical analysis metrics.
The proposed AC discusses statistical principles and methods in an effort to explain appropriate approaches of determining proper sample sizes and population specimens for the purpose of showing equivalency and discrepancies between sample populations of parts.
The AC primarily attempts to address two separate problems: (1) the use of statistical tests designed to show difference rather than equivalency and (2) the use of small sample sizes.
The guidance observes that applicants frequently submit data based on a simple t-test, which is designed to determine difference, rather than equivalency. This is a problem because failure to show a statistically significant difference is not the same as demonstrating the equivalence between an approved part and a replacement part. The guidance states that a methodology for demonstrating equivalency must instead be used.
The guidance also explains the procedure for determining the sample sizes of approved and replacement parts necessary to demonstrate equivalency. This test for determining sample size yields a large sample-size requirement, based on the needs of the statistical test for equivalency as well as the importance of accounting for lot-to-lot variance.
The practical effect of compliance with the suggested sample size would appear to make the cost of conducting the required testing incredibly cost-prohibitive. MARPA seeks comments from its members to determine whether compliance with the sample sizes indicated by the suggested procedure are economically feasible.
The FAA appears to acknowledge the burden of the sample sizes indicated by the formula by including in the proposed guidance a section titled “Practical alternative for sample size requirements.” The section offers alternative suggestions to using the statistically derived sample sizes and methods for analyzing small samples.
The proposed AC also appears to take an unrealistic view of the nature of part procurement when procuring approved parts for the purpose of comparison. The guidance repeatedly emphasizes the importance of drawing sample parts from different lots to account for variance. Determining the source lots of a series of approved parts seems to be an unrealistic burden without substantial compliance from the manufacturer.
It is important that the engine PMA community examine this guidance to determine whether it offers any benefit to safety and whether compliance would even be practicable.
You can find this proposed AC, Statistical Analysis Considerations for Comparative Test and Analysis Based Compliance Findings for Turbine Engine, at http://www.faa.gov/aircraft/draft_docs/media/AC_33-X.doc.
Comments are due to the FAA by July 11, 2013. Comments can be emailed to Marc Bouthillier at Marc.Bouthillier@faa.gov or faxed to (781) 238-7199. We would appreciate copies of your comments, so we can make sure that MARPA‘s own comments supports the industry’s comments and needs.