The FAA has issued a new advisory circular for statistical analysis. This AC only applies to engine and APU parts.
The new advisory circular uses statistical analysis to arrive at correct sample sizes. This sample size formula is introduced for persons who are trying to correlate two populations of parts. MARPA had pointed out to the FAA that typically a PMA applicant does not correlate two different populations of parts – instead they derive the reasonable tolerances on one population of parts, and then design and produce within those tolerances. The FAA would like PMA applicants to arrive at their appropriate sample size, test the parts being reverse-engineered, and then produce an equivalent number of pre-PMA parts to test for the same properties (and then correlate the two populations). This is contrary to current FAA regulatory guidance, which requires the design to be approved and the requires the production quality system to ensure that parts are all produced within the approved design parameters. In essence, ACO engineers will now take control of the quality assurance system through the design process.
One of the problems with the AC is that it relied on statistical analysis for clinical trial sample size as the basis for assessing statistical analysis of reverse-engineering sample size. Clinical trials for pharmaceuticals typically rely on populations of hundreds or even thousands of people. Trying to test hundreds or thousands of parts in order to reverse-engineer them is simply not realistic. Furthermore, the degree of part-to-part difference under modern quality assurance systems does not support such large sample sizes.
For those cases where the equations in the advisory circular give a lower number for the appropriate sample size, the FAA has also established minimum sample sizes. The AC sets some minimum limits for the number of parts that must be tested in order to derive certain values (remember that you need that number of PC parts and also that number of reverse-engineered parts to meet the AC’s requirements):
Minimum Number of Parts to be Sampled
- For basic material properties that are more dependent on alloy constituency than on part manufacture process – 10 approved parts from three separate lots with at least three parts per lot
- For properties affected by how the material is processed during part manufacture such as high-cycle fatigue, low-cycle fatigue, creep, tensile strength, crack growth, etc. – 30 approved parts
- For fatigue testing – at least 25 tested parts that are run until they crack
- For parts exhibiting complex geometry or complex manufacturing variables, additional specimens may be required
- For parts considered to have a high degree of criticality, greater sample sizes may be required
No statistical basis is offered for these minimum sample sizes.
The new advisory circular provides guidance for statistical analysis of sample-size despite the fact that no regulation actually requires such a broad-based sample. The advisory circular appears to potentially change the regulations by increasing the burden on applicants. To the extent that this is true, it is inappropriate.
Our concern is that despite warnings that this is non-mandatory guidance, this guidance may be used as if it were a regulation, with offices refusing to accept PMA applications that are otherwise valid, but that failed to use this AC as a basis for identifying sample size.
If you find that this AC effectively changes the application obligations imposed on you as a PMA applicant, STC applicant, or other FAA-approval applicant, then please contact MARPA so we can raise this concern with the appropriate personnel at the FAA.
AC 33-10 is known by its full title: Statistical Analysis Considerations for Comparative Test and Analysis Based Compliance Findings for Turbine Engine and Auxiliary Power Unit Replacement, Redesign and Repaired Parts.
Some of you may be wondering where to find the FAA’s policy memo on design approval holder restrictions on ICA availability. Older MARPA links to the original position of that guidance are no longer valid because it has been moved. But it is still available if you know where to look!
Many MARPA members look for this policy memo because it clarifies that anti-competitive language in ICAs (restricting use of PMAs or third party repairs) is unacceptable to the FAA:
While not exhaustive, the FAA finds the following practices of using restrictive language in the ICA or through restrictive access or use agreements unacceptable under the provisions of 14 CFR §21.50(b) and related ICA airworthiness requirements:
1) Requiring the owner/operator to only install DAH-produced or authorized replacement parts, articles, appliances, or materials.
2) Requiring that alterations or repairs must be provided or otherwise authorized by the DAH.
3) Requiring the use of only maintenance providers or other persons authorized by the DAH to implement the ICA.
4) Establishing, or attempting to establish, any restriction on the owner/operator to disclose or provide the ICA to persons authorized by the FAA to implement the ICA.
MARPA had a very good meeting today with Mark Bouyer and Ann Azevedo of the FAA’s Engine and Propeller Directorate (EPD).
The focus of the meeting was status on EPD policy that may affect PMA manufacturers.
Azevedo explained that she has responded to the comments on the Statistics Advisory Circular (AC), and hopes to have the final draft of that guidance available to the public by September. The Statistics AC is meant to address FAA concerns that have been recognized in practice, such as misusing statistical methods to show equivalence, and underestimating the appropriate sample sizes.
Bouyer expects the Materials AC to go out for public comment this month. The Materials AC will identify the essential data that is necessary when a PMA applicant is trying to replace the material used in the type design. MARPA members should watch for this one, and be prepared to offer their comments.
The FAA had published the Geometry AC for comment. This AC is meant to enhance awareness of how reverse engineering can introduce dimensional differences in replacement parts. MARPA Board members have expressed that the draft of this proposed guidance appeared to be very helpful to the industy.
The Burner Rig AC, which was also previously out for comment, is expected to be issued by September. It is expected to provide a method for establishing functional equivalence for certain degradation modes in parts such as oxidation, hot corrosion, erosion, etc. The AC is expected to identify existing technology as a means of compliance.
Finally, the FAA is internally reviewing AC 33.8 with a plan to update and clarify the AC. The updates are intended to make the AC easier to use.
The FAA has been very active and diligent in preparing guidance. In the immediate future (before the Conference), MARPA members should expect to see the release version of the Statistics AC and the Burner Rig AC, and they should expect to see the Materials AC go out for comment.
MARPA recently filed comments on the FAA Engine and Propeller Directorate’s draft Advisory Circular 33-Geometry, discussing geometry and dimensional considerations for comparative test and analysis for turbine engine and APU replacement parts. In a previous blog post we observed that AC identified a number dimensional and geometric factors that the FAA expects to be assessed in ensuring the integrity of dimensional characteristics for the purposes of showing similarity.
We requested feedback from our members describing to what extent the FAA’s expectations were reasonable and practicable, and identifying any issues with the proposed guidance on which MARPA should comment. We received several very helpful responses from our members that helped us shape our comments to the proposed AC. Among the issues members identified were:
Feedback from our members is both helpful and valuable to our comments, as it helps us to identify issues that directly affect members’ businesses, and helps us to better focus our resources on those matters that are important to the PMA community. The result is more detailed and on-point responses to the FAA to better help shape the guidance material that will ultimately be issued.
We greatly appreciate the feedback we received from our members on this Advisory Circular, and we hope that our members will continue to answers our requests for responses as additional guidance and rulemaking documents are issued. Together we can work with the FAA to develop the best possible guidance for our industry.
The FAA is currently seeking comments on its Draft Advisory Circular Engine Overtorque Test, Calibration Test, Endurance Test, and Teardown Inspection for Turbine Engine Certification (§§ 33.84, 33.85, 33.87, 33.93).
As the title suggests, the AC offers guidance on compliance with the engine overtorque, calibration, and endurance tests, and teardown inspection called out in Part 33 of the Federal Aviation Regulations. Although the guidance is directed at engine manufacturers, foreign regulatory authorities, applicants for engine type design approval, and FAA designees, it also notes that parts manufacturer approvals “may require running certain endurance testing for compliance with § 33.87″ and refers to AC 33.87-2 for guidance on showing compliance by comparative test methods.
MARPA would like to know to what extent members anticipate this AC might effect them, and whether we should submit comments. If you plan on submitting comments, or have already done so, we would would like your feedback so that we can incorporate member concerns into our comments.
Comments on the Draft AC are due next week, so if you have feedback for us please submit them to firstname.lastname@example.org soon!
Today, the FAA announced the issue of AC 21.3o3-4. The title of this AC will be “Application for Parts Manufacturer Approval Via Tests and Computations or Identicality.”
As of this time, the AC is not yet available to the public. When it is available to the public, it is expected to be posted online at this location:
Note: the issue date published on the FAA’s website is March 21, 2014; however the AC is not yet available through the FAA’s online advisory circulars, nor through the Regulatory and Guidance Library (RGL). The note on the FAA’s website states “This document‘s content is not currently available.”
The FAA Engine and Propeller Directorate has issued a new draft Advisory Circular that could have a significant effect on companies seeking PMAs, STCs, or approval of repair or alteration of turbine engine and APU parts. Draft AC 33-Geometry: Geometry and Dimensional Considerations for Comparative Test and Analysis for Turbine Engine and Auxiliary Power Unit (APU) Replacement, Redesign, and Repaired Parts is directed at applicants who use comparative techniques to reproduce dimensional characteristics of parts during the reverse engineering process.
The AC is intended to provide guidance to help PMA (and other) applicants assess their reverse engineering methods to identify possible causes of dimensional inaccuracies, as well as helping applicants develop adequate criteria for determining dimensional similarity between the PMA design and the sample. To that end, the draft AC identifies several dimensional and geometric factors that the FAA expects should be assessed in seeking to ensure the integrity of dimensional characteristics for the purposes of showing similarity. These factors range from geometric relationships to precision and accuracy to influence on critical parts.
The background section of the draft AC states that because reverse engineering methods vary in their measurement techniques, interpretation and combination of data, and in dimensioning systems, they do not typically produce a design with the exact same dimensional properties as the type design part. It further explains that because dimensional differences may exist, functional assessments–which may include both test and supplemental analytical data–will be necessary to safeguard type design functional properties, as well as capabilities of interfacing and higher level assemblies. Such explanations help to illuminate the FAA’s expectations of an applicant in making showings of dimensional similarity. MARPA would like feedback from our members describing to what extent such test data is currently available and practical within the industry.
Although this AC, like all Advisory Circulars, is neither mandatory nor regulatory, such guidance documents do sometimes become ingrained and relied upon in considering applications. It is therefore important to review the draft and offer comments to the FAA to ensure the guidance reflects the realities of our industry. MARPA will be closely reviewing the guidance and offering comments on behalf of the PMA community. If you identify any particular issues within the guidance, please bring them to our attention so we can incorporate them into our comments. Email your concerns to email@example.com. We also recommend that members who identify concerns file their own comments on the guidance. Comments are due July 3, 2014 and should be submitted to Mark Bouyer of the Engine and Propeller Directorate, at firstname.lastname@example.org
In recent weeks the FAA has released final versions of two important pieces of guidance. FAA Order 8110.42D – Parts Manufacturer Approval Procedures, which cancels revision C, was published on April 10, and FAA AC 21.303-2 – Application For Parts Manufacturer Approval Via Tests and Computations Or Identicality, was published on April 7. Both of these documents have a direct effect on PMA producers.
MARPA submitted comments addressing issues in these guidance documents in September, 2013. Among the issues MARPA noted was the intended use in Order 8110.42D of the FAA’s Risk Based Resource Targeting Tool (RBRT tool) to prioritize PMA projects. MARPA observed that the RBRT tool as intended relied on subjective assessments of project risk that could result in unfair treatment of certain PMA projects. The FAA agreed that the RBRT tool was not quite ready for prime time and has removed it from the Order. MARPA will remain vigilant, however, as guidance for use of the RBRT tool will be addressed in a future Order.
MARPA also worked with its members to offer comments and feedback to the FAA regarding AC 21.303-2. This new AC consists primarily of guidance to PMA applicants that formerly appeared in the now-cancelled Order 8110.42C. MARPA offered comments addressing the expected sample sizes needed for PMA applications based on test and computation, origins of samples, and discretion for establishing sample sizes. The FAA’s adoption of these comments should better help square the advisory guidance with the realities of industry sampling.
MARPA also commented that references to Order 8110.119 – Streamline Process for Parts Manufacturer Approval (PMA) should be included in both documents to encourage use of the Streamlined PMA process when applicable. We are pleased that the FAA agreed and adopted references to the Order in both guidance documents. We are hopeful that the inclusion of references to Order 8110.119 in both Order 8110.42D and AC 21.303-2 will further promote ACO acceptance and PMA applicant use of the Streamlined PMA process.
Both of these guidance documents are now active. Members should familiarize themselves with the guidance, as it will inform the nature of the application process and the relationship with the FAA and your ACOs. In reviewing the two documents, if you notice anything troubling or that seems out of place, please bring it to our attention so that we can work with the FAA to find a solution. Send your thoughts on Order 8110.42 revision D and AC 21.303-2 to email@example.com.
In January, we wrote in this space about two new FAA Policy Statement proposals aimed at increasing coordination between FAA offices in the cases of certain engine PMA and propeller PMA applications. MARPA is always on the lookout for those potential regulations and policies–whether large or small–that could effect our members. In an effort to best support the industry, MARPA solicits feedback from members and files comments addressing the concerns of the PMA community with the FAA. MARPA also works to support the FAA in order to provide industry’s perspective and needs in order to help develop the best rules, policy, and guidance possible.
Last week we filed comments on draft policy statements PS-ANE-33.3-05 and PS-ANE-35.15-02. These two statements were substantially similar in content, and therefore raised similar concerns. The policy statements are intended to create a uniform policy describing the engine and propeller PMA projects for which an ACO is required to use the national certification project notification (CPN) database to alert the Certification Management ACO (CMACO) and Engine and Propeller Directorate (EPD) about the project. The purpose is to enable the CMACO and EPD to provide relevant input when the PMA application is for a project the failure of which may result in a loss of thrust or power.
MARPA noted several possible issues with the proposed Policy Statements. First, our comments noted that the policy as proposed could run the risk of burdening PMA applicants by delaying response time, as each FAA office took the opportunity to review the project. MARPA also noted the risk that differences of opinion between offices could be hashed out using an individual PMA application as the mechanism. This could also result in undue burden to the applicant. Finally, MARPA noted that inquiries by FAA offices about certain PMA projects could inadvertently alert TC holders of possible competition in a particular part market, and give that TC holder the opportunity to lock up the market before the PMA package is ever approved; a clear unfair competitive advantage.
Finally, MARPA noted that a certain provision uniquely included in PS-ANE-33.3-05 (the engine-PMA Policy Statement) was needlessly critical of PMAs and potentially harmful to the PMA industry. The sentence in question reads “[f]ailure of some of these engine PMA parts has resulted in unsafe conditions and the issuance of airworthiness directives.” MARPA explained that this sentence was inappropriate for two reasons. First, it incorrectly implies that PMA parts are abnormally unsafe and that TC products do not have a history of resulting in ADs. Second, the sentence in no way advances the purpose of the Policy Statement, which is to establish uniform standards for use of the CPN database. For these reasons, MARPA recommended the deletion of the sentence.
MARPA will continue to work for its members to advise and cooperate with the FAA in crafting fair and effective policy. Members comments on these issues are always welcome.
You can read MARPA’s comments in their entirety on the MARPA website under the Government Affairs tab at http://www.pmaparts.org/government/.
The FAA is proposing changes to the Part 21 manufacturing rules. A number of these will impact PMA holders. It will be important for the PMA community to provide comments to this NPRM.
The proposed changes include:
With respect to the new permission for PMA holders to issue 8130-3 tags, the FAA confirms that the intent is to permit manufacturers to issue the 8130-3 tags, although the NPRM appears to be missing an explicit authority to issue such a release. Instead, the NPRM as drafted requires a procedure for managing the 8130-3 tags process; but subpart L continues to require application to the FAA for issue of an 8130-3 tag. This is something that will need to be changed if the FAA is to successfully implement this provision.
If the proposal becomes a regulation, the rules would include a requirement for supplier control and a parallel requirement for ensuring conformity of supplier products. This appears to suggest that supplier control must be supplemented with 100% conformity verification of incoming supply; but more importantly it would also mean that non-conforming supplier-provided-parts would represent a regulatory violation even when they are caught in the PMA holder’s incoming inspection! This seems to be inappropriate.
The proposed rule is scheduled to be issued in tomorrow’s Federal Register. An advance copy is available online today at https://s3.amazonaws.com/public-inspection.federalregister.gov/2014-04330.pdf.