The Statistics AC is going to be revised.
We have been working with the FAA to study the Advisory Circular (AC) 33-10, Statistical Analysis Considerations for Comparative Test and Analysis Based Compliance Findings for Turbine Engine and Auxiliary Power Unit Replacement, Redesign and Repaired Parts. We have pointed out a number of concerns with respect to this advisory circular in both our formal comments and in subsequent discussions with the FAA. These can be summarized as saying that the AC does not appear to get the FAA to where the FAA wants to be, and consequently will not appreciably contribute to safety.
The FAA has let us know that they will be rescinding the version of AC 33-10 that was published in late August. It must go through the internal FAA process so the rescission will not be immediate, but applicants should probably treat it as if it no longer existed because the text did not suit the FAA’s intended purposes.
This is not the end for statistics. The FAA still wants to provide useful guidance to the industry. But they have recognized that the current draft of the statistics AC is not what they want and have decided to take a step back and revisit the scope and intent of the AC. They plan to incorporate changes and solicit formal comments on the revised document in order to make sure that they meet their goal of creating a useful document that helps to move the industry towards our mutual safety goals. This will give everyone in the MARPA community an opportunity to contribute to the document.
This action reflects a joint efforts by the FAA and MARPA to understand the true impact of the guidance, and the rescission is a triumph of both good government and good industry efforts.
The FAA has released two new advisory circulars that may affect the PMA community. Both advisory circulars are issued by the Transport Aircraft Directorate and apply to Part 25 aircraft (and parts thereof).
We would be interested in hearing from any MARPA member who is affected by one of these new advisory circulars.
The FAA has issued a new advisory circular for statistical analysis. This AC only applies to engine and APU parts.
The new advisory circular uses statistical analysis to arrive at correct sample sizes. This sample size formula is introduced for persons who are trying to correlate two populations of parts. MARPA had pointed out to the FAA that typically a PMA applicant does not correlate two different populations of parts – instead they derive the reasonable tolerances on one population of parts, and then design and produce within those tolerances. The FAA would like PMA applicants to arrive at their appropriate sample size, test the parts being reverse-engineered, and then produce an equivalent number of pre-PMA parts to test for the same properties (and then correlate the two populations). This is contrary to current FAA regulatory guidance, which requires the design to be approved and the requires the production quality system to ensure that parts are all produced within the approved design parameters. In essence, ACO engineers will now take control of the quality assurance system through the design process.
One of the problems with the AC is that it relied on statistical analysis for clinical trial sample size as the basis for assessing statistical analysis of reverse-engineering sample size. Clinical trials for pharmaceuticals typically rely on populations of hundreds or even thousands of people. Trying to test hundreds or thousands of parts in order to reverse-engineer them is simply not realistic. Furthermore, the degree of part-to-part difference under modern quality assurance systems does not support such large sample sizes.
For those cases where the equations in the advisory circular give a lower number for the appropriate sample size, the FAA has also established minimum sample sizes. The AC sets some minimum limits for the number of parts that must be tested in order to derive certain values (remember that you need that number of PC parts and also that number of reverse-engineered parts to meet the AC’s requirements):
Minimum Number of Parts to be Sampled
- For basic material properties that are more dependent on alloy constituency than on part manufacture process – 10 approved parts from three separate lots with at least three parts per lot
- For properties affected by how the material is processed during part manufacture such as high-cycle fatigue, low-cycle fatigue, creep, tensile strength, crack growth, etc. – 30 approved parts
- For fatigue testing – at least 25 tested parts that are run until they crack
- For parts exhibiting complex geometry or complex manufacturing variables, additional specimens may be required
- For parts considered to have a high degree of criticality, greater sample sizes may be required
No statistical basis is offered for these minimum sample sizes.
The new advisory circular provides guidance for statistical analysis of sample-size despite the fact that no regulation actually requires such a broad-based sample. The advisory circular appears to potentially change the regulations by increasing the burden on applicants. To the extent that this is true, it is inappropriate.
Our concern is that despite warnings that this is non-mandatory guidance, this guidance may be used as if it were a regulation, with offices refusing to accept PMA applications that are otherwise valid, but that failed to use this AC as a basis for identifying sample size.
If you find that this AC effectively changes the application obligations imposed on you as a PMA applicant, STC applicant, or other FAA-approval applicant, then please contact MARPA so we can raise this concern with the appropriate personnel at the FAA.
AC 33-10 is known by its full title: Statistical Analysis Considerations for Comparative Test and Analysis Based Compliance Findings for Turbine Engine and Auxiliary Power Unit Replacement, Redesign and Repaired Parts.
The FAA Aircraft Certification Service has issued their sequencing document. This document provides guidance on how the FAA will prioritize certification / approval projects.
Projects will be provided FAA resources based on a Safety Index, Applicant Showing or Designee Finding (ASDF) value, and a project priority which will be calculated from the first two values. Anyone who plans to seek FAA approval from the Aircraft Certification Service needs to understand this Standard Operating Procedure (SOP) in order to know how to best formulate their project to get the highest score (and the highest priority) when seeking FAA data approval resources.
The first value calculated will be the Safety Index (SI). The sequencing program’s SI calculation puts a tremendous emphasis on safety-related projects needed to prevent an accident (described as “very high/immediate safety benefit”). This is good to the extent that it prioritizes airworthiness directives (ADs). When you look at the numbers, an AD that only affects five small general aviation aircraft will get a safety index of 450, while a national/strategic priority for a typical airline aircraft would be given a safety index of 350. This shows that ADs will always take priority over other projects in other categories.
Unfortunately, this category that strongly emphasizes ADs is broader than just ADs. The descriptive language of “very high/immediate safety benefit” could be interpreted to mean other projects as well. This category could undermine the system if it is misused to describe projects as having “very high/immediate safety benefit” when they don’t really meet the normal parameters for an AD.
The sequencing program also prioritizes large aircraft over small, and large fleets over small. This will put projects designed to support general aviation aircraft or rare aircraft (small fleets) at a distinct disadvantage.
The ASDF is a function of the complexity of the project and the amount of the project that will be approved by one or more designees. You take the total number of findings of compliance that will be performed by the FAA (0, 1-5, 6-15 or more than 16 – any amount that is more than sixteen findings will be capped in the table). Then you identify the percentage of findings of compliance that will be handled by one or more designees (so the finding of compliance does not need to be performed by an FAA employee). These percentages will fit into one of these categories: 100%, 90-99%, 75-89%, 50-74%, or <50%. You cross reference these two values in a table to obtain the ASDF, which will be low, medium or high. A High ADF will be reserved to those projects with nearly all of the findings performed by a designee (or findings based on applicant-only showing which is a theoretical notion that is being developed by an aviation rulemaking committee). For these projects, the FAA will have very little interaction with the application’s approvals.
In projects where the FAA has a high number of retained findings and/or the percentage of FAA-retained findings exceeds a metric established by the FAA (50-75% depending on the circumstances), the FAA will assign a low ASDF rating.
Project priority is then determined by cross referencing SI and ASDF. An AD-level safety index will always lead to a priority one ranking, even when ASDF is low. Everything else will fit into priority status two through four. The FAA will break ties among priorities by using SI, so this means that SI is your most important metric.
Those performing projects with negligible safety impact (which includes interiors projects like passenger entertainment systems and cabin modifications) get a zero value for safety impact. This will mean that the safety index for these projects is always zero, which will place them in the lowest possible project priorities.
Why is project priority so important? Because a project with a priority level one gets processed according to the office’s normal project flow times, but a priority level four project can be processed according to the office’s normal project flow times PLUS 90 DAYS! So (for example) the FAA may delay the response to your application (and thus the start of your project) by an additional 90 days if your project falls into priority level four; and then after that, additional 90 day response delays are permitted for each ‘resource-limited’ response.
These 90-day-delays are on top of office flow times (OFT), which are the amount of time allocated to a response by the local office – this time is set by the local office at the local office’s discretion based on staffing and workload. OFTs will apply to responses to discrete elements like test plans or test reports.
The new guidance also gives an automatic safety index of zero to applications that come in without a certification plan. This will be a tremendous strike against new market entrants, and could stifle innovation, because new market entrants typically do not know how to draft a certification plan.
We have always believed that a certification plan is a good idea. But such a plan is not required by the regulations and thus there is no regulatory guidance in how to create such a plan. The Paperwork Reduction Act (PRA) forbids government agencies from imposing information-collection burdens on the public without first obtaining OMB approval for the collection. The certification plan will become a de facto requirement for anyone that does not want to placed at the back of the line with a zero SI.
Certification plans are good things. They help to focus the applicant’s efforts by identifying which regulations are relevant for the showing of compliance, and how will the applicant show compliance with each. But for the FAA to engage in a de facto refusal of tax-payer-funded services to new market entrants who are unable to produce such a certification plan seems to be fundamentally unfair, as it creates yet another barrier to entry for the new market entrants.
Perhaps most damaging is the FAA’s failure to provide firm guidance concerning a certification plan. Without firm guidelines, any FAA office is free to reject any certification plan as inadequate. The rejected applicant will then have to face a “zero” SI (thus denying him/her timely services) or go back and try again until the applicant is able to meet the requirements of the local office (thus delaying the projects entry into the FAA oversight system). Imposing internal FAA workflow requirements that impose a de facto requirement on the public to produce a document that is not required by regulations (by making it de facto requirement for obtaining timely FAA services) is a back-door way of circumventing the regulatory process that is described by the Administrative Procedures Act.
What should the solution be? If the FAA fails to publish a certification plan guideline to aid applicants in meeting this de facto requirement, then perhaps industry should work with the FAA to develop standard guidelines for a certification plan. This does not relieve us of the problem of using internal work flow documents to impose pseudo-regulatory requirements, but at least it provides some guidance to permit industry to meet the FAA’s de facto requirements.
Many existing approval applicants already have an internal process for creating a certification/approval plan (often one that was worked out with the local FAA office to meet their particular requests as well as regulatory requirements). For these companies, it is important to review and understand this sequencing SOP in order to calculate how best to achieve a high SI in order to get treated earlier, rather than later.
Another important solution for PMA applicants is the streamlined PMA process found in FAA Order 8110.119. That document allows PMA manufacturers with non-safety-sensitive (NSS) parts to obtain an expedited review (which should be outside of the normal sequencing mechanism). It is not without its own burdens, though. The applicant must perform an additional analysis (not-required-by-the-regulations) to show that the parts are not safety sensitive, in order to qualify for NSS treatment, and then must have full evidence of compliance (as with any other application).
Do your export 8130-3 tags have the right language on them? If they don’t then you run the risk that they may be rejected in Europe.
The Bilateral Aviation Safety Agreement (BASA) between the United States and the European Union specifies that the documentation that accompanies a PMA part bound for the European Union must include specific language in order to be acceptable to the European Union’s airworthiness authority, EASA.
The BASA Technical Implementation Procedures (TIP) direct that PMA parts being exported from the United States to the European Union bear appropriate language in block 12 (the remarks block) of the export 8130-3 tag. In order to meet this requirement properly, the export 8130-3 tag must identify the part as falling into one of these three categories (by using the authorized language):
1. For a PMA part which is not a critical component, the remarks block of the 8130-3 should state:
“This PMA part is not a critical component.”
But if the PMA part is a critical component, then there are two options for the language in the remarks block.
2. In the first option for critical components, if the PMA holder also holds an EASA STC design approval which incorporates the PMA part into an EASA certified or validated product, then the language should say:
“Produced by the holder of the EASA STC number [INSERT THE FULL REFERENCE OF THE EASA STC INCORPORATING THE PMA].”
3. In the second option, if the PMA holder holds a licensing agreement from the TC or STC holder (giving the PMA holder the rights to use the TC/STC design for the PMA parts), then the following statement should be written in the remarks block:
“Produced under licensing agreement from the holder of [INSERT TC or STC NUMBER].”
These options 2 and 3 are the only two options for exporting FAA-PMA critical components from the US to the EU.
The PMA “criticality statement” is something that is requested under the technical implementation procedures (TIPs) that accompany the US-EU bilateral aviation safety agreement (BASA).
Under the US-EU TIP, a “Critical Component” is defined as:
“a part identified as critical by the design approval holder during the product type validation process, or otherwise by the exporting authority. Typically, such components include parts for which a replacement time, inspection interval, or related procedure is specified in the Airworthiness Limitations section or certification maintenance requirements of the manufacturer’s maintenance manual or Instructions for Continued Airworthiness.”
The determination of whether a PMA part is critical is made by the design approval holder (the FAA-PMA holder) and confirmed as part of the FAA approval. Section 4.4(c) of Order 8130.21H states that “The determination of a PMA article’s criticality, as required to be entered in Block 12 when exported, can only be determined by the actual design approval holder (that is, the FAA-PMA holder).” This is important language because certain parties (foreign governments and competitors) have attempted to gainsay the FAA-approved “critical part” decisions of the FAA-PMA holders.
we have heard of DARs who thought that this language meant that only the PMA holder could obtain the export 8130-3 tag. This is not so. This language is not meant to prevent a designee issuing an export 8130-3 tag from making a PMA “criticality statement” on the 8130-3 tag that is consistent with the determination of the design approval holder. Thus, any designee issuing an export 8130-3 tag for any FAA-PMA part may rely on the (PMA) design approval holder’s determination as to whether the PMA part is a critical component.
This critical parts language generally does not appear to apply to most bilateral airworthiness agreements – it is a special nuance of the US-EU Agreement. Adding the “criticality” language does not hurt the 8130-3 tag (and may be useful if the end-user is not yet known), but the criticality language generally remains unnecessary unless the part is destined for Europe.
The FAA has released a new draft Policy Statement concerning the vibration surveys and engine surveys required by § 33.83. The new guidance clarifies that this regulation is intended to require a full engine test (for type certificate applicants). The draft guidance, known as “PS‑ANE‑33.83‑01,” is currently available for public comment.
Members will recall that MARPA successfully opposed a final rule that would have applied the “full engine test” standard to PMA and STC applications. At the time, our discussions with FAA representatives revealed that their real concern was applying the full engine test standard to engine type certificate applications, and not to PMAs. The reference to PMAs and STCs, they explained, was an unfortunate mistake.
This draft guidance attempts to more narrowly address the FAA’s concerns about full engine test for type certificate applicants.
In order to ensure that the FAA’s intent is clear, MARPA plans to offer some additional language designed to clarify that this Policy Statement does not supersede the discussion laid out in FAA Advisory Circular 33-8 (Guidance for Parts Manufacturer Approval of Turbine Engine and Auxiliary Power Unit Parts under Test and Computation). We will also look for opportunities to help the FAA meet their policy goals.
Comments are due to the FAA by November 21, 2014. They can be emailed to email@example.com or mailed to her at:
Federal Aviation Administration
Engine and Propeller Directorate
Standards Staff, ANW-111
12 New England Executive Park
Burlington, MA 01803
Please share your comments with MARPA, too, so we can make sure our comments are consistent with the concerns of our members.
Some of you may be wondering where to find the FAA’s policy memo on design approval holder restrictions on ICA availability. Older MARPA links to the original position of that guidance are no longer valid because it has been moved. But it is still available if you know where to look!
Many MARPA members look for this policy memo because it clarifies that anti-competitive language in ICAs (restricting use of PMAs or third party repairs) is unacceptable to the FAA:
While not exhaustive, the FAA finds the following practices of using restrictive language in the ICA or through restrictive access or use agreements unacceptable under the provisions of 14 CFR §21.50(b) and related ICA airworthiness requirements:
1) Requiring the owner/operator to only install DAH-produced or authorized replacement parts, articles, appliances, or materials.
2) Requiring that alterations or repairs must be provided or otherwise authorized by the DAH.
3) Requiring the use of only maintenance providers or other persons authorized by the DAH to implement the ICA.
4) Establishing, or attempting to establish, any restriction on the owner/operator to disclose or provide the ICA to persons authorized by the FAA to implement the ICA.
The FAA has released Notice 8110.116, which stresses the importance of writing clear notes in the Type Certificate Data Sheet (TCDS), and of ensuring that those notes are consistent with FAA policy. This new guidance can be very valuable to everyone in the industry. The value to TC applicants and holders (who create draft TCDS for FAA adoption) is obvious, but less obvious are the subtle and not-so-subtle admonitions against using the TCDS for anti-competitive effect.
One valuable element of this guidance is that it stresses certain things that should not appear in the TCDS. For example, in paragraph 6(a)(6), the guidance advises against language that limits work to a scope narrower than permitted by the regulations:
(6) Do not include requirement that is non-regulatory in the notes. For example: “This aircraft shall be maintained in accordance with the BHT-427 Maintenance manual.” This note implies that BHT-427 is the only source for maintaining the aircraft, and conflicts with 14 CFR 43.13(a) which states “Each person performing maintenance . . . shall use methods . . . or other methods acceptable to the administrator.”
This admonition is consistent with FAA guidance that has prohibited language limiting operators options in seeking repairs. E.g. PS-AIR-21.50-01: Type Design Approval Holder Inappropriate Restrictions on the Use and Availability of Instructions for Continued Airworthiness (March 23, 2012). In the past, there have been problems with manufacturer’s guidance that seeks to limit operators’ repair options, which in turn has had an adverse effect on independent repair stations who are qualified by the FAA to perform the same repairs. Other language in the guidance highlights the prohibition against anti-competitive statements by stating:
Avoid language promoting a TC holder or their suppliers as the sole source for maintenance or overhaul.
It is contrary to 14 CFR parts 43 and 21 to include a note that all repairs or modification schemes must be approved by the TC holder prior to FAA approval.
In a similar vein, the guidance also forbids use of notes that could be interpreted as statements of FAA General Policy. The obvious reason for this is because the TCDS should not be used as a vehicle for establishing FAA policy. The example that is given is useful in its own right, because this is clearly not a statement of FAA policy, despite the fact that some government prosecutors have tried to promote similar policies:
(8) Avoid notes that can be interpreted as FAA general policy. For example: “Reuse of parts and assemblies that have been involved in an accident is not permitted unless approved by FAA Engineering.”
In subsection 6(a)(10), the FAA advises against the use of brand names, but in subsection (11), the FAA requires that when a specific brand name is used, then the note must also advise of the possibility of using FAA-approved alternatives:
(11) If a specific brand name material needs to be listed, then you must state that an equivalent material is also may be approved. For example: “SOHIO BIOBOR JF biocide additive is approved for use in fuel at a concentration not exceeding 270 PPM. Use of other, equivalent material, may be approved by the FAA.”
Aviation is a competitive industry, but some companies have attempted to use FAA-approved and FAA-acceptable documents as levers to inhibit competition. The FAA has repeatedly asserted that its job is safety – not competition – and that it will not allow its approvals to be used as independent inhibitors of competition where no safety interest is served. While the main purpose of this new guidance is to standardize the layout of TCDSs, it also includes important admonitions that should help prevent anti-competitive behavior.
Do you have a topic for the MARPA Technical Committee’s next meeting? We are now collecting items for the agenda for the October meeting!
The MARPA Technical Committee addresses technical issues facing the industry. The Committee review proposed FAA Publications (like advisory circulars and orders) and offer technical advice to improve those proposals. Technical Committee members meet with the FAA to discuss policy issues. Technical Committee members also sit on FAA Advisory Committees and Working Groups in order to develop policy that affects the PMA community.
Appropriate topics for the MARPA Technical Committee Agenda include aviation/manufacturing policy issues, amendment of current aviation/manufacturing policy documents, and new technologies that affect the aviation manufacturing industry. Any technical issue that could affect safety is fair game. MARPA is not just focused on FAA and EASA compliance issues – issues before other agencies (in the US or around the world) are also welcome.
The Technical Committee will meet on October 1, 2014 at 11:00 am. The meeting will take place at the Las Vegas Renaissance Hotel at 3400 Paradise Road
Las Vegas, Nevada 89169 (in conjunction with the MARPA Annual Conference – this occurs during the Workshop Day).
All MARPA members are welcome to attend the Technical Committee meeting.
Please send you Technical Committee Agenda items to Jason Dickstein.
MARPA had a very good meeting today with Mark Bouyer and Ann Azevedo of the FAA’s Engine and Propeller Directorate (EPD).
The focus of the meeting was status on EPD policy that may affect PMA manufacturers.
Azevedo explained that she has responded to the comments on the Statistics Advisory Circular (AC), and hopes to have the final draft of that guidance available to the public by September. The Statistics AC is meant to address FAA concerns that have been recognized in practice, such as misusing statistical methods to show equivalence, and underestimating the appropriate sample sizes.
Bouyer expects the Materials AC to go out for public comment this month. The Materials AC will identify the essential data that is necessary when a PMA applicant is trying to replace the material used in the type design. MARPA members should watch for this one, and be prepared to offer their comments.
The FAA had published the Geometry AC for comment. This AC is meant to enhance awareness of how reverse engineering can introduce dimensional differences in replacement parts. MARPA Board members have expressed that the draft of this proposed guidance appeared to be very helpful to the industy.
The Burner Rig AC, which was also previously out for comment, is expected to be issued by September. It is expected to provide a method for establishing functional equivalence for certain degradation modes in parts such as oxidation, hot corrosion, erosion, etc. The AC is expected to identify existing technology as a means of compliance.
Finally, the FAA is internally reviewing AC 33.8 with a plan to update and clarify the AC. The updates are intended to make the AC easier to use.
The FAA has been very active and diligent in preparing guidance. In the immediate future (before the Conference), MARPA members should expect to see the release version of the Statistics AC and the Burner Rig AC, and they should expect to see the Materials AC go out for comment.