December 2012 Issue
Amateur-Built Aircraft Safety Study
We take an in-depth look at the results and recommendations issued by the NTSB.
Used homebuilts can often be found at attractive prices, but purchasers may not anticipate the operations and handling quirks of an Experimental aircraft. Of these four Fly Babys, three are owned by persons other than the builder.
In May 2012, the NTSB released the results of its study of Experimental/Amateur-Built aircraft accident rates. The report received extensive coverage in the aviation media. Reaction in the homebuilding community was intense, and there was confusion about the source of the data and the basis for the recommended changes. There was little explanation for the reasoning behind the published results.
The reason for the confusion was simple: The NTSB press release was the executive summary, not the report itself. A few weeks later, the full report did get released, a PDF file 169 pages long. The full-report release received little press coverage, and that’s a pity. There’s good information in it, and it does provide the basis for the NTSB’s recommendations.
The report is based mostly on statistical analyses of the NTSB accident database. For any such analysis, the process is the key factor. “Garbage in, garbage out” is one aspect, but even good data can be turned into garbage if the procedures are faulty. A gem cutter can shatter a valuable diamond into worthless dust if he hits it wrong.
Here, the assessment is simple: The NTSB hit it right.
The report’s authors noted problems in the past regarding the inclusion of non-Amateur-Built (AB) aircraft in accident studies, and they made a concerted effort to ensure only AB aircraft were included. They worked with the FAA to validate the certification status of the aircraft included in the report and excluded Special Light Sport (SLSA), Experimental Light Sport (ELSA), Experimental Exhibition and similar non-homebuilt aircraft. Appendix A of the report summarizes the process and notes how the NTSB records themselves are often in error.
Sources were cited (including KITPLANESŪ articles by former Editor Marc Cook and myself), data sources were provided, assumptions were stated and potential errors were discussed. For the portion that went into 2011 statistics, the NTSB provided the accident number for every AB event used in the analysis. NTSB staffers visited kit manufacturers and reviewed government and industry training resources to gain insight into the operation of our sport. It was truly a professional product.
However (and you knew that word was coming), this doesn’t mean that the conclusions the NTSB reached are inarguable. Some may still disagree with their assumptions or how various aspects of the data were assembled.
Figure 1 shows the top 10 causes for homebuilt accidents from 2001 to 2010. The number one cause of accidents, the NTSB says, is issues affecting the engine. It edged out the “Pilot Loss of Control” category, with 23.2% of the accidents. But when I checked this category against my own database, I got a significantly lower result.
Figure 1: An “Occurrence” is an event that happens as part of an accident. It is not the same as “Probable Cause,” because a given accident can exhibit multiple occurrence types.
The difference is apparently in how the NTSB and I differ on cases where engine power was lost for unknown reasons. In more than 10% of homebuilt accidents, the accident sequence began with a loss of engine power, but the NTSB investigator was unable to determine the cause. In some cases, the engine was too badly damaged and no evidence remained. But often, it is still sufficiently intact. In some instances, they’re even able to restart the engine after the accident.
What actually happened? Some may have been subtle mechanical issues in the engine itself. But more were probably due to transient factors such as carburetor icing or vapor lock, a fuel-system problem where the evidence is destroyed.
In my opinion, counting the undetermined cases in the “Powerplant” failure category was not an ideal approach. These events occur much less often in production-type aircraft, so the results for “System/Component (Powerplant)” between the E-AB and non-E-AB graphs in Figure 1 would be much closer and place it much lower in the top 10.
The NTSB found that powerplant problems occurred more often than pilot loss of control.
Pilot Error and Flight-Test Planning
Referring back to Figure 1, E-AB aircraft have a rate of “Loss of Control in Flight” almost twice as high as the non-E-AB world. The “Loss of Control in Flight” category for fatal accidents stood out even more. The NTSB figures showed that 43.9% of fatal homebuilt accidents involved the pilot losing control of the aircraft.
This is an obvious area for improvement. One of the NTSB’s recommendations from this study was to encourage owners, builders and pilots of E-AB aircraft to complete flight-test training. About a third of kit builders did not receive transition training, and more than 60% of plansbuilt owners did not. The NTSB also wants to see more formal flight-test plans. The report references the EAA Homebuilt Owners’ survey performed in 2011. Survey respondents had been asked about the level of detail in their flight-test plans. Only about three out of eight builders (37%) claimed to have used a detailed test plan.
Coupled with the NTSB’s finding that 9% of homebuilt accidents occur within the first 10 hours after completion and the rate of fatal accidents due to “Loss of Control,” the basis for the NTSB’s recommendation is clear.
However, I must quibble on one aspect of the NTSB’s justification. The NTSB compares the FAA’s lack of requirements for detailed flight-test plans with the requirements of the United Kingdom. The U.K. demands submission of a detailed test plan, approval of the flight-test pilot and generation of a flight manual for the aircraft prior to granting permission for the test period to begin.
However, the comparison lacks one important item: There’s no reference to the relative rate of “Loss of Control” accidents in the U.K. versus the U.S. Without this insight, there’s no way of telling how much of a difference more stringent requirements would make.
The NTSB analysts took special care to include only Experimental/Amateur-Built aircraft such as this RV-6A, and exclude aircraft in other Experimental categories, like the Yaks in the background.
Aircraft Flight Manuals
In addition to its emphasis on adequate flight-test plans, the report also makes a strong push for generating accurate aircraft flight manuals. This is in reaction to the higher accident rate among second or later owners of completed homebuilts. In 2011, for instance, there were more accidents during the first flight by purchasers than by builders. Original builders often take specialized training to prepare for their aircraft’s first flight, but purchasers can be caught unaware by unusual flight characteristics or system operating quirks.
The NTSB feels that the generation of detailed, accurate aircraft flight manuals would reduce this problem. This apparently drove the NTSB to one of its more unfortunate recommendations: that the FAA “…require the review and acceptance of aircraft operating limitations and supporting documentation as a condition of registration or re-registration [emphasis added] of an Experimental/Amateur-Built aircraft.”
This would essentially eliminate permanent airworthiness certificates for homebuilts. The purchaser of a flying homebuilt would stand the risk that the FAA might not approve the aircraft for continued operation. Also, as the FAA now requires regular re-registration of all N-numbered aircraft, every homebuilt would thus have to undergo an FAA review every three years.
I’m guessing the report writers were focused on ensuring that detailed flight manuals were available to purchasers of used homebuilts and didn’t consider the implications for existing aircraft.
The flight-manual issue is a thorny one. The big kit manufacturers will easily be able to assemble templates for their customers, but what about the plansbuilt folks? For some, writing an aircraft flight manual offers a much more difficult problem than the building of an aircraft. Will Joe Smith’s Mini-Max aircraft flight manual be held to the same standards as that of a Lancair IV-P? If so, the requirement would be truly unfair to the plansbuilt segment of the sport.
In addition to the flight-test-planning comparison with the U.K., the NTSB makes a similar reference to Canada’s fuel-system testing requirements, again, with no comparison of accident rates. This is related to another of their major recommendations: that the FAA institute a requirement for formal fuel-system testing as a condition for receiving an airworthiness certificate.
This type of fuel-flow test for homebuilts is suggested by FAA Advisory Circular 90-89A, but is required by Transport Canada. The NTSB report recommended that formal documentation of the test be required and approved by the FAA.
This is a tough recommendation to argue against. Of course, fuel-system functional testing is a good idea. Every homebuilder reference, including FAA Advisory Circular 90-89A, recommends a fuel-flow test to ensure that the fuel system can deliver at least 1.5 times the full-throttle fuel flow in climb attitude. This is the same test Transport Canada requires.
The question is whether there are enough people who don’t perform such a test to justify the institutional overhead and overall hassle associated with formally documented testing.
My own homebuilt accident database shows about 94 accidents involving fuel-system problems between 2001 and 2010. Indeed, among hardware-induced accidents, fuel-system issues are second only to powerplant problems. Eleven of these accidents occurred on the aircraft’s first flight—a pretty high proportion.
Figure 2 shows the breakdown of causes for homebuilts accidents during first flight, the rest of the 40-hour test period and beyond the test period. Blockage issues (including filters, vents or lines) and fuel-line problems constitute a large proportion on the first flight—more in that first hour than in the 39 hours that follow!
Figure 2: The single hours of homebuilts’ first flights show more accidents related to blocked or leaking fuel systems than the remaining 39 of the Phase I flight-test period.
Would FAA-mandated fuel-flow testing have prevented these? Possibly. But we’re talking about just a single first-flight accident per year, on average. And of those 11 first-flight accidents in the 10-year time frame, only one resulted in a fatality.
Any aircraft accident is a tragedy, regardless of the degree of injury. But instituting a requirement for formal reporting of homebuilt fuel-flow test results would be of little benefit. Again, all builders should run such tests. It’s just that the additional paperwork won’t produce much safety improvement.
They’re Out to Get Us! (Or Are They?)
Objections are often raised when outsiders are critical of E-AB aircraft safety. Some folks claim, “They’re biased toward the big manufacturers!” or, “They don’t understand homebuilts,” or even, “They don’t like our freedom and are trying to wreck the sport!”
The NTSB report recommended updates for several Advisory Circulars dealing with homebuilt aircraft.
That’s not to say they’re always wrong—after all, even paranoid people have enemies. But the analysis for the NTSB’s “The Safety of Experimental/Amateur-Built Aircraft” report was performed scientifically, with full disclosure of data and processes. It’s a quality product, making some good points with few, if any, signs of anti-homebuilding bias. The recommended re-registration documentation review is a major area of concern, but I believe the NTSB didn’t consider the implications in light of the FAA’s recent requirement for regular re-registration of all aircraft.
We may quibble with some of the assumptions used or the magnitude of the problems uncovered, but the basic findings of the report are hard to argue with: Homebuilts do have a higher accident rate than production aircraft. This rate could be reduced by more detailed flight-test planning and the creation of accurate and complete flight manuals. And purchasers of used homebuilts are not being prepared properly for transition into aircraft that sometimes exhibit unusual flight characteristics.
The report recommended that the FAA and EAA help create a coalition of kit manufacturers and other groups to develop transition-training resources and encourage their use.
A better safety record would benefit us all. We don’t have to accept all of the recommendations, but the NTSB has given us a batch of good, solid advice.
IMPORTANT CUSTOMER SERVICE NOTICE TO KITPLANES SUBSCRIBERS:
All mailed correspondence, including subscription invoices, renewals, and gift notices, will bear our address:
PO BOX 8535
Big Sandy, TX 75755
Third parties claiming to be selling KITPLANES subscriptions are not legitimate.