Office of Aviation Safety Analytical Services Discussion on the NASA National Aviation Operational Monitoring Service (NAOMS) Project Presented to: Aeronautics and Space Engineering Board (ASEB) - NAOMS Study Committee By: Mike Basehore, ASA-100 Date:
Two Main Issues to Address 1. Lack of criteria to guide responses to the questions and the fact that responses are based on the crew s perception of what occurred rather than what actually occurred. 2. Event rates were calculated using analyses that were based upon extrapolation methods to estimate absolute numbers of events occurring within a given time period. Given that the extrapolated numbers deviate significantly from other databases, there is reason to question the validity of this methodology in determining event rates. 2
Questionnaire Review A review of the questionnaire by the FAA and members of the Commercial Aviation Safety Team found many of the questions to be subjective and to rely heavily upon recollections and perceptions of flight crews over a period of 2 to 3 months. In addition, many questions were vague or too broad and as a result are of limited value. 3
Examples from Questionnaire Review ER1. How many times during the last (TIME PERIOD) did an aircraft on which you were a crewmember divert to an alternate airport or return to land because of an aircraft equipment problem? (ER1A. What systems caused the diversion or return to land? ) Comment: Flightcrew members may not know if an equipment problem was just an indication or a confirmed fault. Additionally, often the malfunctioning system cannot be identified until after maintenance review. Specific systems at fault may not be identified by the indication the crew sees, or other factors not related to specific system indication may have caused the fault. Thus, the crew may believe a particular system had a problem, but that may not be the case. 4
Examples from Questionnaire Review ER4. How many times during the last (TIME PERIOD) did an in-flight aircraft on which you were a crewmember experience uncommanded movements of any of the following [elevators, rudder, etc.] Comment: May be based on crew perception rather than measured data. Aircraft equipment measures actual occurrences, and may be the better source for this information. The actual device that had an uncommanded movement may not be known to crew. Note that some aircraft are designed to move these devices without crew input. 5
Examples from Questionnaire Review ER7. During the last (TIME PERIOD) how many times did an inflight aircraft on which you were a crewmember experience a total engine failure? Comment: Total engine failure is not defined, and it is unclear how this relates to IFSD, which is tracked by the FAA. 6
Current Approach to Voluntary Reporting Since the time of the NASA study, the FAA and the industry have developed more robust reporting mechanisms, including the Aviation Safety Action Program (ASAP) which encourages employees to report events without fear of retribution. ASAP requires the report to be submitted within 24 hours of the event and each report is reviewed by a committee with representation from the airline, labor and the FAA. 7
Aviation Safety Action Programs Over 70 operators are participating in ASAP, and over 160 ASAP MOU's have been established for different employee groups (pilots, dispatchers, mechanics, and flight attendants). The FAA Air Traffic Organization is in the process of standing up ATSAP, an ASAP for air traffic controllers. Because events are immediately reported and reviewed by experts to assure they are complete, the credibility of these reports is quite high. 8
Media reports have speculated that a preliminary review of NAOMS data indicates that certain incidents are underreported in FAA data systems. Specific incident types reported are: Runway Incursions Bird Strikes Near Mid Air Collisions Engine Failure Unruly Passengers Aircraft Diversions Smoke in Cockpit 9
Priority Rank Data Source Owner Access High 3Air Traffic Quality Assurance Program - Operational Errors and Deviation System (OEDS) ATO Restricted High 3 Air Traffic Quality Assurance Program - Pilot Deviations System (PDS) ATO Restricted High 3 Air Traffic Quality Assurance Program - Runway Incursions (RI) ATO Restricted High 3 AirClaims CASE Database AirClaims Ltd Licensing High 3 Aviation Safety Reporting System (ASRS) FAA/NASA Public High 3National Transportation Safety Board Aviation Accident and Incident Data System (NTSB) NTSB Public Medium 1 Enhanced Airworthiness Data Mart (EADM) FAA/AirClaims Ltd Restricted Medium 1 Event Monitoring System (EMS) FAA/ATO Restricted Medium 1 FAA Accident/Incident Data System (AIDS) FAA Public Medium 1 Service Difficulty Reporting System (SDRS) FAA Public Medium 2 Bureau of Transportation Statistics (BTS) - Traffic and Capacity Data DOT Public Medium 2 Jet Operating Statistics (JOS) AirClaims Ltd Restricted Medium 2 NTSB Safety Recommendations to the FAA with FAA Responses NTSB/FAA Public Medium 2 Wildlife (WL) FAA Restricted Medium 2 World Aircraft Accident Summary (WAAS) AirClaims Ltd Licensing Medium 3 Air Traffic Quality Assurance Program - Near Midair Collision System (NMACS) ATO Public Medium 3 Air Traffic Quality Assurance Program - Vehicle/Pedestrian Deviation System (VPDS) ATO Restricted Medium 3 AirCraft Analytical System (ACAS) AvSoft Ltd Licensing Low 1 Helicopter Analytical System (HeliCAS) FAA Licensing Low 1 National Program Tracking and Reporting Subsystem (NPTRS) FAA Restricted Low 2 Air Transportation Oversight System (ATOS) FAA Restricted Low 2 ATP Navigator Licensing Low 2 National Vital Information Subsystem - Air Agency FAA Low 2 National Vital Information Subsystem - Air Operators FAA Low 2 National Vital Information Subsystem - Historical Fleet FAA Low 2 Terminal Area Forecast (TAF) FAA/ATO Restricted Low 3 National Flight Data Center (NFDC) Air Route Traffic Control Center (ARTCC) FAA Low 3 National Flight Data Center (NFDC) Landing Facilities (LF) FAA Public Low 3 National Flight Data Center (NFDC) Location Identifier (LI) FAA Public Low 3 National Flight Data Center (NFDC) Navigational Aids (NA) FAA Public Low 3 National Flight Data Center (NFDC) Radio Fix (FX) FAA Public Low 3 National Flight Data Center (NFDC) Special Use Airspace (SU) FAA Public Low 3 SpaceTrak AirClaims Ltd Licensing
Examples of FAA Databases The FAA provides surface incidents information, data, and analysis to the public via the following website: http://www.faa.gov/runwaysafety/stats.cfm The FAA provides wildlife strike information, data, and analysis to the public via the following websites: http://wildlife-mitigation.tc.faa.gov/public_html/index.html, http://wildlife.pr.erau.edu/bash90-06.pdf The FAA provides Near Mid Air Collision (NMAC) information and data to the public via the following website: http://www.asias.faa.gov The FAA provides engine in flight shut down (IFSD) information, data, and analysis to the public via the following website: http://www.faa.gov/aircraft/air_cert/design_approvals/engine_prop/media/caa M2_Report.pdf 11
Runway Incursions 450 400 350 300 250 200 150 100 50 0 2000 2001 2002 2003 2004 2005 2005 12
Wildlife - Bird Strikes 8000 7000 6000 All Aircraft Commercial Aircraft 5000 4000 3000 2000 1000 0 2000 2001 2002 2003 2004 2005 2006 13
Near Mid Air Collision Reports 250 200 150 100 50 0 2000 2001 2002 2003 2004 2005 2006 14
0.050 U.S. Transport Engine Shutdown Rate 3-Month Average per 1000 Engine Flight Hours 0.045 IFSDs per 1000 Eng Flt Hrs 0.040 0.035 0.030 0.025 0.020 0.015 0.010 0.005 0.000 Mar-00 Jul-00 Nov-00 Mar-01 Jul-01 Nov-01 Mar-02 Jul-02 Nov-02 Mar-03 Jul-03 Nov-03 Mar-04 Quarter Jul-04 Nov-04 Mar-05 Jul-05 Nov-05 Mar-06 Jul-06 All Engines (incl. recips) Turbines only ETOPS only (12-mo) Nov-06 Mar-07 15
The FAA and Industry believe that the information contained in other databases, many of which contain data that are not voluntarily submitted, is more accurate in terms of event rates. 16
Mandatory Reporting FAR Parts 121.703 and 135.415 mandate the reporting of many aircraft events, such as fire, fire warning, smoke, engine shutdown; these data are contained in, and publicly available on, the FAA s Service Difficulty Report (SDR) database. 17
The FAA and Industry believe that a peer review and analysis of NAOMS data by people familiar with aviation and stakeholders that provided the data can be used as an indicator or pointer to potential safety areas which can then be evaluated in combination with the other extensive databases. 18
NAOMS Overview The NAOMS survey could be very useful in sampling flight crew perceptions of safety, and complementing other databases such as ASRS. The survey data, when properly analyzed, could be used to call attention to low-risk events that could serve as potential indicators for further investigation in conjunction with other data sources. 19