Analyzing Risk at the FAA Flight Systems Laboratory Presented to: Workshop By: Dr. Richard Greenhaw, FAA AFS-440 Date: 29 November, 2005
Flight Systems Laboratory Who we are How we analyze risk Airbus A380 ICAO OFZ example 2
Flight Systems Laboratory FAA Flight Standards AFS-440 Flight Operations Simulation & Analysis Flight Systems Laboratory 3
Flight Systems Laboratory Analyzes risk associated with implementation of emerging instrument flight operational concepts navigational systems 4
Flight Systems Laboratory Clients Air Traffic Control Airports FAA Regions Aviation Industry FAA Executives 5
Flight Systems Laboratory Safety Study scientific of operational system to determine safety 6
Flight Systems Laboratory Safety Study 1. quantitative 2. traceable 3. transparent 4. operationally valid 7
Flight Systems Laboratory Safety Study a) comparative b) absolute 8
AFS-440 Risk Analysis Approach Safety Study Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 9
AFS-440 Risk Analysis Approach Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 10
AFS-440 Risk Analysis Approach Define the Problem Identify the Problem Problem Background Operational Scenario List List of Questions to Be Answered Create Study Project Plan Safety Study Project Plan 11
AFS-440 Risk Analysis Approach Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 12
AFS-440 Risk Analysis Approach Model the Problem Describe Hazards Determine TCVs Create Data Collection Plan Data Collection Plan List of Hazards & Their Severity Test Criteria Violations Develop Risk Model Risk Model 13
AFS-440 Risk Analysis Approach Risk Model Use: to calculate the probability of a TCV Elements: the geometry of the operational scenario the dynamics of the scenario the patterns of behavior of pilots and controllers the characteristics and performance of the navigation and guidance systems the flight characteristics of the aircraft involved the distribution of types of aircraft involved the environmental patterns 14
AFS-440 Risk Analysis Approach Risk Model For each element: an operational definition & how to measure the type of measurement results required how to gather the data (measurements) what assumptions used to find item s value 15
AFS-440 Risk Analysis Approach Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 16
AFS-440 Risk Analysis Approach Collect & Organize the Data Acquire Data Organize Data Organized Data Data from actual flight operations Data from test flights Data from flight simulators Data from Monte Carlo simulations 17
AFS-440 Risk Analysis Approach Types of Data Aircraft Performance Navigation System Performance Crew Performance Surveillance System Performance Air Traffic Controller Performance Environmental Characteristics 18
AFS-440 Risk Analysis Approach Monte Carlo Simulation Data Airspace Simulation and Analysis Tool ASAT A multifaceted computer tool for aviation related simulations and safety evaluations 19
AFS-440 Risk Analysis Approach ASAT consists of high fidelity models and empirical data representing each component of the real life scenario: Aircraft Geographical Environmental Navigation Systems Surveillance Human Factors 20
AFS-440 Risk Analysis Approach INPUTS: Aircraft (and/or) Flight Simulators Pilot Performance ATC Performance Environment Obstacles Terrain Navigation Avionics ASAT Operational Behavior Tracks 21
AFS-440 Risk Analysis Approach INPUTS: Aircraft (and/or) Flight Simulators Pilot Performance ATC Performance Environment Obstacles Terrain Navigation Avionics ASAT 22
AFS-440 Risk Analysis Approach 23
AFS-440 Risk Analysis Approach 24
AFS-440 Risk Analysis Approach 25
AFS-440 Risk Analysis Approach 26
AFS-440 Risk Analysis Approach Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 27
AFS-440 Risk Analysis Approach Analyze the Data & Evaluate the Risk Calculate TCV Probabilities Evaluate Risk Likelihood of each Hazard / TCV Determine Acceptable Risk Comparative / Absolute Acceptable Risk for each Hazard / TCV Comparison of TCV Probability & Severity with Acceptability to Give Recommendation of Risk Acceptability 28
AFS-440 Risk Analysis Approach Risk Matrix No Safety Effect Minor Major Hazardous Catastrophic Probable Low Medium High Unacceptable High Unacceptable High Unacceptable Remote Low Low Medium High Unacceptable High Unacceptable Extremely Remote Low Low Low Medium High Unacceptable Extremely Improbable Low Low Low Low Medium 29
AFS-440 Risk Analysis Approach No Safety Effect Minor Major Hazardous Catastrophic No effect on flight crew. Has no effect on safety. Inconvenience. Slight increase in workload such as flight plan changes. Slight reduction in safety margin or functional capabilities. Environmental or system damage. Some physical discomfort to occupants of aircraft. Significant increase in flight crew workload. Significant reduction in safety margin or functional capability. Injury, environmental or system damage. Physical distress to Aircraft occupants. Large reduction in safety margin or functional capability. Serious or fatal injury to small number of persons. Physical distress/ Excessive workload such that flight crew cannot perform required tasks accurately. Outcome would result in: -Hull loss - Multiple fatalities - Fatal injury or incapacitation 30
AFS-440 Risk Analysis Approach *Tables are from the FAA Safety Management System Manual, Version 1.0, July 24, 2003 31
AFS-440 Risk Analysis Approach Triggers Define the Problem Model the Problem Collect & Organize the Data Analyze Data & Evaluate Risk Report Results Safety Study Report 32
AFS-440 Risk Analysis Approach Report Results Write the Draft Report Sign Final Report Draft safety Study Report Review & Revise Draft Report Final Safety Study Report Official Signed Safety Study Report 33
Airbus A380 ICAO OFZ Problem Safety Study: 1. Define the Problem 2. Model the Problem 3. Collect & Organize the Data 4. Analyze the Data & Evaluate the Risk 5. Report the Results 34
Airbus A380 ICAO OFZ Problem Safety Study: Define the Problem Background: ICAO OFZ Code E Approach Inner Horizontal 148 ft 33.3 % Inner Transitional 197 ft 197 ft Inner Approach 120 m. Code F Approach Inner Horizontal 148 ft 33.3 % Inner Transitional 254 ft 254 ft Inner Approach 155 m. 35
Airbus A380 ICAO OFZ Problem Safety Study: Define the Problem Scenario List: A380 performing a hand-flown balked landing on a precision runway Crosswinds: 0-10, 10-20, 20-25 knots Initiation Heights: 70, 40, 10 feet 36
Airbus A380 ICAO OFZ Problem Safety Study: Define the Problem List of Questions to Be Answered: What is the risk of the A380 penetrating the ICAO Code E OFZ during a hand-flown balked landing on a precision runway under typical environmental conditions? 37
Airbus A380 ICAO OFZ Problem Safety Study: Model the Problem Hazards & Severity: TCVs: A380 penetration of the OFZ surface during a balked landing Severity = hazardous 1. Penetration of Inner Transitional Surface on left or right by an A380 wingtip during hand-flown balked landing Code E Approach 33.3 % Inner Transitional 197 ft 197 ft 38
Airbus A380 ICAO OFZ Problem Safety Study: Model the Problem Mathematical Model: geometry d Note: 10.35m is the wingtip height at 1g 39
Airbus A380 ICAO OFZ Problem Safety Study: Model the Problem Mathematical Model: geometry Dynamics: limit consideration to time while tips can be < 45 m 3 initiation heights Environmental: wind speeds 40
Airbus A380 ICAO OFZ Problem Safety Study: Collect & Organize the Data Collect Data Airbus A340 simulators in Berlin & Toulouse Professional, current, qualified flight crews 156 runs: Toulouse between May 21 & July 26, 2004 356 runs: Berlin between August 23 & September 3, 2004 Of those 512 runs, 333 were hand-flown balked landings Initiation heights: 70, 40, 10 feet Winds: 0-10 36%, 10-20 32%, 20-25 32% 41
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Preliminary Results Established: 1. Berlin & Toulouse data can be combined 2. Reasonable estimate for balked landing rate: 1.9 E-03 3. Crosswind & initiation height really do affect lateral deviations 4. Test crosswind speeds vs actual distribution 5. Test initiation heights vs actual distribution 42
Berlin & Toulouse s related to x-wind & balk initiation height Airbus A380 ICAO OFZ Problem 20 4310
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper 3 Cases Analyzed: 1. Test crosswinds & test initiation heights 2. Actual crosswinds & test initiation heights 3. Actual crosswinds & actual initiation heights 44
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Case 1 d w s = (w d) / w 45
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Case 1 Generalized Extreme Value Distribution GEV(x) = exp 1 + ξ x µ σ 1 / ξ 46
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Case 1 0.1 0.08 GEV PDF 0.06 0.04 0.02 10 20 30 40 50 47
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Case 1 1.75 E-07 1.25 E-07 P(S > 100) < 4.5 E-06 7.5 E-08 2.5 E-09 95 100 105 110 48
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Data Analysis Working Paper Case 1 probability penetration given balked < 4.5 E-06 probability balked = 1.9 E-03 probability penetration < 8.6 E-09 49
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Risk Acceptability Working Paper Case 1: The TCV is hazardous Risk Matrix Probable No Safety Effect Low Minor Major Hazardous Catastrophic Medium High Unacceptable High Unacceptable High Unacceptable Remote Low Low Medium High Unacceptable High Unacceptable Extremely Remote Low Low Low Medium High Unacceptable Extremely Improbable Low Low Low Low Medium 50
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Risk Acceptability Working Paper Case 1: The TCV is hazardous therefore acceptable is < 1.0 E-07 51
Airbus A380 ICAO OFZ Problem Safety Study: Analyze the Data & Evaluate the Risk Risk Profiles and Evaluations Case 1: The TCV is hazardous therefore acceptable is < 1.0 E-07 Probability of penetration < 8.6 E-09 Therefore, recommend risk is acceptable for this hazard / TCV 52
Airbus A380 ICAO OFZ Problem Safety Study: Report the Results ICAO Circular 301, New Larger Aeroplanes - Infringement of the Obstacle Free Zone 53