Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) 31 st Digital Avionics Systems Conference Williamsburg, VA October 2012 1 Metron Aviation, Inc: NASA Ames Tech Monitors: David Thipphavong Dr. Heinz Erzberger Arash Yousefi, Ph.D., Richard Xie, Ph.D., Shubh Krishna, GMU: John Shortle, Ph.D., Yimin Zhang
Motivation Various NextGen automation concepts are proposed to provide increased levels of airspace capacity by reducing human workload increasing airspace capacity enhancing safety in higher levels of traffic density Different concepts propose varying levels of automation different system architectures (i.e. centralized vs. distributed) different roles/responsibilities for human operators (ATC vs. pilot automation) Two types of SA system architecture: Centralized: Automated Airspace Concept i.e. Advanced Airspace Concept (AAC) Distributed: Automated Flight Rules i.e. Automated Flight Rules (AFR) System level safety-capacity analysis is needed to guide the decision makers in selecting capacity enhancing concepts that maintain target level of safety 2
Objective For selected NextGen concepts: Establish safety-capacity trade off relations Perform phase transition analysis to understand the conditions under which the system should transition from maintaining high capacity to maintaining safety Sensitivity analysis to identify critical points of failure and required redundancies Safety-driven methods for concept design & refinement Required reliability measures of system components? (e.g. required Mean Time to Failure for onboard automated separation software) Optimal system architecture? (e.g. required redundancies and safety nets) 3
Basic Definitions in our Safety Modeling System = Automated Separation Assurance (SA) System System mission= Provide Separation Services System failure = System fails to prevent a conflict Initiating event = There exist at least two aircraft with conflicting 4DTs Conflict = Los of Separation (LOS), Near Mid-Air Collisions (NMAC), or Mid-Air Collisions (MAC) Describe System Identify Hazards Analyze Risk Assess Risk Treat Risk
Advanced Airspace Concept (AAC) Transmit aircraft state (position, velocity, flight plan) to central (ground ) host Detect conflicts and generate resolutions at the central host Transmit conflict resolutions from the central host to the pilot Reference: Erzberger, H., 2009, Separation Assurance in the Future Air Traffic System, 2009 ENRI International Workshop on ATM/CNS, Mar. 5 6.
Automated Flight Rules (AFR) ConOps Transmit aircraft state vector (position, velocity, intent) and flight plan via ADS-B Detect conflicts and generate resolution options onboard for AFR flights The AFR pilot relies on onboard automation for CD&R All aircraft (IFR, AFR) state vectors (position, speed) via ADS-B in All aircraft planned trajectories via ADS-B in Reference: Wing, D., and M. Ballin, 2004, Pilot in command: A feasibility assessment of autonomous flight management operations, 24th International Congress of the Aeronautical Sciences. RTA at TRACON boundary if any issued by the ATS via CPDLC Onboard conflict resolver Ownship aircraft state vector Right-of-way rules Airline/aircraft efficiency considerations (i.e. Fuel, wind aloft, etc) CR: horizontal or vertical maneuvers or speed adjustment. FMS Flight crew Desired resolution action Broadcast via ADS-B out
Calculation of Conflict Probability Pr{Collision} = Pr{Aircraft on course for conflict} Part I Assumes no conflict resolution Establishes safety-capacity relationship Simulation runs (e.g. FACET) x Pr{Collision Aircraft on course for conflict} Part II Analyze effect of intervention by AAC and AFR SafeATAC
Simulation runs to estimate conflict rate Conflict types considered here: Conflict Type Lateral Vertica l Loss of separation 5NM 1000 ft (LOS) Critical loss of 1.1NM 100ft separation (CLOS) Near mid-air collision 500ft 100ft (NMAC) Mid-air collision (MAC) 100ft 30ft Conflict rates provided through NAS simulators (e.g. FACET) Flight trajectories simulated using FACET with 1.5 x traffic schedule Direct routes flown, no conflict resolution Two cases: Great circle routes, airway routes 50 simulation replications, varying departure times Conflict detection for sectors in Chicago center. 8 Reference: Belle, A., J. Shortle, A. Yousefi, and R. Xie, 2012, Estimation of Potential Conflict Rates as a function of Sector Loading, 5th International Conference on Research in Air Transportation (ICRAT 2012), Berkeley, CA, May.
Part II - SafeATAC Model system-level events that lead to CD&R failure or success Reference: Model the functional events that lead to events in DET Model how components work together to perform a function Model how component performance changes over the time Yousefi, A. and R. Xie, 2011, Safety-Capacity Trade-off and Phase Transition Analysis of Automated Separation Assurance Concepts, 30th Digital Avionics Systems Conference (DASC), Seattle, WA, October 16-20. Xie, R. and A. Yousefi, 2012, Safety Analysis of Primary and Secondary Conflicts for Automated Airspace Concepts, 12th AIAA Aviation Technology, Integration, and Operations (ATIO) Conference, Indianapolis, IN, September.
Dynamic Event Tree (DET) for AFR Strategic Timeframe If failed Tactical Timeframe If failed TCAS Timeframe If failed Pilot See-and-Avoid Timeframe If failed Conflict realized
Fault Tree Layer Dynamic Event Tree Fault Trees (an example)
Reliability Block Diagram Airborne ASAS to detect a potential conflict Source of conflicting flight s positions Source of ownship s positions
Component Modeling functional P ww functional functional functional P ww P wn P nw P wn Pnw P nn P nn Non-functional Non-functional Non-functional Non-functional User s input Hardware: P wn is a constant Software: P wn or P nw is a function of time P nw Pilot: P wn or P nw is a function of time Time
Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) Workflow Event Tree Fault Tree Reference: Yousefi, A., R. Xie, and S. Krishna, 2012, Safety Analysis Tool for Automated Airspace Concepts (SafeATAC) User Manual, Metron Aviation Technical Report to NASA Ames Research Center, Rep # NNH09ZEA001N, July. Safety-Capacity Analysis Reliability Block Diagram Component Reliability Data SafeATAC Full Factorial Analysis (Sensitivity of Overall System Safety to Individual Component Reliability Metrics) Conflict Rate 1. Revise System Design 2. Revise Component Reliability
SafeATAC Specifying Input Files DET Structure CD&R Stages Fault Tree Reliability Block Diagram Component Reliability Data Conflict Rate
16 SafeATAC Interactive Visualization Dynamic Event Tree plot Click Event 1.2.2 Event Description Fault Tree diagram Reliability Block Diagram Click to see fault tree Click to reveal Reliability Block Diagram
Automatic Report Generation Click on the menu item to start analysis and generate report 17
Automatic Report Generation HTML Report Sample View 1-18
Automatic Report Generation Report Sample View 2-19
Experimental Results Experimental results for alternative system architectures of AAC and AFR 1. Safety-Capacity analysis and Phase Transition for different ConOps 2. Safety net analysis 3. Impact of strategic and tactical CD&R on overall system safety within AAC and AFR 4. Full Factorial Analysis to Identify safety critical components 5. Transitional phases in implementation of AAC 6. Secondary conflict within distributed separation assurance 20
S-C Tradeoff for the Baseline 1.40E-09 1.20E-09 Prob(NMAC / Flight) 1.00E-09 8.00E-10 6.00E-10 4.00E-10 2.00E-10 0.00E+00 0 10 20 30 40 50 60 70 80 Flight Count per 15 minutes per sector
Safety Performance of AFR Baseline Case Pr {Conflict Not Resolved Trajectory Conflict} = 3.77 x 10-8 Decomposed into each timeframe CD&R Timeframe CD&R failure probability Strategic ASAS 3.24 10-5 Tactical ASAS 5.48 10-2 99.9% of conflict will be solved by Strategic ASAS! TCAS 2.26 10-2 Pilot See-and- Avoid 0.938 Pilot See-and-Avoid is the least effective means for CD&R
System Characteristics Phase Transition 2.00E-09 1.80E-09 1.60E-09 Performance Change in Transponder Baseline TRN Failure = 1e-3 TRN Failure = 1e-5 Prob(NMAC / Flight) 1.40E-09 1.20E-09 1.00E-09 8.00E-10 6.00E-10 4.00E-10 2.00E-10 0.00E+00 0 10 20 30 40 50 60 70 80 Flight Count per 15 minutes per sector
Safety Sensitivity Curve to Effectiveness of ASAS Strategic CD Software 1.00E-04 1.00E-06 Prob(NMAC Not Resolved) 1.00E-08 1.00E-10 1.00E-12 1.00E-14 1.00E-16 1.00E-18 1.00E-20 1.00E-22 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 ASAS Strategic Detection Software Transition Probability
Impact of TSAFE or AR on overall System Safety within AAC Baseline safety level is 1.98 x 10-8, given a trajectory conflict Experiment 1: TSAFE is removed from the baseline AAC model, by setting the initial failure probability as 1, and probability of transitioning from nonworking to working as 0 Safety level becomes 3.6 x 10-8 Risk almost doubled (same order of magnitude) Experiment 2: AR is removed from the baseline AAC model Safety level becomes 2.22 x 10-7 Risk is 12 times worse than the baseline Other factors that can be changed : Length of strategic and tactical time frames Time step size of each time frame Performance of the components in each CD&R system 25
Impact of ASAS on overall System Safety within AFR Baseline safety level is 1.46 x 10-7, given a trajectory conflict Experiment 1: ASAS Tactical Resolution is removed from the baseline AAC model, by setting the initial failure probability as 1, and probability of transitioning from nonworking to working as 0 Safety level becomes 3.94 x 10-7 Risk almost tripled Experiment 2: ASAS Strategic Resolution is removed from the baseline AAC model Safety level becomes 1.32 x 10-6 Risk is 9 times worse than the baseline Factors in play include: Length of strategic and tactical time frames Time step size of each time frame Performance of the components in each CD&R system 26
FFA to Identify Safety-Critical Components Fractional Factorial Analysis (FFA) Sparsity-of-effects principle: a system is usually dominated by single-factor effects or two-factor interactions Based on that principle, FFA is conducted using a subset of all factors to analyze system behavior SSR + Vary one parameter at a time Baseline Transponder + + In one FFA example, following components are chosen ASAS Detection Software, ASAS Strategic Resolution Software, ASAS Intent-based Tactical Resolution Software, ASAS State-based Tactical Resolution Software, and Mode A/C/S Transponder SSR + Full Factorial x x 6 5 x 7 x 2 x 1 x 3 Transponder + x 8 x 4 + 27
FFA Results Increasing reliability of Strategic Resolution Software by 10x increases system safety by about 8x Increasing reliability of other components by 10x (and associated component pairings) have little effect on overall safety 28
Safety Analysis of Transitional Phases in Implementation of AAC A transitional phase bridges the current operation and a future ConOps Current operations are Controller in charge of CD&R in strategic timeframe In tactical timeframe: URET detects, controller resolves Safety level: 1.45 x 10-7, given a trajectory conflict An example of transitional ConOps is Controller still in charge in strategic CD&R In tactical timeframe: TSAFE detects and resolves Safety level: 1.49 x 10-7, given a trajectory conflict AAC Baseline safety level: 1.98 x 10-8 AR improves safety by more than 8 folds 29
Secondary Conflict on Distributed Separation Assurance Possible failure of intent sharing causes sidewalk conflicts in AFR (secondary conflict) A hypothetical AFR model contains perfect intent sharing between flights Perfect intent sharing will eliminate the sidewalk conflict, and increases the probability of resolving the primary conflict Probabilities Baseline (with secondary conflict) AFR w/o secondary conflict CD&R Success 0.9999997514 0.99999985385 without secondary conflict Secondary Conflict 1.024e-7 0 30
Summary Developed tool in Matlab to facilitate safety analyses (SafeATAC) SafeATAC capabilities: User-friendly interactive model implementation (i.e. DET, fault trees, reliability block diagrams & component reliability modeling) Safety-capacity trade off relations Phase transition analysis when transitioning from maintaining high capacity to maintaining safety Sensitivity analysis to identify critical points of failure and required redundancies Sample Experiments Impact of strategic and tactical CD&R on overall system safety for both AAC and AFR Removing strategic CD&R has much more effect on decreasing safety levels Full Factorial Analysis to Identify safety critical components Transitional phases in implementation of AAC Adding TSAFE improves safety by more than 8-fold Secondary conflict within distributed separation assurance Risk was small 31
Future Work Modeling component reliability as a function of traffic load and complexity (i.e. ADS-B jamming as a function of traffic load) Modeling risk factors that distinguish centralized and distributed systems (e.g. roll-call interrogation capability for centralized) Verification and Validation (V&V) of SafeATAC Models Safety analysis of UAS integration in the NAS 32
Questions? 33