UAS Integration Risk Assessment Air Traffic Organization

Transcription

1 UAS Integration Assessment Air Traffic Organization Federal Aviation Administration Presented to: Committee on Assessing the s of UAS Integration Presented by: Federal Aviation Administration Date: Wednesday, September 27, 2017

2 What is a Safety Management System? An integrated collection of processes, policies, procedures, and programs used to assess, define, and manage safety risk in the provision of air traffic control and navigational services Management Safety Policies and Standards SMS Communication, Values, and Culture Data, Analysis, and Reporting 9/27/2017 SMS Overview 2

3 What Does SMS Provide? Data-Informed, Expertise-Driven Management Continuous monitoring, assessment, & mitigation Common framework to identify & address safety hazards and risks Communication/ management of potential & actual risks in the system 9/27/2017 SMS Overview 3

4 What are the SMS Components? Safety Management Safety Assurance Safety Policy Safety Promotion 9/27/2017 SMS Overview 4

5 What is the SRM Process? Describe the System Identify the Hazards Analyze the Assess the Treat the DIAAT: The formalized approach used by a panel of experts and affected stakeholders to identify issues, evaluate their criticality, and determine a means to minimize or eliminate their impact 9/27/2017 SRM Process Overview 5

6 Describe the System Describe the System Identify the Hazards Analyze the Assess the Treat the 9/27/2017 SRM Process Overview: Describe the System 6

7 What is the 5M Model? A tool used to describe the system, operation, or procedures being added or changed (hu)man/ Person Mission Management Machine Media or Environment 9/27/2017 SRM Process Overview: Describe the System 7

8 Identify Hazards Describe the System Identify the Hazards Analyze Assess Treat 9/27/2017 SRM Process Overview: Identify Hazards 8

9 What is a Hazard? Any real or potential condition that can cause injury, illness, or death to people; damage to or loss of a system, equipment, or property; or damage to the environment. A prerequisite to an accident or incident. Tool/Technique Preliminary Hazard List/ What If Analysis Functional Hazard Analysis Bow-Tie Diagram Best Use Procedural Changes (system or operational) Acquisition or modification of equipment All applications when it is difficult to determine hazard or cause/effect relationship 9/27/2017 SRM Process Overview: Identify Hazards 9

10 Analyze Describe the System Identify the Hazards Analyze Assess Treat 9/27/2017 SRM Process Overview: Analyze 10

11 What Is? The composite of predicted severity and likelihood of the potential effect of a hazard, before any of the proposed mitigations are implemented Severity The consequence or impact of a hazard s effect (outcome) in terms of degree of loss or harm (less severe to more severe) While assessed first, determination of severity is independent of likelihood Likelihood The estimated (predicted) probability or frequency, in quantitative or qualitative terms, of a hazard s effect (outcome) Definitions are tailored to ATC Operations, Flight Procedures, and Systems Engineering 9/27/2017 SRM Process Overview: Analyze 11

12 Assess Describe the System Identify the Hazards Analyze Assess Treat 9/27/2017 SRM Process Overview: Assess 12

13 What are Levels? levels are assessed for each hazard based on severity and likelihood Severity / Likelihood High: Unacceptable Medium: Acceptable Low: Acceptable Must be mitigated to a Medium or Low risk prior to implementation May be implemented but safety requirements are recommended to increase the safety margin May be implemented (but safety requirements are recommended) Must have at least one safety performance target 9/27/2017 SRM Process Overview: Assess 13

14 Treat Describe the System Identify the Hazards Analyze Assess Treat 9/27/2017 SRM Process Overview: Treat 14

15 How do we Treat? Identify feasible risk management options 1 Identify and evaluate means to manage the risk or reduce it to an acceptable level The four risk management strategies are: Control, Avoidance, Transfer, and Assumption 2 Determine predicted residual risk The risk that is estimated to exist after the safety requirements are implemented, or after all avenues of risk mitigation have been explored. 3 Define safety performance targets Identify measurable goals used to verify the predicted residual risk of a hazard Determine appropriate metrics Consider controls and safety requirements Pre-SRM panel data analysis serves as the basis for comparison against the post-implementation metrics 4 Develop a monitoring plan Document all hazards and document and verify low-risk hazards at least once Document a plan to implement safety requirements for all risk levels identified in the final Hazard Analysis Worksheet (HAW) Record monitoring activities 9/27/2017 SRM Process Overview: Treat 15

16 Evaluating UAS Using SRM 9/27/2017 Evaluating UAS Using SRM 16

17 Scoping UAS Integration UAS risk analysis efforts focused on the following: Line-of-sight operations Operations beyond visual line of sight Commercial uses and operations (i.e., the Pathfinder Program) Certificate of Authorization (COA) processes Impact on Airspace Class (e.g., operations near and around airports and large population centers) 9/27/2017 Evaluating UAS Using SRM 17

18 Past UAS Safety Studies SRM panels convened for each class of airspace and Pathfinders for the safe integration and testing of UAS in the National Airspace System (NAS) COA process for UAS low-altitude operations under Section 333 (200 feet Above Ground Level (AGL)) COA process for UAS low-altitude operations under Section 333 at or below 400 feet AGL Class B Class A and High E Class A and High E Class C Class D Class E Class G Conditions and limitations for allowing operations in Class E Surface Area 9/27/2017 Evaluating UAS Using SRM 18

19 Common Hazards Across Safety Analyses Lost Link - The loss of real-time command and control (C2) data link. If the link between the Pilot in Command (PIC) and the UAS is lost, the Unmanned Aircraft (UA) utilizes pre-programmed contingency (i.e., lost link) procedures until the link is re-established or until the UAS ends the flight in a safe manner Fly Away - Interruption or loss of the control link, or when the pilot is unable to affect control of the aircraft and, as a result, the UA is not operating in a predicable or planned manner Difficulty to be Seen by Manned Aircraft - Due to size of UA (e.g., Quad Copters or small fixed-wing), speed, body composition, and markings 9/27/2017 Evaluating UAS Using SRM 19

20 Common Hazards Across Safety Analyses (cont.) Loss of Visual Line of Sight - The PIC has lost visual contact with the UA as required by 14 Code of Federal Regulations Section Loss of Communications Between Pilot and Observer because of radio failure and they are not co-located 9/26/2017 Evaluating UAS Using SRM 20

21 Worst Credible Effect SMS Matrix 1973 MOR in NMAC in MAC in 25 years Frequent A Minimal 5 Minor 4 Major 3 Hazardous 2 Catastrophic 1 MOR NMAC MAC 1.00E E E-02 Most UAS operations - NMAC is the Worst Credible Effect Probable B Remote C Extremely Remote D Extremely Improbable E 1.00E E E E E E E E E-11 9/27/2017 Evaluating UAS Using SRM 21

22 Current UAS Effects and Severity Classifications Under 55 pounds and 100 mph or slower Minimal 5 Discomfort to those on the ground UAS Hazard Severity Classification Note: Severities related to ground-based effects apply to movement areas only. Minor Major Hazardous CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING: Low Analysis Medium Analysis High Analysis Event severity, two or Event severity, three Event severity, four fewer indicators fail indicators fail indicators fail Non-serious injury to three or fewer people on the ground A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins UAS crash obstructing Runway Incapacitation to Unmanned Aircraft System crew Non-serious injury to more than three people on the ground Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 100 feet Proximity of less than 100 feet to a manned aircraft Serious injury to persons other than the Unmanned Aircraft System crew Catastrophic 1 A collision with a manned aircraft Fatality or fatal injury to persons other than the Unmanned Aircraft System crew 9/27/2017 Evaluating UAS Using SRM 22

23 Current UAS Effects and Severity Classifications Over 55 pounds or faster than 100 mph Minimal 5 Discomfort to those on the ground Loss of separation leading to a Measure of Compliance greater than or equal to 66 percent UAS Hazard Severity Classification Note: Severities related to ground-based effects apply to movement areas only. Minor Major Hazardous CONDITIONS RESULTING IN ANY ONE OF THE FOLLOWING: Low Analysis Medium Analysis High Analysis Event severity, two or Event severity, three Event severity, four fewer indicators fail indicators fail indicators fail Non-serious injury to three or fewer people on the ground Non-serious injury to more than three people on the ground A reduced ability of the crew to cope with adverse operating conditions to the extent that there would be a significant reduction in safety margins Manned aircraft making an evasive maneuver, but proximity from Unmanned Aircraft remains greater than 500 feet UAS crash obstructing Runway Incapacitation to Unmanned Aircraft System crew Proximity of less than 500 feet to a manned aircraft Serious injury to persons other than the Unmanned Aircraft System crew Catastrophic 1 A collision with a manned aircraft Fatality or fatal injury to persons other than the Unmanned Aircraft System crew 9/27/2017 Evaluating UAS Using SRM 23

24 Pros and Cons of NMACs as Credible Effects NMACs are reported by controllers and pilots NMACs are tracked and categorized using existing data collection It is difficult or impossible to determine the cause of the NMAC involving a UAS UAS operators cannot be identified The existence and criticality of a NMAC is subjective based on its current definition (requires third-party verification) 9/27/2017 Evaluating UAS Using SRM 24

25 Monitoring UAS Encounters Quarterly, AJI collects data on UAS encounters; demographics include: Altitude Nearest ATC facility Proximity to manned aircraft Type of manned aircraft involved UAS position in relation to manned aircraft Class of airspace Phase of flight (manned aircraft) Severity of NMAC 9/27/2017 Evaluating UAS Using SRM 25

26 Trend in UAS Encounters (April July 2017, FY17 Q3) Total Linear (Total) /27/2017 Evaluating UAS Using SRM 26

27 FY17 Trends in UAS Encounters Data set all 500ft NMACs (1/1/17-9/19/17) 41% are 100ft NMACs 5% occurred under 500ft altitude 53% occurred over 3000ft altitude Reports Identifying Airspace Class A: 0.5% Class D: 14% Class B: 34% Class E: 19% Class C: 15% Class G: 5% 9/27/2017 Evaluating UAS Using SRM 27

28 Future of UAS Assessment 9/27/2017 Future of UAS Assessment 28

29 UAS Integration Collision with manned aircraft, people on the ground, or other UAS Damage to property Environment Issues The primary goal of the ATO is to prevent collision with manned aircraft in controlled airspace, while FAA lines of business/other agencies are responsible for risks not related to air traffic 9/27/2017 Future of UAS Assessment 29

30 Target Level of Safety The NAS is a very safe system, but it is not risk-free The probability of collision is greater than zero The goal is to maintain a low probability of collision, but how low? Most air navigation service providers have a target level of safety. The ATO s minimum acceptable level of safety for a catastrophic outcome is 1E-9 Recently, the ATO adopted an Acceptable Level of Safety concept for rare operations, such as commercial space efforts 9/27/2017 Future of UAS Assessment 30

31 Quantitative Assessment should be assessed in relation to its proximity to collision All incidents and accidents should be analyzed with respect to their probability of collision Need for a Continuum concept Employ modeling: Use objective data Analytical approaches; geometry and density Minimize reliance on subject matter expert input by employing modeling (e.g., Monte Carlo modeling) 9/27/2017 Future of UAS Assessment 31

32 Factors in Modeling Not all collisions are created equal. This is dependent upon: UAS size UAS speed Vulnerability of aircraft Credit : Controls and mitigations should be accounted for (e.g., sense and avoid) Penalty : All credible sources of errors should be accounted for (e.g., human error) 9/27/2017 Future of UAS Assessment 32

33 Current Modeling Effort Under the Low-Altitude Authorization and Notification Capability (LAANC) initiative, the FAA has developed maps with pre-approved flight zones and maximum altitudes for operating drones near airports With the near-term focus of supporting LAANC, Volpe is working on a quantitative model to examine small UAS operating in controlled airspace around airports Input data: Manned aircraft tracks Output 1: Collision probability Output 2: -Adjusted Altitude Collision probability is negligible below, and non-negligible above, -Adjusted Altitude Output 3: Sample Size Used to assign confidence level to Outputs 1 and 2 9/27/2017 Future of UAS Assessment 33

34 Collision Probability Flight swath modeled by aircraft frontal profile area extruded over distance traveled Relevant airspace examined over a given period of time, number of aircraft, etc. Aircraft frontal profiles Small UAS randomly located in airspace Manned aircraft Airport Unmitigated Collision Probability per Unit Time = Ratio of manned aircraft swath volumes to total airspace volume Length of analysis time period 9/27/2017 Future of UAS Assessment 34

35 -Adjusted Altitude Controlled airspace divided into 1-min x 1-min grid volumes -Adjusted Altitude is defined as 200 feet below routine operations in each grid volume: Manned aircraft Highest Maximum altitude of grid volume (400 feet) Altitude of routine manned a/c operations in grid volume Ground level 200 feet Safety Buffer -Adjusted Altitude Operate with caution 9/27/2017 Future of UAS Assessment 35

36 Routine Operations Volpe s model is in the early stages of development -Adjusted Altitude is calculated on a grid-by-grid basis, rounded down to the nearest 50 feet in each grid, and presented two ways Worst-case = Lowest altitude reached manned aircraft tracks (points within tracks) over analysis time period, minus 200-foot safety buffer Fifth Percentile below ceiling : 1,200 feet selected as ceiling, below which is considered shared airspace Filter out lowest 5 percent as outliers or rare operations 200 feet below fifth percentile of points below 1,200 feet in each grid volume 9/27/2017 Future of UAS Assessment 36

37 Challenges for Analysis Insufficient data collection to establish reliable performance monitoring Lack of procedures for UAS Communication issues (e.g., nonstandard terminologies) Difficult to quantify mitigations Lack of licensing requirements to maintain a level of design reliability Insufficient operator proficiency 9/27/2017 Future of UAS Assessment 37

38 Takeaways ATO s primary safety focus is on preventing collision between manned aircraft Current State Safety and operational experts have used the SRM process to assess UAS risk potential. AJI is collecting data on UAS encounters with manned aircraft. Future State Modeling approaches focused on collision probability to determine the true risk from UAS operations in the NAS. Improved data (collected and simulated) for monitoring performance targets. 9/27/2017 Future of UAS Assessment 38

39 Back-Up Slides 9/27/2017 Back-Up Slides 39

40 SRM Panels Carry out the SRM process and produce safety documents with risks and recommendations to mitigate them. Stakeholders and Process Participants: Safety engineers ATC experts Pilots Bargaining unit representatives Data analysts Industry subject matter experts 9/27/2017 Back-Up Slides 40

41 SRM Process Review Current System Review Proposed Change or Existing Issue Develop Preliminary Hazard List Develop Hazard Analysis Worksheet Develop Monitoring Plan SRM Document If No Hazards Are Identified 9/27/2017 Back-Up Slides 41

42 Link Between Cause, Hazard and Effect Identify the Hazards Cause(s) Possible Effect Ice Hazard 9/27/2017 Back-Up Slides 42

43 Preliminary Hazard List Hazard identification tool that lists potential hazards The result of brainstorming (may include hazards, causes, effects, and sometimes system state or controls) Used to list all possible hazards within scope Stepping stone to HAW Identify the Hazards Ultimately, the list will be a combination of hazards, causes, and effects that will be later categorized in the SRM process. Within/out of scope is not addressed at this time. 9/27/2017 Back-Up Slides 43

44 HAW What is a HAW? An analysis tool used to document the risk assessment of the system or change Inputs are developed from a PHL (or other Hazard Identification tool) Why complete a HAW? Required as part of the SRM process Links identified hazards to controls and risk mitigation activities To document risks, hazards, system states, and safety requirements Identify the Hazards 9/27/2017 Back-Up Slides 44

45 HAW: Template (Key Definitions) Identify the Hazards (1) Hazard ID (2) Hazard Description (3) Causes (4) System State (5) Controls (6) Control Justification (7) Effects (Credible Effects) (8) Severity (9) Severity Rationale (10) Likelihood (11) Likelihood Rationale (12) Initial (13) Safety Requirements (14) Organization Responsible for Implementing Safety Requirements (15) Predicted Residual (16) Safety Performance Targets * Refer to Quick Reference Guide for Additional Information. 9/27/2017 Back-Up Slides 45

46 HAW Ice Example: Identify Hazards (1) Hazard ID (2) Hazard Description (3) Causes (4) System State November-March ABC-01 Ice accumulation on the sidewalk (slipping hazard) Freezing rain Snow Temperatures below 32 F 9/27/2017 Back-Up Slides 46

47 Controls: Guidelines Analyze Understand the impact of the control Must be associated with the change, hazard, cause and system state Cite the specific version, paragraph and/or section number(s) when using FAA Orders Include information explaining how the control mitigates the risk 9/27/2017 Back-Up Slides 47

48 Effects (Credible Effects) Credible effect refers to the reasonable expectation that the assumed combination of conditions that define the system state will occur within the operational lifetime of a typical ATC system. Hazard assessments consider all credible effects (outcomes). Less severe effects may pose a higher risk than the worst credible effect. Analyze 9/27/2017 Back-Up Slides 48

49 SMS Likelihood Table Analyze Operations: Expected Occurrence Rate (per operation / flight hour / operational hour) Quantitative (ATC / Flight Procedures / Systems Engineering) Frequent (A) (Probability) 1 per 1,000 Probable (B) 1 per 1000 > (Probability) 1 per 100,000 Remote (C) 1 per 100,000 > (Probability) 1 per 10,000,000 Extremely Remote (D) 1 per 10,000,000 > (Probability) 1 per 1,000,000,000 Extremely Improbable (E) 1 per 1,000,000,000 > (Probability) 1 per /27/2017 Back-Up Slides 49

50 HAW Ice Example: Analyze Analyze HAZARD: ABC-01 (5) Control(s) (6) Control(s) Justification (7) Effects (8) Severity (9) Severity Rationale (10) Likelihood (11) Likelihood Rationale Current building safety/security plan ABC regulations BLD PLN 001 ABC Reg 002 Spraine d Wrist Major (3) Slipping on ice can cause joint injuries, bodily harm, muscle strains Probable (B) 10,000 people use the sidewalks, and approximately 5 injuries occur due to slips on ice, giving a rate of 5x10-4 per operation (one per 2000). Intervention by facility personnel Facilitate personnel intervene by placing warning signs and spreading salt/ice melting agents. Head injury Hazardous (2) Slipping on ice can cause accidental head injury. Extremely Improbable (E) Over the past 10 years 36 head injuries have been reported. Estimate that a typical person makes about 100 trips per winter on foot. Total population of US is about 300 million, so rate is 36 / (10*100*3x108) = 1.2x10-11 per operation. 9/27/2017 Back-Up Slides 50

51 HAW Ice Example: Assess Assess (7) Effects (8) Severity (10) Likelihood (12) Initial risk Initial Hazard Sprained Wrist Head injury Major (3) Hazardous (2) Probable (B) Extremely Improbable (E) 3B - High 2E - Medium 3B - High 9/27/2017 Back-Up Slides 51

52 HAW Ice Example: Initial Assess ABC -01 HAZARD: ABC-01 9/27/2017 Back-Up Slides 52

53 HAW Ice Example: Treat (13) Safety Requirements Establish sand buckets at each end of the sidewalk. Custodian will sand the sidewalk each hour when the hazard is present. Post facility-wide warning of icing Each employee will receive a briefing on the alternate route to follow if the hazard is present. Facility management will budget the cost of sidewalk anti-icing materials. HAZARD: ABC-01 (14) Organization Responsible for Implementing Safety Requirements Jeene Smith, ABC Custodian Joe Smith, ABC Manager Jane Smith, Director (15) Predicte d Residual 3C Medium Treat (16) Safety Performance Targets Fewer than two falls each winter due to slipping on ice. 9/27/2017 Back-Up Slides 53

54 HAW Ice Example: Predicted Residual Treat ABC -01 9/27/2017 Back-Up Slides 54

55 Monitoring Plan Template 55 9/27/2017 Back-Up Slides 55