Performance Metrics for Oceanic Air Traffic Management Moving Metrics Conference Pacific Grove, California January 29, 2004 Oceanic Metrics Team
Agenda Metrics Team Michele Merkle, FAA AUA-600 Lynne Hamrick, MITRE/CAASD Yueh-Shiou Wu, MITRE/CAASD Tamara Karakis, CSSI Introduction: Purpose & Background of Oceanic Air Traffic Control (ATC) Background of Oceanic Metrics Air Carrier Meetings Oceanic Metrics Overview Metrics Based on Priorities Sample Dashboard Charts Current Challenges Related to Oceanic Metrics Baseline Performance Results Summary 2
Introduction Purpose of Briefing Provide an overview of oceanic performance metrics Describe challenges related to measuring oceanic Air Traffic Control (ATC) service qualities Discuss initial results and trends Background of Oceanic Air Traffic Control (ATC) Non-radar procedural separation Communications via Controller-Pilot Data Link (CPDLC) for Future Air Navigation System 1/A (FANS 1/A)-equipped aircraft High Frequency (HF) Radio Operator for non-equipped aircraft Oakland Oceanic Center (ZOA) controls 21.3 million square miles New York Oceanic Center (ZNY) controls 3.3 million square miles 3
Oakland Oceanic Airspace 4
New York Oceanic Airspace Gander Oceanic Shanwick Oceanic Moncton Boston New York Washington Non-WATRS Bermuda Santa Maria Oceanic WATRS New York Oceanic Miami San Juan 5
Background of Oceanic Metrics 1993 Government Performance and Results Act (GPRA) Required federal agencies to measure their performance and effectiveness FAA moves towards Performance Based Organization (PBO) and Air Traffic Organization (ATO) formed Goal to develop a more efficient and businesslike air traffic system AUA goal to continue improving oceanic service, while measuring the effect of new automation/procedures on the service provided 6
Air Carrier Meetings Onsite Air Carrier Meetings in 1999 and 2003 Dialogue with air carriers coordinated via Air Transport Association (ATA) meeting Onsite air carrier visits in 1999 and 2003 Air Canada Corporation, American Airlines, Continental Airlines, Delta Airlines, Federal Express Corporation, Northwest Airlines, United Airlines, United Parcel Service Corporation, US Airways Corporation Air Carrier Personnel Operational Analysts, Dispatchers, Meteorologists, Pilots, Instructors, ATC, Operational Managers Summary of Air Carriers Visits Received an overview of air carrier operations operating environment, route structure, fleet mix Consolidated and compared lists of priorities Discussed data sources Established baseline metrics 7
Oceanic Metrics Overview Purpose of the Dashboard Provides visual summary of performance of pertinent metrics for facility and airspace regions within the Flight Information Region (FIR) Tracks customer demand and level of service provided by the FAA oceanic ATC Establishes baseline to determine affect of automation and/or procedure changes (e.g., Advanced Technology and Oceanic Procedures or ATOP) Identify anomalies and areas that need more tracking Provide monthly charts for monitoring trends in oceanic service qualities Established data exchange process with Centers Primary data source: Oceanic Display and Planning System (ODAPS) Other data sources: Oceanic Data Link (ODL), Track Advisory (TA) Generated programs to process and analyze data Oceanic Data Repository (ODR) Oceanic Analysis Tool Set (OATS) Oceanic Metrics Generator (OMG) 8
Metrics Based on Priorities Air Carrier Priorities and Oceanic Metrics Air Carrier Priorities Operating Environment Communication Safety Efficiency - Flexibility Requests Granted Metrics Flight count Avionics equipment Fleet mix Avionics equipment Altitude requests Response times Operational errors Altitude requests due to WX Deviation requests Altitude requests granted Response time Data Availability and Sources Available Flight Plans from ODAPS Available Flight Plans from ODAPS HF messages from ODAPS CPDLC messages from ODL Available AAT-200 HF messages from ODAPS CPDLC messages from ODL Available Plans from ODAPS HF messages from ODAPS CPDLC messages from ODL 9
Metrics Based on Priorities (Continued) Air Carrier Priorities and Oceanic Metrics Air Carrier Priorities User Satisfaction - Predictability Optimal vs. Actual Planned vs. Actual User Satisfaction - Predictability Delay / On-time Performance User Satisfaction Fuel Consumption Metrics First preference granted Entry altitude flown as filed Data Needed Available Flight Plans from ODAPS Position reports from ODAPS Track Advisory reports In Development with Aviation System Performance Metrics (ASPM) and Enhanced Traffic Management System (ETMS) Planned departure and arrival times Actual departure and arrival times In Development Preferred route and altitude in ICAO Flight Plan (FPL) vs. Actual route and altitude flown 10
11 Sample Dashboard Charts ZOA Avionics Equipment Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 100% 80% 60% 40% 20% 0% ZOA First Preference Granted Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 50 45 40 35 30 25 20 15 10 5 0 100% 80% 60% 40% 20% 0% Total 1st Option Percentage ZOA Altitude Requests due to Weather - HF Flights 100% 90% 24 80% 70% 20 60% 16 50% 12 40% 30% 20% 8 4 10% 0% 0 Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 Cleared Modified Unable % Cleared % Modified % Unable Response Time in Minutes ZNY Oceanic Flights ZNY Entry Altitude Flown as Filed 700 100% 80% 60% 504 479 468 429 432 450 441 445 398 381 395 401 378 384 600 500 400 40% 300 20% 200 100 0% 0 Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 Entry Alt Lower Entry Alt Same Entry Alt Higher Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 ZOA Altitude Requests - HF Flights 100% 20 90% 80% 16 70% 60% 12 50% 40% 8 30% 20% 4 Response Time in Minutes 10% 0% 0 Oct-03 Sep-03 Aug-03 Jul-03 Jun-03 May-03 Apr-03 Mar-03 Feb-03 Jan-03 Dec-02 Nov-02 Oct-02 Sep-02 Cleared Modified Unable % Cleared % Modified % Unable Datalink Required Nav. Perf. Reduced Vert. Sep. Min. ADS In Use Avg Number of Flights per Day
Current Challenges Related to Oceanic Metrics US oceanic performance metrics are affected by actions taken by non-us oceanic ATC Limited end-to-end data available Variations in operations and priorities across different geographic and domain sub-regions Processing HF messages 12
Baseline Performance Results Trends observed Response time for HF flights are longer than that for FANS 1/A flights Percent of positive or negative response to request are basically the same regardless of the aircraft communication capabilities (i.e., FANS 1/A or HF) Daily traffic varies more than 30% (e.g., May and August); but variation of performance level is small Most flights (80%) received preferred entry altitude for New York airspace or first preference for the Pacific Organized Track System (PACOTS) Average response time to altitude change requests has decreased from 10-50 minutes in 1998 to 5-15 minutes for HF flights and 3-6 minutes for FANS 1/A flights in 2003 Plausible reasons for the above trends Introduction of data link not only allow FANS 1/A flights to communicate with ATC faster, it also reduced congestion on the channel allowing HF flights to get better services Oceanic Data Link enhanced controller productivity for all flights, not just FANS 1/A flights Whether a positive response can be granted is dependent on traffic situation, not on communication means Implementation of Reduced Vertical Separation Mimima (RVSM) allowed more flights to fly their preferred altitude profile Sample slides follow 13
Baseline Performance Results (Continued) ZOA Oceanic Flight Count 560 540 520 500 480 460 440 1998 1999 2000 2001 2002 2003 420 400 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 14
Baseline Performance Results (Continued) ZOA - HF Flights: Altitude Change Requests Ave ra g e Re qs / Da y 227 215 218 206 198 220 225 239 222 225 221 228 Average Re sp o n se Time (Minute s ) 60 50 40 30 20 10 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct No v Dec 100% 80% 60% 40% 20% 0% Percent Year of 1998 Cleared Time Mo d ifie d Time Una ble Time % Cleared % Modified % Unable 15
Baseline Performance Results (Concluded) 20 Average Reqs / Day ZOA - HF Flights: Altitude Requests 285 258 241 201 188 182 196 216 196 201 245 100% Average Response Time (Minutes) 15 10 5 80% 60% 40% 20% Percent 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Year of 2003 Cleared Time Modified Time Unable Time % Cleared % Modified % Unable 0% 20 ZOA - FANS 1/A Flights: Altitude Requests Average Reqs / Day 130 159 154 139 119 135 166 190 186 175 178 100% Average Response Time (Minutes) 15 10 5 0 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec 80% 60% 40% 20% 0% Percent Year of 2003 Cleared Time Modified Time Unable Time % Cleared % Modified % Unable 16
Summary Oceanic Performance Metrics Assesses the operating environment and quality of oceanic service provided to the airspace users Summary of performance Tracks customer demand and level of service provided Provides a foundation for making sound business decisions Baseline comparison Anomalies Trends Oceanic Metrics are evolving and expanding to meet the challenges of measuring a complex system and the performance of ATC service in a meaningful way Different data sources (e.g., ATOP is replacing ODAPS and ODL) Additional facilities (e.g., Anchorage) Changing priorities 17