Many thanks to our host

Using the PBN Approach Norma Campos Operations Research Analyst FAA ATO

2012 CANSO Caribbean and Latin America Conference: Using the Performance Based Approach (PBA) Speakers: Jurandyr de Souza Fonseca (ANS Performance Analyst / Consultant, Brazil s DECEA) John Gulding (Manager, Forecast Analysis, FAA ATO) Moderator: Norma V. Campos-Delory (Operations Research Analyst, FAA ATO) Date: September 26, 2012

What is the Performance Based Approach? ICAO Document 9883 PBA is a decision-making method based on three principles: Strong focus on desired / required results Informed decision-making driven by those desired / required results Reliance on facts and data for decision-making PBA is part of the transition towards a performance-based global air navigation system (ANS) as envisaged in: If you cannot measure it, you cannot manage it!

What are the advantages of PBA? Results-oriented. Customer focus and accountability Transparency in policy-making by stating goals in terms of outcomes rather than solutions Optimize resource allocation and allows for greater flexibility and cost-effectiveness in solutions adopted Rigorous quantitative and qualitative methods replace decisions based on anecdotal evidence Improves predictability of benefits

How can PBA be used? Management and Planning Activities Policy-making Regulation Transition planning and system changes System design and validation Economic and operational management System optimization Operational management Areas of Application Safety, security, environment Economic performance of airspace users, airports, and ANSPs Operational performance of flight operations, airport operations, and ANSP services Human resources Performance of technical systems within the air navigation system

The Performance Management Process 1. Define / review scope, context and general ambitions and expectations Performance monitoring and review 6. Assess achievement of objectives Detailed implementation plan 5. Implement solutions Performance Gap Analysis Strengths Weaknesses 2. Identify opportunities, issues, and set performance objectives (specific improvements) Opportunities Measurable Threats Achievable 3. Quantify objectives through Key Performance Indicators (KPIs) Relevant Time-Bound 4. Select solutions to exploit opportunities and resolve issues

Key Performance Areas (KPAs) Access and Equity Capacity Cost Effectiveness Efficiency Environment Flexibility Global Interoperability Security Participation by the ATM Community Safety Predictability Identified on Appendix D of the ICAO Global Air Traffic Management Operational Concept (Doc 9854) Many of KPAs are mutually interdependent. Performance trade-offs of objectives and targets to reach a balanced outcome

What do we need to implement PBA? Commitment and buy-in from senior management Agreement on performance objectives and targets Organization of process, roles, and responsibilities Proper allocation of resources Data collection, processing, storage, and reporting Coordination to ensure an agreed vision of the expected results, accountability, and use of compatible approaches Management of cost implications associated with PBA What are the major challenges for your organization? What are the major challenges for the LAC region?

Some questions for the audience Do you think s PBA a valid tool for ANS planning? If so, what areas would profit the most from adopting the PBA framework? Does your organization already use PBA principles in its decision-making process, if so, what concepts and practices? Does your organization have any formally documented PBAbased process? Do you think it to be profitable to develop them? Does your organization have a process for producing, analyzing, and reporting KPIs? What challenges has your organization faced while implementing PBA OR developing its performance monitoring tools?

Using the PBN Approach John Gulding Manager, Forecast Analysis FAA ATO

FAA and the Federal Aviation Administration Performance Based Approach LATCAR 2012 Buenos Aires Argentina By: Date: John Gulding, FAA September 26, 2012

FAA Organization Office of the Administrator & Deputy Administrator 46,539 Employees FAA Lines of Business Air Traffic Organization 35,384 Employees Aviation Safety Commercial Space Transportation Airports Human Resource Management FAA Staff Offices Policy, International Affairs & Environment Government & Industry Affairs Security & Hazardous Materials Communications NextGen Chief Counsel Civil Rights Finance and Management Audit and Evaluation FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 14

PBA Requirements ICAO Guidance (Doc 9883) 1. Define policy in terms of performance objectives 2. Make objectives measurable by defining appropriate indicators 3. Develop the data and methodologies necessary to calculate the indicators 4. Have the expertise to maintain data quality and to assess the link between indicator trends and management actions. Challenges Making objectives measurable Outcomes or enablers Hard to quantify all activities Inter-dependencies outside ATM: flight schedules, airport capacity, weather Data integrity, large scale processing Maintaining core expertise: can take time to build up core analysis capability FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 15

PBA Process ICAO Guidance (Doc 9883) 1. Define/review general ambitions and expectations 2. Identify opportunities, issues and set performance objectives 3. Quantify objectives through Key Performance Indicators (KPIs) 4. Select solutions to exploit opportunities and resolve issues 5. Implement Solutions 6. Assess achievement of objectives FAA Practice Destination 2025 Metrics that drive strategic goals FAA FY2012 Portfolio of Goals Metrics that drive tactical approach and accountability FAA FY2012 Business Plans One business plan for each line of business and staff offices OSI Goals Scorecard Performance based compensation FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 16

FAA Strategic Plan (Aspirations) Move to the Next Level of Safety Deliver Aviation Access through Innovation Create the Workplace of Choice Improve Global Performance through Collaboration Sustain Our Future FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 17

2012 - Access through Innovation - Metrics Optimize airspace and Performance Based Navigation (PBN) procedures to improve efficiency an average of 10 percent across core airports by 2018 Improve flight predictability by reducing variances in flying time between core airports based on a 2012 baseline Increase throughput at core airports by 12 percent to reduce delays by 27 percent using a 2009 operations baseline Improve throughput at core airports during adverse weather by 14 percent by 2018 Achieve a 5 percent reduction in average taxi-time at Core airports FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 18

FAA Tactical Goals for 2012 For each metric used to track an aspirational goal: Metric Description and Target Specific Metric Calculation Definition Data Sources used for Metric FAA and Public Benefit of Metric External Inter-Dependencies FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 19

12 OSI Performance Metric Categories The Organizational Success Increase (OSI) includes 30 target measures 10 Performance Areas FAA Organizations 8 6 4 2 0

FAA Data Sources for Performance Flight Trajectory Data Position Data, Flight Plan Airline Reported Performance Key Surface Event Times, Delay Cause Codes FAA Data Capacity values, ATM reported Delay Weather Data FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 21

Performance Based Approach in the ATO ATO s metrics were redesigned in 2012 to apply to four key areas: safety, efficiency, cost effectiveness and community Track metrics that identify our highest priorities and that can change operational behavior through management actions. ATO Efficiency Metrics Adjusted Operational Availability NAS Equipment Impact Terminal Arrival Efficiency Rate (TAER) Average Daily Capacity (ADC) ATO Efficiency Metrics Under Development Airport Actual Arrivals and Departures (AAAD) Filed versus Flown En-Route Distance (FFED) FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 22

Equipment Impact & Operational Availability Availability - percentage of hours where equipment is available with respect to maximum facility/service hours Target: 99.70%, Outcome: 99.78% Equipment impact - failure or outage which causes reduced capacity at the facility (airport or en-route) Number of FAA equipment related delays per 100,000 operations Target: 3.0, Outcome: 1.22 100.00 Adjusted Operational Availability for Core Airports - FY12 99.75 99.50 99.25 99.00 98.75 ATL BOS BWI CLT DCA DEN DFW DTW EWR FLL HNL IAD IAH JFK LAS LAX LGA MCO MDW MEM MIA MSP ORD PHL PHX SAN SEA SFO SLC TPA FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 23

Terminal Arrival Efficiency Rate (TAER) Unimpeded Trajectory TAER ETA Arrival Fix x Actual Trajectory 40 nmi The TAER measures the extent to which the airport facility handles the number of arrivals (demand) they indicated they could accommodate (capacity) Requires a method for estimating demand. Requires an arrival capacity value for the airport Arrival Airport 100 nmi Capacity value may change based on weather or other conditions TAER Targets are facility specific FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 24

CLT 21-JAN-2012 18:45-19:00 5.53 excess min, 80 th percentile FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 25

CLT 23-APR-2012 14:00-14:15 4.42 excess min, 80 th percentile FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 26

Average Daily Capacity (ADC) Sum of the number of flights the facilities plan as capability for landings and take-offs in a month, divided by the number of days in the month This is a dynamic measure that changes during the day based on factors such as weather and runway availability. The ADC and the TAER are designed to work together ADC metric provides incentive to declare a high capacity FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 27

Terminal Services (TAER) Core Airports Insight: Variation from FY12 TAER Targets FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 28

Management Actions Tracking metrics improves overall understanding of NAS performance NAS Equipment Availability Assess Failure and Repair Rates of Equipment Improve Training/Education Terminal Arrival Efficiency Rate Educate on opportunities to improve performance Prioritize areas for improvement Acknowledge/reward operational excellence FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 29

Flight Efficiency Indications Under Development # Flts Avg. Direct (D) Avg. Flt Plan (FP) A-D Avg. A-FP Avg. 5,757 813.9 839.1 26.3 1.1 FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration (A-D)/ D (A-FP)/ FP 30 3.2% 0.13%

For Further Information CANSO Operations Standing Committee Reviews current practice on assessing operational performance David Maynard david.maynard@canso.org Gabor Nemes gabor.nemes@hungarocontrol.hu Doug Stoll douglas.a.stoll@boeing.com FAA Continuing Outreach through CANSO, ICAO and bi-lateral agreements John Gulding john.gulding@faa.gov Norma Campos norma.v.campos@faa.gov FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 31

Terminal Services (ADC) Core Airports - Average Daily Capacity (ADC) 90000 88000 86000 84000 Core Airports Insight: Variation from FY12 ADC Targets FAA and the Performance Based Approach September 26, 2012 Federal Aviation Administration 32 Average Daily Capacity Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep FYTD Target FY12 FY11 Quarterly Target FYTD FY12 Target 300 200 100 0-100 -200 MCO BWI DCA EWR DEN PHX MIA SAN MDW LAX LAS SFO DFW PHL FLL SEA JFK BOS SLC ORD MSP CLT LGA TPA IAH MEM DTW IAD ATL HNL

Using the PBN Approach Jurandyr Fonseca ANS Performance Analyst DECEA

Performance-Based Approach in the Brazilian Department of Airspace Control Jurandyr de Souza Fonseca ANS Performance Analyst

The Brazilian Department of Airspace Control DECEA is the central agency of the Brazilian-Airspace Control- System (SISCEAB), which integrates all resources to exercise airspace control, provide civilian ANS and Brazilian Airspace Control System Airspace Control DECEA Central Agency DECEA s Organizations Other ANSPs Air Navigation Services support military and air defense operations in Brazil s sovereign (Continental) and jurisdictional (Atlantic) airspace. Military Air Operations Brazilian Aerospace Defense System

DECEA (cont.) It regulates ANS in Brazil, as well as plans, coordinates and supervises its country-wide development.

DECEA (cont.) By means of its organizations, it is the country s largest ANSP.

Use of PBA in DECEA DECEA follows the three PBA principles. Its main performance drivers are: airspace control, air defense and national integration needs, and air traffic growth prediction and resulting demand. It prioritizes Safety, Access and Equity, Flight Efficiency, Capacity, Global Interoperability, Security and Environment KPAs. It methodically invests to retain its capacity to meet user demand for ANS, improve performance levels and keep up with advancements in CNS/ATM technology.

PBA in DECEA (cont.) Performance expectations and objectives are stated in several documents. Brazil s Multiannual Plan 2012-2015. Promote the safe and efficient movement of civil and military air traffic in the airspace under the jurisdiction of Brazil [...] expand capabilities of air defense, airspace control, flight safety [...] comply with its international commitments. Command of Aeronautics Policy for the Control of Brazilian Airspace, Jan. 2010 (restricted). Airspace Control System Development Plan, Oct. 2010.

PBA in DECEA (cont.) Performance objectives for the Brazilian Airspace Control System: Ensure the safety of air traffic without compromising the fulfillment of civil and military operations [...] [...] accomplish greater fluidity, regularity and economy to air traffic in the area of responsibility of Brazil. [...] Achieve excellence in the formation and training of personnel [...] as well as their allocation according to the organizational needs and individual training [...] [...] ensure adequate logistics support to existing equipment and systems, by means of the continuous improvement of [...] processes; and [...] actions to raise the technical level of maintenance professionals.

PBA in DECEA (cont.) Operational Improvement National ATM Operational Concept, Dec. 2011. National ATM Implementation Performance Objectives Indicators & Supporting Metrics Plan, Apr. 2012. Elementary Changes

Advancement of PBA in DECEA Increased interaction with the ATM Community, including high-level, Decision Group meetings for performance assessment and review, as well as for evolving the National ATM Implementation Plan. Deployment and ongoing improvement of a high-level, core performance data base (Performance Indicators Management System). Current effort on operations performance data (e.g. air traffic, safety, MET, SAR, ATCO). Expected leap in data availability from new automated systems (AIS, ATC, ATFM and Logistics).

External Data ETL Performance Indicators Management System Relational Data Bank ETL Data Warehouse Access & Publication Dashboards Reports

Relational Data Bank s Entity-Relationship Diagram Status on 3 Sep 2012 to DW

WHAT from RDB WHEN Data Warehouse s Snowflake Schema Diagram WHERE WHY HOW HOW MUCH WHO Dimensions WHO WHAT WHERE WHEN WHY Facts HOW HOW MUCH Status on 3 Sep 2012

WHAT KPIs and Supporting Metrics Descriptions, Properties, Analyses etc. WHY KPAs, Functions, Areas, Norms, Requirements, Orders etc. from RDB Measured, Projected and Planned (Target) Data Values. HOW HOW MUCH Year, Semester, Month, Day, Week- Day, Time Period etc. WHEN ANSPs, Organizations, ANS Facilities, Installations etc. Data Warehouse s Snowflake Schema Diagram WHERE FIR/CTA/UTA, TMA, AD, Geographical Region, Federative Unit, City, Locality etc. WHO Dimensions Facts WHO WHAT WHERE WHEN WHY HOW HOW MUCH Status on 3 Sep 2012

Sample Dashboard for Performance Data Output WHY WHAT WHO HOW HOW MUCH (WHERE) WHEN

Aeronautical Information DB BIA (to be fielded) Brazilian Institute of Geography and Statistics Reports IBGE DECEA s ERP SISPLOG (to be developed) DECEA s Annual Budgetary Plan PLANSET ANS Charges DB BIMTRA ATFM System s DB Sigma (to be fielded) ATFM Center Daily Auxiliary Air Economic s Air Traffic Situation Reports (CGNA) Routes Publication ROTAER Localitie s Performance Personnel DB (SDAD) Operational Meteorology DB OPMET MET Indicators Management System Personne l English Proficiency Exam s Reports Matériel TAF Evaluation System s Reports PCOAMET Safety SAR Matériel and Services Logistics System s DB SILOMS Semestrial Safety Indicators and Statistics Reports (ASEGCEA) IN USE PLANNED SAR Operations Reports (SDOP)

Aeronautical Information DB BIA (to be fielded) Auxiliary Air Routes Publication ROTAER Operational Meteorology DB OPMET Brazilian Institute of Geography and Statistics Reports IBGE Localitie s MET DECEA s ERP SISPLOG (to be developed) DECEA s Annual Budgetary Plan PLANSET Economic s Cost- Effectiveness Capacity Availability Predictability ANS Charges DB BIMTRA Air Traffic Efficiency Productivity Personne l Global Interoperability ATFM System s DB Sigma (to be fielded) ATFM Center Daily Situation Reports (CGNA) Personnel DB (SDAD) English Proficiency Exam s Reports TAF Evaluation System s Reports PCOAMET Safety Safety SAR Quality of Service Matériel Matériel and Services Logistics System s DB SILOMS Semestrial Safety Indicators and Statistics Reports (ASEGCEA) IN USE PLANNED SAR Operations Reports (SDOP)

Goals Provide ANS so as to ensure that all users fly safely and efficiently Access & Equity Efficiency Safety Means By maintaining the ability to satisfy the demand Flexibility Using and making available reliable, interoperable and protected resources Capacity Predictability Availability Global Interoperability Security Personnel Matériel Rules Information Conditions Having in view the expectations and possibilities of all parties AIS MET Traffic And their need to operate in an environmentally and economically effective way. Participation by the ATM Community Environment Cost Effectiveness Productivity

Sample KPIs and Supporting Metrics Safety ATS safety index (risk situations per 105 air traffic movements) nation-wide < region < airspace type Occurrences of ATS-related factor in air traffic incidents factor group < factor type flight rules other views/taxonomies... Runway incursions Aborted landings, interrupted/delayed take-offs due to runway incursions Conformity to safety inspection protocols

Sample Indicators and Metrics (cont.) Efficiency Average flight time between city pairs on selected routes (minutes) Quantity of flights per year between on the same routes (supporting metric) City Pair Route A > B B > A

Sample Indicators and Metrics (cont.) Efficiency Daily percentage of regular flights departures and arrivals with delays larger than 30 minutes in selected airports Christmas New Year s Day

Sample Indicators and Metrics (cont.) Availability Personnel Percentage of ATCOs in Operations proficient in English communication (ICAO s Level 4 English or higher)

Sample Indicators and Metrics (cont.) Availability Information Aerodrome forecast (TAF) accuracy Wind direction is difficult to predict... Other parameters not that much...

Sample Indicators and Metrics (cont.) Productivity Quantity of SAR Operations (supporting metric)

Challenges to Advance PBA Translating high-level performance objectives into binding performance targets is a daring socio-political process. Apprehension of undue/unfair accountability is not a small issue. Decision-making driven by required results is altogether different from decision-making oriented by intended results. Reliance on measured/projected data is what sets performance-based approach apart from more common objective-based approaches. Risk of reverse-engineering the PBA process. Difficulty to implement performance measurement and data collection capabilities can be an obstacle.

Challenges (cont.) Performance management, review and assessment are complex, costly activities. Highly diversified source data bases; massive amount of data to be prepared, presented, analyzed, interpreted and used. Intensive Business Intelligence activity, requiring a properly staffed team of skilled professionals (ANS analysts; DBA-, ETL-, OLAP-qualified professionals). Socially-intensive decision-making process, requiring the involvement of other segments of the ATM Community.

Challenges (cont.) Down-to-earth factors that may hinder the advancement of PBA: PBA can be very time-consuming, vis-à-vis the pressure to field available ANS resources known to satisfy urging performance demands. ANS is a technology-driven industry, so global interoperability requirements may limit the value of PBA for choosing technical solutions. PBA can be used in its essence by investing where the demand is higher. Follow the traffic : measure it, project it, and prepare for it.