Estimating Sources of Temporal Deviations from Flight Plans Ms. Natasha Yakovchuk (yakovn2@rpi.edu) Prof. Thomas R. Willemain (willet@rpi.edu) Department of Decision Sciences and Engineering Systems Rensselaer Polytechnic Institute Sponsored by Free Flight Program, Federal Aviation Administration Contract # DTFA01-98-00072
Agenda FAA interests: system predictability, assessing interventions Problem introduction Proposed methodology Evaluation of alternative estimation methods System implementation in SAS for annual and next-day reporting Ongoing research Operational validation Statistical analysis of results 2
Motivation and FAA sponsorship Client: Free Flight Program of the Federal Aviation Administration Need: improve system predictability and decrease unexpected flight delays More specifically: trace flight delays to their sources, and quantify them Intended use: next-day and annual reporting, special studies Potential use: evaluating impact of FAA initiatives 3
Problem introduction airtime 150 140 130 120 110 Average daily airtime for flights from Atlanta to Boston for year 2001 Average airtime fluctuates (due to winds aloft, weather and congestion in airports, etc.) Flight plans anticipate normal problems 100 date Shift attention to Deviation = Actual Airtime Flight Planned Time 2 types of Deviations: ETE and G2G 4
Problem introduction: Deviations deviation from flight plan 20 10 0-10 Deviations from flight plans for flights from Atlanta to Boston for year 2001 Deviations from flight plans measure unanticipated problems, or surprises Common factors for different flights are considered as systemic sources of deviations -20 date Decompose deviations into four sources: System + Origin airspace + Destination airspace + En route airspace 5
FAA data as a two-way table Origin airport Destination airport 31 major US airports Each table represents one day of operations Each cell contains an average deviation from flight plan One observation per cell, averaging over multiple flights Data available for January 01- March 03 Presence of structural holes structural holes 6
Fragment of the table Destination airport Origin airport 7
Row + Column Analysis origin 1 2 destination 1 2 m y 11 y 12 y 1m y 21 y 22 y 2m linear additive model y ij = µ + α i + β j + ε ij 1 2 en route effects (residuals) 1 2 m ε 11 ε 12 ε 1m ε 21 ε 22 ε 2m origin effects α 1 α 2 n y n1 y n2 y nm n ε n1 ε n2 ε nm α n destination effects β 1 β 2 β m µ µ = system effect (e.g., September 11 th ) α i = origin effect (e.g., restricted departure gates) β j = destination effect (e.g., fog) ε ij = en route effect (e.g., convective weather, MIT, circular holding) system effect 8
Which estimation method to use? Methods: Ordinary Least Squares Least Absolute Deviations (LAD) Median Polish Full factorial design: Factors (at 3 levels each): table size percentage of holes percentage of outliers Responses (comparison criteria): accuracy of estimates (RMSE and MAE for effects) outlier detection capability (sensitivity and specificity) 9
Modeling FAA data Normal probability plot for BWI:IAD effect Can use estimates from LAD Generate origin, destination and en route effects independently Most effects can be modeled by N(µ, σ 2 ) after removing outliers Scatterplot of µ against σ for en route effects µ and σ of effects can be modeled independently µ is modeled by Normal σ is modeled by Lognormal 10
Major findings Root mean square error for en route effects, 10% outliers 1.34 Root mean square error for terminal effects, 10% outliers 0.94 1.0 Sensitivity for enroute effects, 10% outliers rmse_enroute 1.32 1.30 1.28 1.26 1.24 rmse_term 0.92 0.90 0.88 0.86 sensitivity 0.9 0.8 0.7 0.6 1.22 LAD least median squares polish 0.84 LAD least median squares polish 0.5 Benchmark LAD least median squares polish All error measures are on the order of only one minute for all three methods! Since FAA data have up to 10% outliers, we choose resistant methods (better in accuracy and outlier detection capability) LAD is slightly better in estimating terminal effects than median polish Choose LAD for estimation 11
System implementation for the FAA A turnkey system implemented in SAS that produces: Next-day estimates of effects Map-based displays Statistical graphics Datasets for use in one-off statistical studies 12
Timeplot of system effects, 2001 Timeplot of system effect for year 2001 Computations based on ETE 13
Boxplots of destination effects, 2001 PHL SFO LGA LAS 14
15 Map of destination effects
16 Map of en route effect outliers
Ongoing research #1: Operational validation Validate the results against other databases: Aviation System Performance Metrics (ASPM) Operations Network (OPSNET) Post Operations Evaluation Tool (POET) Strategic Plans of Operation (SPO) National Oceanic and Atmospheric Administration (NOAA) Conduct at two levels: Macroscopic validation (compare statistics for a certain time period) Microscopic validation (detailed validation for selected days) 17
Macroscopic validation: ASPM airport: ORD percentage of late arrivals 80 60 40 percentage of late arrivals 20 0 80 60 40-20 -10 0 10 20 30 destination effect airport: SFO Strong correlation between destination effects (calculated from G2G data) and ASPM percentage of late arrivals (Jan 2001-March 2003) 20 0-20 -10 0 10 20 30 destination effect 18
Microscopic validation: Weather February 15, 2001 19
Ongoing research #2: Statistical analysis of effects Objective: Use the estimated effects to study the NAS Origin versus destination effects for LAX (G2G): negative correlation airport: LAX Origin versus destination effects for MEM (G2G): no correlation airport: MEM 20 15 10 10 dest.effect 0 dest.effect 5-10 0-20 -5 20-20 -10 0 10 orig.effect -5 0 5 10 15 20 orig.effect
Statistical analysis of effects Heteroscedasticity (ETE) Positively correlated origin effects (G2G) 15 10 5 0-5 -10-15 -20 Destination effects for LGA (year 2001) las lax san phx 21
Acknowledgements FAA/Free Flight Program Mr. Dave Knorr Mr. Ed Meyer CNA Corporation Mr.Joe Post Mr.James Bonn Metron Aviation Dr. Bob Hoffman 22