Validation of Runway Capacity Models

Validation of Runway Capacity Models Amy Kim & Mark Hansen UC Berkeley ATM Seminar 2009 July 1, 2009 1

Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 2

Introduction Many different runway capacity models commercially available and in use today Model developers usually claim that their models have been validated, but Information on the validation & calibration processes can be vague or unclear to users Validation is performed at differing levels of rigor Initial validation exercises were often carried out by the developers themselves 3

Purpose Validation information or processes used are often vague or unclear Validation is performed at differing levels of rigor Develop a standard validation process that can be used to test any runway capacity model results against data Initial validation exercises were often carried out by the developers themselves Perform a third party validation of two commonly used models, the Airfield Capacity Model (ACM) and Runway Simulator (rs) 4

Capacity We define it as the average maximum sustainable throughput (arrivals or departures) per hour at a given airport. 5

Factors Affecting Airport Capacity Weather & MC designation Aircraft fleet mix & performance Controller environment & workload Runway occupancy times Overall arrival/departure split Number of runways in use, geometric layout, location of exits to taxiways State/performance of ATM system ATC separation requirements Mix and sequencing of arrivals & departures on runways 6

Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 7

Description of Models Airfield Capacity Model (ACM) Developed by the FAA & Mitre CAASD in the late 1970s/early 1980s Analytical, deterministic model Calculates hourly capacity of runway systems under continuous demand Simple to use; scenarios can be generated quickly Limited capability Validated in the early 1980s by the FAA Used mainly by the FAA and consultants 8

Description of Models Runway Simulator (rs) Developed at Mitre CAASD Discrete event simulation Intermediate between analytical model and a discrete event simulation model Based on link node system: blocking rules Estimates capacity and delay Simulates aircraft on runways & terminal airspace Some stochasticity Validated by Mitre Used by Mitre, and most recently the FAA 9

Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 10

ASPM Data Quarter hourly airport and individual flight data 2006 data for SFO January March 2005 data for LAX, supplemented with PDARS data Includes: Demand metrics Throughput metrics Other operational data & geometric characteristics Throughput Demand 11

Methodology SFO Runways Predominant configuration 28L,28R 1L,1R 12 Source: http://www.faa.gov/about/office_org/headquarters_offices/ato/publications/bench/2004download.htm

Methodology LAX Runways Predominant configuration 24R,25L 24L,25R 13 Source: http://www.faa.gov/about/office_org/headquarters_offices/ato/publications/bench/2004download.htm

Presentation Outline Introduction Purpose Description of Models Data Methodology Experimental Procedure Metrics for Validation Conclusions & Future Work 14

Methodology Experimental Procedure 15 1. Data grouped into hours; candidate analysis hours filtered based on the following: Predominant runway configuration is in use during the entire hour. The weather is VMC or IMC for the entire hour. The hour falls within the period of the day with the highest average demands (9 am 2 pm, at both SFO & LAX). 2. 50 hours are randomly chosen from filter (approximately 30 VMC, 20 IMC) 3. Obtain model capacity estimates for each hour Hours distinguished by MC, fleet mix, arr/dep split

Metrics for Validation I (Theil, Methodology 1966) 1. Prediction realization diagram 2. Inequality coefficient U = i (P 3. Inequality proportions 1= 1 n i i A ( P A ) (P i 2 A Bias Proportion (U m ) A 2 i i ) i 2 ) 2 + 1 n (s P s (P i A ) A 2 Variance Proportion (U s ) i ) 2 + 2 (1 r)s P s 1 (P i A n Covariance Proportion (U c ) i A ) 2 16 Use data only when throughput < demand

Methodology Metrics for Validation II Employ the Tobit censored regression model Assume counts are Upper censored by capacity or demand. Distributed censored normal. Q o (t) = min[d o (t), C o (t)] (1) C o (t) = β 0 + β 1 *M o,m (t) + ε (2) Arrivals or departures 17 Q o (t) count for operation o in t (obtained from ASPM). D o (t) demand for operation o in t (obtained from ASPM). C o (t) capacity for operation type o, in time t. M o,m (t) model capacity estimate for o, from m,in t. ε error term, normal IID with mean 0, variance σ o 2 ACM or rs

In a perfect model We would expect that 1. Theil U=0 2. Tobit regression, C o (t) = β 0 + β 1 *M o,m (t) + ε β 0 = 0 and β 1 = 1, indicating that model (ACM or rs) predictions are identically equal to the expected values of capacity; and σ 0 0, indicating that the variability of capacity around model predictions is low. 18

Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 19

SFO: Capacity Estimates vs. Unconstrained Counts 70 70 60 60 50 50 ACM model capacity 40 30 20 rs model capacity 40 30 20 VMC Arrivals IMC Arrivals VMC Departures IMC Departures 10 10 0 0 10 20 30 40 50 60 70 0 0 10 20 30 40 50 60 70 Operations (per hour) Operations (per hour) 20

LAX: Capacity Estimates vs. Unconstrained Counts 100 100 90 90 80 80 ACM model capacity 70 60 50 40 30 rs model capacity 70 60 50 40 30 VMC Arrivals IMC Arrivals VMC Departures IMC Departures 20 20 10 10 0 0 10 20 30 40 50 60 70 80 90 100 0 0 10 20 30 40 50 60 70 80 90 100 Operations (per hour) Operations (per hour) 21

Prediction Realization Analysis 22 ACM SFO: RMS error Inequality Inequality Proportions Coefficient (U) Arr Dep with lower U values, rs is the better predictor Air MC Arr Dep Arr Dep U m U s U c U m U s U c SFO VMC 21 21 0.65 0.66 0.94 0.00 0.06 0.92 0.01 0.06 IMC 4Primary 4 0.14 source 0.15of 0.36 inequality 0.06 are 0.58 different 0.32 0.09 0.59 SFO Total 17 16 0.54 0.54 0.48 0.25 0.27 0.47 0.22 0.31 LAX VMC 20(Bias 18 (U m 0.38 ) for 0.30 ACM, 0.76 covariance 0.09 0.15(U ) 0.80 for rs) 0.06 0.14 IMC 10 11 0.20 0.20 0.24 0.41 0.35 0.28 0.40 0.32 LAX: Model U values comparable LAX Total 17 15 0.32 0.27 0.56 0.02 0.43 0.59 0.19 0.22 Total 17 16 0.39 0.35 0.51 0.03 0.46 0.52 0.09 0.38 Inequality Inequality Proportions overall: rs RMS rs error is the better predictor Coefficient (U) Arr Dep Air MC Arr Dep Arr Dep U m U s U c U m U s U Arrival & departure capacity predictive c SFO VMC 7 7 0.20 0.22 0.46 0.00 0.54 0.58 0.04 0.38 IMC 4performance 3 0.13 0.12 is very 0.01similar 0.06 0.93 0.01 0.06 0.93 SFO Total 6 6 0.18 0.19 0.21 0.08 0.71 0.26 0.00 0.74 LAX VMC 17 20 0.32 0.33 0.74 0.10 0.15 0.81 0.07 0.11 IMC 6 6 0.11 0.12 0.37 0.14 0.48 0.40 0.02 0.58 LAX Total 14 16 0.26 0.27 0.56 0.10 0.34 0.60 0.00 0.40 Total 10 12 0.24 0.26 0.37 0.22 0.41 0.40 0.13 0.47

SFO: Model I C o (t) = β 0 + β 1 *M o,m (t) + ε (aircraft/hour) ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 21.0 2.60 8.04 22.2 2.00 11.06 β 1 0.2 0.06 4.22 0.2 0.05 4.03 σ o 1.7 0.10 17.12 1.6 0.11 15.08 rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 7.8 2.58 3.02 12.4 3.02 4.10 β 1 0.7 0.08 8.61 0.6 0.09 5.98 σ o 1.6 0.10 16.20 1.4 0.08 17.80 23

LAX: Model I C o (t) = β 0 + β 1 *M o,m (t) + ε (aircraft/hour) ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 22.7 2.71 8.35 60.7 13.71 4.43 β 1 0.5 0.04 11.58 0.1 0.21 0.56* σ o 1.7 0.08 20.52 2.2 0.08 26.66 rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 21.0 5.57 3.77 22.3 3.56 6.27 β 1 0.5 0.09 5.82 0.5 0.06 8.53 σ o 2.1 0.09 24.84 1.8 0.09 20.10 24 * are not significant at the 95% level.

SFO: Model II C o (t) = β 0 + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 11.9 2.70 4.41 12.9 4.02 3.22 β 1 0.7 0.10 6.87 0.6 0.15 3.97 β 2 14.2 2.78 5.12 12.3 3.78 3.24 σ o 1.5 0.09 15.72 1.4 0.11 13.77 rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 1.4 3.26 0.41* 11.7 3.17 3.58 β 1 1.0 0.12 8.20 0.6 0.11 5.64 β 2 6.2 1.70 1.70* 1.5 1.32 1.15* σ o 1.4 0.08 0.08* 1.7 0.08 16.85 * are not significant at the 95% level. 25

SFO: Capacity Estimates vs. Unconstrained Counts 70 70 60 60 50 50 ACM model capacity 40 30 20 rs model capacity 40 30 20 VMC Arrivals IMC Arrivals VMC Departures IMC Departures 10 10 0 0 10 20 30 40 50 60 70 0 0 10 20 30 40 50 60 70 Operations (per hour) Operations (per hour) 26

LAX: Model II C o (t) = β 0 + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 22.0 2.48 8.8 95.4 32.73 2.92 β 1 0.5 0.04 12.64 0.7.57 1.29* β 2 3. 6 1.56 2.28 8.9 7.18 1.24* σ o 1.6 0.09 18.86 2.2 0.08 27.07 27 rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 5.1 6.61 0.77* 19.6 3.33 5.89 β 1 1.0 0.12 8.69 0.6 0.06 10.59 β 2 13.7 2.63 5.20 6.7 1.75 3.83 σ o 2.0 0.10 19.78 1.7 0.08 22.05 * are not significant at the 95% level.

LAX: Capacity Estimates vs. Unconstrained Counts 100 100 90 90 80 80 ACM model capacity 70 60 50 40 30 rs model capacity 70 60 50 40 30 VMC Arrivals IMC Arrivals VMC Departures IMC Departures 20 20 10 10 0 0 10 20 30 40 50 60 70 80 90 100 0 0 10 20 30 40 50 60 70 80 90 100 Operations (per hour) Operations (per hour) 28

Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 29

Conclusions 1. rs estimates are generally better than those of ACM Theil U, Tobit regression model coefficients 2. Validation I (Theil) High bias in ACM results 3. Validation II (Tobit regression) Tend to over predict the differences between IMC and VMC capacities VMC capacities are high 30 Validation II advantage over Validation I

Future Work Test other (more complex) capacity models Model other runway configurations and/or airports Compare model results against empirical capacity estimates from other sources (i.e. PDARS). 31

32 Thanks for your attention!

SFO & LAX: Model II C o,m (t) = β o + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 8.62 1.488 5.80 5.83 2.065 2.82 β 1 0.77 0.036 21.73 0.84 0.059 14.24 β 2 11.89 1.580 7.53 14.69 2.064 7.12 σ o 1.88 0.061 30.81 2.04 0.075 27.31 33 rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o 9.41 1.695 5.55 11.37 1.320 8.61 β 1 0.75 0.037 20.12 0.70 0.031 22.88 β 2 6.59 1.140 5.78 4.96 1.110 4.47 σ o 1.82 0.076 23.97 1.62 0.062 26.17 33

SFO & LAX: Model II 100 ACM Regression II 100 ACM Regression II 90 90 80 80 70 70 Empirical Capacity 60 50 40 30 20 Empirical Capacity 60 50 40 30 20 10 10 0 0 10 20 30 40 50 60 70 80 90 100 0 0 10 20 30 40 50 60 70 80 90 100 Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 34 34

Methodology I Empirical Capacity Estimation SFO Quarter hourly data 35

Methodology I Empirical Capacity Estimation LAX Quarter hourly data 25 20 Average Count 15 10 5 0 0 5 10 15 20 25 30 35 Demand 36 VMC Departures VMC Arrivals IMC Departures IMC Arrivals

SFO: Model II 70 ACM regression results II 70 rs regression results II 60 60 50 50 Tobit Capacity 40 30 Tobit Capacity 40 30 20 20 10 10 0 0 10 20 30 40 50 60 70 0 0 10 20 30 40 50 60 70 Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 37

LAX: Model II 100 ACM regression results II 100 rs regression results II 90 90 80 80 70 70 Tobit Capacity 60 50 40 Tobit Capacity 60 50 40 30 30 20 20 10 10 0 0 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 38