Validation of Runway Capacity Models

Transcription

1 Validation of Runway Capacity Models Amy Kim & Mark Hansen UC Berkeley ATM Seminar 2009 July 1,

2 Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 2

3 Introduction Many different runway capacity models commercially available and in use today Model developers usually claim that their models have been validated, but Information on the validation & calibration processes can be vague or unclear to users Validation is performed at differing levels of rigor Initial validation exercises were often carried out by the developers themselves 3

4 Purpose Validation information or processes used are often vague or unclear Validation is performed at differing levels of rigor Develop a standard validation process that can be used to test any runway capacity model results against data Initial validation exercises were often carried out by the developers themselves Perform a third party validation of two commonly used models, the Airfield Capacity Model (ACM) and Runway Simulator (rs) 4

5 Capacity We define it as the average maximum sustainable throughput (arrivals or departures) per hour at a given airport. 5

6 Factors Affecting Airport Capacity Weather & MC designation Aircraft fleet mix & performance Controller environment & workload Runway occupancy times Overall arrival/departure split Number of runways in use, geometric layout, location of exits to taxiways State/performance of ATM system ATC separation requirements Mix and sequencing of arrivals & departures on runways 6

7 Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 7

8 Description of Models Airfield Capacity Model (ACM) Developed by the FAA & Mitre CAASD in the late 1970s/early 1980s Analytical, deterministic model Calculates hourly capacity of runway systems under continuous demand Simple to use; scenarios can be generated quickly Limited capability Validated in the early 1980s by the FAA Used mainly by the FAA and consultants 8

9 Description of Models Runway Simulator (rs) Developed at Mitre CAASD Discrete event simulation Intermediate between analytical model and a discrete event simulation model Based on link node system: blocking rules Estimates capacity and delay Simulates aircraft on runways & terminal airspace Some stochasticity Validated by Mitre Used by Mitre, and most recently the FAA 9

10 Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 10

11 ASPM Data Quarter hourly airport and individual flight data 2006 data for SFO January March 2005 data for LAX, supplemented with PDARS data Includes: Demand metrics Throughput metrics Other operational data & geometric characteristics Throughput Demand 11

12 Methodology SFO Runways Predominant configuration 28L,28R 1L,1R 12 Source:

13 Methodology LAX Runways Predominant configuration 24R,25L 24L,25R 13 Source:

14 Presentation Outline Introduction Purpose Description of Models Data Methodology Experimental Procedure Metrics for Validation Conclusions & Future Work 14

15 Methodology Experimental Procedure Data grouped into hours; candidate analysis hours filtered based on the following: Predominant runway configuration is in use during the entire hour. The weather is VMC or IMC for the entire hour. The hour falls within the period of the day with the highest average demands (9 am 2 pm, at both SFO & LAX) hours are randomly chosen from filter (approximately 30 VMC, 20 IMC) 3. Obtain model capacity estimates for each hour Hours distinguished by MC, fleet mix, arr/dep split

16 Metrics for Validation I (Theil, Methodology 1966) 1. Prediction realization diagram 2. Inequality coefficient U = i (P 3. Inequality proportions 1= 1 n i i A ( P A ) (P i 2 A Bias Proportion (U m ) A 2 i i ) i 2 ) n (s P s (P i A ) A 2 Variance Proportion (U s ) i ) (1 r)s P s 1 (P i A n Covariance Proportion (U c ) i A ) 2 16 Use data only when throughput < demand

17 Methodology Metrics for Validation II Employ the Tobit censored regression model Assume counts are Upper censored by capacity or demand. Distributed censored normal. Q o (t) = min[d o (t), C o (t)] (1) C o (t) = β 0 + β 1 *M o,m (t) + ε (2) Arrivals or departures 17 Q o (t) count for operation o in t (obtained from ASPM). D o (t) demand for operation o in t (obtained from ASPM). C o (t) capacity for operation type o, in time t. M o,m (t) model capacity estimate for o, from m,in t. ε error term, normal IID with mean 0, variance σ o 2 ACM or rs

18 In a perfect model We would expect that 1. Theil U=0 2. Tobit regression, C o (t) = β 0 + β 1 *M o,m (t) + ε β 0 = 0 and β 1 = 1, indicating that model (ACM or rs) predictions are identically equal to the expected values of capacity; and σ 0 0, indicating that the variability of capacity around model predictions is low. 18

19 Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 19

20 SFO: Capacity Estimates vs. Unconstrained Counts ACM model capacity rs model capacity VMC Arrivals IMC Arrivals VMC Departures IMC Departures Operations (per hour) Operations (per hour) 20

21 LAX: Capacity Estimates vs. Unconstrained Counts ACM model capacity rs model capacity VMC Arrivals IMC Arrivals VMC Departures IMC Departures Operations (per hour) Operations (per hour) 21

22 Prediction Realization Analysis 22 ACM SFO: RMS error Inequality Inequality Proportions Coefficient (U) Arr Dep with lower U values, rs is the better predictor Air MC Arr Dep Arr Dep U m U s U c U m U s U c SFO VMC IMC 4Primary source 0.15of 0.36 inequality 0.06 are 0.58 different SFO Total LAX VMC 20(Bias 18 (U m 0.38 ) for 0.30 ACM, 0.76 covariance (U ) 0.80 for rs) IMC LAX: Model U values comparable LAX Total Total Inequality Inequality Proportions overall: rs RMS rs error is the better predictor Coefficient (U) Arr Dep Air MC Arr Dep Arr Dep U m U s U c U m U s U Arrival & departure capacity predictive c SFO VMC IMC 4performance is very 0.01similar SFO Total LAX VMC IMC LAX Total Total

23 SFO: Model I C o (t) = β 0 + β 1 *M o,m (t) + ε (aircraft/hour) ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β σ o rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β σ o

24 LAX: Model I C o (t) = β 0 + β 1 *M o,m (t) + ε (aircraft/hour) ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β * σ o rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β σ o * are not significant at the 95% level.

25 SFO: Model II C o (t) = β 0 + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β β σ o rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o * β β * * σ o * * are not significant at the 95% level. 25

26 SFO: Capacity Estimates vs. Unconstrained Counts ACM model capacity rs model capacity VMC Arrivals IMC Arrivals VMC Departures IMC Departures Operations (per hour) Operations (per hour) 26

27 LAX: Model II C o (t) = β 0 + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β * β * σ o rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o * β β σ o * are not significant at the 95% level.

28 LAX: Capacity Estimates vs. Unconstrained Counts ACM model capacity rs model capacity VMC Arrivals IMC Arrivals VMC Departures IMC Departures Operations (per hour) Operations (per hour) 28

29 Presentation Outline Introduction Purpose Description of Models Data Methodology Conclusions & Future Work 29

30 Conclusions 1. rs estimates are generally better than those of ACM Theil U, Tobit regression model coefficients 2. Validation I (Theil) High bias in ACM results 3. Validation II (Tobit regression) Tend to over predict the differences between IMC and VMC capacities VMC capacities are high 30 Validation II advantage over Validation I

31 Future Work Test other (more complex) capacity models Model other runway configurations and/or airports Compare model results against empirical capacity estimates from other sources (i.e. PDARS). 31

32 32 Thanks for your attention!

33 SFO & LAX: Model II C o,m (t) = β o + β 1 *M o,m (t) + β 2 *I o (VMC=1) + ε ACM Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β β σ o rs Departure Arrival Parameter Estimate Error t stat Estimate Error t stat β o β β σ o

34 SFO & LAX: Model II 100 ACM Regression II 100 ACM Regression II Empirical Capacity Empirical Capacity Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 34 34

35 Methodology I Empirical Capacity Estimation SFO Quarter hourly data 35

36 Methodology I Empirical Capacity Estimation LAX Quarter hourly data Average Count Demand 36 VMC Departures VMC Arrivals IMC Departures IMC Arrivals

37 SFO: Model II 70 ACM regression results II 70 rs regression results II Tobit Capacity Tobit Capacity Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 37

38 LAX: Model II 100 ACM regression results II 100 rs regression results II Tobit Capacity Tobit Capacity Predicted Capacity (ACM) Predicted Capacity (rs) VMC Arrivals IMC Arrivals VMC Departures IMC Departures VMC Arrivals IMC Arrivals VMC Departures IMC Departures 38