Todd Keech CSC 600 Project Report Background Predicting Flight Delays Using Data Mining Techniques According to the FAA, air carriers operating in the US in 2012 carried 837.2 million passengers and the aviation industry supported 11.8 million jobs resulting in a 5.4 percent contribution to the US gross domestic product. Despite positive contributions to the US economy, nearly all commercial airlines experience flight delays and cancellations which can negatively impact economic growth. A 2010 study by researchers at University of California, Berkley concluded that flight delays, cancellations and missed connections cost the US economy approximately $32.9 billion resulting in a $4 billion reduction to the US gross domestic product. Researchers found that $16.7 billion of the $32.9 billion cost is borne by passengers in the form of lost time and additional expenses such as food and lodging. Problem The purpose of this project is to use publicly available data and data mining techniques to construct an analytical model to predict flight delays based on flight attributes such as origin, destination, date/time, distance, etc. Additional models will be created to determine the most likely cause of a flight delay and to predict the approximate length of the delay. Motivation There a number of practical uses for flight delay modeling. The FAA and commercial carriers can use flight delay analytical models to identify the most frequently congested network paths and determine appropriate solutions to address the most common delay causes. Commercial carriers can use predictive models as an input into scheduling optimization algorithms which seek to minimize aircraft ground-time and maximize flight time. Retailers and consumers can also benefit from predictive flight delay modeling. Retailers such as Kayak and Travelocity may provide flight analysis information to end consumers as a means to empower informed decision-making. For example, a customer booking a flight on Kayak may reconsider travel arrangements based on the predicted likelihood of a delay. Retailers looking to increase their customer base can offer information services to enable consumer to make better decisions. Consumers can use this information to avoid the costs and inconvenience of flight delays. Fewer flight reservations across congested paths will force carriers to adjust prices downward or discontinue such flights resulting in greater efficiency in the overall airline system. Data Data for this project is available on the Bureau of Transportation Statistics (BTS) website. The BTS On- Time Performance database contains flight information, including scheduled and actual arrival times, as reported by US commercial airlines in CSV-formatted data files. Each row in the data set represents an individual flight. Data files are available by month dating back to 1987. Each monthly file contains approximately 500,000 observations. The following attributes are included in each observation: Flight date Quarter Month Carrier Flight Distance CRS elapsed time
CRS arrival time CRS departure time Day of month Day of week Origin Destination Destination city Destination state Origin city Origin state Arrival delay CARRIER_DELAY LATE_AIRCRAFT_DELAY NAS_DELAY WEATHER_DELAY SECURITY_DELAY In the BTS data set, flights are characterized as on-time or delayed based on the arrival delay attribute. This attribute contains the total number of delay minutes relative to the scheduled arrival time. The data set also contains five delay category attributes that quantify the reason for the delay: CARRIER_DELAY Delays due to circumstances under the carrier s control (mechanical, administrative, etc.) LATE_AIRCRAFT_DELAY The aircraft scheduled to service the route was late in arriving NAS_DELAY National Airline System delays (volume, air traffic, control, etc.) SECURITY_DELAY Security-related incidents resulting in a late departure or arrival WEATHER_DELAY Weather-related delays Each of these fields contains the total number of delay minutes for each category. Since a flight can be delayed for multiple reasons, Arrival delay is the summation of the five delay categories. In some cases, the delay cause is not reported. In addition to the On-time Performance data set, this study utilized two additional data sets. The BTS holiday travel data set is used to identify peak travel periods based on airline industry standards. The NOAA storm events database dataset is used to identify severe weather periods. Peak travel periods and sever weather are both used as predictors for determining flight delays, delays lengths and delay causes. Preprocessing The BTS On-Time Performance data contains over 10 million observations based on flight data dating back to 1987. To limit the amount of data used in this study, observations were limited to 2014 domestic flight data, including flights to US territories. This project also limits the study area to flights arriving or departing Philadelphia International Airport (PHL) via Southwest Airlines. The rationale behind targeting a narrow band of locational and carrier-specific flight data was to isolate local delay patterns with the intention of improving the accuracy of the models. A data set covering a large geographic area and multiple carriers may be more generalizable, but local delay patterns may be lost in the noise resulting in a less accurate model. Preprocessing consisted of the following steps: Create a binary field, IS_DELAYED, to indicate whether or not a flight was delayed based on the arrival delay attribute Map NOAA severe weather data to flight data based on flight date. The end result was a binary indicator, SEVERE_WEATHER, for each flight record that specified whether or not the flight
occurred on a severe weather day. This study included severe weather that occurred in the greater Philadelphia area. Map peak travel data to flight data based on flight date. The end result was a binary indicator, PEAK_TRAVEL, for each flight record that specified whether or not the flight occurred on a peak travel day Create a categorical attribute, DELAY_TYPE, to capture the leading flight delay cause. o DELAY_TYPE = max(carrier_delay, LATE_AIRCRAFT_DELAY, NAS_DELAY, SECURITY_DELAY, WEATHER_DELAY). o In the event of a tie, the delay type was chosen randomly. o In some instances, all values for CARRIER_DELAY, LATE_AIRCRAFT_DELAY, NAS_DELAY, SECURITY_DELAY, WEATHER_DELAY were 0 even though the Arrival delay field was non-zero. In these instances, DELAY_TYPE= UNKNOWN. Create a new fields to capture the CRS arrival hour and CRS departure hour based on the CRS arrival time and CRS departure time fields After preprocessing, the total number of observations used in the study was 17,477. The data was randomly sampled to produce a training set containing 13,107 observations and a test set containing 4,370 records. Data Exploration Exploration of the training set revealed that 47.2% of Southwest flights destined for or originating from PHL were delayed in 2014. Further exploration of the data revealed that flight date and flight origin may be good predictors of flight delay given the difference in overall delay percentage compared to the population.
Examination of flight delay length revealed that the majority of flight delays are relatively short. 61.9% of delays were less than 30 minutes in length. The following graph shows the breakdown of flight delay reasons. Learning Algorithm Summary Predicting Flight Delays This project employed the use of several different machine learning techniques to predict whether or not a flight would be delayed. The accuracy, precision and recall scores for each model are given in the table below. Method Accuracy % Precision Recall Simple 53.0 0 0 GLM 66.1 64.8 61.4 Tree 65.0 61.2 70.2 Bagging 67.5 66.0 63.6 AdaBoost 67.3 65.6 64.5 Random Forest 65 61.6 68 Naïve Bayes 66.2 64.1 64.3 K-Nearest Neighbor 70 68.6 71.4 The Simple model predicted no flights would be delayed resulting in a 53% accuracy rate on the test data set. The simple predictor is used as a reference for comparing more complicated models. Based on data exploration, the Naïve Bayes model was hypothesized to produce the best model while GLM was hypothesized to produce the lowest quality model. The author presumed that a non-linear
model between flight delays and the predictor variables would produce the best-fit. Overall, the linear and non-linear models performed similarly with accuracy rates ranging between 65% and 70%. The ROC curve comparison between the GLM model and the Naïve Bayes model actually indicates that the GLM model is a slightly better fit since the area under the curve is greater by.003. Surprisingly, the k-nearest neighbor model performed slightly better than other models in terms accuracy rate and precision and recall scores. The k-nearest neighbor model used the nearest three neighboring points (Euclidean distance) to determine whether or not a flight would be delayed. The accuracy of the model may be explained due to clustering of the flight date, departure time and origin attributes. The Adabag R package contains an importance function which graphs the relative importance of each variable in the model. The importance function used the decrease in GINI as the metric to determine the impact of a variable on the model. Based on the importance graph, flight date and departure hour are the most important predictors in the model. Removing predictors such as severe weather and peak travel did not impact the model significantly. Predicting Flight Delay Length The Random Forest algorithm produced the best-fitting model for predicting the length of a flight delay. Logistic regression resulted in a model with an adjusted R-squared of 20, meaning that the model could only explain 20% of the variability in flight delay length based on the predictors. Similarly, a standard rpart regression tree resulted in a high RSE relative to the mean error indicating a poorly fit model. Method RSE Mean Error Adj R-Squared Linear 38.5 13.5 20
Regression Tree 35.7 13.35 NA Random Forest 26.81 9.75 NA Predicting Flight Delay Reason The Random Forest model produced an accuracy rate of 55% for predicting the cause of a flight delay. K- nearest neighbor was slightly less accurate with a 54% accuracy rate. A simple model that predicted no flight delays had an accuracy rate of 52%. Method Accuracy % Simple 52 Random Forest 55 K-Nearest Neighbor 54 Conclusion In this study, flight attributes were used to predict flight delays, flight delay lengths and the flight delay causes. Several non-flight attributes such as weather and peak travel data were also used in each model, although removing these predictors did not change the overall model drastically. Flight delay models resulted in 65-70% accuracy rates. Flight delay length and delay type models were far less accurate. One of the goals of this study was to create a forward looking flight prediction model. The models that resulted in the highest accuracy rates relied on flight date as an important predictor. Flight date, however, is a backward looking predictor in that it relies on historical events to predict future events. When flight date was removed from the model, accuracy rates fell by 4-5%. Extension The commercial aviation system is complex. Events occurring in one area of the system can cause ripple effects that impact events in other areas. Consequently, a model that relies solely on flight attributes will fail to account for complex interactions across the system. As an extension to this study, future researchers may choose to include several additional pieces of data to account for these factors: Weather This study used local Philadelphia weather to predict flight delay impacts at PHL. Inclusion of weather conditions at the origin, destination and along the flight path may better capture the impacts of weather. Aircraft age Aircraft age might be used to determine the likelihood of a mechanical issue as the cause of a delay. System and airport congestion Airport capacity metrics and scheduled flight data may be used to calculate expected system and airport congestion level as a means of predicting delays. Project Data Sources Bureau of Transportation Statistics, On-Time Performance Database, http://www.transtats.bts.gov/dl_selectfields.asp?table_id=236&db_short_name=on-time Bureau of Transportation Statistics, On-Time Performance Database, http://www.transtats.bts.gov/holidaydelay.asp National Oceanic and Atmospheric Administration, Storm Events Database, http://www.ncdc.noaa.gov/stormevents/ References Flight delays cost $32.9 billion, passengers foot half the bill, University of California, Berkley, October 2010, http://newscenter.berkeley.edu/2010/10/18/flight_delays/
The Impact of Civil Aviation on the US Economy, Federal Aviation Administartion, June 2014, https://www.faa.gov/air_traffic/publications/media/2014-economic-impact-report.pdf