Anomaly Detection in airlines schedules. Asmaa Fillatre Data Scientist, Amadeus

Transcription

1 Anomaly Detection in airlines schedules Asmaa Fillatre Data Scientist, Amadeus

2 AMADEUS PRESENTATION 1. IT company that develops business solutions for the travel and tourism industry 2. Operates globally in the travel and technology market Travel buyers Consumers/ General public Corporate travel departments Travel providers 711 airlines (over 420 bookable) 24 Insurance companies 50+ cruise and ferry lines 207 tour operators 110,000+ hotel properties 30 car rental companies 95 railways IT SOLUTIONS Including direct distribution technology Common / overlapping platforms & applications Common data centre Common customers Common sales & marketing infrastructure DISTRIBUTION BUSINESS Provision of indirect distribution services Travel agencies Travel Management companies Business travel agencies Leisure travel agencies Online travel agencies Consolidators Single-site agency Travel search companies Airline sale offices and airline websites connected to Amadeus direct sell technology Page 2

3 1 Airline Schedules

4 Airline schedules Flight Connections & Network Analysis Market Capacity Evolution and Trend Airline schedules Airport Hub Analysis Market Competition Analysis Page 4

5 Airline schedules data daily flights One year = flights Flight number Departure time Arrival time Aircraft type Departure airport Arrival airport Airline code Flight schedules Airline code Airline country Airlines Airport code Airport name Airport location Longitude latitude Airports Aircraft code Aircraft capacity Aircrafts 5

6 Motivations 1. The airline schedules contain many errors. 2. It is important to identify outliers prior to modelling and analysis. 3. Detect anomalies automatically 4. Overcome the issue of non prior knowledge (no ground truth) Page 6

7 Anomalies examples (1) Airlines use wrong IATA airport codes Airlines missing Merger between two companies Flown distance much higher than aircraft average Flown distance much higher than the aircraft average Elapsed time/distance not appropriate New routes traffic Sports event (OG, FIFA World Cup, etc) Sudden grow in monthly Aircraft capacity for United Airlines Page 7

8 Anomalies examples (2) Page 8

9 4 Unsupervised Anomaly detection Goal: Process unlabelled data and detect anomalies

10 Machine learning Labeled data (normal/abnormal) Direct feedback Predict outcome/future Supervised No labels No feedback Find hidden structure Unsupervised Learning Semisupervised Some labelled data : Supervised learning + additional unlabelled data Unsupervised learning + additional labelled data Page 10

11 Residuals-based anomaly detection in three steps Input data Low-rank approximation Residuals generation Anomaly detection Page No Anomaly Anomaly

12 Residual and Anomaly Detection Residual R i = Input Reconstrucion Residual normalization Residual thresholding Z i = (R i μ) σ Three sigma rule Any data sample outside the interval μ 3σ, μ + 3σ is considered to be potential anomaly Page 12

13 6 Deep learning: Stacked Autoencoder Goal: Learn the internal structure and features of the data itself

14 Autoencoder One hidden layer Minimize X X w.r.t. all W e (l), Wd (l) and be (l), bd (l) Trained with Backpropagation Self-supervised technique Learn a meaningful representation of the data in some other dimensionality where and Encoding Decoding Page 14

15 PCA Input W b + Output Autoencoder Introduce non linearity Input W b + W b +. Output Page 15

16 Regularization Deep Autoencoder or stacked autoencoder Cost function Average sum of squared error Weight decay Sparsity Penality Constraints on the activation ρ Regularization by λ which should be close to ρ Page 16

17 Stacked Autoencoder training Training one hidden layer at a time Example with 2 hidden layer Page 17

18 Input images Hello world of deep learning Anomaly Detection on MNIST Autoencoder lowest reconstruction error highest reconstruction error Learned features Output images Page 18

19 7 Autoencoder based Anomaly detection for airlines schedules

20 Normalized number of flights Raw data: multivariate time series airlines R2R nb flights weeks Preprocessing For more natural representations of data The Autoencoder can learn some patterns Some region to region time series 2012 week 10 Page 20

21 Autoencoder for time series Anomaly detection Data preparation Data transformation Autoencoder configuration TS preprocessing Data normalization Training set Train Autoencoder Testing set W, B 100 Reconstruction error thresholding β, λ and ρ Outlier detection Page 21

22 8 United Airlines (UA) schedules data processing Goal: highlight how does the Autoencoder perform in practice

23 UA anomaly detection (1) World normalization of Input data World Min Max Normalization. Autoencoder Outpuṯ Residuals R2Ri to 2016 Input from UA Page 23

24 UA anomaly detection (2) Regional normalization of Input data Min Max Normalization per region. Autoencoder Outpuṯ Residuals R2Ri to 2016 Input from UA Page 24

25 8 Air France (AF) Goal: highlight how does the Autoencoder perform in practice

26 AF anomaly detection (1) World normalization of Input data World Min Max Normalization. Autoencoder Outpuṯ Residuals R2Ri to 2016 Input from AF Page 26

27 AF anomaly detection (1) Regional normalization of Input data Min Max Normalization per region. Autoencoder Outpuṯ Residuals R2Ri to 2016 Input from AF Page 27

28 Autoencoder pros and cons Pros Cons Page 28

29 Conclusion Unsupervised machine learning (no ground truth) o Well adapted to the absence of labels o Hard to interpret: the review process of outliers relies on domain experts Deep learning/feature engineering Page 29

30 Thanks for your attention