Fusion of Flight Data with Social Media

Transcription

1 Fusion of Flight Data with Social Media STO-MP-IST-160-W Dr. Axel Tanner IBM Research Zurich Säumerstrasse 4 CH-8803 Rüschlikon Dr. Albert Blarer armasuisse Science + Technology Feuerwerkerstrasse 39 CH-3602 Thun Dr. Vincent Lenders armasuisse Science + Technology Feuerwerkerstrasse 39 CH-3602 Thun

2 Datasources Semantic Gap ADS-B & Mode S data generates StateVectors icao24, callsign : identity time : time lat, lon, : position, baro-/geoaltitude direction velocity, heading, vertrate squawk : status Big Data aspects high volume high velocity high veracity low variety opensky-network.org Semantic gap plane identity: icao24 registration 3443d5 EC-LVU flight identity: callsign VLG7985 flightno VL7985 Twitter twitter.com Users and activity 100M active users/day estimated 23M bots 9% of active users (2014) 500M day 6000/sec! all topics... ambient awareness, but often informal and very noisy 1% stream freely accessible per API Big Data aspects high volume high velocity low veracity high variety 2 specific, detailed, explicit & concrete problem coverage, also somewhat noisy (not all squawks are created equal) very broad range extremely noisy Motivation see whether combination of data sources enhances finding and understanding of events

3 Datafusion Flight data & Twitter ADS-B/Mode S 2 anomalous flight path background data * Twitter 1 squawk /7700 OpenSkyAlerts icao24 callsign previous/current flight data flight number start/destination geo coordinates icao24 callsign flight number callsign Information extraction Tweets 3 plane diverted flight overdue... Datafusion selected terms medical, technical,... Situation Situation extended with context (flight path, start/destination, Twitter-comments) annotated (geo, medical emergency ) Filter alerting 3 *various (paid and free) services exist, e.g. flightaware.com, flightrader24.com; DBpedia

4 Datafusion flight data & Twitter: Trigger Squawk After a Marseille medical diversion #VY just into Zurich 2hrs later

5 Understanding tweets creating annotations explicit matching of airport codes (extracted from dbpedia). Caveat: e.g. Anderson Regional Airport, NC, US = AND flight numbers (per regex) Caveat: general scheme for flight numbers is quite general false positives some special terms (like medical, technical, radio, emergency, Notfall,...) Named Entity Recognition with DBpedia-spotlight * annotations (de/en) summarization of tweets removing repeated information ranking top-tweets via a scheme based on word-frequencies using the NLTK ** package giving higher weight to recognized terms 5 * ** Natural Language Toolkit (NLTK)

6 Datafusion flight data & Twitter: Trigger Squawk After a Marseille medical diversion #VY just into Zurich 2hrs later

7 Twitter as main trigger source Testing with Tweet searches for english #planecrash (flight OR plane) delayed (flight OR plane) diverted (flight OR plane) overdue... german #absturz abgestürzt (flug OR flugzeug) umgeleitet (flug OR flugzeug) verspätet (flug OR flugzeug)... Semantic gap requires to get to concrete flight from United Airlines from Chicago to Hong Kong diverted to Anchorage 7

8 Datafusion flight data & Twitter: Trigger Woman discovers husband s affair while checking his phone on QR s DOH-DPS flight... went crazy shouting, hitting..... why has flight qr962 been diverted to chennai and has it landed yet?

9 Summary data sources have very different characteristics & semantic gap good: data from multiple sources can enrich each other trigger from one source could lead to additional data collection (where complete data is impossible to gather, e.g., Twitter) more complete picture allows better filtering downstream future work more sophisticated Twitter event analysis/understanding collective flight path anomalies other data sources to fuse Examples show that combination is interesting but G-WACG G-JAMM trigger OpenSky alerts semantic ambiguity (flight number TN8 = TV channel in Nicaragua) precise conclusions are difficult to extract from Twitter data trigger OpenSky flight path anomaly individual flight paths show incidents very seldom trigger Twitter streams too many/overlapping events (Twitter flight overdue ) information extraction from natural (resp tweet ) language difficult