Overview of the TREC 2009 Entity track. What is the track about? Information need. Airlines that currently use Boeing 747 planes

Transcription

1 Overview of the TREC 29 Entity track Krisztian Balog, Arjen P. de Vries, Pavel Serdyukov, Paul Thomas, Thijs Westerveld What is the track about? The Eighteenth Text REtrieval Conference (TREC 29) Gaithersburg, Maryland, USA, November 29. Information need Airlines that currently use Boeing 747 planes Airlines that currently use Boeing 747 planes Airlines that currently use Boeing 747 planes

2 Airlines that currently use Boeing 747 planes

3 Envisaged interface Aims for the track Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com Entity-oriented search on the Web Related entity finding is the first step KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. What is an entity? Related entity finding Entity = something with a homepage Given Input entity, defined by its name and homepage Type of the target entity (person, organization, or product) Narrative (describing the nature of the relation in free text) Return homepages of related entities Example An answer record <query> <num>7</num> <entity_name>boeing 747</entity_name> <entity_url>clueweb9-en </entity_url> <target_entity>organization</target_entity> <narrative>airlines that currently use Boeing 747 planes.</narrative> </query> HP Homepages of the entity (up to 3) WP Wikipedia page of the entity NAME String answer that represents the entity concisely SUPPORT Supporting documents (up to 1)

4 Example Related tasks 7-HP1 Q clueweb9-en runid 7-HP2 Q clueweb9-en runid 7-WP Q clueweb9-enwp runid 7-NAME Q United_Airlines 1.98 runid SUPPORT... QA list questions They don t have a relationship component No homepages for entities Entity Different task Only entities in Wikipedia No restrictions on entity types Informal reformulation of the task for TREC regulars Mixture of QA + Homepage finding Return homepages of answers to questions Introduction Topic development Judgments Results Approaches Outlook Outline Data collection Topic development Category B subset of ClueWeb "The hardest track I have ever developed topics for." (Ian Soboroff)

5 Topic development Main problem: No proper tools exist today Related problems Not many entities have homepage in CatB What is a homepage, anyway? Trivial topics QA list topics could not be reused Topic development Hunt for entities on the Web, then map them back to ClueWeb Burn rate: 1/5-1/1 Topics Assessment procedure Number of primary HPs per topic Target type 3 Organization Person Product 11 Decide if HP{1,2,3} are primary or relevant Decide if WP is primary or relevant Decide if NAME is correct or inexact for something in the record Clustering pages and names that belong to the same entity Metrics Runs NDCG@R Normalized discounted cumulative gain at rank R R is the number of primaries and relevants for the topic P@1 Fraction of primary pages in the top 1 results #rel Number of relevant pages #pri Number of primary pages Max. 4 runs/team allowed 13 participating teams 41 runs 33 automatic / 8 manual

6 Results HP{1..3} Gain Contains primary 2 Contains relevant 1 All non-relevant Group Typ NDCG@R P1 #rel #pri Purdue Ae uogtr A CAS A NiCT A UAms (Amst.) A TUDelft A BUPTPRIS M UALR_CB A UIUC A Waterloo A EceUdel A BIT M UAms (ISLA) A Purdue uogtr BUPTPRIS TUDelft UAms (Amst) NiCT CAS Results NDCG@R P1 #rel #pri EceUdel Waterloo UIUC UALR_CB UAms (ISLA) BIT Results HP{1,2,3} WP Group #rel #pri #rel #pri Purdue uogtr CAS NiCT UAms (Amst.) TUDelft BUPTPRIS UALR_CB 15 4 UIUC Waterloo EceUdel BIT UAms (ISLA) Purdue uogtr TUDelft UAms (Amst) NiCT CAS Results #rel HP #pri HP #rel WP #pri WP Waterloo UIUC UALR_CB BUPTPRIS UAms (ISLA) BIT EceUdel A typical approach Additional approaches Input (entity, target type, relation) Query: entity or entity+relation Standard IR for doc.ret. Related entities Named entity detection Documents or document snippets Homepages of related entities Bootstrapping Wikipedia Name variant detection Exploiting lists and tables Using Google

7 Some dos and don ts What you should not do Take qrels and report results in your SIGIR paper Topic set is too small What you should do Look at results topic by topic What worked and what didn t eval_entity.pl num_ret 7 27 num_rel rel_ret 7 15 num_pri 7 69 pri_ret 7 2 P ndcg_r eval_entity.pl -v Envisaged interface # 1 7-HP1 Q clueweb9-en KMR1PU # 1 7-HP2 Q clueweb9-en KMR1PU # 1 7-HP3 Q clueweb9-en KMR1PU # 2 7-WP Q clueweb9-en KMR1PU # 2 7-NAME Q Asiana_Airlines KMR1PU # Record gain: 3 # 1 7-HP1 Q clueweb9-en KMR1PU # 2 7-WP Q clueweb9-enwp KMR1PU # 2 7-NAME Q Cargolux KMR1PU # Record gain: 3 # 1 7-HP1 Q clueweb9-en KMR1PU # 1 7-HP2 Q clueweb9-en KMR1PU # 1 7-HP3 Q clueweb9-en KMR1PU # 2 7-WP Q clueweb9-enwp KMR1PU # 2 7-NAME Q Aerolineas_Argentinas KMR1PU # Record gain: 3 # 1 7-HP1 Q clueweb9-en KMR1PU # 1 7-HP2 Q clueweb9-en KMR1PU # 2 7-WP Q clueweb9-enwp KMR1PU # 2 7-NAME Q Aerosur KMR1PU Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. Entity track 21 Pilot task 1: summary generation Entity track 21 Pilot task 2: aspect identification Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations.

8 Questions?