Overview of the TREC 29 Entity track Krisztian Balog, Arjen P. de Vries, Pavel Serdyukov, Paul Thomas, Thijs Westerveld What is the track about? The Eighteenth Text REtrieval Conference (TREC 29) Gaithersburg, Maryland, USA, November 29. Information need Airlines that currently use Boeing 747 planes Airlines that currently use Boeing 747 planes Airlines that currently use Boeing 747 planes
Airlines that currently use Boeing 747 planes
Envisaged interface Aims for the track Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. www.jal.co.jp/en/ British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com Entity-oriented search on the Web Related entity finding is the first step KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. www.klm.com What is an entity? Related entity finding Entity = something with a homepage Given Input entity, defined by its name and homepage Type of the target entity (person, organization, or product) Narrative (describing the nature of the relation in free text) Return homepages of related entities Example An answer record <query> <num>7</num> <entity_name>boeing 747</entity_name> <entity_url>clueweb9-en5-75-2292</entity_url> <target_entity>organization</target_entity> <narrative>airlines that currently use Boeing 747 planes.</narrative> </query> HP Homepages of the entity (up to 3) WP Wikipedia page of the entity NAME String answer that represents the entity concisely SUPPORT Supporting documents (up to 1)
Example Related tasks 7-HP1 Q clueweb9-en11-75-3688 1.98 runid 7-HP2 Q clueweb9-en11-75-35827 1.98 runid 7-WP Q clueweb9-enwp1-93-13838 1.98 runid 7-NAME Q United_Airlines 1.98 runid SUPPORT... QA list questions They don t have a relationship component No homepages for entities Entity ranking @INEX Different task Only entities in Wikipedia No restrictions on entity types Informal reformulation of the task for TREC regulars Mixture of QA + Homepage finding Return homepages of answers to questions Introduction Topic development Judgments Results Approaches Outlook Outline Data collection Topic development Category B subset of ClueWeb "The hardest track I have ever developed topics for." (Ian Soboroff)
Topic development Main problem: No proper tools exist today Related problems Not many entities have homepage in CatB What is a homepage, anyway? Trivial topics QA list topics could not be reused Topic development Hunt for entities on the Web, then map them back to ClueWeb Burn rate: 1/5-1/1 Topics Assessment procedure Number of primary HPs per topic 4 3 2 1 6 Target type 3 Organization Person Product 11 Decide if HP{1,2,3} are primary or relevant Decide if WP is primary or relevant Decide if NAME is correct or inexact for something in the record Clustering pages and names that belong to the same entity Metrics Runs NDCG@R Normalized discounted cumulative gain at rank R R is the number of primaries and relevants for the topic P@1 Fraction of primary pages in the top 1 results #rel Number of relevant pages #pri Number of primary pages Max. 4 runs/team allowed 13 participating teams 41 runs 33 automatic / 8 manual
Results HP{1..3} Gain Contains primary 2 Contains relevant 1 All non-relevant Group Typ NDCG@R P1 #rel #pri Purdue Ae.361.235 126 61 uogtr A.2662.12 347 79 CAS A.213.235 8 7 NiCT A.197.155 99 64 UAms (Amst.) A.1773.45 198 19 TUDelft A.1351.95 18 42 BUPTPRIS M.892.15 48 3 UALR_CB A.666.2 15 4 UIUC A.575.1 64 3 Waterloo A.531.1 55 5 EceUdel A.488. 12 1 BIT M.416.2 81 9 UAms (ISLA) A.161. 3 1.4.3.2.1 Purdue uogtr BUPTPRIS TUDelft UAms (Amst) NiCT CAS Results NDCG@R P1 #rel #pri EceUdel Waterloo UIUC UALR_CB UAms (ISLA) BIT 4 3 2 1 Results HP{1,2,3} WP Group #rel #pri #rel #pri Purdue 126 61 3 9 uogtr 347 79 6 73 CAS 8 7 4 63 NiCT 99 64 6 67 UAms (Amst.) 198 19 63 2 TUDelft 18 42 32 8 BUPTPRIS 48 3 7 14 UALR_CB 15 4 UIUC 64 3 2 13 Waterloo 55 5 1 9 EceUdel 12 1 18 45 BIT 81 9 6 47 UAms (ISLA) 3 1 9 4 3 2 1 Purdue uogtr TUDelft UAms (Amst) NiCT CAS Results #rel HP #pri HP #rel WP #pri WP Waterloo UIUC UALR_CB BUPTPRIS UAms (ISLA) BIT EceUdel 9 67.5 45 22.5 A typical approach Additional approaches Input (entity, target type, relation) Query: entity or entity+relation Standard IR for doc.ret. Related entities Named entity detection Documents or document snippets Homepages of related entities Bootstrapping Wikipedia Name variant detection Exploiting lists and tables Using Google
Some dos and don ts What you should not do Take qrels and report results in your SIGIR paper Topic set is too small What you should do Look at results topic by topic What worked and what didn t eval_entity.pl num_ret 7 27 num_rel 7 125 rel_ret 7 15 num_pri 7 69 pri_ret 7 2 P1 7 1. ndcg_r 7.2422 eval_entity.pl -v Envisaged interface # 1 7-HP1 Q clueweb9-en3-3-2826 1.98742226523746 KMR1PU # 1 7-HP2 Q clueweb9-en1--6792 1.98742226523746 KMR1PU # 1 7-HP3 Q clueweb9-en-71-585 1.98742226523746 KMR1PU # 2 7-WP Q clueweb9-en1-5-2366 1.98742226523746 KMR1PU # 2 7-NAME Q Asiana_Airlines 1.98742226523746 KMR1PU # -------------------- Record gain: 3 # 1 7-HP1 Q clueweb9-en1-7-19878 2.98742226523746 KMR1PU # 2 7-WP Q clueweb9-enwp-29-2268 2.98742226523746 KMR1PU # 2 7-NAME Q Cargolux 2.98742226523746 KMR1PU # -------------------- Record gain: 3 # 1 7-HP1 Q clueweb9-en3-3-28249 3.9723293483488 KMR1PU # 1 7-HP2 Q clueweb9-en9-14-2389 3.9723293483488 KMR1PU # 1 7-HP3 Q clueweb9-en3-5-371 3.9723293483488 KMR1PU # 2 7-WP Q clueweb9-enwp-5-1128 3.9723293483488 KMR1PU # 2 7-NAME Q Aerolineas_Argentinas 3.9723293483488 KMR1PU # -------------------- Record gain: 3 # 1 7-HP1 Q clueweb9-en5-75-3866 4.966957444242745 KMR1PU # 1 7-HP2 Q clueweb9-en7-56-2812 4.966957444242745 KMR1PU # 2 7-WP Q clueweb9-enwp-14-11292 4.966957444242745 KMR1PU # 2 7-NAME Q Aerosur 4.966957444242745 KMR1PU Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. www.jal.co.jp/en/ British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. www.klm.com Entity track 21 Pilot task 1: summary generation Entity track 21 Pilot task 2: aspect identification Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. www.jal.co.jp/en/ British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. www.klm.com Japan Airlines Japan Airlines International Co., Ltd. is the flag carrier of Japan... The group has a fleet of 279 aircraft, consists of only Boeing aircraft for its long-haul operations; and Airbus, Boeing and McDonnell Douglas aircraft for its short-haul operations. www.jal.co.jp/en/ British Airways British Airways is the flag carrier airline of the United Kingdom. It is headquartered in Waterside near its main hub at London Heathrow Airport and is the largest airline in the UK based on fleet size, international flights and international destinations. ba.com KLM Royal Dutch Airlines KLM is the national airline of the Netherlands and is part of Air France-KLM. KLM operates worldwide scheduled passenger and cargo services to more than 9 destinations. www.klm.com
Questions? http://ilps.science.uva.nl/trec-entity/guidelines/plans-for-21/