Airborne Validation Results Part A

Transcription

1 EUROPEAN AIRPORT MOVEMENT MANAGEMENT BY A-SMGCS, Part 2 Contract No. TREN/04/FP6AE/S / Airborne Validation Results Part A Thomas Wittig FAV Document No: 2-D6.6.1a Version No Classification: Public Number of pages: 142 Project Funded by European Commission, DG TREN The Sixth Framework Programme Strengthening the competitiveness Contract No. TREN/04/FP6AE/S / Project Manager M. Röder Deutsches Zentrum für Luft und Raumfahrt Lilienthalplatz 7, D Braunschweig, Germany Phone: +49 (0) , Fax: +49 (0) fp6-emma@dlr.de Web page: , - All rights reserved - EMMA Project Partners The reproduction, distribution and utilization of this document as well as the communication of its contents to other without explicit authorization is prohibited. This document and the information contained herein is the property of Deutsches Zentrum für Luft- und Raumfahrt and the EMMA project partners. Offenders will be held liable for the payment of damages. All rights reserved in the event of the grant of a patent, utility model or design. The results and findings described in this document have been elaborated under a contract awarded by the European Commission.

2 EMMA2 Airborne Validation Results Part A Distribution List Member Type No. Name POC Distributed 1 Web Internet Intranet 1 DLR Joern Jakobi 2 AENA Guilio Maira 3 AIF Marianne Moller 4 SELEX Giuliano D'Auria 5 ANS_CR Jan Kubicek 8 DSNA Philippe Montebello 9 ENAV Antonio Nuzzo 10 NLR Jürgen Teutsch 11 PAS Alan Gilbert 12 TATM Corinne Heinrich Contractor 13 THAV Marc Fabreguettes 15 AUEB Konstantinos G. Zografos 16 CSL Prague Airport Libor Kurzweil 17 DAS Joachim Bader 18 DFS Klaus-Ruediger Täglich 19 EEC Stéphane Dubuisson 20 ERA Jan Hrabanek 21 ETG (FAV) Thomas Wittig 23 SICTA Mariacarmela Supino 24 TUD Carole Urvoy 25 SOF Lionel Bernard-Peyre Sub-Contractor CSA Karel Muendel Körte Max Körte Customer EC Doris Schroecker Additional EUROCONTROL Bengt Collin 1 Please insert an X, when the PoC of a company receives this document. Do not use the date of issue! Save date: Public Page 2

3 EMMA2 Airborne Validation Results Part A Document Control Sheet 2-SP6 Project Manager Marcus Biella Responsible Author Thomas Wittig FAV Additional Authors Marcus Biella Jörn Jakobi Thomas Ludwig Carsten Wehrstedt Christian Drege Carole Urvoy Ronny Friebel Subject / Title of Document: Airborne Validation Results Part A Related Task('s): 2-WP6.3 Deliverable No. 2-D6.6.1a Save Date of File: Document Version: 1.00 Reference / File Name 2-D661a_VO-TR_V1.0.doc Number of Pages 142 Dissemination Level Public Target Date DLR DLR DLR DLR TUD TUD FAV Change Control List (Change Log) Date Release Changed Items/Chapters Comment Structure by DLR Initial draft: Results for RTS Pt.1 in GECO (Validation of GTD and TAXI- CPDLC) by DLR Addition of pilot comments Results for RTS Pt.2 in GECO (Validation of GTD and TAXI- CPDLC) by DLR Update of DLR and TUD results No EC comments. Approved Final Version 1.0 Save date: Public Page 3

4 EMMA2 Airborne Validation Results Part A Table of Contents Distribution List... 2 Document Control Sheet... 3 Change Control List (Change Log)... 3 Table of Contents Executive Summary Introduction Scope of Document Validation Approach Schedule of 2-WP6.6 Activities Ground Traffic Display & TAXI-CPDLC Trials (DLR) Real Time Simulation Results Set Up Results to Operational Feasibility (RTS) Results to Operational Improvements (RTS) Operational Flight Trials Results Set Up Results to Technical Feasibility (OST) Surface Movement Awareness and Alerting System SMAAS; Traffic Conflict Detection; Ground- Air Data Base Upload (TUD) Real Time Simulation Results Set Up Results to Operational Feasibility (RTS) Results to Operational Improvements (RTS) Operational Flight Trials Results Set Up Technical and Operational approval (FAV / TUD trials) Results to Technical Feasibility (FAV Trials) Results to Operational Feasibility (OST) Conclusions Conclusions to the new A-SMGCS services Conclusions to Operational Improvements Annex I References Abbreviations List of Figures List of Tables Annex II Raw Data (DLR RTS) QE-OF QE-OI (Safety) QE-OI (Capacity/Efficiency) SUS ASSA Save date: Public Page 4

5 EMMA2 Airborne Validation Results Part A 1 Executive Summary This document reports the results of the A-SMGCS airborne validation activities carried out in EMMA2 WP6.6 by the green team (DLR, TUD, FAV, and DAS). There were four test campaigns: Real time simulations (RTS) o at DLR facilities (in June and October 2008) o at TUD facilities (in July 2008) Prague Airport on-site trials (OST) o by TUD & FAV (in August 2008) o by DLR & TUD (in November 2008) During the airborne validation trials the following EMMA2 higher A-SMGCS components were implemented in different platforms and under evaluation. Ground Traffic Display (GTD) o RTS (DLR platform, TUD platform) o OST (DLR platform, TUD platform) Ground Air Data Base Upload o RTS (TUD platform) Surface Movement Alerting (SMA) o RTS (TUD platform) o OST (TUD platform) Traffic Conflict Detection (TCD) o RTS (TUD platform) o OST (fed by TIS-B) (TUD platform) TAXI-CPDLC o RTS (DLR platform) o OST (DLR platform) They were operated in different trials in total by twenty different pilots. Additionally, a tower simulation with ANS_CR controllers was linked to the DLR cockpit simulator during the DLR trials, and at the on-site trials, the DLR test aircraft ATTAS enabled a comprehensive and realistic testing of the TAXI-CPDLC service carried out in the test bed room of the Prague Tower in shadow mode. Generally speaking, the overall system provided a medium maturity and could fulfil the high operational expectations. The pilots approved its operational feasibility by accepting most of the operational requirements and new procedures. Operational improvements were measured. Subjectively the pilots rated significantly that - GTD is regarded as potential means for increased safety which eases pilots work considerably. With its indication of the surrounding traffic the GTD is an additional information source for the pilots to navigate and to manage the aircraft speed more safely and efficiently. - GTD significantly increases the pilots situation awareness (by allowing an intuitive use, providing the appropriate amount of information, and as an extraordinary aid in locating traffic) especially under low visibility conditions. - Pilots workload is not affected negatively when using the GTD. - Furthermore, pilots report that the use of a GTD will improve their performance. - TAXI-CPDLC is regarded as potential means for increased safety which will ease pilots work considerably with appropriate input devices. The graphical presentation of the cleared Save date: Public Page 5

6 EMMA2 Airborne Validation Results Part A taxi route on the EMM (compared to textual clearances only) will enable pilots to follow an assigned taxi route more safely and efficiently. - An increase of workload while using the TAXI-CPDLC service could be observed and was explained by the pilots due to the fact that a novel and not-well designed input device was used. - Situation Awareness was not affected by the use of TAXI-CPDLC. - The Ground to Air Database Upload function is regarded as potential means for increased safety which will ease pilots work especially in large airports. - The Ground to Air Database Upload function significantly increases the pilots situation awareness by enabling the loading of and the interaction with electronic Preflight Information Bulletin Data (e.g. NOTAM information). - Pilots workload is not affected negatively when using the Ground to Air Database Upload function. - The SMA and the TCD are regarded as potential means for increased safety as they will diminish the risks of accidents and incidents on the surface movement area especially in low visibility conditions. Save date: Public Page 6

7 EMMA2 Airborne Validation Results Part A 2 Introduction 2.1 Scope of Document This document is positioned within the framework of activities for the European airport Movement Management by A-SMGCS, part 2 (EMMA2) project. Based on the technical annex, the sub-project 2-SP6 deals with the validation A-SMGCS services implemented at four ground sites (Paris Charles de Gaulle, Prague-Ruzynĕ, Toulouse-Blagnac, and Milano-Malpensa) and the airborne site. Specific validation test plans and test reports were developed at each site of the project, presenting the sitespecific validation process and strategy and their respective results. This test report bases on 2-D6.1.6, Validation Test Plan Airborne [6] and follows the E-OCVM guidelines [1]. In accordance to the EMMA2 High Level Objectives, which were identified in the 2- D6.1.1a Validation Plan [5], it covers following aspects. 1. Operational Feasibility 2. Operational Improvements 3. Additional Results 4. Conclusions. This document reports the results of the airborne validation activities carried out in WP6.6 by the green team (DLR, TUD, FAV, and DAS). It comprises results from real time simulations (RTS) and on-site trials (OST) at Airport Prague with Pilots from DLH, CSA, and Germania (and ATCOs from ANS CR), by support of the industry DAS, FAV, Airtel ATN, PAS and the R&D enterprises EEC, TUD and DLR. There were two DLR test campaigns: RTS in June and October 2008 at DLR premises, and the on-site trials at Airport Prague in November RTS1 focussed exclusively on the technical and operational feasibility of the new A-SMGCS services. Results were mainly used to improve the services for the on-site trials. In RTS the operational feedback of the pilots was compiled and is documented in this report. In DLR RTS a Tower simulation environment was linked with cockpit simulator with commercial pilots in order to get more realistic results to the interaction between ATCOs and pilots supported by new A-SMGCS services like TAXI-CPDLC and on-board ground traffic displays (GTD). Results compiled by the tower simulation (incl. feedback of the pilots) can be found in a separate EMMA2 document (2-D6.3.1, [13]); the present document focuses on the pilot perspective exclusively. There were two TUD test campaigns: RTS in July 2008 at TUD premises, and the on-site trials at Airport Prague in August 2008 with FAV and November Both campaigns focussed on the technical and operational feasibility of the SMA, Ground Air Data Base upload and TCD. On-site trials were used to prove technical and operational feasibility aspects in real life that could not be assessed in a simulated environment. The details to the planning of those test campaigns can be found in the EMMA2 SP6 Validation Test Plan Airborne (2-D6.1.6) [6] and Validation Test Plan Prague (2-D6.1.3) [15]. This document describes the results achieved and derived conclusions but also repeats some core aspects of the test plan [6] to make the document readable as a stand alone document. The document delivers important input to the overall EMMA2 2-D6.7.1 Validation Comparative Analysis Report [14], which will provide an overall analysis of all EMMA2 validation results. Save date: Public Page 7

8 EMMA2 Airborne Validation Results Part A 2.2 Validation Approach During the proposal phase of EMMA2, it was decided to use the Master European Validation Plan (MAEVA) project approach to validation as the basis for EMMA2 validation activities. With the start of EMMA2 it was decided to use E-OCVM. The E-OCVM constitutes a widely accepted Validation Methodology which has been based on the experience gained in ATM R&D activities within the framework of EC and EUROCONTROL funded projects [1]. In EMMA2 the E-OCVM approach has been applied and adapted to the EMMA2 needs. The EMMA2 Validation Plan [5] and the Generic Experimental Test Plan [4] are the outcome documents of this process. They are the valid generic validation plans for all EMMA2 validation activities (except the validation activities by THAV and AIF, the so-call blue team ). Based on those generic EMMA2 validation documents the specific Validation Test Plan Airborne [6] has been produced to plan all validation activities carried out in SP6 WP6.6 by the green team and THAV of the blue team. 2.3 Schedule of 2-WP6.6 Activities 2-WP6.6 of EMMA2 focuses on airborne A-SMGCS validation activities, implemented by the green team (DLR, TUD, FAV, and DAS) in the sub-project 2-SP2 of EMMA2. Here, two validation platforms will be used: Real-time simulation platform, to simulate safety-critical events, validate A-SMGCS procedures, and measure operational improvements in a realistic environment On-site trials with a test aircraft or test vehicle, to validate requirements and procedures in the real operational environment and show some potential operational improvements. The series of tests started with real-time simulations at DLR-Braunschweig s Cockpit Simulator, concentrating on Ground Traffic Display (GTD), and TAXI CPDLC. Here the whole system was to be validated, by measuring operational feasibility and operational improvements (safety and human factors issues in the first place). These real-time simulations were a preparatory step for the flight trials with the ATTAS at Prague Airport, which are aimed at validating the higher A-SMGCS services under real operational conditions. It should be borne in mind that even though simulation re-creates a realistic environment, and permits the safe testing of safety-critical events and repetitive testing of rare events, at the end operational trials must be performed to prove the operational feasibility of new services and procedures. Save date: Public Page 8

9 EMMA2 Airborne Validation Results Part A 3 Ground Traffic Display & TAXI-CPDLC Trials (DLR) 3.1 Real Time Simulation Results Set Up The validation platform DLR s Generic Experimental Cockpit (GECO) is designed as a fixed-base flight simulator based on the Airbus A 320 architecture using the flight dynamics of the VFW614 (in accordance to DLR s test aircraft ATTAS) in a fly by wire version (Figure 3-1). It is capable for testing and evaluating new onboard systems giving a maximum of flexibility. Figure 3-1: DLR s Generic Experimental Cockpit (GECO) The simulator is equipped with a collimated outside view and widespread hardware equipment (Figure 3-2): Two seated-flightdeck including Primary Flight Display (PFD) und Navigation Display (NAV) (providing HSI during in flight and EMM during taxiing) at both sides for CM1 and CM2; and Engine Display (ENG) Several components of the Flight Management System (FMS; with RAD/NAV page in use during EMMA2 trials) including Flight Control Unit (FCU) and Multipurpose Control and Display Unit (MCDU) HMI-Interfaces (e.g. trackball, touchpad, touchscreen) Head-Up Display (HUD) High-Resolution Airport Environment Save date: Public Page 9

10 EMMA2 Airborne Validation Results Part A Figure 3-2: PFD, NAV, ENG and FCU in GECO The GECO is part of an integrated network; it can be used in co-operation with other simulation platforms e.g. Apron & Tower Simulator (ATS) or Air Traffic Management and Operations Simulator (ATMOS). Further equipment can be connected via ISDN or internet. The communication between the GECO and ATS is realised via the TCP/IP protocol. Experimental conditions and weather parameters will be coordinated and controlled by the instructor. The architecture between the cockpit and the tower simulator is shown in Figure 3-3. Save date: Public Page 10

11 EMMA2 Airborne Validation Results Part A Figure 3-3: Architecture of Cockpit & Tower Simulation, Pseudo Pilots and DMAN Save date: Public Page 11

12 EMMA2 Airborne Validation Results Part A Experimental Design The experimental design is based on the use of real experiments. In real experiments, the same scenarios are used for the Baseline System and the A-SMGCS set-up in order to achieve ceteris paribus conditions. In this way, results from the Baseline system can be directly compared with the A-SMGCS within a traffic scenario. For TAXI-CPDLC different scenarios will be used. Therefore no direct experimental comparison can be made. Nevertheless comparisons will be drawn and considered as well and the results are used to check the operational feasibility initially since the maturity level was not to be expected to be V3 (in accordance to E-OCVM [1], and to give first hints (!) for operational improvements. The independent variable (IV) for on-board activities is therefore defined as follows: [1] IV SYS the system version (EMM vs. higher A-SMGCS services) with four levels of the treatment A-SMGCS Services : Level 1 Baseline (EMM), Level 2 GTD, (EMM + GTD) [Level 3 TAXI-CPDLC standard (EMM + GTD + standard TAXI-CPDLC)] [Level 4 TAXI-CPDLC advanced (EMM + GTD + advanced TAXI-CPDLC)] The independent variable is a treatment factor that is operated by the experimenter and is supposed to cause expected effects. It is described and argued in the table 3-1. The second independent variable is the role of the pilot in the cockpit. He will act in half of the scenarios as CM1; these scenarios are later repeated to have more statistical power, and then he will act as CM2 (and vice versa for the other half of the scenarios). [2] IV ROLE subjects role with two levels (CM1, resp. CM2). Independent Variables (IV) Description IV SYS System Version= Technical System varied by different functions and their provided services Levels This variable has four different technical system versions corresponding to an A-SMGCS (= EMM) and A-SMGCS higher services: Level 1 EMM (baseline) Level 2 EMM + GTD Level 3 EMM + GTD + standard TAXI-CPDLC Level 4 EMM + GTD + advanced TAXI-CPDLC IV ROLE Role of Subject This variable has two different levels. Pilots will act as: CM1 CM2 Table 3-1: Independent variable and levels in RTS TAXI-CPDLC Procedures Standard TAXI-CPDLC: Procedures Outbound Given below is a TAXI-CPDLC example for an outbound flight for RWY06. Save date: Public Page 12

13 EMMA2 Airborne Validation Results Part A The aircraft is parked at apron north (Gate 34) RWY06 has been assigned for departure The scheduled taxi route will be H F E, with crossing the RWY 13/31 in between A-SMGCS provides expected route information by TAXI-CPDLC on pilot s request, after the departure clearance has been issued CDD issues start up-clearance via TAXI-CPDLC CDD transfers the strip to GEC, and the system transmits the CONTACT message by TAXI- CPDLC PNF contacts GEC by voice GEC issues pushback clearance by TAXI-CPDLC GEC issues taxi clearance up to RWY 13/31 by TAXI-CPDLC GEC transfers the strip to TEC, and the system transmits the CONTACT message by TAXI- CPDLC. PNF contacts TEC by voice TEC issues crossing RWY 13/31 clearance, taxi to RWY 06 clearance, line-up and take-off clearance by voice Table 3-2 shows the communication between ATCO and PNF in detail. (Both GECO Staff columns can be neglected here as they refer only to the scenarios which will be used in RTS2.) DM Pilot non flying GECO staff UM ATC GECO staff Taxi Route Information: Aircraft Status: Parking at Apron North, Stand 34, Departure Clearance received, before Start-up 132 REQUEST TAXI ROUTING INFORMATION 3 ROGER Start-Up: Aircraft Status: 25 REQUEST START-UP CLEARANCE 0 WILCO Ready for Start-up A-SMGCS: TAXI-CPDLC 243 EXPECT ROUTING TO HOLDINGPOINT E RWY 06 VIA TWY H F E 239 CDD: START-UP APPROVED Send expected Routing to GECO Save date: Public Page 13

14 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff Handover to GEC: Aircraft Status: Performing or having performed Start-up 0 WILCO Push-Back Aircraft Status: 25 REQUEST PUSHBACK CLEARANCE 0 WILCO Initial call to GEC made by voice, ready for pushback Taxi Clearance: Aircraft Status: Pushback finished, ready for taxiing 25 0 WILCO REQUEST TAXI CLEARANCE Handover to TEC: Aircraft Status: Reaching RWY 13/31 0 WILCO Send Request to ATS Crossing of Runway 13/31, Taxiing to Runway 06, Line-up and Take-off: Aircraft Status: Initial call to TEC made by voice, Request crossing RWY 13/31 Cleared to cross RWY 13/31 and CDD: Save date: Public Page CONTACT LKPR GROUND GEC: PUSHBACK APPROVED GEC: Note: The Taxi Clearance is a multielement clearance transmitted in one go. TAXI TO HOLDINGPOINT E RWY 06 VIA TWY H F HOLD SHORT OF RWY 13 NEXT EXPECT VIA TWY F E GEC: CONTACT LKPR TOWER TEC: voice Cleared to cross RWY 13/31 and proceed to holding point RWY 06 via Foxtrott and Echo

15 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff proceed to holding point RWY06 via Foxtrott and Echo Ready Departure Runway 06 for on Lining-Up Runway 06 Cleared for take- Off Runway 06 Handover to DEP: (optionally) Aircraft Status: Climb-out CONTACT DEPARTURE Line-Up 06 Runway Cleared for Take- Off Runway 06 A-SMGCS: voice CONTACT DEPARTURE Table 3-2: TAXI-CPDLC standard procedures for an outbound flight Standard TAXI-CPDLC: Procedures Inbound The aircraft approaches Prague Airport on RWY 06 The assigned stand is on apron north (Gate 34) The scheduled taxi route will be L G A-SMGCS provides expected route information by TAXI-CPDLC before final approach. TEC issues landing clearance by voice. TEC issues handover to GEC by voice. GEC issues taxi clearance by TAXI-CPDLC Table 3-3 shows the communication between ATCO and PNF in detail. Save date: Public Page 15

16 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff Taxi Route Information: Aircraft Status: Before Final Approach for RWY REQUEST TAXI ROUTING INFORMATION 3 ROGER Landing clearance: Aircraft Status: Approaching RWY 24 TOWER, ILS RWY 06 established, request landing clearance Cleared to land RWY 06 Handover to GEC: Aircraft Status: After landing on RWY 06 Contact Ground on 121,900 Taxi Clearance: Aircraft Status: Initial call to GEC made by voice. GROUND, on your frequency 0 WILCO Leaving Frequency (optionally via data link): Aircraft Status: Parked on stand. 136 ON-BLOCKED A-SMGCS: 243 EXPECT ROUTING TO STAND 34 VIA TWY L G 241 TEC: voice Wind knots, cleared to land RWY 06 TEC: voice Contact Ground 121,900 LKPR on Send expected Routing to GECO GEC: Note: GEC will issue taxi clearance without pilot s request. Follow data link TAXI TO STAND 34 VIA TAXIWAY L G GEC: No acknowledgement by GEC via TAXI- CPDLC required Save date: Public Page 16

17 EMMA2 Airborne Validation Results Part A Table 3-3: TAXI-CPDLC standard procedures for an inbound flight Advanced TAXI-CPDLC: Procedures Outbound The aircraft is parked at Apron North (Gate 34). Runway 24 has been assigned for departure The scheduled taxi route will be H Z A-SMGCS provides expected route information by TAXI-CPDLC on pilot s request, after the departure clearance has been issued CDD issues start up-clearance via TAXI-CPDLC CDD transfers the strip to GEC, and the system transmits the CONTACT message by TAXI-CPDLC. PNF contacts GEC by voice. GEC issues pushback clearance by TAXI-CPDLC GEC decides for a different taxi route via H and A and selects this route at the routing function. GEC issues revised taxi clearance by TAXI-CPDLC. GEC transfers the strip to TEC, and the system transmits the CONTACT message by TAXI-CPDLC. Pilot contacts TEC by voice TEC issues line-up and take-off clearance by voice Table 3-4 shows the communication between ATCO and PNF in detail. (Both GECO Staff columns can be neglected here as they refer only to the scenarios which will be used in RTS2.) DM Pilot non flying GECO staff UM ATC GECO staff Taxi Route Information: Aircraft Status: Parking at Apron North, Stand 34, Departure Clearance received, before Start-up 132 REQUEST TAXI ROUTING INFORMATION 3 ROGER A-SMGCS: TAXI-CPDLC 243 EXPECT ROUTING TO HOLDINGPOINT Z RWY 24 VIA TWY H Z Send expected Routing to GECO Save date: Public Page 17

18 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff Start-Up: Aircraft Status: 25 REQUEST START-UP CLEARANCE 0 WILCO Ready for Start-up Handover to GEC: Aircraft Status: Performing or having performed Start-up 0 WILCO Push-Back Aircraft Status: 25 REQUEST PUSHBACK CLEARANCE 0 WILCO Initial call to GEC made by voice, ready for pushback Taxi Clearance: Aircraft Status: Pushback finished, ready for taxiing 25 0 WILCO REQUEST TAXI CLEARANCE Handover to TEC: Aircraft Status: Reaching RWY 24 Holding Point A 0 WILCO Line-up and Take-off: Aircraft Status: Initial call to TEC made by voice, Ready Departure Runway 24 for on CDD: START-UP APPROVED CDD: CONTACT LKPR GROUND GEC: PUSHBACK APPROVED GEC: Note: The Taxi Clearance is a multielement clearance transmitted in one go. REVISE TAXI TO HOLDINGPOINT A RWY 24 VIA TWY H A GEC: CONTACT LKPR TOWER TEC: voice Save date: Public Page 18

19 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff Lining-Up Runway 24 Cleared for take- Off Runway 24 Handover to DEP: (optionally) Aircraft Status: Climb-out CONTACT DEPARTURE Line-Up 24 Runway Cleared for Take- Off Runway 24 A-SMGCS: voice CONTACT DEPARTURE Table 3-4: TAXI-CPDLC advanced procedures for an outbound flight Advanced TAXI-CPDLC: Procedures Inbound The aircraft approaches Prague Airport on RWY 24 The assigned stand is on apron north (Gate 31). The scheduled taxi route will be D F G with crossing of RWY 13/31 in between A-SMGCS provides expected route information by TAXI-CPDLC before final approach TEC issues landing clearance by voice TEC issues taxi clearance, crossing-rwy13/31-clearance and handover to GEC by voice The assigned stand is changed from Gate 31 to Gate 34 and thus a different taxi route via F G has to be cleared. GEC issues revised taxi clearance by TAXI-CPDLC Table 3-5 shows the communication between ATCO and PNF in detail. (Both GECO Staff columns can be neglected here as they refer only to the scenarios which will be used in RTS2.) Save date: Public Page 19

20 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff Taxi Route Information: Aircraft Status: Before Final Approach for RWY REQUEST TAXI ROUTING INFORMATION 3 ROGER Landing clearance: Aircraft Status: Approaching RWY 24 TOWER, ILS RWY 24 established, request landing clearance Cleared to land RWY 24 Taxi Clearance up to RWY 13/31, Crossing of RWY 13/31 and Handover to GEC: Aircraft Status: After landing on RWY 24. Taxi to RWY 13/31 via TWY D F cleared to cross RWY 13/31 and contact Ground on 121,900 after crossing Taxi Clearance: Aircraft Status: RWY 13/31 crossed, initial call to GEC made by voice. GROUND, on your frequency A-SMGCS: 243 EXPECT ROUTING TO STAND 31 VIA TWY D F G TEC: voice Wind knots, cleared to land RWY 24 TEC: voice Taxi to RWY 13/31 via TWY D F cross RWY 13/31 and contact Ground on 121,900 after crossing GEC: Note: Follow data link Send expected Routing to GECO Assigned stand on flight strip has been changed from 31 to 34. GEC will issue taxi clearance without pilot s request. Save date: Public Page REVISETAXI TO STAND 34 VIA TAXIWAY F G

21 EMMA2 Airborne Validation Results Part A DM Pilot non flying GECO staff UM ATC GECO staff 0 WILCO Leaving Frequency (optionally via data link): Aircraft Status: Parked on stand. 136 ON-BLOCKED GEC: No acknowledgement by GEC via TAXI- CPDLC required Table 3-5: TAXI-CPDLC advanced procedures for an inbound flight Traffic Scenarios A description of the scenarios for the GECO is listed in table 3-6 and 3-7. EXE04 will be a standard scenario for TAXI-CPDLC; EXE02 will be an advanced TAXI-CPDLC scenario. Different versions of EXE06 will be a baseline scenarios resp. GTD scenarios. Save date: Public Page 21

22 EMMA2 Airborne Validation Results Part A Start Gate 34 EXE 02 EXE 04 EXE 06 13NM before RWY24 Gate 34 13NM before RWY06 Gate 34 13NM before RWY24 Gate S6 Destination RWY24 Gate 34 RWY06 Gate 34 RWY24 Gate S6 RWY24 Taxi Route Visibility Wind Time of day TAXI- CPDLC (standard) TAXI- CPDLC (advanced) H B Expected taxi-out route TWY Z revised to TWY A D F G H F E CAT I Calm L G H A D F L P Day Day Day n.a. Expected taxi-in route revised after crossing by receiving actual taxi clearance granted by the GEC: preliminary Gate 31 revised to Gate 34 Expected Taxi Route (congruent to actual clearance) Duration ~ 20 ~ 20 ~ 60 n.a. Table 3-6: Scenario Description Overview RTS (Part 1, June 2008) n.a. n.a. P L H A Save date: Public Page 22

23 EMMA2 Airborne Validation Results Part A Start Gate 18 EXE 02 EXE 04 EXE 06 13NM before RWY24 Gate 18 13NM before RWY06 Gate 55 13NM before RWY24 Gate S6 Destination RWY24 Gate 34 RWY06 Gate 34 RWY24 Gate S6 RWY24 Taxi Route Visibility Wind Time of day TAXI- CPDLC (standard) TAXI- CPDLC (advanced) H B Expected taxi-out route TWY Z revised to TWY A n.a. D F G Expected taxi-in route revised after crossing by receiving actual taxi clearance granted by the GEC: preliminary Stand 34 revised to Gate 18 H F E CAT I Calm Day L G Expected Taxi Route (congruent to actual clearance) H1 H A Duration ~ 20 ~ 20 ~ 60 Table 3-7: Scenario Description Overview RTS (Part 2, October 2008) n.a. D F L P n.a. n.a. R L H A Taxi routes are depicted in red in Figure 3-4 Figure 3-10; preliminary taxi routes are coloured in blue. Save date: Public Page 23

24 EMMA2 Airborne Validation Results Part A Figure 3-4: Taxi Route in EXE02, Pt. 1 outbound (Gate 34 RWY24) Figure 3-5: Taxi Route in EXE02, Pt. 2 inbound (RWY24 Gate 34) Save date: Public Page 24

25 EMMA2 Airborne Validation Results Part A Figure 3-6: Taxi Route in EXE04, Pt. 1 outbound (Gate 34 RWY06) Figure 3-7: Taxi Route in EXE04, Pt. 2 inbound (RWY06 Gate 34) Save date: Public Page 25

26 EMMA2 Airborne Validation Results Part A Figure 3-8: Taxi Route in EXE06, Pt. 1 outbound (Gate 34 RWY24) Figure 3-9: Taxi Route in EXE06, Pt. 2 inbound (RWY24 Gate S6) Save date: Public Page 26

27 EMMA2 Airborne Validation Results Part A Figure 3-10: Taxi Route in EXE06, Pt. 3 outbound (Gate S6 RWY24) The GECO will appear as an Airbus A320 with a DLH-callsign in the ATS. Callsigns will change from trial to trial to make the GECO less extraordinary for the ATCOs (see table 3-8). Exercise Callsign Action SID/STAR ETA/ETD Stand EXE02 DLH72C Departure MEDOV 1A 00:10 34 EXE02 DLH73C Arrival on final 00:28 34 EXE04 DLH471 Departure TABEM 2D 00:12 34 EXE04 DLH472 Arrival on final 00:28 34 EXE06_01 DLH621 Departure MEDOV 1A 00:11 34 EXE06_01 DLH622 Arrival on final 00:28 S6 EXE06_01 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_02 DLH621 Departure MEDOV 1A 00:10 34 EXE06_02 DLH622 Arrival on final 00:28 S6 EXE06_02 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_03 DLH621 Departure MEDOV 1A 00:10 34 EXE06_03 DLH622 Arrival on final 00:28 S6 EXE06_03 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_04 DLH621 Departure MEDOV 1A 00:10 34 EXE06_04 DLH622 Arrival on final 00:28 S6 EXE06_04 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_05 DLH621 Departure MEDOV 1A 00:10 34 EXE06_05 DLH622 Arrival on final 00:28 S6 Save date: Public Page 27

28 EMMA2 Airborne Validation Results Part A EXE06_05 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_06 DLH621 Departure MEDOV 1A 00:10 34 EXE06_06 DLH622 Arrival on final 00:28 S6 EXE06_06 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_07 DLH621 Departure MEDOV 1A 00:10 34 EXE06_07 DLH622 Arrival On final 00:28 S6 EXE06_07 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_08 DLH621 Departure MEDOV 1A 00:10 34 EXE06_08 DLH622 Arrival On final 00:28 S6 EXE06_08 DLH623 Departure MEDOV 1A 01:04 S6 EXE06_09 DLH621 Departure MEDOV 1A 00:10 34 EXE06_08 DLH622 Arrival On final 00:28 S6 EXE06_09 DLH623 Departure MEDOV 1A 01:04 S6 Table 3-8: List of GECO call signs for every test run Participants There were eight pilots, seven of them are working currently for the Deutsche Lufthansa, Czech Airlines, and Germania. Their average age is 42,4 years (SD: 15,3 years) and the mean reported flight hours are 7471 (SD: 7731). Age Sex Nationality Amount of flight hours Hierarchi cal Company 33 Male German 1700 F/O Germania A/C currently operated Fokker 100 PRG Experience No Base Airport 39 Male German 6000 SF/O DLH 744 Yes FRA 59 Male German >20000 CPT DLH A330, A Male German 3000 F/O DLH A320 Yes MUC 67 Male German CPT ex DLH / Daimler Mercedes n.a. Yes MUC 28 Male German 1500 F/O DLH A No FRA 47 Male Czech CPT CSA A32F Yes LKPR 43 Male Czech 6500 CPT CSA B737 Yes LKPR Yes MUC FRA Table 3-9: Demographical Description of the pilots participating at the DLR RTS Save date: Public Page 28

29 Schedule of Executed Test Runs ATS GECO Validation Area Traffic Scenario Day No. Treatment TEC GEC CDD CM1 CM2 Treatment Mo, 1 Training (EFS/DMAN) A B Training Training EXE04 RWY06 6/23 2 Training(TAXI-CPDLC) B A Training TAXI-CPDLC Training EXE02 RWY24 3 Training(TAXI-CPDLC) B A TAXI-CPDLC Op. Feasibility EXE04 RWY06 standard 4 Baseline A B Baseline Op. Improvement EXE06 1 RWY24 5 EFS A B GT Op. Improvement EXE06 5 RWY24 6 Baseline B A Baseline Op. Improvement EXE06 2 RWY24 Tu, 7 EFS B A GT Op. Improvement EXE06 6 RWY24 6/24 8 TAXI-CPDLC + Routing A B TAXI-CPDLC Op. Feasibility EXE04 RWY06 standard We, 6/25 Th, 6/26 9 TAXI-CPDLC + Routing B A TAXI-CPDLC revised expected taxi-out: H Z H A 10 TAXI-CPDLC + Routing A B TAXI-CPDLC revised actual taxi-out H Z H A, taxi-in Op. Feasibility Op. Feasibility EXE02 RWY24 EXE02 RWY24 11 Without Tower Simulation C D Training Training EXE04 RWY06 12 Baseline D C Training Op. Improvement EXE06 3 RWY24 13 EFS C D Baseline Op. Improvement EXE06 4 RWY24 14 EFS+DMAN D C Baseline Op. Improvement EXE06 7 RWY24 15 EFS+DMAN C D GT Op. Improvement EXE06 8 RWY24 16 EFS+DMAN D C GT Op. Improvement EXE06 9 RWY24 17 TAXI-CPDLC + Routing C D TAXI-CPDLC Op. Feasibility EXE04 RWY06 standard 18 TAXI-CPDLC + Routing D C TAXI-CPDLC standard 19 TAXI-CPDLC + Routing C D TAXI-CPDLC rev. exp. taxi-out: H Z H A 20 TAXI-CPDLC + Routing D C TAXI-CPDLC revised actual taxi-out H Z Op. Feasibility Op. Feasibility Op. Feasibility EXE04 RWY06 EXE02 RWY24 EXE02 RWY24 Save date: Public Page 29

30 ATS GECO Validation Area Traffic Scenario Day No. Treatment TEC GEC CDD CM1 CM2 Treatment H A, taxi-in Mo, 10/13 Tu, 10/14 We, 10/15 Th, 10/16 21 Training (EFS/DMAN) E F Training Training EXE04 RWY06 22 Training(TAXI-CPDLC) F E Training TAXI-CPDLC Training EXE02 RWY24 23 Training(TAXI-CPDLC) F E TAXI-CPDLC stand. Op. Feasibility EXE04 RWY06 24 Baseline E F Baseline Op. Improvement EXE06 1 RWY24 25 EFS E F GT Op. Improvement EXE06 5 RWY24 26 Baseline F E Baseline Op. Improvement EXE06 2 RWY24 27 EFS F E GT Op. Improvement EXE06 6 RWY24 28 TAXI-CPDLC + Routing E F TAXI-CPDLC stand. Op. Feasibility EXE04 RWY06 29 TAXI-CPDLC + Routing F E TAXI-CPDLC revised exp. taxi-out route: G A H A 30 TAXI-CPDLC + Routing E F TAXI-CPDLC revis. actual taxi-out H1 H Z A H1 H A, in Op. Feasibility Op. Feasibility EXE02 RWY24 EXE02 RWY24 31 Without Tower Simulation G H Training Training EXE04 RWY06 32 Baseline H G Training Op. Improvement EXE06 3 RWY24 33 EFS G H Baseline Op. Improvement EXE06 4 RWY24 34 EFS+DMAN H G Baseline Op. Improvement EXE06 7 RWY24 35 EFS+DMAN G H GT Op. Improvement EXE06 8 RWY24 36 EFS+DMAN H G GT Op. Improvement EXE06 9 RWY24 37 TAXI-CPDLC + Routing G H TAXI-CPDLC stand. Op. Feasibility EXE04 RWY06 38 TAXI-CPDLC + Routing H G TAXI-CPDLC stand. Op. Feasibility EXE04 RWY06 39 TAXI-CPDLC + Routing G H TAXI-CPDLC revised expected taxi-out route: H1 H Z A H1 H A 40 TAXI-CPDLC + Routing H G TAXI-CPDLC revised actual taxi-out H Z A H A, taxi-in TAXI-CPDLC + Routing G H TAXI-CPDLC revised actual taxi-out H Z H A, in Op. Feasibility Op. Feasibility Op. Feasibility EXE02 RWY24 EXE02 RWY24 EXE04 RWY06 Table 3-10: Test Schedule including treatment, allocation of Pilots to the Cockpit and ATCOs to the CWP positions, Validation Area and the used traffic scenario Save date: Public Page 30

31 Metrics and Indicators In accordance to the EMMA2 high and low level objectives, the following table outlines the respective indicators and metrics that were used in 2-WP6.6 on-board validation activities by the DLR. They were derived from the general EMMA2 validation documents: the Validation Plan [5] and the Generic Experimental Test Plan [4]. Area HLO LLO Indicator Metric Operational Feasibility Verification of Fulfilment of 68 items of the QE-OF Questionnaire [IND_1.1.1] a Operational Improvements EMMA2 Operational Requirements and Procedures [HLO_1] Increase of Safety [HLO_2] Suitability Behaviour Working Performance [HLO_4] of and EMMA2 Operational Requirements and Procedures [LLO_1.1] Situational awareness according to the operator will be preserved improved [LLO_2.2] 10 items of the SUS [IND_1.1.2] Mid-run questionnaire I.S.A. Situation Awareness [IND_2.2.1] 6 point Likert scale to each item proved for their significance by non-parametric binominal test [M_ ] 5 point Likert scale to each item proved for their significance by Save date: Public Page 31 or Appropriate level of user s workload [LLO_4.1] Improved situational awareness [LLO_4.2] Post-trial assessed Situation Awareness [IND_2.2.2] Mid-run questionnaire I.S.A. workload [IND_4.1.1] Mid-run questionnaire I.S.A. Situation Awareness [IND_4.2.1] for self a non-parametric binominal test [M_ ] Mid-run self assessed situational awareness (after specific events) -11 point Likert scale - ANOVA of mean per test run are compared [M_ ] Questionnaire ASSA (15 items) a 5 point Likert scale to each item proved for their significance by a non-parametric binominal test [M_ ] Mid-run self assessed workload for (after specific events) -5 point Likert scale - ANOVA of mean per test run are compared [M_ ] Mid-run self assessed situational for awareness (after specific events) -11 point Likert scale - ANOVA of mean per test run are compared [M_ ]

32 Area HLO LLO Indicator Metric Post-trial self Questionnaire ASSA assessed Situation Awareness (15 items) a 5 point Likert scale to each item [IND_4.2.2] proved for their significance by a non-parametric binominal test Less human errors [LLO_4.3] Self assessment of human errors [IND_4.3.1] [M_ ] Debriefing [M_ ] Table 3-11: Metrics and Indicators (RTS) Save date: Public Page 32

33 3.1.2 Results to Operational Feasibility (RTS) This chapter describes the results gained to decide on the operational feasibility of the new A-SMGCS services, which were validated. In particular, it is aimed to decide on the verification of the EMMA2 operational requirements from an operational and also from a technical point of view. The operational part is covered by debriefings and the EMMA2 Operational Feasibility Questionnaire for Pilots (QE- OF) asking the users for their acceptance to service or performance requirements. Additionally, the QE-OF gives pilots room to provide recommendations, how to improve the new procedures. All results can be found in the following two sections. However some operational requirements had also to be verified by the decision of the system designer: the system designer were asked by the EMMA2 checklist whether a specific requirement had been fulfilled or not. Sometimes both technician and user had to decide to exhaustively decide upon an operational requirement Operational Feasibility Questionnaire (QE-OF) After finishing off all of test runs, the Operational Feasibility Questionnaire (QE-OF) was given to the pilots. The questionnaire consisted of 72 items ordered by the EMMA2 services and with reference to their operational requirements and procedures described in the SPOR [3]. Not all items could be answered because not every feature of a service was implemented or could be tested. Sometimes pilots were not even affected by a situation that was asked (for instance a runway conflict) in this case the Controller could answer not affected (n.a.). Additionally they could answer with not important (n.i.) when they felt that a requirement is not that important anymore after experiencing the whole service. The following instruction has been given to them before they were requested to fill in the questionnaire: Introduction: Below you find questions/statements for which we are interested in your personal opinion how far you can agree to or not (answers from 1 Strongly disagree to 6 Strongly agree ). Your answers will help us to decide if the system design, system performances or new procedures have met your demands. There are two additional columns where you can make a cross: If you feel that the content of a statement is of no value for you NOT IMPORTANT or you were not affected by this item during the test: NOT AFFECTED, make your cross in those columns. For instance, you are asked to an alert performance but you wasn t affected by such an alert. Please refer your answers to the experiences you gained while you were using the new A- SMGCS services (GTD, TAXI-CPDLC). If you want to provide additional comments/explanations you can use the whole row. Particularly when you do not agree to a statement we would be interested in the reason why. Your data will be kept confidential. Thank you in advance. Save date: Public Page 33

34 The following hypotheses pair was applied to each of the item of the QE-OF: Identifier OF-H0 OF-H1 Hypothesis The users opinion does not agree to the operational feasibility aspects of a specific item. The users opinion agrees to the operational feasibility aspects of a specific item Results In section all answers (raw data) of the pilots in the QE-OF can be found. Since the sample size is only eight different pilots per item, the binominal test as a non-parametric statistic was used to prove the results for its statistical significance [8]. The binominal distribution can be seen as a rather conventional and robust statistic, which is supporting the H0 hypothesis, meaning that results become harder significant in the expected direction (H1). The advantage is that results, which become significant, are more likely to be really meaningful. By use of a binominal test for a single sample size, each item was proven for its statistical significance by following conditions: Binominal Test Answers from 1 (disagreement) through 6 (agreement) Expected mean value = 3.5 Test ratio:.50 N = 8 α = 0.05 A star (*) attached to the p-value means that a questionnaire item has been answered significantly because the p-value is equal or less than the critical error probability α, which is Additionally, such items were coloured green. When comments were given to an item, they are reported directly below the statement. Rows coloured in green indicate a statistical significance amber indicate a negative attitude to this statement, what means that average answers were below the expected mean value of 3.5 Table 3-12 shows the respective results. Save date: Public Page 34

35 General requirements ID Questions / Statements M N SD p EMMA2 OR 2-G When working with the new cockpit services, procedures, responsibilities and functions are clearly defined. GEN_Serv-06 3-G a) Ground Traffic Display c) TAXI-CPDLC I think the new cockpit services were well integrated into the existing systems. GEN_Serv-10 5-G a) Ground Traffic Display ** c) TAXI-CPDLC The new cockpit services are capable of being used appropriately when operating within the movement area. GEN_Serv-13 a) Ground Traffic Display ** c) TAXI-CPDLC Comments: Pilot 5: TAXI-CPDLC operation requires too much head down times while using the CDTI. This means either a/c needs to be stopped (time is lost) or redundancy of second crew member operating the CPDLC is lost G Even in case of a failure of an element of a new cockpit services, the failure effect was such that the status was always in the "safe" condition. GEN_Serv-16 a) Ground Traffic Display ** c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly disagree ; for inbound disagree G The new cockpit services enable me to interface and function efficiently. GEN_Serv-22 a) Ground Traffic Display ** Save date: Public Page 35

36 c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly sagree ; for inbound disagree G The design of the new cockpit services excludes failures that result in erroneous data for operationally significant time periods. GEN_Serv-24 a) Ground Traffic Display c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly disagree ; for inbound disagree Pilot 3: it s not easy to decide for pilots if this is really the case G The new cockpit services have the ability to provide continuous validation of data and timely alerts to the user when the system must not be used for the intended operation. GEN_Serv-25 a) Ground Traffic Display c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly sagree ; for inbound disagree Pilot 6: no error messages received in the scenarios G The new cockpit services provide a continuous service. GEN_Serv-27 a) Ground Traffic Display ** c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly sagree ; for inbound disagree G Any unscheduled break in operations was sufficiently short or rare as not to affect the safety of aircraft. GEN_Serv-28 Save date: Public Page 36

37 a) Ground Traffic Display ** c) TAXI-CPDLC Comments: Pilot 1: for outbound slightly sagree ; for inbound disagree Pilot 3: breaks wouldn t have been necessary if CDTI hadn t been so novel G I was able to detect significant failures and could initiate remedial action to restore the service or provide a reduced level of service. GEN_Serv-29 a) Ground Traffic Display * c) TAXI-CPDLC Comments: Pilot 6: Failures with the CDTI occurred G The new cockpit services allowed me for a reversion to adequate back-up procedures if failures in excess of the operationally significant period occur. GEN_Serv-32 a) Ground Traffic Display Comments: Pilot 6: Yes, by chart ** c) TAXI-CPDLC Comments: Pilot 6: Yes, by voice G The new cockpit services are capable of supporting operations of aircraft and vehicles within their minimum and maximum speeds and any heading. GEN_Perf-01 a) Ground Traffic Display c) TAXI-CPDLC Comments: Pilot 6: Rather disturbs than helps Save date: Public Page 37

38 Surveillance requirements ID Questions / Statements M N SD P 1-S 2 The ground traffic display provided accurate position information on all movements within the movement area that I could work safely and efficiently. Comments: Pilot 1: very effective ** EMMA2 OR SURV_Ser v-01 2-S The ground traffic display provided identification and labelling of the surrounding traffic that I could work safely and efficiently ** SURV_Ser v-02 3-S The ground traffic display was coping with moving and static aircraft and vehicles that I could work safely and efficiently ** SURV_Ser v-03 4-S The ground traffic display was sufficiently capable of updating surveillance data of the surrounding traffic that I could work safely and efficiently. 5-S (While I was using it) The ground traffic display was unaffected by operationally significant effects such as adverse weather and topographical conditions ** * SURV_Ser v-04 SURV_Ser v-05 Routing/planning requirements ID Questions / Statements M N SD P EMMA2 OR 2 In a simulation environment it is not possible to address the operational feasibility of a surveillance service since the performance of the service itself is simulated. Save date: Public Page 38

39 10-R The used terminology or symbology to represent a taxi route was easy to interpret. Comments: Pilot 4: RWY Identification markers could have been larger ** ROUT_Serv -12 Requirements related to HMI ID Questions / Statements M N SD P EMMA2 OR 2-H The new display maintained a balance between human and machine tasks ** HMI_Serv-02a 3-H The new display permitted me to retain the power to make decisions as to those functions for which I was responsible ** HMI_Serv-02b 4-H The new display provided a convenient mix of visual, audio and tactile inputs and responses HMI_Serv-02c 5-H Input devices were functionally simple - involving me in a minimum number of input actions. Comments: Pilot 6: Totally un-ergonomically HMI_Serv-03 7-H The new display allowed me to operate the new services in a safe and efficient manner. Comments: Pilot 4: For Ground Traffic Display agree ; for TAXI-CPLDC slightly disagree HMI_Serv-07 8-H The new display was harmonised where possible with existing cockpits components. Comments: Pilot 4: For Ground Traffic Display agree ; for TAXI-CPLDC slightly disagree ** HMI_Serv-08 9-H The new display design took into account my working environment under various operational conditions, means, the new display was adaptable to the various circumstances. Comments: Pilot 4: For Ground Traffic Display slightly agree ; for TAXI-CPLDC slightly disagree HMI_Serv-09 Save date: Public Page 39

40 10-H The new display allowed me to utilise the display capabilities (e.g. range scale selection, brightness, map overlays). Comments: Pilot 7: Brightness control knob would be appreciated ** HMI_Serv H The new display provided me with the complete airport traffic situation, allowing a rapid situation assessment. Comments: Pilot 3: This is the main message from the trials! ** HMI_Serv H I was presented with a clear 'picture' to easily locate and identify aircraft. Comments: Pilot 7: Yes, for close proximity (up to 500m) ** HMI_Serv H Only the minimum traffic information was permanently displayed HMI_Serv H I was provided with clear and visible indication when I was deviating from my cleared taxi route. Comments: Pilot 6: Additional hints / alerts would be more appreciated HMI_Serv H All relevant surrounding traffic was presented in clear and pre-defined formats that helped me operating my a/c in a safe and efficient manner. Comments: Pilot 5: E.g. follow TWY A B to R [line break] WY should be follow TWY A B to R WY in ONE line, not to mistake it with TWY R HMI_Serv H Depending on my operational needs, the new display was highly configurable with regard to layout, size, shape, fonts, colours and interaction capability. 30-H TAXI-CPDLC assisted me in handover operations. Comments: Pilot 7: Low experience with the new device HMI_Serv-40 HMI_Serv-48 Save date: Public Page 40

41 35-H I was reliably presented with a means to input TAXI CPDLC messages. Comments: Pilot 7: It will need some training and some new listing of messages to operate the onboard system efficiency HMI_Serv H It was possible to easily correct a mistaken action. Comments: Pilot 3: Not easy because CDTI is novel for pilots in the trials HMI_Serv H I was provided with a quick and efficient means to modify a previous ATC assigned route after receiving a revised taxi clearance. 39-H I could always easily intervene with the new display to set additional constraints unknown to the new on-board function (e.g. closed taxiway). Comments: Pilot 7: Not installed HMI_Serv-61 HMI_Serv H The indication of stop bars was well integrated into the new display. Comments: Pilot 6: I wasn t able to delete the stop bar HMI_Serv H When the crossing was operated by TAXI- CPDLC, the status of the displayed stop bar was always in accordance with ATCO clearances. Comments: Pilot 6: Had to be confirmed manually HMI_Serv H The presentation of other surrounding movements was not delayed to an extent where it is no longer operationally acceptable. 48-H The response time of the HMI was adequate to allow making inputs without having to wait unduly for the system to process and validate the input HMI_Perf-04 HMI_Perf-05 Save date: Public Page 41

42 TAXI-CPDLC requirements ID Questions / Statements M N SD P EMMA2 OR 2-T When needed, it was always possible to switch back from data link communication to voice communications in a safe and efficient manner TAXI- CPDLC_Serv T I was provided with an effective human-machine interface to permit efficient TAXI-CPDLC communication with ATC. 6-T In the event of an unexpected termination of a data link application I was sufficiently notified of the failure TAXI- CPDLC_Serv- 06a ATCO TAXI- CPDLC_Serv T I was always provided with the capability to adequately respond to TAXI-CPDLC messages and to issue requests, as appropriate TAXI- CPDLC_Serv T TAXI-CPDLC messages were appropriately displayed and stored in a manner that permitted timely and convenient retrieval when such actions were necessary. Comments: Pilot 3: Difficult to find old messages especially if new message is sent. The listing should be upside down with the newest message appearing on the top of the list ** TAXI- CPDLC_Serv T I was always under the control of only one ATC unit at a time. Comments: Pilot 7: Sometimes it seemed that CDD and GEC worked simultaneously TAXI- CPDLC_Serv T Each TAXI-CPDLC message transmission was followed by a positive technical acknowledgement, which informed me that the message has completely been transmitted and is available on the ATCO s display. Comments: Pilot 2 & 6: No feedback in some cases TAXI- CPDLC_Serv- 19 Save date: Public Page 42

43 14-T The time I needed to look outside was not considerably impaired by operating TAXI- CPDLC. Comments: Pilot 6: Consequently, the a/c needs to be stopped while using the CDTI because at least one pilot will look head down. Pilot 7: Especially revised instruction on apron! Anticipated constraint, SPOR T The amount of inputs to operate TAXI-CPDLC was reasonably low. Comments: Pilot 6: CDTI menu unnecessarily convoluted, unconventional commands (WILCO) Anticipated constraint, SPOR T The total time required for selecting a TAXI- CPDLC message, transmission of the message, reading and interpretation of the message was adequate to communicate in a safe and efficient manner. Comments: Pilot 6: Safe, but not efficient ICAO Doc9694 PART IV, T ATCO s TAXI-CPDLC response time was sufficient short to work in a safe and efficient manner. Comments: Pilot 7: For the given status of the system, yes! Anticipated constraint, SPOR T The handling of other aircraft by TAXI-CPDLC did not lead to loss of important information, which is usually provided by a R/T partyline effect. Comments: Pilot 6: Party line effect is missed. Pilot 6: No queue info during push-back. Party line for additional situation awareness no longer possible Anticipated constraint, SPOR 3.7 Save date: Public Page 43

44 19-T The mix of TAXI-CPDLC and voice communication with different clearances did not lead to confusion and safety critical communication errors. Comments: Pilot 6: Different clearances via voice resp. TAXI- CPDLC were experienced. 22-T The TAXI-CPDLC messages were easy to understand and could be handled in a safe and efficient way. If not please write down which one should be improved and how: Comments: Pilot 6: Use ACCEPT or READ- BACK/CONFIRM instead of WILCO listing should be upside down with the newest message appearing on the top of the list Anticipated constraint, SPOR 3.7 EMMA2 message set 23-T It was easy to recognise an incoming TAXI- CPDLC message. Comments: Pilot 3: No, should be improved (more volume). Pilot 6: Use ACCEPT or READ- BACK/CONFIRM instead of WILCO EMMA2 SPOR 3.7.3f 25-T Receiving TAXI-CPDLC taxi route information in advance of the real taxi route clearance is an appropriate procedure to provide me with important preliminary information. Comments: Pilot 2: But not in 2000ft alt during approach Pilot 6: Preliminary route deviates from the actual route too often EMMA2 SPOR 3.7.3j 26-T Receiving TAXI-CPDLC taxi route information in advance of the real taxi route clearance did not exceed my available mental and time resources. Comments: Pilot 7: Not in case of message arriving when ILS established ** EMMA2 SPOR 3.7.3j Save date: Public Page 44

45 27-T TAXI-CPDLC communication while the aircraft was taxiing could be performed in a safe and efficient way. Comments: Pilot 3&6: Consequently, the a/c needs to be stopped while using the CDTI because at least one pilot will look head down EMMA2 SPOR 3.7.3m 28-T The frequency of the next control position can be transmitted silently by a TAXI-CPDLC message, but the initial call from the pilot at the next control position should be retained by voice. Comments: Pilot 6: Yes. Otherwise it s not guaranteed if there s contact with the tower. E.g. in case of emergency if a revised clearance ( Hold position ) is given EMMA2 SPOR 3.7.3p Table 3-12: QE-OF - Means, N, SD, and P-Value Conclusions There were 72 questionnaire items asking the pilots acceptance to specific operational requirements and procedures of the new higher-level A-SMGCS services. Finally, 20 operational requirements or procedures could be verified in the DLR RTS test campaign, proven by a non-parametric binominal test o (highlighted in green in table 3-12), 36 operational requirements or procedures were accepted but not with a probability it can doubtlessly relied on o (statements in black in table 3-12), and 16 operational requirements or procedures were not answered positively o (highlighted in amber in table 3-12). The pilots disagreed on 16 of the 72 items, and respectively could not accept the experienced performance. Those 16 negative answers are mainly caused by the use of a novel and unfamiliar input device which served for selecting CPDLC messages to be sent and for the textual displaying of received clearances, the CDTI (cf. chapter ) SUS Questionnaire In addition to the tailor-made QE-OF for pilots the standard questionnaire Standard Usability Scale (SUS) [9] was given. While it is difficult to answer the question is system A more usable than system B, (due to the fact that measures of effectiveness and efficiency may be very different), it can be argued that a sufficiently high-level definition of subjective assessments of usability, makes comparisons between systems possible. SUS has generally been seen as providing a type of high-level subjective view of usability and is thus often used in carrying out comparisons of usability between systems (here: baseline vs. higher A- SMGCS on-board services). It contains five items in favour of a new system (item 1, 3, 5, 7, and 9) and five items to the disadvantage of a new system (item 2, 4, 6, 8, and 10). Save date: Public Page 45

46 The SUS was given to the pilots after the very last trial. Answers range from 1 Strongly disagree to 5 Strongly agree. Again, the following hypotheses pair was applied to each of the item of the SUS: Identifier OF-H0 OF-H1 Hypothesis The users opinion does not agree to the operational feasibility aspects of a specific item [in terms of usability]. The users opinion agrees to the operational feasibility aspects of a specific item [in terms of usability] Results In section all answers (raw data) of the pilots in the SUS can be found. Since the sample size is only eight different pilots per item, the binominal test as a non-parametric statistic was used to prove the results for its statistical significance [8]. The binominal distribution can be seen as a rather conventional and robust statistic, which is supporting the H0 hypothesis, meaning that results become harder significant in the expected direction (H1). The advantage is that results, which become significant, are more likely to be really meaningful. By use of a binominal test for a single sample size, each item was proven for its statistical significance by following conditions: Binominal Test Answers from 1 (disagreement) through 5 (agreement) Expected mean value = 3.0 Test ratio:.50 N = 8 α = 0.05 A star (*) attached to the p-value means that a questionnaire item has been answered significantly because the p-value is equal or less than the critical error probability α, which is Additionally, such items were coloured green. When comments were given to an item, they are reported directly below the statement. Rows coloured in green indicate a statistical significance in favour of the new system (p < 0.05). Table 3-13 shows the respective results. Answers range from 1 Strongly disagree to 5 Strongly agree. ID Questionnaire Item M N SD P 1 I think that I would like to use this system frequently I found the system unnecessarily complex. 3 I thought the system was easy to use I think that I would need the support of a technical person to Save date: Public Page 46

47 ID Questionnaire Item M N SD P be able to use this system. 5 I found the various functions in this system were well integrated. 6 I thought there was too much inconsistency in this system. 7 I would imagine that most people would learn to use this system very quickly. 8 I found the system very difficult to use ** ** 9 I felt very confident using the system. 10 I needed to learn a lot of things before I could get going with the system ** Table 3-13: SUS- Means, N, SD, and P-Value Conclusions The pilots answered all ten items in favour of the new system. Three of them (highlighted in green in table 3-13) became even statistically significant, proven by a non-parametric binominal test Debriefing Comments This chapter summarizes statements made by the pilots at the debriefings after the RTS trials. The comments reflect a common opinion of all eight pilots. The comments mainly address aspects of the operational feasibility and proposals for improvements of the new A-SMGCS services. When such proposals could already be implemented within EMMA2 they are coloured in green, and blue when it could not be solved yet Ground Traffic Display The GTD was appreciated and highlighted by all pilots. The added value of the GTD is seen especially under low visibility conditions. The intuitive approach allows the pilots to work in a simple and interactive way. According to the pilots, the design fits completely the pilots needs. In the opinion of the pilots, the GTD worked reliable in the simulation trials (= with an update rate of 5Hz) The different modes (PLAN, ARC and NAV) and the zoom stages are highly appreciated and intuitive to use. The GTD allows pilots to utilise the display capabilities (e.g. zoom and mode selection) appropriately. This was realised by both the mode resp. range selector in the EFIS and soft buttons on the GTD s touchscreen. Pilots preferred to use the knobs in the EFIS. Save date: Public Page 47

48 Pilots reported that the head down times while using the GTD display (located on the navigation display) are acceptable. Nevertheless, the outside view through the cockpit windows remains their main source of information. Minor additional comments: Two pilots propose to display only traffic with transponder on. Two pilots propose to display landing traffic on short final. Two of eight pilots expressed their wish to have 3D buildings in the display. Not considered to integrate 3D buildings by now One pilot wants to integrate EMM on the PFD; and blanking out the NAV display. Instead of merging the information with the PFD, the use of a HUD is considered Taxi-CPDLC TAXI-CPDLC is regarded as potential means which will ease pilots work considerably with appropriate input devices. The graphical presentation of the cleared taxi route on the EMM (compared to textual clearances only) will enable pilots to follow an assigned taxi route in an intuitive way. The pilots feedback for requesting and receiving clearances by TAXI-CPDLC for startup, push-back, taxi-in, and taxi out was predominantly positive. It must be distinguished clearly between o the positively rated graphical information on the EMM display (represented in this chapter) and o the rating of the usability of the CDTI, which served for selecting CPDLC messages to be sent and for the textual displaying of received clearances (cf. chapter ). Pilots highly appreciated and trusted the displayed taxi routes and the clearances. They agree that the used terminology and symbology to represent a taxi route were easy to interpret. Paper charts were only held ready as a back-up as expected. The pilots stated that they were provided with a very effective human-machine interface in terms of the EMM display to operate a data link communication with ATC. The visualisation of taxi routes on the EMM was appealing for them. Especially the exocentric perspective view on the EMM was used by the CM2 to gain advance information during taxiing. Six pilots appreciate the initial call at the beginning of each TAXI-CPDLC dialogue at each new ATC position. Pilots responded that they would like to use the TAXI-CPDLC in the future and that they agree in general with its concept given and realised in EMMA2. Yet some of the proposed procedures are perceived as being clumsy. o Pilots recommend that the CDD should issue departure clearance, start up and change to ground frequency in one step, when the situation allows it. Fulfilled in the RTS in October as far as possible Save date: Public Page 48

49 o o In the trials the crossing was performed by voice communication, due to its nature of a safety critical clearance. This led to the situation that the pilots in the cockpit had to switch off the red crossbar (received from the Ground controller as the clearance limit) on their EMM. This procedure is of course criticised by the pilots. They agree that this task must be performed by an ATCO. Further testing of adapted procedures would be needed here, e.g. crossing clearance by TAXI-CPDLC, or by a voice clearance combined with an automatic deletion of the displayed stop bar by the ATCO s pressing of the crossing clearance button on their EFS. In addition to the initial call a dedicated radio frequency for emergency situations is recommended by two pilots. Pilots agree that a preliminary taxi route will not bring added value. The assigned route can change o at short notice during initial approach (inbound) or o preparing the a/c for departure (outbound), and thus is not a reliable information in the opinion of the pilots. During arrival o no preliminary taxi route should be sent after pilots preparation of the landing (approx. before ft; preliminary information at ft is too late) o resp. a preliminary information about the expected gate would be sufficient. Nevertheless receiving a TAXI-CPDLC taxi route information in advance did not exceed the available mental and time resources in the pilots opinion. The audio signal for an incoming message is choked until the CM1 or CM2 push the button. No other message can be detected before the button is pressed. o Therefore an additional signal ins necessary and o the CDTI must automatically return into the main page so that the oncoming message can be seen easily. o Furthermore, if one message button is pressed, both lights must go off automatically. Solved during the June 2008 trials. Labelling of the buttons must be integrated in harmonisation with existing cockpits components. o ACCEPT or READ-BACK/CONFIRM is a better choice than WILCO Considered for trials in the future. When receiving a revised taxi route (or the taxi route approved by the ground controller) all other taxi routes shown before (= first route before revised; or preliminary taxi route) must be deleted to avoid confusions and an information overflow. Pilots reported that the colour o of an old route remained green and the revised route appeared in yellow (happened twice in the June trials). o Solved during the June 2008 trials by reprogramming. that the colour of revised route appeared in green on the left, but still in yellow on the right ND (happened three times in the June trials). Two different displays were in use resulting in different latencies; solved as far as possible during the June 2008 trials; Furthermore it happened that the CM1 pushed the cross button on the ND and the red lined crossbar in his ND disappeared as expected, but not on the CM1 side. Solved during the June 2008 trials by reprogramming. Four pilots explained that ATCO s handling of other aircraft by TAXI-CPDLC led to loss of important information, which is usually provided by a R/T partyline effect. Save date: Public Page 49

50 A minority of pilots proposed following changes in the system: Use of a larger font (especially necessary during turbulences) (two pilots) Implementation of a printer (one pilot) Louder aural signal (one pilot) CDTI Using the CDTI as an additional input device for data link communication (instead of using the FMS) is appreciated by the majority of the pilots. In the opinion of most of the pilots (= six) it might be too confusing to integrate the data transfer into the FMS resulting in a difficult handling of the TAXI-CPDLC. Nevertheless the novelty of the menu navigation in the CDTI and the resulting high complexity is not acceptable for the pilots. Therefore the handling of the messages could not always be successfully performed. Pilots experiences are reflected here together with their recommendations. Pilots had difficulties to find new messages in the system as long as they were working on the previous message. It was not easy or intuitive enough to find the new message in the log page. In some cases, pilots were not able to find the logical acknowledgment from the tower indicating that the pilots message was sent successfully to the tower. o One crew preselected request taxi and was not able to receive any new messages because the system was blocked. This problem was solved after the June 2008 trials. Therefore they have concerns that new message will get lost especially if the pilot is occupied. Pilots recommend o that a new message must be suppressed as long as the last message is not been processed by the pilots. o A clear screen button for the last message or a o refresh button (the last two recommendations by two pilots) Furthermore, they had problems to find old messages on the log page. o Additional problems occurred when old messages remained as not affirmed on the CDTI display after their clearance. this problem was solved during the June 2008 trials. The CDTI was used in the validation trials for data link communication as a test-release resp. prototype to test a new user menu in the simulation. This new menu concept was not approved by the pilots and led to longer head-down times than acceptable. The CDTI needs important upgrades if a further use of the CDTI is foreseen in simulation and flight trials. As indicated in the pilots feedback, additional and crucial recommendations for the CDTI were made: To provide a menu similar to FANS B To integrate LLSK (Left Line Select Keys) and RLSK (Right Line Select Keys) To ensure accessibility of the main menu To guarantee clear arrangement of the messages received in the CDTI s log function, e.g. the listing should be upside down with the newest message appearing on the top of the list. TAXI-CPDLC messages were not appropriately displayed and stored in a manner that permitted timely and convenient retrieval when such actions were necessary. Nevertheless it must be highlighted that the use of CDTI was necessary to have a flight-worthy and certified interface for preparing and conducting the subsequent flight trials. Save date: Public Page 50

51 3.1.3 Results to Operational Improvements (RTS) This chapter describes the results gained to decide on the operational improvement of the new A-SMGCS services, which were under test. Subjective data were collected by opinions of the pilots, who were asked to estimate the potential of a new service to provide operational improvement. Such are a useful resource to decide on a hypothesis and finally to validate a new service. Subjective data were gained by the EMMA2 Operational Improvement Questionnaire for Pilots (QE-OI) [6], Airport Surface Situation Awareness Questionnaire (ASSA) [11]; [12] Instantaneous Self Assessment (ISA) [10] o of Workload o of Situation Awareness Safety (HLO_2) Operational Improvements Questionnaire (QE-OI), Sub-Scale Safety After finishing all test runs the Operational Improvement Questionnaire for Pilots (QE-OI) was given to the pilots. Five of its eight items address operational improvements in terms of safety that could be expected due to the implementation and operational use of the higher-level A-SMGCS services. The items refer either to expected benefits that were identified in the 2-D1.1.1 SPOR document [3] or refer directly to a low level objective (cf. [5]; [4] and ). Following instruction has been given to them before they were requested to fill in the questionnaire: Introduction: Below you find statements for which we are interested in your personal opinion how far you can agree to or not (answers from 1 Strongly disagree to 6 Strongly agree ). Your answers will help us to decide if the new services are able to provide operational improvements in terms of supporting you in aerodrome operations. Please refer your answers to the experiences you gained while you were using the new EMMA2 A-SMGCS services (GTD and TAXI-CPDLC) and compare with the current situation in the cockpit. If you want to provide additional comments/explanations you can use the whole row. Your data will be kept confidential. Thank you in advance. The following hypotheses pair was applied to each of the safety items of the QE-OF: Identifier SF1-H0 SF1-H1 Hypothesis An increment of safety while using [higher services of A-SMGCS] is not perceived by the pilots. An increment of capacity while using [higher services of A-SMGCS] is perceived by the pilots. Save date: Public Page 51

52 Results In section all answers (raw data) of the pilots in the QE-OF can be found. Since the sample size is eight different pilots per item, the binominal test as a non-parametric statistic was used to prove the results for their statistical significance [8]. By use of a binominal test for a single sample size, each item was proved for its statistical significance by following conditions: Binominal Test Expected mean value = 3.5 Test ratio:.50 Answers from 1 (disagreement) through 6 (agreement) N = 8 α = 0.05 A star (*) attached to the p-value means that a questionnaire item has been answered significantly because the p-value is equal or less than the critical error probability α, which is Additionally, such items are coloured green. When pilots gave comments to an item, they are reported directly below the statement. Table 3-14 shows the respective results. Answers range from 1 (disagreement) through 6 (agreement). Save date: Public Page 52

53 ID Questions / Statements M N SD p EMMA2 OR 1P-OI 2P-OI The communication via TAXI-CPDLC would reduce potentially safety-critical misunderstandings between ATCO and Flight Crew compared to using radio communication only. The display of graphical taxi clearances on the Electronic Moving Map (EMM) would enable me to follow an assigned taxi route more safely. Comments: Pilot 5: The potential danger exists that you follow the EMM which is not certified as primary means of navigation ** 4-OI 6P-OI 8P-OI The receipt of only textual taxi clearances (without a graphical display) via TAXI-CPDLC would also enable me to follow an assigned taxi route more safely. The indication of the surrounding traffic is an additional information source to navigate and manage the aircraft speed more safely. Particularly in high workload situations, a Traffic Conflict Detection (TCD) service would clearly indicate and better draw my attention to a potential safety critical situations. Comments: Pilot 7: I strongly agree but had no [real] opportunity to test the system ** * Table 3-14: QE-OI - Means, N, SD, and P-Value for safety-related items Capacity / Efficiency (HLO_3) Operational Improvements Questionnaire (QE-OF), Sub-Scale Efficiency As mentioned in section the Operational Improvement Questionnaire for Pilots (QE-OI) was given after the last trial. Three of eight items address operational improvements in terms of efficiency that could be expected due to the implementation and operational use of the higher-level A- SMGCS services. The items refer either to expected benefits that were identified in the 2-D1.1.1 SPOR document [3] or refer directly to a low level objective (cf. [4] and ). The following hypotheses pair was applied to the capacity/efficiency items of the QE-OI: CE1-H0 Hypothesis An increment of capacity & efficiency while using [higher services of A-SMGCS] is not perceived by the pilots. Save date: Public Page 53

54 CE1-H1 An increment of capacity while using [higher services of A-SMGCS] is perceived by the pilots Results In section all answers (raw data) of the pilots in the QE-OF can be found. The analysis of data is explained in section Table 3-15 shows the respective results. Answers range from 1 (disagreement) through 6 (agreement). ID Questions / Statements M N SD P EMMA2 OR The display of graphical taxi clearances on the Electronic Moving Map (EMM) would enable me to follow an assigned taxi route more efficiently. 3P-OI Comments: Pilot 7: it may reduce taxi times, allows higher taxi speeds, increases TWY capacity ** 5P-OI 7P-OI The receipt of only textual taxi clearances (without a graphical display) via TAXI-CPDLC would also enable me to follow an assigned taxi route more efficiently. The indication of the surrounding traffic is an additional information sources to navigate and manage the aircraft speed more efficiently ** Table 3-15: QE-OI - Means, N, SD, and P-Value for capacity/efficiency related items Suitability of Behaviour and Working Performance (HLO_4) Workload (LLO4.1) ISA The ISA [10] measuring workload was applied twice in each segment (both inbound and outbound) of each test run. Pilots were asked approx. in the middle of the segment (e.g. after a runway cross) and at the end of the segment (before reaching the gate resp. before line-up). The scale ranges from underutilised (1) to excessive (5). The following hypotheses pair was applied to the ISA results: Identifier HF1-H0 HF1-H1 Hypothesis An increment of workload while using [higher services of A-SMGCS] is perceived by the users. An increment of workload while using [higher services of A-SMGCS] is not perceived by the users. Save date: Public Page 54

55 Results In table 3.19 all answers (raw data) of the pilots in the workload ISA can be found. Answers range from under-utilised (1) to excessive (5). The means of each test run were analysed in a 4 x 2 (A-SMGCS treatment x pilot role) two-way repeated measures analysis of variance (ANOVA) [8]. The ANOVA revealed a highly significant main effect of the A-SMGCS treatment (F (3,21) = 5.418; p =.006) with a mean of M = 2.20 for the (EMM) baseline, respectively M = 2.12 for GTD and M = 2.48 for standard TAXI-CPDLC (incl. GTD) and M = 2.36 for advanced TAXI- CPDLC (incl. GTD) on a scale reaching from under-utilised (1) to excessive (5). No significant main effect could be shown for the pilot role (F (1,7) = 1.932; p =.207), both CM1 and CM2 seem to have a similar level of workload. Yet there is a significance for the interaction between treatment and pilot role (F (3,21) = 3.743; p =.027), with the highest mean M = 2.52 in the condition CM2 using advanced TAXI- CPLDC. Factors df Square sum F-value Significance p-value A-SMGCS treatment ** error pilot role error Treat * role interaction * error Table 3-16: Two-way repeated measures ANOVA ISA Workload Situation Awareness (LLO_4.2) ASSA The questionnaire for Airport Surface Situation Awareness [11]; [12] was given to the pilots after the very last trial. Answers range from 1 Strongly disagree to 5 Strongly agree. The following hypotheses pair was applied to each of the item of the ASSA: Identifier HF3-H0 HF3-H1 Hypothesis An increment of situation awareness while using [higher services of A-SMGCS] is not perceived by the users. An increment of situation awareness while using [higher services of A-SMGCS] is perceived by the users Results In section all answers (raw data) of the pilots in the ASSA can be found. Answers range from 1 (disagreement) through 5 (agreement). Since the sample size is eight different pilots per item, the binominal test as a non-parametric statistic was used to prove the results for their statistical significance [8]. Save date: Public Page 55

56 By use of a binominal test for a single sample size, each item was proved for its statistical significance by following conditions: Binominal Test Expected mean value = 3 Test ratio:.50 Answers from 1 (disagreement) through 5 (agreement) N = 8 α = 0.05 A star (*) attached to the p-value means that a questionnaire item has been answered significantly because the p-value is equal or less than the critical error probability α, which is Table 3-17 shows the respective results. ID Questions / Statements M N SD P 1 Display use did not increase head down time Display clutter was not a problem Aided in surface traffic acquisition ** 4 Aided in surface visual traffic acquisition ** 5 Increased time for checklist Increased time for crew duties Surface map easy to bring up ** 8 Ownship symbol easy to see ** 9 Accurately shows my position ** 10 More info should be displayed ** 11 Useful info provided ** 12 Easy to get orientated to MMS Display ** 13 Aid in determining position of traffic ** 14 Aid in understanding ATC communication Aid in locating traffic ** Table 3-17: ASSA - Means, N, SD, and P-Value ISA (Situation Awareness) The ISA [10] in an adapted version measuring situation awareness was applied twice in each segment (both inbound and outbound) of each test run. Pilots were asked approx. in the middle of the segment (e.g. after a runway cross) and at the end of the segment (before reaching the gate resp. before lineup). The scale ranges from low (1) to high (10). The following hypotheses pair was applied to the ISA results: Identifier Hypothesis Save date: Public Page 56

57 HF3-H0 HF3-H1 An increment of workload while using [higher services of A-SMGCS] is perceived by the users. An increment of workload while using [higher services of A-SMGCS] is not perceived by the users Results In table 3.20 all answers (raw data) of the pilots in the situation awareness ISA can be found. The scale ranges from low (1) to high (10). The means of each test run were analysed in a 4 x 2 (A-SMGCS treatment x pilot role) two-way repeated measures analysis of variance (ANOVA) [8]. The ANOVA revealed no significant main effect of the A-SMGCS treatment (F (3,21) =.146; p =.931) with a mean of M = 8.09 for the (EMM) baseline, respectively M = 9.03 for GTD and M = 8.70 for standard and M = 8.74 for advanced TAXI-CPDLC (incl. GTD) on a scale ranging from low (1) to high (10). There is no significant main effect for the pilot role (F (1,7) = 1.337; p =.286), both CM1 and CM2 seem to have a similar level of situation awareness. There is no significance for the interaction between treatment and pilot role (F (3,21) = 1.391; p =.273). Factors df square sum F-value Significance p-value A-SMGCS treatment error pilot role error Treat * role interaction error Table 3-18: Two-way repeated measures ANOVA ISA Situation Awareness Save date: Public Page 57

58 Table 3-19: Pilots raw data, M, and SD of the ISA (Workload) Table 3-20: Pilots raw data, M, and SD of the ISA (Situation Awareness) Save date: Public Page 58

59 Debriefing Comments Ground Traffic Display In the pilots opinion the use of a GTD will improve their performance significantly. GTD is regarded as potential means for increased safety which eases pilots work considerably. GTD significantly increases the pilots situation awareness (by allowing an intuitive use, providing the appropriate amount of information, and as an extraordinary aid in locating traffic) especially under low visibility conditions. Pilots workload is not affected negatively when using the GTD. With its indication of the surrounding traffic the GTD is an additional information source for the pilots to navigate and to manage the aircraft speed more safely and efficiently Taxi-CPDLC TAXI-CPDLC is regarded as potential means for increased safety. The graphical presentation of the cleared taxi route on the EMM (compared to textual clearances only) will enable pilots to follow an assigned taxi route more safely and efficiently. An appropriate input device is necessary. An increase of workload while using the TAXI-CPDLC service could be observed and was explained by the pilots due to the fact that a novel and not-well designed input device (CDTI) was used. Situation Awareness was not affected by the use of TAXI-CPDLC. Save date: Public Page 59

60 3.2 Operational Flight Trials Results Set Up The validation platform The on-site validation exercises used the A-SMGCS test-bed components at Prague-Ruzynĕ airport, established under 2-SP3 & 2-SP2 (green team) of the EMMA2 project. ATTAS test aircraft: o EMM, o Ground traffic display (fed by TIS-B), o TAXI-CPDLC (CDTI + Graphical presentation on the EMM), o ADS-B out/in, and o Walkie-Talkie (provided by Airport Prague). The ATCOs could listen to the three real different control radio telecommunication (CDD, GEC, TEC) but worked in shadow mode, that is, they did interact with the ATTAS but not with the other a/c. They moved the EFS in accordance to the real world for all a/c. Identically to the set-up in the ATS, there were three CWPs equipped: TEC, GEC, and CDD. All the EMMA2 systems were linked to the real environment: surveillance and flight plan data. Tower Test Bed: o 3 CWPs: TEC, GEC, CDD o TSD, EFS, DMAN, Routing, TAXI-CPDLC, RIMCAS, basic conformance monitoring o Frequency monitors (black box with loudspeaker and volume control) o Handsets with predefined buttons for communication with operational TWR Figure 3-11: DLR ATTAS Test Aircraft (D-ADAM) during OST in PRG Save date: Public Page 60

61 For more detailed information see the SP6 Validation Test Plan Airborne (2-D6.1.6, [6]) Experimental Design There is no real experimental design possible in field trials since very less can be controlled to perform real experiments. Nevertheless, flight trials provide an excellent opportunity to check aspects of the technical and operation feasibility that cannot be checked in simulation trials, where for instance the traffic and real interfaces are simulated. Therefore, two different experimental designs has been chosen: GTD and TAXI-CPDLC trials. ATTAS / TUD Van test Ground Traffic Display (GTD via TIS-B) (between 1,5 hour in between of ) TAXI-CPDLC trials o the ATTAS is controlled by voice via the regular ATCOs o in parallel the ATTAS communicates via TAXI-CPDLC with the EMMA2 ATCOs in the test bed the respective clearances / messages o Procedure would be, as tested in simulation (cf. chapter ) o ATTAS performs an outbound / inbound scenario (two aerodrome circling, approx. 80 minutes) (four IFR flight plans (two outbound, two inbound)) o responsibilities: During the Tests, the regular controllers have the ATTAS (D-ADAM) under control, but The actual ATCO (CDD/GEC) would only perform the initial call with the ATTAS pilots, they are briefed about the silent START-UP, PUSHBACK, and TAXI clearance given by TAXI-CPDLC from the EMMA2 ATCOs, stay in contact with EMMA2 ATCOs, monitor the control of the ATTAS and intervene by R/T in case of critical situations Traffic Scenarios ATTAS parking position during the trials was S17 on APRON SOUTH. Figure 3-12: DLR ATTAS Test Aircraft (D-ADAM) parking position during OST in PRG The two main scenarios for TAXI-CPDLC trials were: Save date: Public Page 61

62 ATTAS taxiing out from Apron South to RWY 24 performing an outbound scenario. After aerodrome circling and arrival on RWY 24, ATTAS performed an inbound scenario back to S Participants Three DLR test pilots and a CSA pilot from the RTS performed the trials in the ATTAS cockpit Schedule of Executed Test Runs Pilot Tu, We, Th, TIS-B CPDLC TIS-B CPDLC TIS-B CPDLC A X X X (X) X X B X X X X C X X D X X Table 3-21: Pilots executed Test Runs in the ATTAS Test Aircraft Metrics and Indicators As real experiments with real baseline comparisons cannot be done in the field, the test focussed on the assessment of the technical feasibility of the new A-SMGCS services, concentrating on those aspects that could not be covered yet by the real time simulation trials, mainly the GTD fed by TIS-B and the exchange of real TAXI-CPDLC messages in shadow mode trials in parallel with real traffic. Therefore interviews were carried out at the end of the day to allow the pilots to address aspects of the technical and as far as possible operational feasibility of the new A-SMGCS services Results to Technical Feasibility (OST) This chapter describes the main results in terms of technical feasibility gained by the on-site trials. Results mainly base on observations by the test team and open discussions with the pilots after the trials. All technical results are documented in the 2-D2.5.2 Assessment Report for the Ground CPDLC Clearance function PRG/MXP [17] Ground Traffic Display (GTD) After tuning of the TIS-B function it was proven that TIS-B is capable to fulfil the needs to provide the complete traffic situation of the airport for the GTD. DLR test pilots were able to use the display during taxiing on PRG airport in its full functionality resulting in general acceptance of the GTD. The update rate was sufficient enough to provide the pilots with the complete current traffic situation. o Partially, during the first TIS-B trials the integration of the TIS-B was unreliable some false targets were displayed and own A/C symbol was skipped on the displayed taxi route. The system worked very well and will be a great step ahead in taxi procedures according to the pilots. Save date: Public Page 62

63 Automatic switching from AERODROME to Flight NAV mode is highly desirable (approx. 100 ft after lift off and vice versa during approach and landing) to improve availability of NAV information. According to the pilots, mentioned comments are associated with the experimental installation and could be easily solved more or less with today s technology. One pilot stated o that he is not in favour of the exocentric 3-D-esque mode of the EMM. In his opinion there is no added value and prefers to use the 2-D mode. o that the use of colours on the EMM should be reconsidered. In general it is very appreciated by the pilots and there were no further comments to it TAXI-CPDLC In general, the TAXI-CPDLC service worked perfectly. There were some initial problems with respect to doubled EFS caused by following aerodrome circuits and one link lost due to the ATTAS leaving of the VDLm2 range, but this could be solved easily. In detail, there were five aerodrome circuits performed by the ATTAS test aircraft: o 1 st circuit, , 14:20 15:00 unintentionally there were created to departure EFS for the ATTAS (call sign D-ADAM), which caused trouble with the log on the log on was done on the former created but already deleted EFS after retrieving the deleted EFS and binning the actual EFS this problem was solved and the test could started the transmission of all TAXI-CPDLC clearances run perfectly during the flight phase, when the ATTAS was 15,5 miles the log off was performed With inbound the ATTAS logged on again (green log on sign was seen on the D-ADAM inbound EFS) After crossing RWY13/31 and handover to the regular GEC, the regular GEC instructed follow data link and the test bed GEC provided the inbound taxi route to the final stand by TAXI-CPDLC o 2 nd circuit, , 12:20 13:15 all TAXI-CPDLC outbound clearances worked perfectly (see also a screenshot of the EFS display) during the aerodrome circuit the link was lost as the ATTAS distance to the VDLm2 antenna was too far with inbound the log on failed o 3 rd circuit, , 13:15 15:00 Call sign ADAM was chosen instead of D-ADAM in order to avoid problems with the log on All out- and inbound clearances worked perfectly o 4 th circuit, , 12:35 13:25 All out- and inbound clearances worked perfectly o 5 th circuit, , 13:30 14:10 All out- and inbound clearances worked perfectly Pilots made additional comments: The system worked technically very well and will be a great step ahead in taxi procedures. In general the principle and the concept are appreciated by the pilots. Save date: Public Page 63

64 Nevertheless, in their opinion, any CDU or keyboard will improve work with listing and sending messages on CPDLC - compared to the use of the CDTI. A CREW MESSAGE amber light for better crew awareness was missing in the cockpit/on the front panel. According to the pilots, mentioned comments are associated with the experimental installation and could be easily solved more or less with today s technology. Save date: Public Page 64

65 4 Surface Movement Awareness and Alerting System SMAAS; Traffic Conflict Detection; Ground-Air Data Base Upload (TUD) 4.1 Real Time Simulation Results Set Up The validation platform The fixed-base Research Flight Simulator of TUD s Institute of Flight Systems and Automatic Control is a highly modular and configurable research simulator, featuring a sophisticated collimated visual system consisting of a three-channel retro-projection with a viewing frustum of 180 horizontally and ±20 vertically. Pilots perceive the resulting image, which seems to be located at infinity, through a mirror. Consequently, refocusing occurs whenever pilots change their view from head-down activities inside the cockpit to the outside visual, and this approach guarantees an excellent approximation of reality for human factors evaluations [16]. The cockpit is not an exact reproduction of the flight deck of a specific aircraft, but deliberately kept at the more generic level of a modern glass cockpit with two flight crew members. The inside dimensions of the cockpit correspond to those of a modern widebody aircraft of the Airbus A330/340 family. While the flight simulation employs the flight mechanical model of an Airbus A300 B4, a flyby-wire flight control system with sidesticks was chosen because it represents the state of the art and allows an unobstructed view of the flight guidance displays. Figure 4-1: View of the TUD fixed-base Research Flight Simulator The cockpit features all primary and secondary controls; in some cases actual aircraft parts, such as an original A320 Flight Control Unit (FCU) and genuine thrust levers, contribute to enhance the level of immersion and realism. In other cases, as for the MCDU, parts obtained from a commercial supplier of flight simulation equipment have been used, and even some developments of the Institute were installed. As an example, the simulator features active side-sticks, which can also be operated in a standard mode. Flight guidance and system displays are presented on large 15 LCD screens, which can either, be arranged in portrait or landscape mode. Cockpit Configuration for the experiment Save date: Public Page 65

66 The First Officer (CM-2) crew station will be used for evaluation, mainly because this position is already equipped with an MCDU hardware mock-up. The two LCD displays for PFD and ND are arranged in portrait configuration, thus resembling the A380 display arrangement. The outer screen is used to display a standard Airbus style PFD. The inner screen will feature a conventional Airbus style ND supplemented by the Airport Moving Map and additional HMI features as described subsequently. Display control will be achieved via the conventional EFIS Control Panel, which features a workaround for the ranges below 10 nm: When selecting the constraints (CSTR) button, pilots can use the QNH selector to control AMM range Objectives and Hypotheses Introducing a new display system in the cockpit with all of its information could seduce the pilots to increase their head down time, due to the overload of information presented by the system while the pilots needs undue time to recognize the important information. Objective O1: Verify that the introduction of SMAAS does not tend to result in more Head down time for the pilots. Hypothesis H1-1: The SMAAS does not cause excessive extra head down time for the crew. With the introduction of the SMAA System in the cockpit displaying most of the important information for the crew directly on the ND, the crew could start trusting the system too much. Mainly the display of traffic information is dependent on the availability of corresponding equipment in the other aircraft (ADS-B) or on the airport (TIS-B). The crew must be aware at all times that the SMAAS only gives out the information available to the cockpit and therefore not trust the system completely. Objective O2: Assess the confidence the pilots have in the SMAAS. Hypothesis H2-1: The crew maintains a good level of confidence in the System the crew trusts what it sees on the display but stays aware of the limits of the System. The increase in situation awareness comes along with the increase in safety, because crew failures will be minimized. To ensure that the SMAAS increases situation awareness with its presentation of surface restrictions as well the associated alerts, Objective 3 was defined. Objective O3: Verify that the presentation of surface restrictions as well as alerts in addition with the Airport Moving Map will increase situation awareness. Hypothesis H3-1: Presenting closed runways and other pertinent runway or airport restrictions from NOTAM or similar sources is an essential feature to increase situational awareness on an airport moving map. The alerting features associated are operationally relevant. Similarly, to ensure that the SMAAS increases situation awareness with its presentation of traffic information as well as the associated alerts the Traffic Conflict Detection function, Objective 4 was defined. Objective O4: Verify that the presentation of traffic information as well as the associated alerts in addition with the Airport Moving Map will increase situation awareness. Hypothesis H4-1:: Presenting traffic information is an essential feature to increase situational awareness on an airport moving map. The alerting features associated are operationally relevant. Objective 5 was defined to verify that the overall design principles that should be used in the developing of aircraft systems will be adhered. Objective O5: Verify that the introduction of SMAAS follows the overall design principles used in the cockpit. Hypothesis H5-1: The HMI is consistent with existing standards/implementations. Taxiing along the airport belongs to the phases with the most workload for the pilots. Introducing a new system that will increase their workload in this phase is not desirable. Because it is unlikely to develop an information system that will have no effect on the workload it is desirable to develop a system with just a small increase of workload. Save date: Public Page 66

67 Objective O6: Verify that the introduction of SMAAS does not lead to any undue increase of workload. Hypothesis H6-1: The introduction SMAAS does not lead to any undue increase of workload Traffic Scenarios For the fixed base research simulator, seven different scenarios were used. The two first scenarios were used mostly to familiarize the pilot with the TUD Simulator and with the SMAAS System. Two locations were chosen for the scenarios. Frankfurt Airport (EDDF) was used for the familiarization scenarios: FAM 01 and FAM 02 as well as the first two SMAAS scenarios: SMA 01 and SMA 02. The following three SMAAS scenarios take place at the Charles-de-Gaulle Airport (LFPG) in Paris: SMA 03 to SMA 05. The scenarios are described in detail in the following paragraphs FAM 01 This scenario was developed, to get the pilot familiar with the simulator and the Airport Moving Map Display. For this reason no traffic was provided, while the pilot had to taxi out from Gate A16 to Runway 25R at Frankfurt Airport (EDDF). Figure 4-2: Scenario FAM 01 EMMA-FAM-01 Simulator & Airport Moving Map Familiarisation Remarks / ATC Location Frankfurt/Main Airport (EDDF) Familiarisation scenario in daytime VMC, CAVOK Objectives Background Crew Task Weather Primary: To familiarise the crew with the handling of TUD s flight simulator and to acquaint them with the airport moving map functionality. Secondary: To collect first comments on the overall HMI from the crew. Fictitious airline flight from Frankfurt (EDDF) to Paris Charles-de-Gaulle (LFPG). Taxi from Gate A16 to RWY 25R via M, N, G1, A and D. Daytime VMC, CAVOK Duration: ca. 20 min Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): M N G1 A D 07L.25R, activate hold short Save date: Public Page 67

68 Traffic Start of Scenario No traffic. Aircraft has been pushed back from parking position A16, the engines are running. Scenario Script ATC instructs the crew to taxi to 25R via M, N, G1, A and D, holding short of E. RWY 25R is marked as FMS RWY on the airport moving map. The SID/flight plan is visualized on the ND. The crew requests taxi instructions from ATC. Upon review/readback of the assigned route, the crew releases the parking brake and taxis to RWY 25R following the assigned route, with intermediate holdshort of E The crew holds short of RWY 25R and is subsequently cleared for line-up and takeoff. Alternative taxi clearance: M, A, D. Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Two-Five Right via Mike, November, Golf One, Alpha and Delta. Controller: Lufthansa Four Tango Uniform, proceed via Alpha and Delta, hold short of Runway Two-Five Right. Controller: Lufthansa Four Tango Uniform, line up and wait Runway Two- Five Right. Controller: Lufthansa Four Tango Uniform, cleared for takeoff Runway Two-Five Right, wind is calm, report airborne. Subjective Measurements Objective Measurements None for this scenario (familiarisation). None for this scenario (familiarisation) FAM 02 In this scenario a first alert will be provided. The pilot had to taxi from Gate A16 to Runway 07R. At the transit from Taxiway A to Taxiway W, which is the assigned taxi route, the pilot was asked, to stay on Taxiway A and head to Runway 18, which has been marked as a closed Runway, slowly. After that he had to resume the assigned taxi route (Figure 4-3). Figure 4-3: Scenario FAM 02 For this scenario no traffic movement will occur at the airport. EMMA-FAM-02 Simulator & Airport Moving Map Familiarisation Remarks / ATC Location Frankfurt/Main Airport (EDDF) Familiarisation scenario in night VMC, CAVOK Save date: Public Page 68

69 Objectives Background Primary: To familiarise the crew with the handling of TUD s flight simulator and to acquaint them with the airport moving map functionality. Secondary: To collect first comments on the overall HMI from the crew. Fictitious airline flight from Frankfurt (EDDF) to Paris Charles-de-Gaulle (LFPG). Duration: ca. 20 min Crew Task Taxi from Gate A16 to RWY 07L via M, A, W and R. Divert from assigned route and proceed straight onto closed RWY 18. Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): M A W R 07R.25L Weather Traffic Start of Scenario Scenario Script Subjective Measurements Objective Measurements Night VMC, CAVOK No traffic. Aircraft has been pushed back from parking position A16, the engines are running. ATC instructs the crew to taxi to 07R via M, A, W and R. At any rate, the assigned taxi route will be displayed on the airport moving map. RWY 07R is marked as FMS RWY on the airport moving map. The SID/flight plan is visualized on the ND. The crew releases the parking brake and follows the assigned route. The crew, however, does not make the turn into W, but continues straight onto the completely closed RWY 18. None for this scenario (familiarisation). None for this scenario (familiarisation). Alternative taxi clearance: M, N, L. Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Zero-Seven Left via Mike, Alpha, Whisky and Romeo SMA 01 The assigned Route is from Gate A16 to Runway 25L (Figure 4-4). Runway 18 is completely closed, while Runway 25R, which must be crossed by the pilot is in use only as Taxiway. Neither a highlighting nor the alerting systems (SMAAS) are enabled. Save date: Public Page 69

70 Figure 4-4: Scenario SMA 01 EMMA-SMA-01 Taxiing to RWY 07R and departure Remarks / ATC Location Frankfurt/Main Airport (EDDF) Line-oriented scenario Objectives Background Crew Task Weather Traffic Start of Scenario Primary: Line-oriented scenario without the use of the epib/sma function: no display of NOTAM information, only read at the beginning of the scenario. Secondary: Assessment of Situational Awareness with respect to NOTAM information. Fictitious airline flight from Frankfurt (EDDF) to Paris Charles-de-Gaulle (LFPG). Taxi from Gate A16 to RWY 25L via M, N, F, C and D. Night VMC, CAVOK Up to 10 traffic items Aircraft has been pushed back from parking position A16, the engines are running. Duration: ca. 15 min Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): M N F * F C D 07R.25L Scenario Script Subjective Measurements Objective Measurements The crew reads through a stack of NOTAM information including the complete closure of RWY 18 and the use of RWY 07L only as taxiway (not take-off/landing). ATC instructs the crew to taxi to 07R via M, N, F, C and D. The crew releases the parking brake and follows the assigned route. Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. None for this scenario. Alternative taxi clearance: M, A, D. Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Twenty-Five Left via Mike, November, Foxtrot, Charlie and Delta. Save date: Public Page 70

71 SMA 02 This scenario (Figure 4-5) is the same as the previous scenario (SMA 01), except, that the highlighting of active or closed Runways and the alerting system is enabled. Figure 4-5: Scenario SMA 02 During the taxiing, the pilot gets the information, that Runway 25R is not longer in use just as a Taxiway but can be used as a regular Runway. The pilot should use this information, to interact with the MCDU to unlock the Taxiway restriction from Runway 25R. After that he should proceed along his assigned route. EMMA-SMA-02 Taxiing to RWY 07R and departure Remarks / ATC Location Frankfurt/Main Airport (EDDF) Line-oriented scenario Objectives Background Crew Task Weather Traffic Start of Scenario Primary: Line-oriented scenario with assessment of the epib/sma function on a smaller/more familiar airport. Secondary: Assessment of Situational Awareness with respect to NOTAM information. Fictitious airline flight from Frankfurt (EDDF) to Paris Charles-de-Gaulle (LFPG). Taxi from Gate A16 to RWY 25L via M, N, F, C and D. Night VMC, CAVOK Up to 10 traffic items Aircraft has been pushed back from parking position A16, the engines are running. Duration: ca. 15 min Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): M N F * F C D 07R.25L Scenario Script The crew loads NOTAM information including the complete closure of RWY 18 and the use of RWY 07L only as taxiway (not take-off/landing) with the epib on the RFMS. Alternative taxi clearance: M, A, D. Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Twenty-Five Left via Mike, November, Save date: Public Page 71

72 ATC instructs the crew to taxi to 07R via M, N F, C and D. At any rate, the assigned taxi route will be displayed on the airport moving map. RWY 07R is marked as FMS RWY on the airport moving map. The SID/flight plan is visualized on the ND. The crew releases the parking brake and follows the assigned route. Foxtrot, Charlie and Delta. Subjective Measurements Objective Measurements Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. None for this scenario SMA 03 This scenario is related to scenario SMA 01. The crew had to taxi from Gate Z4 to Runway 09L, while Runway 09R and Runway 08L/26L are completely closed. Neither a highlighting nor the alerting system is enabled (Figure 4-6). The pilot can interact with the MCDU but does not get any feedback of his interaction visually on the displays since the highlighting and alerting system SMAAS is not on. Figure 4-6: Scenario SMA 03 EMMA-SMA-03 Taxiing to 09L and departure Remarks / ATC Location Paris Charles-de-Gaulle (LFPG) Line-oriented Scenario Objectives Primary: Line-oriented scenario without the use of the epib/sma function: no display of NOTAM information, only read at the beginning of the scenario. Secondary: Assessment of Situational Awareness with respect to NOTAM information. Duration: ca. 20 min Save date: Public Page 72

73 Background Fictitious airline flight from Paris Charles-de-Gaulle (LFPG) to Frankfurt (EDDF) or Amsterdam (EHAM). Crew Task Taxi out to RWY 09L, perform takeoff and fly SID. Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): NA1 N B DB D K2 Z2 09L.27R Weather Traffic Start of Scenario Scenario Script Subjective Measurements Objective Measurements Night VMC, CAVOK Up to 10 traffic items After a fictitious pushback from Gate Z4 at Terminal 1, the aircraft is on taxiway NA1. The crew reads through a stack of NOTAM information including the complete closure of RWY 09R/27L and the use of RWY 08L/26R only as taxiway (not take-off/landing). The crew is instructed to taxi to RWY 09L via NA1, N B, DB, D, K2, Z2 and to hold short of RWY 09R. If the crew notices that RWY 09R is completely closed, ATC excuses itself and says: yes this runway is now used as a taxiway. The crew is cleared for take-off on RWY 09L. Post-run and debriefing questionnaires Post-run and debriefing pilot comments Does the crew notice 09L: taxiway? Alternative taxi clearance: - Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Zero-Niner Left via November, Bravo, Delta-Bravo, Kilo Two and Zulu Two, hold short of Runway Zero-Niner Right. Controller: Lufthansa Four Tango Uniform, crossing Runway Zero-Niner Right is approved, hold short of Runway Zero-Niner Left. Controller: Lufthansa Four Tango Uniform, line up and wait Runway Zero- Niner Left. Controller: Lufthansa Four Tango Uniform, cleared for takeoff Runway Zero-Niner Left, wind is calm, report airborne SMA 04 Save date: Public Page 73

74 Figure 4-7: Scenario SMA 04 This scenario is similar to scenario SMA 03, except the active state of the highlighting and alerting system. Like in SMA 03 the pilot should interact with the MCDU to unlock the complete closure of Runway 09R (Figure 4-7) EMMA-SMA-04 Taxiing to 09L and departure Remarks / ATC Location Paris Charles-de-Gaulle (LFPG) Line-oriented Scenario Objectives Background Primary: Line-oriented scenario with assessment of the epib/sma function on a larger/less familiar airport. Secondary: Assessment of Situational Awareness with respect to NOTAM information. Fictitious airline flight from Paris Charles-de-Gaulle (LFPG) to Frankfurt (EDDF) or Amsterdam (EHAM). Duration: ca. 20 min Crew Task Taxi out to RWY 09L, perform takeoff and fly SID. Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): NA1 N B DB D K2 Z2 09L.27R Weather Traffic Start of Scenario Night VMC, CAVOK Up to 10 traffic items After a fictitious pushback from Gate Z4 at Terminal 1, the aircraft is on taxiway NA1. Scenario Script The crew loads NOTAM information including the complete closure of RWY 09R/27L and the use of RWY 08L/26R only as taxiway (not take-off/landing) Alternative taxi clearance: - Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Save date: Public Page 74

75 Subjective Measurements on the epib. The crew is instructed to taxi to RWY 09L via NA1, N B, DB, D, K2, Z2 and to hold short of RWY 09R. If the crew notices that RWY 09R is completely closed, ATC excuses itself and says: yes this runway is now used as a taxiway. The crew is cleared for take-off on RWY 09L. Post-run and debriefing questionnaires Post-run and debriefing pilot comments Zero-Niner Left via Bravo, Delta-Bravo, Kilo Two and Zulu Two, hold short of Runway Zero-Niner Right. Controller: Lufthansa Four Tango Uniform, crossing Runway Zero-Niner Right is approved, hold short of Runway Zero-Niner Left. Controller: Lufthansa Four Tango Uniform, line up and wait Runway Zero- Niner Left. Controller: Lufthansa Four Tango Uniform, cleared for takeoff Runway Zero-Niner Left, wind is calm, report airborne. Objective Measurements SMA 05 In this scenario, the pilot had to taxi from Gate Z4 to Runway 09R. The visibility was set to IFR conditions. At Runway 09R the crew was instructed, to line up after another aircraft. This aircraft was simulated as being without an ADS-B transponder. Thus it was present in the outside view but not shown on the Airport Moving Map Display, while other traffic was shown. Figure 4-8: Scenario SMA 05 The first aircraft, being number one for takeoff, started Takeoff, rejected it and remained with damaged tyres on the Runway. Meanwhile the pilot got his Takeoff clearance. EMMA-SMA-05 Runway Incursion on RWY 09R (Initial takeoff phase) Remarks / ATC Location Paris Charles-de-Gaulle (LFPG) Event scenario (Taxiway and Runway Incursion) Objectives Primary: Evaluation of Runway Incursion Alerting vs. Traffic display only Secondary: Assessment of airport moving map and traffic display. Duration: ca. 20 min Background Fictitious airline flight from Paris Charles-de-Gaulle (LFPG) to Frankfurt (EDDF) or Amsterdam (EHAM). Save date: Public Page 75

76 Crew Task Taxi out to RWY 09R, perform takeoff and fly SID. Display Configuration: Airbus Standard PFD and Airport Moving Map on ND SMAAS Route (if applicable): NA1 N B BD Y1 09R.27L Weather Traffic TCD Configuration Start of Scenario Scenario Script Night VMC, CAVOK Up to 10 traffic items All known Traffic is displayed. One aircraft (AFR7780) is simulated as not having an ADS-B Transponder and is thus only visible in the outside view and not known by either the Displays or the Alerting System. Runway and Taxiway alerts enabled After a fictitious pushback from Gate Z4 at Terminal 1, the aircraft is on taxiway NA1. The crew is instructed to taxi to RWY 09R via NA1, N, B, BD and Y1, and to hold short of RWY 09R. The crew is instructed to line up, and No. 2 for takeoff, the other aircraft (AFR7780) is cleared for takeoff, but rejects takeoff at 130 kts due to an engine failure and remains on the runway with damaged tyres. The alert will occur as soon as the ownship crew commences takeoff. If they notice that the RWY is not clear, they should be encouraged to commence takeoff all the same in order to demonstrate the alert. Alternative taxi clearance: NA1, N, B, DB, D, DY, Y3. Controller: Lufthansa Four Tango Uniform, taxi to Holding Position Runway Zero-Niner Right via November, Bravo, Bravo-Delta and Yankee One. Controller: Lufthansa Four Tango Uniform, line up and wait Runway Zero- Niner Right, You are Number Two for takeoff. Controller: Lufthansa Four Tango Uniform, cleared for takeoff Runway Zero-Niner Right, wind is calm, report airborne. Subjective Measurements Post-run and debriefing questionnaires Post-run and debriefing pilot comments Objective Measurements Participants Evaluation Crew A total of six male pilots, all from Lufthansa, participated in the trials. The mean age of these pilots was 33,8 years (between 27 and 41 years old) an they had a mean total number of flight hours logged in their entire career in commercial aviation of 6333h (between 1000h and 11000h). Among the pilots were two Captains, one Senior Flight Officers and three First Officers. Three of them had a Flight Instructor education. Three of these pilots are flying an Airbus 320 family (A318, A319, A320, A321) airplane as their current type, one a Boeing 737 classic version (-300, -500) and two indicated a Boeing as their current type. Save date: Public Page 76

77 One of the pilots had experience with airport moving map displays from previous campaigns at TUD. The pilots were asked to rate their familiarity with the airport at which the tests took place on a sixstep scale with text statements, ranging from It is my first time here to I know the layout of this airport almost better than that of my flat. All of the pilots participating in the scenarios at Frankfurt (EDDF) named, that they are familiar ( I know this airport very well. ) with this airport. At Paris (LFPG) most of them indicated, that they had been several times there. Just two of them stated that they were familiar with the airport or knew it very well Assessment Team The core assessment team for all the trials consisted of two TUD research scientists. These two research scientist were systems engineers who participated in the functional and HMI design of the Surface Movement Awareness and Alerting System (SMAAS) and who were also responsible for the creation and extension of the questionnaires used in the experiment. They acted as the experiment leaders, did the briefing as well as the familiarisation, took down pilots comments and assisted pilots in filling the questionnaires. Furthermore, in the scenarios were it was necessary one of the scientists took the role of an ATC controller, giving the crew the instructions concerning taxi route clearances as well as take off and line up Schedule of Executed Test Runs The pilots came to TUD for the whole day and were first invited to fill out the Pilot-Intake- Questionnaire [6]. They were then briefed on the motivation and the functionalities of the Surface Movement Awareness and Alerting with a short power-point presentation. After a brief exchange with the attending pilot concerning the presentation, he was brought to the fixed-base Research Simulator. The two familiarization scenarios at Frankfurt Airport (EDDF) were run as described in chapter Then the two SMAAS scenario taking place in Frankfurt were run before the lunch break. During the break, one of the TUD scientists prepared the simulator for the next scenarios taking place in Paris (LFPG). After the lunch, the pilot resumed the run of the scenarios and did scenarios SMA 03 through to SMA 05. After that, the pilot was asked to fill out the questionnaires described in chapter and a debriefing session followed. The typical course of an evaluation day was the following, where T0 was an individual start time for each pilot. The time is given in minutes. T0 - T0+30 Presentation - Briefing T T0+55 Training T T0+150 System evaluation EDDF scenarios T T0+195 Lunch T T0+285 System evaluation LFPG scenarios T T0+345 Debriefing Metrics and Indicators In general, there are two different methods of gathering information for a validation process. The first method consists in the quantitative measurement of some given indicators for the system and the objective assessment of the gathered data. The second method relies on the qualitative assessment of Save date: Public Page 77

78 subjective measurements, such as user opinions, comments, feedbacks. Since the objectives concerning the Surface Movement Awareness and Alerting System that was to be assessed during the Real Time Simulation trials are all linked to subjective notions such as the crew s situation awareness, the perceived workload or the human machine interface, it was decided to use only the second method: qualitative assessment of questionnaires and debriefing interviews [6]. During the scenarios the pilots comments were documented by one of the TUD scientists. After the exercise, data was collected by means of several debriefing questionnaires and a debriefing interview. The gathered data was then methodically analysed and evaluated Debriefing Questionnaire A set of three dedicated debriefing questionnaires was used as well as two Cooper Harper Rating Scale sheets. The dedicated questionnaires can be found in the EMMA2 document 2-D616 Validation plan airborne in chapter 9.6. Dedicated questionnaires The detailed questionnaires enabled the pilot to rate individual elements using 7 or 10 step Likert scales. The first questionnaire details the pilot s impressions on the airport moving map. The second questionnaire deals with the surface traffic display (CDTI) used in the system. The last questionnaire enables the pilot to state his views on the Surface Movement Awareness and Alerting System as a whole, including mostly the display of operational, clearance and traffic information, as well as the associated alerting mechanisms. Cooper Harper System (CHS) The CHS is used by starting at the lowest maturity level (i.e. the question Is the system usable? at the bottom left-hand corner) and progressing up the table while the question can be answered with a yes. When a question has to be answered with a no, the actual pilot rating is determined by the detailed nature of the identified deficiency, as defined on the right hand side of the table. This process results in the evaluator selecting a rating in a ten-point scale ranging from 10 (the system is unusable) to 1 (the system is fully mature). Save date: Public Page 78

79 Figure 4-9: Cooper Harper System Two Cooper Harper rating scales were used. One was used to assess the Ground to Air Database Upload Function including the electronic Pre-flight Information Bulletin (epib) and more particularly the MCDU interaction used in the trials. The other Cooper Harper rating scale was used to assess the Surface Movement Awareness and Alerting System as a whole Debriefing Interview The aim of the debriefing interview was to go deeply into the key points noticed in the observations during the trials or in the debriefing questionnaires. It helped to bring out the operational improvement possibilities of the system through different human factor aspects: - System usability - Confidence in the system - Situation awareness The debriefing interview also helped the team understand why some pilots may have a diverging answer from the majority of pilots, or from the expected answer. This was used intensively to understand the questionnaire results in the analysis phase. Save date: Public Page 79

80 4.1.2 Results to Operational Feasibility (RTS) Operational Feasibility Results The Operational Feasibility of the Surface Movement Awareness and Alerting Systems was assessed with detailed questioning of the pilots concerning the user acceptability of the system and also concerning their impressions on the Human Machine Interface used in the trials. The following figures show the given answers for six pilots from the RTS and two further pilots from the OST on a scale from 1 (disagreement) to 10 (agreement) Human Machine Interface To assess the HMI, the satisfactory of the AMM display colours was evaluated. Figure 4-10 shows that most of the pilots are satisfied with the colours on the display. Two of the pilots mentioned that one point that could be improved is the contrast for a FMS chosen Runway (the FMS chosen Runway is white outlined while the Runway is light grey). One of the pilots remarked that the red alert which overlays a runway where the conflict is predicted, should stay until the danger is not a factor anymore. This answer was taken up at the Prague trials and resulted by an improper work of the software at this time. Figure 4-10: AMM display colours are satisfactory Also the interpretation of the different colour codes (Figure 4-11) or symbols (Figure 4-12) affects the HMI. In general the interpretation of both easily pleased the pilots. Just for the symbols, improvements could be made. One pilot desires that traffic on runway should be displayed directly in red. In his opinion that would increase situational awareness. One pilot felt that the different symbols for traffic with know and traffic with unknown traffic should not be different. There should be a difference by the traffic type. Save date: Public Page 80

81 Figure 4-11: The different colour codes used are easy to interpret Figure 4-12: The used symbols are easy to interpret Also the information arrangement influences the HMI feeling for the pilots. In Figure 4-13 the answers are shown. A trend to the agreement with the current arrangement is visible while two pilots are not fully satisfied with the arrangement. But there had been no additional comments by the pilots about the improvement of the current system. Figure 4-13: Information is arranged conveniently Save date: Public Page 81

82 One point for the result shown in Figure 4-13 could be that some pilots do not fully agree with the statement that different types of information are easy to find which is shown in Figure Figure 4-14: Different types of information are easy to find The interaction with the system is also an important point for HMI assessment. Most of the pilots thought that the interaction through the use of the MCDU is satisfactory for them. This shows that the idea of using the MCDU for this type of interaction with the electronic Pre-Flight Information Bulletin goes in the right direction. Figure 4-15: AMM interaction concept is satisfactory Figure 4-16 shows that the amount of the information presented on the AMM is not too large. Additionally essential information and information that could be removed if necessary should be noted. Just one pilot answered this question by noting that taxiway names are essential information. Two pilots kept it more generally and noted that this depends on the traffic amount and that the display is a very nice adaption of info with range selection. Save date: Public Page 82

83 Figure 4-16: AMM information amount is too large A main problem using computer displays with textual information on it could be that this information could not be readable well. Therefore Figure 4-17 shows that most of the pilots thought that on the tested display the textual information are very well readable. Figure 4-17: Textual information is very well readable Pilots are familiar with ND which just displays the Runways on it. Therefore the use of an AMM on the ND could lead to an overload of information for some pilots. Error! Reference source not found. shows that just one pilot tend to see an overload of information with the use of AMM. His reason for that is noted below: Display on ND is very good, but overload is possible when a route (flight) is inserted. Save date: Public Page 83

84 Figure 4-18: With AMM the ND is overloaded with information As for safety the closed runway symbology must be intuitive for the HMI as well. Figure 4-19 shows that the pilots thought that the symbols used here are intuitive. Figure 4-19: Closed Runway symbology is intuitive and easy to understand Also for the colour presentation in Figure 4-20 the thoughts are nearly the same as in Figure Figure 4-20: Closed Runway colour presentation is consistent with overall cockpit concept Save date: Public Page 84

85 To complete the view of symbology and colour presentation Figure 4-21 shows the opinions of the pilots according the closed taxiway presentation of information. We have exactly the same result here as we have for Runways. Figure 4-21: Closed Taxiway representation is consistent with the closed Runway concept As mentioned before one pilot thought about a different symbol for traffic with know and traffic with unknown heading. Figure 4-22 aims to this consideration and the pilots opinion is visible in this figure as well. Although he has this opinion he thought that the traffic symbol is acceptable. That was because he had a briefing before the trails started, presenting him these symbols. And in all day business he will get familiar with it. It was also mentioned before that the colour of the traffic symbols is acceptable but one pilot noted that it will be easier for him if traffic on a runway will turn red (Figure 4-23). Figure 4-22: The shape of the traffic symbol is intuitive and acceptable Save date: Public Page 85

86 Figure 4-23: The colour of the traffic symbol is intuitive and acceptable User Acceptability For user acceptability it is essential that the user will see an increase in safety, situational awareness and that the presented information is consistent with existing information. One essential point to gain user acceptability is that the system is easy to handle. A lot of pilots agree with the statement in Figure 4-24 that the AMM is easy to handle, although the interaction is not the same as it probably will be in future aircrafts. Figure 4-24: Handling of the AMM As mentioned in the Safety paragraph the number of selectable ranges is sufficient for most of the pilots (Figure 4-25). They can comfortably choose their preferred range setting or change it during taxiing to fulfil the requirements during different phases (taxi along taxiway, approaching runway, etc.) Save date: Public Page 86

87 7 21. The number of selectable ranges is sufficient c y n e u q fre Figure 4-25: The number of selectable ranges is sufficient Also the display size is acceptable for them. Although there is a larger screen in the simulator (see chapter ) some pilots thought that a larger screen could be better because the amount of information will increase The size of the display is acceptable. 5 4 c y n e u 3 q fre 2 1 Figure 4-26: Display size is acceptable Also the AMM is accepted by the pilots as was already mentioned in HMI paragraph. 4,5 23. The ND is overloaded with information if the Airport Moving Map is added. 4 3,5 3 2,5 c y n e u q fre 2 1,5 1 0,5 Save date: Public Page 87

88 Figure 4-27: With AMM the ND is overloaded with information 4,5 2) I like the way of presenting the FMS selected runway information on th display. 4 3,5 3 2,5 c y e n u q fre 2 1,5 1 0,5 Figure 4-28: Presentation of FMS-selected runway on the display 4,5 14. This traffic presentation is definitely an improvement over the system we have today. 4 3,5 3 2,5 c y e n u q fre 2 1,5 1 0,5 Figure 4-29: Traffic presentation is an improvement over the systems we have today It is essential that aircraft approaching the runways and aircraft taking o are displayed as well. 5 4 c y e n u 3 q fre 2 1 Figure 4-30: as well? It is essential that aircraft approaching the runways and aircraft taking off are displayed Save date: Public Page 88

89 4.1.3 Results to Operational Improvements (RTS) Safety (HLO_2) Safety depends on situational awareness and workload. So a full picture about safety could only occur after the survey of these two points. At the beginning stands a general question about the support the AMM gives. The pilots feel heterogeneously supported by the system compared to the current system. Although a majority of the pilots totally agreed the AMM gives them support they miss with current systems, some pilots were not as enthusiastic. It must be mentioned that the pilots on the trials used different systems. Some of the Lufthansa pilots use the RAAS function (Runway Awareness Advisory System) and thus didn t have the same baseline system to compare with as the others. 4,5 2. The Airport Moving Map gives me support I miss with current systems 4 3,5 3 2,5 c y n u e q fre 2 1,5 1 0,5 Figure 4-31: Does AMM give missed support? As mentioned, a good situational awareness is necessary to operate safely. To allow an improvement in situational awareness with the Surface Movement Awareness and Alerting System, the number of selectable ranges must be sufficient so that the pilots can build a good mental picture of all the information depicted on the displays. This is all the more necessary when alerts come so that the pilot has a sufficient overall view of the situation The number of selectable ranges is sufficient c y n u e q fre Figure 4-32: The number of selectable ranges is sufficient Save date: Public Page 89

90 Figure 4-32 shows that the number of selectable ranges was deemed sufficient for the pilots to have all information they needed. But not only should all necessary information be available. Also a quick recognition of that information is essential. That means that the pilots should get important information with a short head down time. Therefore an overloaded ND is not desirable the pilot should not have to filter out unnecessary information. This is all the more important to ensure that an unnecessary overload of information doesn t diminish the reactivity to an alert. Most of the pilots thought that the tested ND implementation is not overloaded with information. But they mostly noted that it will be necessary to limit the objects that will be displayed on the ND particularly the traffic at some big airports. It has to be noted here that the traffic simulated in the real time simulation was much less than it would have been in real life. One pilot thought that the display is overloaded because in his opinion an overload of information could occur when a flight route is inserted in addition to the displayed information but it is important to note here that the ranges at which the flight route can be observed, and the ranges used for the SMAAS are quite different. Indeed the AMM would be practically not readable when the pilot is studying his flight route and vice versa. 4,5 23. The ND is overloaded with information if the Airport Moving Map is added. 4 3,5 3 2,5 c y e n q u fre 2 1,5 1 0,5 Figure 4-33: With AMM the ND is overloaded with information Surface Movement Alerting When improvement of situation awareness isn t enough to prevent a potential incident, alerting occurs in the Surface Movement Awareness and Alerting System. This alerting function s goal is also to increase safety when the increase in awareness thanks to the Operational, Clearance, Traffic and Position Awareness Functions still isn t enough to prevent wrong behaviour or when another aircraft has an unexpected and potentially dangerous behaviour. So for that reason, a question concerning an alert when attempting to take off from a runway other than the selected one in FMS was listed (Figure 4-34) Most of the pilots thought that it will be operationally relevant to have this alert because own mistakes will not lead to an endangerment of others. Save date: Public Page 90

91 7 20) An alert when attempting to take off from a runway other than the on selected in the FMS is operationally relevant c y u e n q fre Figure 4-34: Operational relevance of alerts when attempting to take off other than FMS selected A similar question is whether the pilots want to be alerted when entering a runway that is completely closed or when trying to land on it (Figure 4-35). The necessity of an alert was mostly agreed fully because in this case an error preceded and the pilots thought that they could roll or land on the runway. 7 24) I would like to have an alert when I am entering a runway that is completely closed, e.g. due to construction, or trying to land on c y e n u q fre Figure 4-35: Alert when entering closed runway or try to land on it Also for closed Runway alerting in general was asked. That means not just for closed runways due to constructions that will cause damage in the a/c. Here also an alerting for operationally closed runways is demanded. Figure 4-36 shows that the pilots thought similar to the preceding question but the relevance is not this high like it is for closed runway due to constructions. Save date: Public Page 91

92 6 25) This type of alert will make flying safer in the future. 5 4 e n c y u q fre Figure 4-36: This type of alert will make flying safer in the future Also following correctly a clearance and route given by ATC is a safety feature. For that reason the taxi route cleared by ATC is taken into account in the system: the route is displayed in green on the AMM. The pilot can thus monitor his own position and compare it with the defined taxi route. If he leaves this taxi route the route starts to blink on the display to attract the attention of the pilot. Most of the pilots thought that this is a useful feature that increases safety Figure ,5 28) The taxi route monitoring function is a useful feature; the blinking rout attracts my attention to the display. 4 3,5 3 2,5 c y e n u q fre 2 1,5 1 0,5 Figure 4-37: Taxi route monitor function is useful; blinking route attracts attention Similar to a takeoff from a Runway other than the FMS selected, a takeoff from a taxiway bears a high risk. So an alert when a crew tries to take off from a taxiway will contribute a high value to flight safety. Most of the pilots thought the same and fully agreed with the statement (Figure 4-38). Save date: Public Page 92

93 6 31) Taxiway takeoff alerting is a valuable contribution to flight safety. 5 4 e n c y u q fre Figure 4-38: Taxiway takeoff alerting is a valuable contribution to flight safety Preventing a takeoff from a taxiway just by displaying an AMM is not enough for some of the pilots as Figure 4-39 shows. That shows that an alert mentioned before is necessary to increase flight safety. 4,5 32) If my moving map display shows that I am currently on a taxiway, I wi never try to take off. 4 3,5 3 2,5 c y u e n q fre 2 1,5 1 0,5 Figure 4-39: When AMM displays that I m on taxiway I will never try to takeoff The pilots comments about the improvement of the traffic presentation were mostly positive. Just one pilot was undecided. This pilot missed speed information for traffic items. This feature would probably lead to clutter if it was shown for all traffic, but could be implemented to show the speed of traffic going faster than the advised taxi speed, or for traffic in the immediate vicinity of ownship. The pilot particularly missed this information during an alert and felt it could, in that case, help him stay in the loop CDTI Save date: Public Page 93

94 4,5 2. How would you rate the contribution of this traffic presentation to fligh safety, i.e. safety of ground operations? 4 3,5 3 2,5 c y u e n q fre 2 1,5 1 0,5 Figure 4-40: Rate of the contribution of this traffic presentation to flight safety 4,5 14. This traffic presentation is definitely an improvement over the system we have today. 4 3,5 3 2,5 c y e n u q fre 2 1,5 1 0,5 Figure 4-41: This traffic presentation is definitely an improvement over the systems we have today In Figure 4-42 the pilots comments about the usage of the system especially at night prior to or during takeoff is listed. Most of them will use it as additional information. But mostly prior to takeoff or used by the second pilot. One pilot mentioned that he will never use it during takeoff because his eyes should be outside all the time and also the second pilot should watch the outside. If the runway is clear of traffic or not must be cleared before takeoff. It was indeed shown in scenario SMA 05 that all the pilots looked outside during the take off and did not solely depend on what they saw in the CDTI. Indeed in this scenario one traffic item was simulated as not being equipped with ADS-B and was thus not assessed by the system: neither to be seen on the display nor to cause an alert. This traffic item was seen in the outside world and despite Low Visibility Conditions and the fact that no alert came from the system, the pilots still noticed the danger. This is a key issue showing that CDTI can already be used to increase safety even if not all traffic items are properly equipped and confirms hypotheses H2-1. Save date: Public Page 94

95 6 20. Especially at night and when visibility is low, I would like to use this display prior to and during takeoff to check whether the runway is really cle of traffic. 5 4 c y e n 3 q u fre 2 1 Figure 4-42: Use of CDTI in Low Visibility and Night to check Runway before Take Off Although the danger of an overload of the display due to high information density is given, most of the pilots agree with the necessity to display approaching or aircraft taking off (Figure 4-43). That is because these aircraft are a real risk for them due to the high speeds involved in those flight phases It is essential that aircraft approaching the runways and aircraft taking o are displayed as well. 5 4 c y u e n 3 q fre 2 1 Figure 4-43: Aircraft approaching or taking off should essentially be displayed Traffic Conflict Detection 7 18) Runway incursion alerting contributes to an increase in flight safety e n c y u q fre Figure 4-44: Runway Incursion alerting increases flight safety Save date: Public Page 95

96 In figure 4-44 we can see that the pilots all felt that Runway Incursion alerting will contribute to an increase in flight safety. This alerting function was deemed by the pilots as the most important part of the traffic conflict detection function. 6 19) With taxi route and runway related clearance (like lineu p, ta clearance) displayed on the moving map, it is unlikely that there be a runway in cu rsion. 5 4 c y e n u 3 q fre Figure 4-45: Necessity of Runway Incursion Alerting despite augmented Situational Awareness As shown in Figure 4-45, a majority of pilots thought that the Runway Incursion alerting function will stay necessary even if complete CPDLC upload to the displays is available, i.e. taxi route and runwayrelated clearances. This confirms the concept of the Surface Movement Awareness and Alerting System: the system must give all the information it has to the pilot so that the situational awareness increases. This leads in the best case to an absence of alerts. But in case the crew still isn t fully aware of the situation and/or another aircraft makes a mistake, the alerting is still necessary. 3,5 23) Runway incursion alerting does not require a change of current procedures. 3 2,5 2 c y n e u q fre 1,5 1 0,5 Figure 4-46: Runway incursion alerting doesn't require new procedures Although all the pilots agreed that the runway incursion alerting function would increase flight safety, not all of them thought it could be introduced without changing the current procedures, as can be seen in figure The alerts triggered by the system are informative ( runway incursion callout) and thus do not give the pilots an instruction as to what they should do. Some pilots noted this should then be given through new procedures, to tell them how to act when the given alerts occur. Save date: Public Page 96

97 3,5 24. Traffic alerting on the ground should be limited to conflicting traffic on the runway. 3 2,5 2 c y u e n q fre 1,5 1 0,5 Figure 4-47: Traffic Alerting limited to Runway Figure 4-47 shows that pilots largely differ on their opinions regarding traffic alerting outside the runway. While some fully disagreed that traffic alerting should be limited to conflicting traffic on the runway, some did agree with the sentence, while others had no particular opinion. All pilots agreed that this function is not as safety critical as the prevention of runway incursion, and found the system s option to limit alerts on taxiways to a maximum of level 2 (master caution), while runway incursion alerting reaches level 3 (master warning). But some even found that it would cause nuisance alerts and would not increase safety, while others found the function useful Suitability of Behaviour and Working Performance (HLO_4) Situation Awareness (LLO4.2) The pilots were asked about their feeling about the increasing of situational awareness provided by the Airport Moving Map. Figure 4-48 shows their answers. Most of the pilots appreciate the introduction of AMM concern in the increase of situational awareness. The question if it is possible to get lost on an airfield while using the AMM gives more adequate answers to situational awareness (Figure 4-49). Here the increase is not this distinct as shown in Figure But this is because some of the experimental constrains could create such a meaning by the pilots. One pilot for example gave two answers. One he mentioned, and this is the answer that is given in this document, represents his opinion about the actuality of the AMM underlying database. This opinion shows a neutral attitude to the statement. But if the database will be up to date, as it must be to be established in all day airline operation, he will agree with the statement that it won t be possible to get lost. To increase situational awareness it could be reasonable to highlight the FMS selected takeoff or landing runway. Nearly all pilots fully agree with that point because on complex airports this information will attract attention out of the user structures (Figure 4-50). Also the distinction of different closure levels is desired by the pilots to increase their situational awareness (Figure 4-51). Furthermore, the alerting mechanisms which occur when the pilot has lost the necessary situational awareness were deemed very useful by the pilots. Indeed all pilots agreed with the concept of alerting in case the pilot takes off on another runway as the one selected in the FMS (Figure 4-52). All the pilots also agreed that they would like to be alerted in case they were either landing on or attempting to enter a completely closed runway, e.g. because of constructions works (Figure 4-53). To know the own route to prevent search for them on maps, etc is essential for situational awareness (Figure 4-54). The pilots agree with it because it could be a problem for pilots which are new at an airport to find the correct taxiways. The pilots mostly do not fully agree with the statement, because at Save date: Public Page 97

98 the time the pilots are familiar with an airport the taxi route will just help them in critical situations and is not mainly responsible for increasing situational awareness because the pilots have the layout in their mind The Airport Moving Map helps to increase situational awareness. 5 4 c y n e q u fre Figure 4-48: AMM increase situational awareness? 2,5 8. With the Airport Moving Map, it is not possible to get lost on an airfield any more. 2 1,5 e n c y q u fre 1 0,5 Figure 4-49: With AMM it is not possible to get lost on airfield 7 1) The presentation of the FMS selected takeoff or landing runway on the airport moving map is relevant from an operational point of view c y e n u q fre Figure 4-50: FMS selected runway presentation is operationally relevant Save date: Public Page 98

99 4,5 7) The distinction of different closure levels is operationally relevant. 4 3,5 3 e n c y u q fre 2,5 2 1,5 1 0,5 Figure 4-51: The distinction of different closure levels is operationally relevant 7 20) An alert when attempting to take off from a runway other than the on selected in the FMS is operationally relevant c y e n u q fre Figure 4-52: An alert when attempting to take off from runway other than the FMS selected is operationally relevant 7 24) I would like to have an alert when I am entering a runway that is completely closed, e.g. due to construction, or trying to land on c y e n u q fre Figure 4-53: Alert when entering a completely closed runway Save date: Public Page 99

100 4,5 28) The taxi route monitoring function is a useful feature; the blinking rout attracts my attention to the display. 4 3,5 3 2,5 c y u e n q fre 2 1,5 1 0,5 Figure 4-54: Usefulness of taxi route monitoring function In Figure 4-55 the answers to the overall increase in situational awareness with ground traffic on the moving map is listed. All pilots agree or fully agree with the statement to display traffic on AMM to increase situational awareness How would you rate your overall traffic situational awareness with groun traffic on the moving map? c y u e n q fre Figure 4-55: Overall situation awareness with ground traffic on AMM In Figure 4-56 the answers according to the question if the presented traffic labels are operational useful is given. This information could help the pilots in identifying traffic and/ or to have additional information about them. Most of the pilots wish to have traffic labels or they think that they are intuitive to use. The pilot who disagrees with the statement wants no abbreviations for the callsigns, because in his opinion the pilots will not know all abbreviations but they can identify a company so that the company name on the display will be more useful. Save date: Public Page 100

101 3,5 9. The presented traffic labels are intuitive and operationally useful. 3 2,5 e n c y u q fre 2 1,5 1 0,5 Figure 4-56: Usefulness of presented traffic labels Figure 4-57 strengthens the answers that were given in Figure Most of the pilots see an improvement over the systems used today. 4,5 14. This traffic presentation is definitely an improvement over the system we have today. 4 3,5 3 2,5 c y u e n q fre 2 1,5 1 0,5 Figure 4-57: Traffic presentation as improvement for the pilots One important point to show that the tested system increases situational awareness is shown in Figure Here the statement is given that the display allows the pilot to establish a correspondence between the traffic outside and the traffic on the map. Most of the pilots agree with this statement just the transformation from a two-dimensional to a three-dimensional system takes some time for them so that not a full agreement for some of them is given. But the most important point is that if display clutter occurs or many items are next to one another, the identifying could be hindered. Save date: Public Page 101

102 3,5 15. The display allows me to establish a correspondence between the traffi see outside and the traffic on the map. 3 2,5 2 c y u e n q fre 1,5 1 0,5 Figure 4-58: Correspondence between traffic outside and traffic on map Workload (LLO4.1) Information that are not needed and which are displayed could increase the pilots workload because their head down time increases while they are searching for the relevant information. Most of the pilots disagreed with the statement that the system displays too much information, although they do not fully disagree due to the low traffic rate in the simulator (Figure 4-59). One pilot mentioned that the flight phases which are displayed are not really necessary for him. 4,5 13. Sometimes information I do not need is displayed. 4 3,5 3 2,5 c y n u e q fre 2 1,5 1 0,5 Figure 4-59: Sometimes information not needed is displayed Save date: Public Page 102

103 3,5 14. Different types of information are easy to find. 3 2,5 2 c y u e n q fre 1,5 1 0,5 Figure 4-60: Different types of information are easy to find 2,5 15. The concept of interaction with the Airport Moving Map is satisfactory 2 1,5 c y n u e q fre 1 0,5 Figure 4-61: AMM interaction concept is satisfactory Figure 4-62 shows a similar diagram than Figure Although the pilot in Figure 4-59 thought that there are some information which are not necessary he also thought that the amount of information are not too much. 4,5 16. The amount of information on the Airport Moving Map is too large. 4 3,5 3 2,5 c y n u e q fre 2 1,5 1 0,5 Figure 4-62: AMM information amount is too large Save date: Public Page 103

104 As for safety the taxi route feature will also be useful to assure the workload will not increase by the system (Figure 4-63). It will be easier to follow the taxiway due to the mental picture the AMM will allow and the taxi route must not bear in mind all the time. Also the correct taxiway could be found more easily. 4,5 28) The taxi route monitoring function is a useful feature; the blinking rout attracts my attention to the display. 4 3,5 3 2,5 c y e n u q fre 2 1,5 1 0,5 Figure 4-63: Taxi route monitoring is useful; blinking route attracts attention Figure 4-64 shows the results to the question if the presentation of ground traffic increases the workload. Two pilots mentioned that there will be a small increase in workload because they will see traffic on their display that is probably out of their visual sight. That means that they get information they normally don t have and they have to decide if they have to be aware of them or not. 3,5 3. Does the presentation of ground traffic increase your perceived workloa 3 2,5 n c y e u q fre 2 1,5 1 0,5 Figure 4-64: Does presentation of ground traffic increase workload? Save date: Public Page 104

105 EMMA2 Airborne Validation Results Part A 4.2 Operational Flight Trials Results Set Up The validation platform TUD s Navigation Test Vehicle, a Volkswagen LT 28 van, is operated by the Institute of Flight Systems and Automatic Control and was originally conceived as a mobile test platform for the development and validation of navigation sensors and equipment. It features an array of high-quality sensors suited for precise navigation and a measurement data acquisition and logging system. Postprocessing software allows the generation of highly precise reference trajectories with a CEP95 of 0.44 m based on sensor logs. Figure 4-65: TUD Van during tests on closed runway 04/22 on Prague airport (LKPR), February 6th, 2006 Furthermore, as a by-product, the high-precision navigation solution also provides the prerequisites for the verification and validation of equipment used on aircraft, particularly in the domain of cockpit displays for surface movement, either in the form of software prototypes or real hardware. Consequently, TUD s Navigation Test Vehicle can be used to simulate a taxiing aircraft and enables low-cost live field tests of such on-board functions in a real airport environment, and thus an assessment of the potential impact of factors such as database and navigation accuracy (e.g. IRS drift and GPS outages or multi-path effects) or the quality of data link date (e.g. ADS-B) Navigation System A Honeywell H764 inertial reference system forms the core of the van s precision navigation system, which is supplemented by an array of various GPS receivers ranging from off-the-shelf to high accuracy measurement models. The sensor mounting positions were measured within centimetre accuracy to correct errors caused by different sensor locations. An overview to the sensor equipment is given in Table 4-1. A Wiener filter is used for the implementation of an optimum sensor data fusion combining inputs from the H764 and a Novatel RT20 GPS receiver with D-GPS capabilities. To provide differential correction data to the on-board receivers, a portable reference station can be installed at points of which the exact position is known. The range of the D-GPS broadcasts depends on the surrounding environment and can vary between several dozens and some hundred meters, provided there is a free line of sight between rover- and reference station. As a backup, D-GPS can be obtained from the ALF Save date: Public Page 105 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

106 EMMA2 Airborne Validation Results Part A service, which is available within 600 km of Frankfurt/Main, Germany. For the EMMA trials, however, no D-GPS correction data were used, because the shakedown trials had shown that a sufficient real-time navigation precision in the order of a few meters could be achieved with GPS only. Figure 4-66: Antenna mounting plate on the roof of TUD's Navigation Test Vehicle Equipment In the back of TUD s Navigation Test Vehicle, one row of seats has been removed to make room for two custom-built, interconnected racks that hold the hardware and computer equipment. Two inverters in the back are used to convert the 12V DC supplied by the vehicle s electric bus to 250V/50 Hz AC required to drive the PC equipment. Save date: Public Page 106 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

107 EMMA2 Airborne Validation Results Part A Figure 4-67: Hardware and computer racks in the back of TUD s Navigation Test Vehicle In its standard configuration, TUD s Navigation Test Vehicle is equipped with at least two PCs interconnected by Ethernet. The Navigation PC is responsible for collecting sensor data and the realtime calculation of the GPS/IRS sensor data fusion. It is equipped with both a MIL-1553 and an ARINC 429 card to interface with common aircraft avionics. The other PC is used for multiple purposes, such as data logging, acquiring traffic data and calculating a traffic data fusion or hosting software prototypes of on-board functions such as an Airport Moving Map. Space permitting, further PCs can be added and connected to the vehicle s Ethernet network. ADS-B live traffic as well as TIS-B data issued from Prag-Airport during the OST in November can be received using a Filser RT60 ADS-B receiver (purchased via Eurotelematik) that sends out traffic data via Ethernet. Furthermore, the vehicle is equipped with Wireless LAN (WLAN), which can be used to access TIS-B traffic acquired through Frankfurt airport s Cooperative Area Precision Tracking System (CAPTS), a Mode-S multilateration surveillance system developed by Thales and Sensis. The TIS-B over WLAN data link was developed for the ETNA project, where this feature was designed to bring traffic awareness to airport vehicles, particularly the airport fire engines. The passenger seat served as a very simple cockpit mock-up for the pilot evaluations and was therefore fitted with a 15" off-the-shelf LCD display (conventional PC monitor) and a Logitech trackball as a basic Crew Control Device (CCD), both provisionally mounted by the help of Velcro tape. Although the LCD used is far from the ruggedized, high-contrast AMLCD screens used in aircraft cockpits, the display readability was very good even in back light conditions. Another limitation of the screen used is that its diagonal is a factor of 1.5 larger than the 6" x 8" screens currently used in the Airbus A380. Save date: Public Page 107 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

108 EMMA2 Airborne Validation Results Part A Figure 4-68: LCD screen installation in front of passenger seat For evaluations on the manoeuvring area, the van was equipped with a turning light, and licensed drivers with the required radio permissions which were kindly supplied by Prague airport authorities. In Prague, additionally, an autonomous ADS-B transponder requiring only power supply was installed on the van s roof. Sensor/ HW Manufacturer Technical Data Remarks H764 (INU) Honeywell Position <1.0 [nmi/hr] CEP velocity <1.0 [m/s] RMS Heading 0.1 [deg] RMS Pitch, Roll 0.05 [deg] RMS RT20 L1 (GPS) Novatel 12 channel L1 20 cm CEP50 (RT2 DGPS) RT2 L1/L2 (GPS) Novatel 12 CH L1/L2 2 cm CEP50 (RT2 DGPS) SK8 (GPS) Trimble 8 CH L1 2 m CEP50 (DGPS) GPS 35 Garmin 12 CH L1 5 m RMS (DGPS) Ashtech G12 Thales 12 channel L1 40cm CEP50 (DGPS) reference RT2, station Hermes Novatel, Phillips L1/L2 RTCM Type 59N INU medium accuracy High accuracy High accuracy Low cost receiver Low cost receiver GA certified reference required. Save date: Public Page 108 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04 point

109 EMMA2 Airborne Validation Results Part A modem (DGPS, RTCA Type 7 VHF link) VHF Modem: 459,57Mhz Odometer Bosch 56 pulse/round ABS controller ALF receiver DeTex RTCM-104 v2.0 Available within (DGPS ) 122,5 khz 600 km radius of Frankfurt MS5535 Intersema -40 C to +125 C Parport PC (Barometer/ Thermometer) 1-12 bar Interface RT60 Filser 1090 Mhz, ADS-B and TIS- Enhanced Squitter B receiver Table 4-1: Sensor equipment of TUD s Navigation Test Vehicle Traffic Scenarios The scenarios were divided in two packages of scenarios. One package was developed to provide scenarios with the General Aviation Aircraft available and the other simulating traffic with two ground vehicles. For the scenarios with the ground vehicles, the south east located area was used, containing runway 04/22 and the surrounding taxiways M and P. TUD Van simulates the own aircraft, with the flight crew for the trials. A second vehicle from the Prague Airport simulates other traffic. For the scenarios with the General Aviation Airplane, trying to land on a Runway, Runway 31 was used, with the TUD van trying to takeoff from Runway 13. Save date: Public Page 109 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

110 EMMA2 Airborne Validation Results Part A Figure 4-69: Overview of Prague airport (LKPR) EMMA-VAN-01: Take Off RWY 04 Traffic On/Approaching RWY This scenario describes a takeoff from the active Runway 22. TUD Van stands cleared for takeoff on Runway 22. Meanwhile a vehicle simulating other traffic taxiing over Taxiway P and crosses the stop bar to Runway 22 and roll onto the Runway. EMMA-VAN-01 Take Off RWY 04 Traffic On/Approaching RWY Remarks / ATC Location Crew Task Prague Airport (LKPR) Van: Start T/O (simulated through flight phase) from RWY 04 at the level of TWY R. Display Configuration: Airport Moving Map on ND Traffic LKPR Vehicle: Roll onto RWY 04 using Taxiway P. Scenario Script The crew is cleared for takeoff on RWY 04. Traffic rolls onto RWY 04 while ownship is already in take off mode. Controller: Lufthansa Four Tango Uniform, cleared for takeoff Runway Zero Four, wind is calm, report airborne. Subjective Measurements Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Save date: Public Page 110 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

111 EMMA2 Airborne Validation Results Part A Objective Measurements Logging of Traffic position and Alerts. Figure 4-70: Take Off RWY 04 Traffic On/Approaching RWY EMMA-VAN-02: Take Off RWY 04 without clearance This scenario describes a takeoff from Runway 22. TUD Van received the line up clearance on Runway 22. The crew starts taking off from Runway 22. EMMA-VAN-02 Take Off RWY 04 without clearance Remarks / ATC Location Crew Task Traffic Scenario Script Subjective Measurements Objective Measurements Prague Airport (LKPR) Van: Start T/O (simulated through flight phase) from RWY 04 at the level of TWY R. Not relevant. The crew starts taking off from RWY 04 after receiving the line up clearance Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Alerts. Display Configuration: Airport Moving Map on ND Controller: Lufthansa Four Tango Uniform, line up and wait Runway Zero Four. Save date: Public Page 111 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

112 EMMA2 Airborne Validation Results Part A Figure 4-71: EMMA-VAN-02: Take Off RWY 04 without clearance EMMA-VAN-03: Collision Hazard with other Traffic on Taxiway This scenarios describes a taxiing situation, where the own airplane rolls in front of a second airplane. EMMA-VAN-03 Collision Hazard with other Traffic on Taxiway Remarks / ATC Location Prague Airport (LKPR) Crew Task Van: Roll along Taxiway M. Display Configuration: Airport Moving Map on ND Traffic LKPR Vehicle: Roll along Taxiway M. Scenario Script Subjective Measurements Objective Measurements The crew rolls on Taxiway M (middle) towards RWY 22. Traffic rolls behind the van and starts getting closer. Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Traffic position and Alerts. If necessary, the speed of the traffic will be artificially increased (simulation) to cause the alert. TUD van rolls along Taxiway M to Taxiway P via Taxiway L. The vehicle simulating the following airplane rolls along the same Taxiways and starts getting closer during the scenario. Save date: Public Page 112 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

113 EMMA2 Airborne Validation Results Part A Figure 4-72: EMMA-VAN-03: Collision Hazard with other Traffic on Taxiway EMMA-VAN-04: Collision with other Traffic on Taxiway In this scenario the own airplane taxi along Taxiway L from East to West and is cleared to cross Runway 04. The intruder rolls along Taxiway R from East to West. At the intersection of Taxiway R and Taxiway L the intruder causes a collision hazard. EMMA-VAN-04 Collision with other Traffic on Taxiway Remarks / ATC Location Crew Task Traffic Prague Airport (LKPR) Van: Roll along Taxiway L from East to West, having been cleared to cross RWY 04. LKPR Vehicle: Roll along Taxiway R from East to West. Display Configuration: Airport Moving Map on ND Scenario Script The crew rolls on Taxiway L towards RWY 04. Traffic rolls on Taxiway R, causing a collision hazard at the intersection. Subjective Measurements Objective Measurements Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Traffic position and Alerts. Save date: Public Page 113 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

114 EMMA2 Airborne Validation Results Part A Figure 4-73: EMMA-VAN-04 Collision with other traffic on taxiway EMMA-VAN-05: Crossing RWY 04 without clearance In this scenario TUD Van rolls on Taxiway L towards Runway 04. The crew has not received any clearance. The crew crosses Runway 04. EMMA-VAN-05 Crossing RWY 04 without clearance Remarks / ATC Location Crew Task Traffic Scenario Script Subjective Measurements Objective Measurements Prague Airport (LKPR) Van: Roll along Taxiway L from East to West and cross RWY 04. Not relevant. The crew rolls on Taxiway L towards RWY 04 and crosses the runway without having received any clearance. Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Alerts. Display Configuration: Airport Moving Map on ND Save date: Public Page 114 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

115 EMMA2 Airborne Validation Results Part A Figure 4-74: EMMA GA-01 Takeoff of RWY 13, Traffic on Approach RWY 31 Figure 4-75: EMMA-VAN-05: Crossing RWY 04 without clearance EMMA GA-01 EMMA GA02 EMMA-VAN-GA-01 Runway Incursion #1 / GA Final Approach Remarks / ATC Location Van Crew Task Prague Airport (LKPR) Roll along Taxiway L from West to East and cross RWY 22. Display Configuration: Airport Moving Map on ND Save date: Public Page 115 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

116 EMMA2 Airborne Validation Results Part A GA Crew Task Final Approach on RWY 13. Scenario Script The GA crew is in final approach on RWY 13. The van crew rolls on RWY 22 towards RWY 13 and crosses the runway while GA is at 1000ft. CPDLC is not available. As soon as the alert is triggered in the GA, the crew executes a Go Around. As soon as the alert is triggered in the Van, the crew exits the runway. If no alert has come when the GA is at 700ft, the GA crew executes a go around. Subjective Measurements Objective Measurements Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Alerts from the Van and the GA. EMMA-VAN-GA-02 Runway Incursion #3 / GA in Take Off mode Remarks / ATC Location Van Crew Task GA Crew Task Prague Airport (LKPR) Roll along Taxiway M from West to East and cross RWY 04. Simulate Take-Off on RWY04 by accelerating to a ground speed above 20kts. Display Configuration: Airport Moving Map on ND Scenario Script Subjective Measurements Objective Measurements The GA crew starts Take Off from RWY 04 after having received the clearance from ATC. The van crew rolls on Taxiway M towards RWY 04 and crosses the runway. As soon as the alert is triggered in the GA, the crew aborts take off. As soon as the alert is triggered in the Van, the crew exits the runway. Post-run and debriefing questionnaires. Post-run and debriefing pilot comments. Logging of Alerts from the Van and the GA. If no alert has come when the GA is at 40kts, the GA crew aborts take off. During these trials it was foreseen to perform in principle two different scenarios. 1. An aircraft approaching a runway blocked by a vehicle. 2. An aircraft taking-off, but again with blocking vehicle on the present runway. Initially the approach scenario was planned to happen on the closed Runway 22 with the intention to not disturb the normal traffic. The initial planning for the Take-Off scenario was to start the aircraft on an operating runway (RWY13) or alternatively also on the closed RWY22 and locate the ground vehicle in a blocking way either by crossing RWY13 on RWY22 or with the other option by crossing RWY22 on a taxiway. After discussion with the local ATC and the airport operator the scenarios were adjusted in terms of impact on safety and traffic capacity. It is against controller rules to allow an approach or take-off on a closed runway, furthermore the blocking of an operational runway needs to be avoided. The finally proposed scenario outline for the approach was to simulate a landing on RWY 13, while the ground vehicle (TUD Van) crosses RWY 13 on RWY22. The aircraft would finish the approach with a go- Save date: Public Page 116 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

117 EMMA2 Airborne Validation Results Part A around and land afterwards. In the case of take-off, the scenario was limited to a simulation only. Since the airborne conflict detection algorithm identifies the take-off phase by a certain increase of the ground speed. More details on the algorithms applied for approach and take-off as well can be found in the SP2-Specification-GA-MovingMap_V1.0.doc. No further planning had been made, it was foreseen to have a briefing with pilots, drivers, coordinators and tower controllers before the start of activities. This meeting was deemed to consider operational restriction etc. as well Participants Evaluation Team Two male pilots, all from Czech Airlines, participated in the trials. The mean age of these pilots was 38 years (between 34 and 42 years old) an they had a mean total number of flight hours logged in their entire career in commercial aviation of 4950h (between 3400h and 6500h). Among the pilots were 1 Captain, who had a Flight Instructor Education and 1 Senior Flight Officer. The types the pilots are logged actual are the Airbus 320 family (A318,A319,A320,A321) and Boeing 737 classic version (-300,-500). Two of the pilots have experiences with airport moving map displays from previous campaigns at TUD respectively EMMA. Pilots were asked to rate their familiarity with the airport at which the tests took place on a six-step scale with text statements, ranging from It is my first time here to I know the layout of this airport almost better than that of my flat. The pilots at Prague named that they are very familiar ( I know this airport very well., I know the layout of this airport almost better than that of my flat. ) with their airport. The GA Team consisted of one pilot and one research scientist. The pilot is a flight trainer employed by the flight club Assessment Team The core assessment team for all the trials consisted of two TUD research scientists. These two research scientist were systems engineers who participated in the functional and HMI design of the Surface Movement Awareness and Alerting System (SMAAS) and who were also responsible for the creation and extension of the questionnaires used in the experiment. They acted as the experiment leaders, did the briefing as well as the familiarisation, took down pilots comments and assisted pilots in filling the questionnaires. At the Prague Airport Trials two more research scientist from TUD joined the assessment crew. One of them was mainly responsible for the operation of the navigation equipment in the Navigation Test Vehicle. The other one drove the van at the Prague Airport. In addition to the four TUD engineers, one engineer from Diehl Aerospace and one engineer from Funkwerk Avionics attended the trials at Prague Airport. For the trials without the G/A Aircraft, an engineer from the Prague Airport Authority played the role of the traffic conflicting with the TUD van Schedule of Executed Test Runs First TUD OST The first TUD on-site trials at the Prague Airport (LKPR) took place from the 25 th to the 29 th of August The time schedule of the trials can be seen in the figure below. Save date: Public Page 117 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

118 EMMA2 Airborne Validation Results Part A CW 35 in 2008 Arrival Trial Preparations Pilot trial Pilot trial G/A Trial Preparations ADS-B traffic monitoring Departure G/A trials Figure 4-76: Time schedule for first TUD OST During each trial without the G/A Aircraft the five scenarios described in chapter were tested. Two pilots from Czech Airlines (CSA) took part in those trials. The pilots first invited to fill out the Pilot-Intake-Questionnaire (see annex). They were then briefed on the motivation and the functionalities of the Surface Movement Awareness and Alerting with a short power-point presentation. After a brief exchange with the attending pilot concerning the presentation and a brief training showing the interaction possibilities with the display, the scenarios EMMA2-VAN-01 through to EMMA2-VAN-05 were run. During these trials, the conflicting traffic was simulated by a vehicle from the Prague Airport Authority equipped with an ADS-B transponder and driven by one of their engineers. The typical course of an evaluation trial was the following, where T0 was an individual start time for each pilot. T0 - T0+30 Presentation Briefing T T0+40 Training T T0+130 System evaluation T T0+195 Debriefing The trials with the G/A Aircraft were conducted without the presence of a pilot in the TUD van. Only the assessment team, as described above, was present. Before those trials, a briefing took place in the tower of the Prague Airport to discuss the trials with the responsible controllers and the pilot of the G/A Aircraft. For a detailed report on the trials with the G/A aircraft please see the following chapter Safety Net for On-Site Testing For the approach scenario an abort point was defined during preparatory discussions with the Tower Controller. That abort point was chosen to be at the runway intersection with taxiway P (Papa). This intersection was defined and shall never be transgressed by the aircraft, meaning the cusp of the turn being well in front or in any case not beyond an imagined line through that point. Save date: Public Page 118 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

119 EMMA2 Airborne Validation Results Part A The Take-Off scenario only required an acceleration to a speed above 20kts, in order to simulate a take-off and trigger an alert. However, during the test while moving on the south-western part of the closed runway 22 no crossing of the operating RWY31/13 was allowed Second TUD OST The second TUD on-site trials at the Prague Airport (LKPR) took place from the 17 th to the 21 st of November The time schedule of the trials can be seen in the figure below. The goal of these trials was mainly to test the TIS-B traffic data at the Prague Airport which were not available in the first TUD OST and also to assist the DLR OST with the ATTAS (see chapter ). On the 18 th of November a briefing took place in the tower of the Prague Airport for all the participants of the DLR OST and TUD OST. After this briefing session, the TIS-B functionality was switched on in the tower for the whole duration of the TUD OST. CW 48 in Arrival Trial Preparations ADS-B/TIS-B monitoring + trial preparations with ATTAS ADS-B/TIS-B monitoring + trial preparations with ATTAS ADS-B/TIS-B monitoring Departure Figure 4-77: Time schedule for second TUD OST Save date: Public Page 119 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

120 EMMA2 Airborne Validation Results Part A On the rest of the 18 th of November and on the 19 th of November the TUD Team assisted the ATTAS Team for the preparation of the Traffic Conflict Detection Trials and in particular in getting proper traffic information on-board. In parallel, the TUD Team made necessary modifications to its software to be able to correctly monitor and record ADS-B as well as TIS-B traffic information. The TUD Van was escorted all along the northern part of the airport by an engineer from the Prague Airport Authority. It was only possible to receive the TIS-B broadcast in the northern part of the airport. On the 20 th of November, the TUD Van was posted outside the airport premises north of the tower so as to be able to have a good view of runway 24 and be able to receive ADS-B traffic information as well as TIS-B traffic information Metrics and Indicators In general, there are two different methods of gathering information for a validation process. The first method consists in the quantitative measurement of some given indicators for the system and the objective assessment of the gathered data. The second method relies on the qualitative assessment of subjective measurements, such as user opinions, comments, feedbacks. Since the objectives concerning the Surface Movement Awareness and Alerting System that was to be assessed during the Real Time Simulation trials are all linked to subjective notions such as the crew s situation awareness, the perceived workload or the human machine interface, it was decided to use only the second method: qualitative assessment of questionnaires and debriefing interviews. During the first on-site trials at Prague, the TUD team noticed quite a lot of problems linked to the use of ADS-B traffic information. This is why it was decided to use quantitative measurements for this particular aspect. During the scenarios the pilots comments were documented by one of the TUD scientists. After the exercise, data was collected by means of several debriefing questionnaires and a debriefing interview. The gathered data was then methodically analyzed and evaluated Debriefing Questionnaire A set of three dedicated debriefing questionnaires was used as well as a Cooper Harper Rating Scale sheet. The dedicated questionnaires can be found in the EMMA2 document 2-D616 Validation plan airborne [6] in the annex. Dedicated questionnaires The detailed questionnaires enabled the pilot to rate individual elements using 7 or 10 step Likert scales. The first questionnaire details the pilot s impressions on the airport moving map. The second questionnaire deals with the surface traffic display (CDTI) used in the system. The last questionnaire enables the pilot to state his views on the Surface Movement Awareness and Alerting System as a whole, including mostly the display of operational, clearance and traffic information, as well as the associated alerting mechanisms. Cooper Harper System (CHS) The CHS is used by starting at the lowest maturity level (i.e. the question Is the system usable? at the bottom left-hand corner) and progressing up the table while the question can be answered with a yes. When a question has to be answered with a no, the actual pilot rating is determined by the detailed nature of the identified deficiency, as defined on the right hand side of the table. This process results in the evaluator selecting a rating in a ten-point scale ranging from 10 (the system is unusable) to 1 (the system is fully mature). Contrary to the Real Time Simulator Trials, only one Cooper Harper rating scale was used, since there was no MCDU mock-up available in the TUD Van, and thus no way of assessing the Ground to Air Database Upload Functionality in particular the MCDU interaction with the electronic Pre-flight Information Bulletin (epib). The Cooper Harper rating scale was used to assess the Surface Movement Awareness and Alerting System as a whole. Save date: Public Page 120 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

121 EMMA2 Airborne Validation Results Part A Debriefing Interview The aim of the debriefing interview was to go deeply into the key points noticed in the observations during the trials or in the debriefing questionnaires. It helped to bring out the operational improvement possibilities of the system through different human factor aspects: System usability Confidence in the system Situation awareness The debriefing interview also helped the team understand why some pilots may have a diverging answer from the majority of pilots, or from the expected answer. This was used intensively to understand the questionnaire results in the analysis phase Traffic logs For the assessment of the quality of traffic information coming from ADS-B data and from TIS-B data, a logging function was implemented and used during the second on-site trials at the Prague airport. The function logged all available information about every traffic item, and whether the information was coming from the tower (TIS-B) or the aircraft itself (ADS-B) Technical and Operational approval (FAV / TUD trials) The trials took place at the 27 th of August. After a flight trip of 2 hours the Piper reached Prague airport at 10:45am local time. The next item on the agenda was a meeting with the tower controllers, in order to operationally decide upon order, timing and procedures of the tests. It was decided to have the Approach scenario first because the traffic density at Prague was low at that time. Hence, the Take-Off simulation followed. Figure 4-78: Approach Scenario - Entry positions (FAV aircraft and TUD Van) Save date: Public Page 121 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

122 EMMA2 Airborne Validation Results Part A After that briefing the scenario outline was nearly similar to the planned one. The major change was that the ground vehicle (TUD VAN) was not moving but standing on a fix position on the runway for both scenarios. Another change was that the position of the TUD Van for the Approach scenario was now between intersections of RWY31 with taxiway D and F. The entry positions of the scenarios can be seen in the picture above. Approach Scenario: At 1255 UTC the Piper started its engine in order to perform low radar approaches on RWY 31. As initiation point the compulsory report point TANGO (see picture below) was selected. Figure 4-79: Approach Scenario (FAV aircraft) At that point a permission for a low radar approach had been requested which shortly after confirmation by ATC began. For safety reasons ATC instructed, the approaching aircraft must never exceed the intersection of RWY31 with taxiway P (PAPA). By consideration of this abort point the Piper turned right heading back to TANGO for requesting the next approach loop. In this way 4 missed approaches and one landing had been performed always staying in contact with the TUD Van. The contact was established by LKPR owned handheld radios. Save date: Public Page 122 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

123 EMMA2 Airborne Validation Results Part A Figure 4-80: Recorded ADS-B track during approaches The picture above is a presentation of the recorded ADS-B track during the approaches. While closing the blocked runway the CDTI triggered alerts as soon as the distance to the as intruder identified target was less than 2 NM or the approach angle to the Runway direction diverted less than 15. In the picture above the activation of the incursion alerts are shown by red rectangles, while the deactivation is indicated by green rectangles. The indication of a runway incursion on the CDTI can be seen below. Save date: Public Page 123 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

124 EMMA2 Airborne Validation Results Part A Figure 4-81: CDTI in FAV OST All runway incursion are logged by the CDTI and attached with the actual timestamp synchronized by GPS, supporting the post-trial evaluation. So it is possible to visualize the position at which the alert was raised and discarded as well. The picture below shows the ADS-B broadcasted position at which the runway incursions were triggered. Figure 4-82: ADS-B broadcasted positions at which the RWY incursions were triggered Take-Off Scenario: Save date: Public Page 124 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04

125 EMMA2 Airborne Validation Results Part A After landing and a short briefing the take-off simulation followed. For this kind of scenario the closed runway 22 was used. While the aircraft was accelerating to above 20 kts on the south-western part of the runway in direction 04, the TUD-Van was standing between taxiways L and P. While performing the simulations the runway incursion was triggered normally as seen in the case of the approaches. At one time the indicated position of the target was overdue, because of signal losses. After a change of the Van s position the signal was received again. Most likely the signal losses were due to the altitude profile of the runway and also the single antenna installation at the bottom of the aircraft. From six runs one was unsuccessful because of the overdue position, so in this case no runway incursion alert was triggered. The on-board system has been approved by deploying the ADS-B receiver (RTH60) as ground station in Egelsbach. The ADS-B receiver is the same as it is used within the TUD Van. The SMA functionality of the CDTI has been verified by using test equipment at the facilities of Funkwerk Avionics. In this way several scenarios could be simulated and the software was approved as ready for on-site testing. The final check if TUD Van and GA-aircraft work together happened at the Prague airport shortly before the trials. During the on-site trials the function of the SMA algorithm has been approved, the scenarios were set up to comply with the respective conditions of the following parameters: height above runway, speed and direction as well as distance to the target vehicle. As a baseline for the post-trial evaluation the logging function of the CDTI-2000 has been applied. The log file contains records of ADS-B target positions as well as records of the own state. Furthermore the CDTI recorded the triggering time of runway alerts. All the records in the log are accompanied by a timestamp derived by the CDTI system clock which itself is synchronised with the GPS time derived by the Trimble 2000 GPS device Results to Technical Feasibility (FAV Trials) During the trials TUD realized some significant delays in position transmission via ADS-B, not between the TUD Van and the GA-aircraft but with other real life traffic. The aircraft installation comprises two GNSS data sources, one for the CDTI ownship depiction and one for the ADS-B out functionality. As the installations allowed to receive the ADS-B messages broadcasted by the own transponder, this enabled to compare the different positioning sources by GPS timestamp. It was also possible to attempt the comparison between the timestamps of the different records at the same position (the delay) but this would be more significant using one single source for both (ADS-B Out and ownstate recording). The following chart shows the result of this analysis: Figure 4-83: Comparison between the positioning sources Save date: Public Page 125 File Name: 2-D661a_VO-TR_V1.0.doc Version: 0.04