ESA Telecom Startup Projects ITT AO-5114 SATWAYS 6/023. Final review

ESA Telecom Startup Projects ITT AO-5114 SATWAYS 6/023 Final review Presentation of the Pilot Results 1

Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call evaluation procedure and results Radio call evaluation procedure and results

Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call testing procedure and results Radio call testing procedure and results

The SATWAYS solution network architecture

The users of the SATWAYS solution The users at the heliport: The pilots Ground personnel, either HCAA maintenance staff or local authorities with proper permissions to use the duty telephones The flight controllers, who may reside: At the Flight Information Center position, responsible for Visual Flight Rules (VFR) flights (helicopters and small airplanes) At other positions, responsible for Instrument Flight Rules (IFR) flights (high-level flights) The user at the Coordinating Center (CC), which belongs to HCAA personnel

The evaluation strategy Prior to the initiation of the pilot, the solution was tested in the presence of HCAA, so as to get a green light For the pilot evaluation, 2 sources of data were used: User interviews Logs collected through the SATWAYS solution A common timebase was used (Network Time Protocol using a GPS clock as source) All components of the solution were synchronized with it The main focus of the pilot evaluation was on the voice service and particularly radio calls

Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Since the functionality of these services relied on commercial products, the only aspect in need of evaluation was the use of satellite transmission Heliport and weather monitoring, as well as video surveillance are uni-directional services, so the delay impact was neither important nor perceivable Control commands fulfilled the original requirements set forth by HCAA These services were tested extensively prior to the pilot phase As a result, they were accepted as is

Evaluation procedure for telephone calls First used during the first installation at the heliport of Patmos, a big part of which has been carried out over the phone Used from time-to-time, all users that have used it reported toll quality and no differences from the experience provided by PSTN This result has convinced HCAA to move on to the deployment of the voice service to radio calls. Formal testing has been carried out during August 2010, based on questionnaires derived from ITU-T s E.125 and P.82 Recommendations

Evaluation results for telephone calls The user group involved employees of the Municipality of Patmos Short-lived testing phase, since after 10 tries everybody has responded: Excellent to the question Which of these four words comes closest to describing the quality of the connection during conversation? No, to the question Did you have any difficulty talking or hearing over that connection? No point in asking more questions Other comments included: Are you sure you are using a satellite? I do not perceive any echo or noise I hear loud and clear

Evaluation procedure for radio calls (1) Initiated on 20/07/2010 Official request of the Communications, Navigation and Surveillance (CNS) Directorate of HCAA towards the Air Traffic Control (ATC) Directorate and acceptance by the latter Flight controllers agreed to test the voice service in a best-effort manner until at least the end of September 2010 Their extreme workload during the summer period due to increased air traffic in the area had to be considered

Evaluation procedure for radio calls (2) It was agreed that both VFR and IFR flights had to be contacted for testing purposes to enlarge the test sample and create more representative results Officially, the pilot involved just 2 controllers (one for VFR and one for IFR flights) More than 15 have actually operated the solution up to now All other services were operational during the User Acceptance Testing

Evaluation procedure for radio calls (3) The controllers were requested to evaluate the quality of their communications using questionnaires The results were then cross-examined with logs collected by the SATWAYS solution The questionnaires were based on the Controller Acceptance Rating Scale (CARS) method, developed by FAA Questions were posed regarding: Target data (time of the communication, callsign, flight level, target position, flight type IFR/VFR) The quality of the voice The interference of the delay on: The effectiveness of the communications The timing of the control instructions The use of correct phraseology The speech quality (or clarity) The speech rate General comments (including feedback from the pilot)

Evaluation procedure for radio calls (4) The results were gathered in lots, so that possible problems were identified and resolved quickly 4 such problems, documented in the ensuing, were discovered Successive versions of the results were made available to ESA

Radio call evaluation results An overview The SATWAYS solution has been used numerous times for operational activities The controllers were rating it in writing only when their workload was reduced and flight safety was not compromised by their actions 21 evaluation reports were gathered The evaluation is ongoing The remainder of the presentation records: The output of the pre-processing of the controllers evaluation reports The evaluation results (both from reports and logs)

Pre-processing of the radio call reports (1) Report 1, Report 2 & Report 3 indicated a problem, which was described as: The frequency does not get cut off, so this creates an uncertainty as to whether the transmitter is emitting or not. Also, I cannot hear a feedback of my voice when I emit and this creates an uncertainty The problem has been identified as a configuration error in the DSP firmware of the VEF The problem was quickly resolved and testing was resumed

Pre-processing of the radio call reports (2) Within Report 4, Report 6, Report 7, Report 8, Report 9, Report 10, Report 11, Report 14, Report 17 and Report 18, the controller indicated a light hiss (or noise) The pilot did not mention something similar All controllers identified this impairment as minor HCAA has been contacted and no further action was advised

Pre-processing of the radio call reports (3) Many comments within the reports had to do with radio coverage issues. This means that: The controllers began trusting the SATWAYS solution and using it for control purposes instead of a pure radio check, as was the initial plan The controllers focused on the radio coverage issues and forgot all about satellite transmission, i.e. they started treating the SATWAYS solution like any other landline communication medium they are accustomed to. HCAA was provided with the details of the installation and asked to provide a radio coverage map in order for the authors to crosscheck the controllers comments All remaining reported problems concerned flights that were marginally within the radio coverage of the VHF transceiver

Pre-processing of the radio call reports (4) The call logs from the SATWAYS solution indicated that at some point in time between the 5:00 AM and 9:00 AM UTC on the 1st of August 2010, the packet loss of the satellite link was multiplied by a factor of 5, indicating a potential problem A periodic pinging facility between the 2 ends of the link was installed and unveiled the following: 1 out of approximately 20 ping packets got lost and the pattern was deterministic Once every x minutes, we experience a 20-second outage of the satellite service. This behaviour can be predicted to the second, though x varies, most often being 50 minutes The circular grade drops gradually to below 80, then returns gradually to 100, then back to 80 and so on and so forth. This situation is also depicted on the top led on the front of the master station The header CRC error and the Frame Plan Missed Total counters increase This indicated an interference from another station: Both stations were turned off on the 20th of August with the help of the municipality of Patmos and Hellas Sat provided the evidence that this is interference from an unknown source The interference has disappeared on the same day

Results of the processing of radio call reports ID Flight level Responsible controller Type of flight Time Date Question 1 Question 2 Question 3 Question 4 Question 5 Question 6 1 32000 CP IFR 9:10 20/7/2010 9 2 2 2 2 2 2 32000 GW IFR 9:20 20/7/2010 9 2 2 1 1 1 3 36000 SE IFR 9:30 20/7/2010 8 3 4 3 3 2 4 20000 PO IFR 9:40 20/7/2010 8 2 2 2 2 2 5 20000 MR IFR 9:40 20/7/2010 8 2 2 2 2 2 6 24000-9000 AJ IFR 21:10 21/7/2010 8 2 2 2 2 2 7 25000-8500 GX IFR 21:20 21/7/2010 8 2 2 2 2 2 8 16500 CU VFR 11:00 25/7/2010 9 2 2 2 2 2 9 6900 PD VFR 11:10 25/7/2010 9 2 2 2 2 2 10 4800 TN VFR 6:25 26/7/2010 9 1 1 1 1 1 11 4500 MP VFR 6:30 26/7/2010 9 1 1 1 1 1 12 12500 TS VFR 5:00 1/8/2010 9 1 1 1 1 1 13 9000 XD VFR 5:20 1/8/2010 9 1 1 1 1 1 14 5500-12000 CK VFR 5:30 1/8/2010 8 1 1 1 1 1 15 9000 VO VFR 5:20 1/8/2010 9 1 1 1 1 1 16 10400 SC VFR 9:00 1/8/2010 9 1 1 1 1 1 17 6500 PV VFR 8:10 13/8/2010 9 1 1 1 1 1 18 Ground-2000- CP VFR 8:20 13/8/2010 9 1 1 1 1 1 ground 19 13500 PV VFR 10:50 30/8/2010 9 1 1 1 1 1 20 11500 XD VFR 10:50 30/8/2010 9 1 1 1 1 1 21 9500 GW VFR 10:55 30/8/2010 9 1 1 1 1 1 Average: 8.71 1.48 1.52 1.43 1.43 1.38 Standard deviation: 0.46 0.60 0.75 0.60 0.60 0.50

Results of the processing of radio call logs Report ID Flight level Responsible controller Type of flight Time Date Average number of Tx buffer frames - Patmos Average number of Tx buffer frames - HCAA Maximum jitter - Patmos (ms) Maximum jitter - HCAA (ms) Average packet loss - Patmos (%) Average packet loss - HCAA (%) 1 32000 CP IFR 9:10 20/7/2010 53.5 20.625 20.375 0.25612 0.00590 2 32000 GW IFR 9:20 20/7/2010 53.5 20.625 20.375 0.25612 0.00590 3 36000 SE IFR 9:30 20/7/2010 53.5 20.625 20.375 0.25612 0.00590 4 20000 PO IFR 9:40 20/7/2010 53.5 20.625 20.375 0.25612 0.00590 5 20000 MR IFR 9:40 20/7/2010 53.5 20.625 20.375 0.25612 0.00590 6 24000-9000 AJ IFR 21:10 21/7/2010 34 20.625 20.625 0.63408 0.53583 7 25000-8500 GX IFR 21:20 21/7/2010 34 20.625 20.625 0.63408 0.53583 8 16500 CU VFR 11:00 25/7/2010 64 20.625 20.625 0.65196 0.01011 9 6900 PD VFR 11:10 25/7/2010 64 20.625 20.625 0.65196 0.01011 10 4800 TN VFR 6:25 26/7/2010 50 20.625 21.25 0.43343 1.14303 11 4500 MP VFR 6:30 26/7/2010 50 20.625 21.25 0.43343 1.14303 12 12500 TS VFR 5:00 1/8/2010 66 20.75 22.375 0.43950 0.73815 13 9000 XD VFR 5:20 1/8/2010 66 20.75 22.375 0.43950 0.73815 14 5500-12000 CK VFR 5:30 1/8/2010 66 20.75 22.375 0.43950 0.73815 15 9000 VO VFR 5:20 1/8/2010 66 20.75 22.375 0.43950 0.73815 16 10400 SC VFR 9:00 1/8/2010 64 20.625 18.375 0 3.50432 17 6500 PV VFR 8:10 13/8/2010 65 16.5 79.625 5.42629 5.75644 18 Ground- CP VFR 8:20 13/8/2010 65 16.5 79.625 5.42629 5.75644 2000- ground 19 13500 PV VFR 10:50 30/8/2010 64 N/A 24.25 N/A 0.00263 20 11500 XD VFR 10:50 30/8/2010 64 N/A 24.25 N/A 0.00263 21 9500 GW VFR 10:55 30/8/2010 64 N/A 24.25 N/A 0.00263

Summary of the evaluation results (1) The users rated the solution with: An average of 8.71 on a scale from 1 to 10 (10 is best) for voice quality An average of 1.38-1.52 on a scale from 1 to 10 (1 is best) regarding the effect of delay on their perception, effectiveness, timing, correct phraseology and speech rate Patmos logs for reports 19, 20 and 21 have been lost due to the combination of a power failure and a misconfiguration of the RSC-10 The recorded maximum jitter values are slightly higher than the larger ones of those originally required The maximum jitter values are almost constant and equal per transmission direction The packet loss values: Are considerably higher with respect to those originally required sometimes considerably larger than the 0.5% requirement (more than tenfold increase) After the termination of the interference, they have dropped to nearly zero These deviations do not seem to affect the radio call quality at all, since exceptional grades are awarded in any case

Summary of the evaluation results (2) The following report summarizes everything the SATWAYS project was initially targeting at: This was a flight with destination the Patmos heliport. The frequency was checked upon departure from the heliport of Mykonos (translation note: another island about 45 miles West of Patmos) and throughout the flight until landing to the heliport of Patmos. At a distance of 15 nautical miles east of the RIPLI region (translation note: controller jargon for an area within the Greek airspace), the pilot reported a 5/5 radio communications quality and so did the flight controller with little noise. At 20 nautical miles west of Patmos and at a flight level of 1500 feet, both the pilot and controller reported 5/5 quality. Beginning the descent to the Patmos heliport, both the transmission and reception were exceptional. On the ground, the communication quality was also exceptional and the flight plan has been closed using radio comms. The pilot has expressed to me his admiration, since traditionally this area had no radio coverage. I consider that the use of satellite communications is a success and covers our requirements for the heliport up to ground level

Thank you for your attention