ESA Telecom Startup Projects ITT AO-5114 SATWAYS 6/023. Final review

Transcription

1 ESA Telecom Startup Projects ITT AO-5114 SATWAYS 6/023 Final review Presentation of the Pilot Results 1

2 Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call evaluation procedure and results Radio call evaluation procedure and results

3 Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call testing procedure and results Radio call testing procedure and results

4 The SATWAYS solution network architecture

5 The users of the SATWAYS solution The users at the heliport: The pilots Ground personnel, either HCAA maintenance staff or local authorities with proper permissions to use the duty telephones The flight controllers, who may reside: At the Flight Information Center position, responsible for Visual Flight Rules (VFR) flights (helicopters and small airplanes) At other positions, responsible for Instrument Flight Rules (IFR) flights (high-level flights) The user at the Coordinating Center (CC), which belongs to HCAA personnel

6 The evaluation strategy Prior to the initiation of the pilot, the solution was tested in the presence of HCAA, so as to get a green light For the pilot evaluation, 2 sources of data were used: User interviews Logs collected through the SATWAYS solution A common timebase was used (Network Time Protocol using a GPS clock as source) All components of the solution were synchronized with it The main focus of the pilot evaluation was on the voice service and particularly radio calls

7 Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call evaluation procedure and results Radio call evaluation procedure and results

8 Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Since the functionality of these services relied on commercial products, the only aspect in need of evaluation was the use of satellite transmission Heliport and weather monitoring, as well as video surveillance are uni-directional services, so the delay impact was neither important nor perceivable Control commands fulfilled the original requirements set forth by HCAA These services were tested extensively prior to the pilot phase As a result, they were accepted as is

9 Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call evaluation procedure and results Radio call evaluation procedure and results

10 Evaluation procedure for telephone calls First used during the first installation at the heliport of Patmos, a big part of which has been carried out over the phone Used from time-to-time, all users that have used it reported toll quality and no differences from the experience provided by PSTN This result has convinced HCAA to move on to the deployment of the voice service to radio calls. Formal testing has been carried out during August 2010, based on questionnaires derived from ITU-T s E.125 and P.82 Recommendations

11 Evaluation results for telephone calls The user group involved employees of the Municipality of Patmos Short-lived testing phase, since after 10 tries everybody has responded: Excellent to the question Which of these four words comes closest to describing the quality of the connection during conversation? No, to the question Did you have any difficulty talking or hearing over that connection? No point in asking more questions Other comments included: Are you sure you are using a satellite? I do not perceive any echo or noise I hear loud and clear

12 Presentation contents The SATWAYS solution users and evaluation strategy Evaluation procedure and results for the control/monitoring/video surveillance and weather monitoring services Telephone call evaluation procedure and results Radio call evaluation procedure and results

13 Evaluation procedure for radio calls (1) Initiated on 20/07/2010 Official request of the Communications, Navigation and Surveillance (CNS) Directorate of HCAA towards the Air Traffic Control (ATC) Directorate and acceptance by the latter Flight controllers agreed to test the voice service in a best-effort manner until at least the end of September 2010 Their extreme workload during the summer period due to increased air traffic in the area had to be considered

14 Evaluation procedure for radio calls (2) It was agreed that both VFR and IFR flights had to be contacted for testing purposes to enlarge the test sample and create more representative results Officially, the pilot involved just 2 controllers (one for VFR and one for IFR flights) More than 15 have actually operated the solution up to now All other services were operational during the User Acceptance Testing

15 Evaluation procedure for radio calls (3) The controllers were requested to evaluate the quality of their communications using questionnaires The results were then cross-examined with logs collected by the SATWAYS solution The questionnaires were based on the Controller Acceptance Rating Scale (CARS) method, developed by FAA Questions were posed regarding: Target data (time of the communication, callsign, flight level, target position, flight type IFR/VFR) The quality of the voice The interference of the delay on: The effectiveness of the communications The timing of the control instructions The use of correct phraseology The speech quality (or clarity) The speech rate General comments (including feedback from the pilot)

16 Evaluation procedure for radio calls (4) The results were gathered in lots, so that possible problems were identified and resolved quickly 4 such problems, documented in the ensuing, were discovered Successive versions of the results were made available to ESA

17 Radio call evaluation results An overview The SATWAYS solution has been used numerous times for operational activities The controllers were rating it in writing only when their workload was reduced and flight safety was not compromised by their actions 21 evaluation reports were gathered The evaluation is ongoing The remainder of the presentation records: The output of the pre-processing of the controllers evaluation reports The evaluation results (both from reports and logs)

18 Pre-processing of the radio call reports (1) Report 1, Report 2 & Report 3 indicated a problem, which was described as: The frequency does not get cut off, so this creates an uncertainty as to whether the transmitter is emitting or not. Also, I cannot hear a feedback of my voice when I emit and this creates an uncertainty The problem has been identified as a configuration error in the DSP firmware of the VEF The problem was quickly resolved and testing was resumed

19 Pre-processing of the radio call reports (2) Within Report 4, Report 6, Report 7, Report 8, Report 9, Report 10, Report 11, Report 14, Report 17 and Report 18, the controller indicated a light hiss (or noise) The pilot did not mention something similar All controllers identified this impairment as minor HCAA has been contacted and no further action was advised

20 Pre-processing of the radio call reports (3) Many comments within the reports had to do with radio coverage issues. This means that: The controllers began trusting the SATWAYS solution and using it for control purposes instead of a pure radio check, as was the initial plan The controllers focused on the radio coverage issues and forgot all about satellite transmission, i.e. they started treating the SATWAYS solution like any other landline communication medium they are accustomed to. HCAA was provided with the details of the installation and asked to provide a radio coverage map in order for the authors to crosscheck the controllers comments All remaining reported problems concerned flights that were marginally within the radio coverage of the VHF transceiver

21 Pre-processing of the radio call reports (4) The call logs from the SATWAYS solution indicated that at some point in time between the 5:00 AM and 9:00 AM UTC on the 1st of August 2010, the packet loss of the satellite link was multiplied by a factor of 5, indicating a potential problem A periodic pinging facility between the 2 ends of the link was installed and unveiled the following: 1 out of approximately 20 ping packets got lost and the pattern was deterministic Once every x minutes, we experience a 20-second outage of the satellite service. This behaviour can be predicted to the second, though x varies, most often being 50 minutes The circular grade drops gradually to below 80, then returns gradually to 100, then back to 80 and so on and so forth. This situation is also depicted on the top led on the front of the master station The header CRC error and the Frame Plan Missed Total counters increase This indicated an interference from another station: Both stations were turned off on the 20th of August with the help of the municipality of Patmos and Hellas Sat provided the evidence that this is interference from an unknown source The interference has disappeared on the same day

22 Results of the processing of radio call reports ID Flight level Responsible controller Type of flight Time Date Question 1 Question 2 Question 3 Question 4 Question 5 Question CP IFR 9:10 20/7/ GW IFR 9:20 20/7/ SE IFR 9:30 20/7/ PO IFR 9:40 20/7/ MR IFR 9:40 20/7/ AJ IFR 21:10 21/7/ GX IFR 21:20 21/7/ CU VFR 11:00 25/7/ PD VFR 11:10 25/7/ TN VFR 6:25 26/7/ MP VFR 6:30 26/7/ TS VFR 5:00 1/8/ XD VFR 5:20 1/8/ CK VFR 5:30 1/8/ VO VFR 5:20 1/8/ SC VFR 9:00 1/8/ PV VFR 8:10 13/8/ Ground CP VFR 8:20 13/8/ ground PV VFR 10:50 30/8/ XD VFR 10:50 30/8/ GW VFR 10:55 30/8/ Average: Standard deviation:

23 Results of the processing of radio call logs Report ID Flight level Responsible controller Type of flight Time Date Average number of Tx buffer frames - Patmos Average number of Tx buffer frames - HCAA Maximum jitter - Patmos (ms) Maximum jitter - HCAA (ms) Average packet loss - Patmos (%) Average packet loss - HCAA (%) CP IFR 9:10 20/7/ GW IFR 9:20 20/7/ SE IFR 9:30 20/7/ PO IFR 9:40 20/7/ MR IFR 9:40 20/7/ AJ IFR 21:10 21/7/ GX IFR 21:20 21/7/ CU VFR 11:00 25/7/ PD VFR 11:10 25/7/ TN VFR 6:25 26/7/ MP VFR 6:30 26/7/ TS VFR 5:00 1/8/ XD VFR 5:20 1/8/ CK VFR 5:30 1/8/ VO VFR 5:20 1/8/ SC VFR 9:00 1/8/ PV VFR 8:10 13/8/ Ground- CP VFR 8:20 13/8/ ground PV VFR 10:50 30/8/ N/A N/A XD VFR 10:50 30/8/ N/A N/A GW VFR 10:55 30/8/ N/A N/A

24 Summary of the evaluation results (1) The users rated the solution with: An average of 8.71 on a scale from 1 to 10 (10 is best) for voice quality An average of on a scale from 1 to 10 (1 is best) regarding the effect of delay on their perception, effectiveness, timing, correct phraseology and speech rate Patmos logs for reports 19, 20 and 21 have been lost due to the combination of a power failure and a misconfiguration of the RSC-10 The recorded maximum jitter values are slightly higher than the larger ones of those originally required The maximum jitter values are almost constant and equal per transmission direction The packet loss values: Are considerably higher with respect to those originally required sometimes considerably larger than the 0.5% requirement (more than tenfold increase) After the termination of the interference, they have dropped to nearly zero These deviations do not seem to affect the radio call quality at all, since exceptional grades are awarded in any case

25 Summary of the evaluation results (2) The following report summarizes everything the SATWAYS project was initially targeting at: This was a flight with destination the Patmos heliport. The frequency was checked upon departure from the heliport of Mykonos (translation note: another island about 45 miles West of Patmos) and throughout the flight until landing to the heliport of Patmos. At a distance of 15 nautical miles east of the RIPLI region (translation note: controller jargon for an area within the Greek airspace), the pilot reported a 5/5 radio communications quality and so did the flight controller with little noise. At 20 nautical miles west of Patmos and at a flight level of 1500 feet, both the pilot and controller reported 5/5 quality. Beginning the descent to the Patmos heliport, both the transmission and reception were exceptional. On the ground, the communication quality was also exceptional and the flight plan has been closed using radio comms. The pilot has expressed to me his admiration, since traditionally this area had no radio coverage. I consider that the use of satellite communications is a success and covers our requirements for the heliport up to ground level

26 Thank you for your attention