Canadian Aviation Safety Seminar 2006

Transcription

1 Canadian Aviation Safety Seminar 2006 Improving Safety and Performance With SMS Proactive Analysis Alex Richman AlgoPlus Consulting Limited

2 Summary Improving safety and business performance Merging the goals Proactive analysis SMS requires two cultures S2

3 One Image or Two? S3

4 SMS Brings the Two Together SMS S4

5 Proactive Analysis What is it? -Finding potential problems before they occur Where? External data sources: -Formal, structured (e.g. Service Difficulty Reports, CADORS) How? Identify new, unusual or persisting problems in fleets Compare own performance with external benchmarks S5

6 Reactive/ Proactive Dichotomy Reactive Post facto after the crash Tombstone mentality Investigation of event Microscopic Smoking gun Forensic cause Proactive Identify potential hazards Wariness General analysis of system Macroscopic Chain of risk factors Probabilistic Richman & Murphy 2005 S6

7 Sources for Proactively Identifying Potential Hazards Discretionary Company experience: reports, minutes Managers perceptions; Workforce opinions Audit reports; Previous hazard analyses Hazards identified by other organizations Systematic System Safety and reliability data recording systems: CADORS SDR, MMIR TC TP 1381E S7

8 Proactive Analysis Advantages 1. Because of their larger size, system-wide databases are far more likely to yield information about events which might occur later in a smaller fleet 2. Benchmark comparisons are possible S8

9 No Significant Trends but Significantly Above System Wide Average S9

10 Aviation Safety Event Classification Accident Substantial damage to aircraft and // or serious injury Fatal Resulting in death Non Fatal No deaths Safety Event Incident All other safety events which are not accidents Minor injuries No injuries S10

11 Proactive Safety Event Analysis Safety Performance Sequence Boeing Aircraft Mechanical Malfunctions (SDR) Significant Safety Events No Injury No Damage Operational Difficulties (CADORS) Aircraft Airport Air Traffic System S11

12 Use the Shared Experience of Others Manufacturer Bulletins and Airworthiness Directives take time to get to the operator Over 80% of unscheduled maintenance is not predicted from Manufacturers manuals or Airworthiness Directives Service Difficulty Report: Website databases have fleet-wide data for your models S12

13 Database Search Steps REVIEW EXTERNAL DATE BASE FIND RELEVANT RECORDS CLEAN DATA IF NECESSARY ATTACH TRAFFIC DENOMINATIONS UPDATE DATABASE REMOVE DUPLICATES ANALYSE FIND RESULTS RELEVANT TO AlgoPlus ORGANIZATION Consulting Ltd. S13

14 Survey of 15 Safety Officers Too much time spent on routine tasks Little time for proactive safety efforts Follow-up is not always well tracked Need outside benchmarking Need better analytical tools Clear safety strategy infrequent GAIN Initial Survey S14

15 Difficulties in Analyzing Databases Database skills and experience needed Analysis requires training and time Complex results must be reviewed Information sharing is erratic Knowledge management is difficult S15

16 AviationSOS (Safety Overview System) Automated system providing: Early identification of mechanical malfunctions Risk assessment, likelihood and hazard Prioritization Traffic based denominators Statistical significance Source narratives and other data Historical review Benchmark comparisons S16

17 Automated Screening of Databases Mechanical Malfunctions -Service Difficulty Reports Canada and USA (Half the worlds fleet) -MMIR (Helicopter Association International) Operational Difficulties (CADORS) -Aircraft -Aerodromes CADORStat -Air Traffic System JetStats HeliStat EngineStat S17

18 ALERTS: JetStats, HeliStat, EngineStat S18

19 JetStats Priority - Features ed day after FAA updates website Priority reports only - higher risk/hazard Past 5 weeks Link to full 6 months, all reports Traffic light summary of risk assessment Specified model(s) only Not yet viewed reports are flagged S19

20 Priority JetStats, HeliStat, EngineStat Traffic Light Component Reason for ALERT Date Reported Flag for not previously viewed S20

21 Source Reports: Details and Narrative Date incident happened Date incident reported S21

22 Source Reports: Details and Narrative Manufacture, Series, Model S22

23 Source Reports: Details and Narrative Part Name, Location, Condition S23

24 Source Reports: Details and Narrative Operator, Registration (coded) S24

25 Source Reports: Details and Narrative Stage of Operation S25

26 Source Reports: Details and Narrative JASC code and name S26

27 Source Reports: Details and Narrative Affecting safety of Flight (FAA) S27

28 Source Reports: Details and Narrative Nature Condition, Hazard level S28

29 Source Reports: Details and Narrative Precautionary Procedure, Hazard level S29

30 Source Reports: Details and Narrative Anomaly: Full text S30

31 Source Reports: Details and Narrative S31

32 Source Reports: Details and Narrative Number of other reports with the same aircraft model and JASC in the past 3.5 years S32

33 3 General Questions 1. What s new or different? Are there new mechanical problems? Has the likelihood rate changed significantly? Are there historically unusual reports? 2. Which problems persist in the fleet? Are they potential audit topics for our fleet? Do they help us identify performance targets? Should we check our fleet? 3. Do other airlines have our problems? S33

34 Has the Likelihood Rate Changed? S34

35 Are There New Mechanical Problems? S35

36 Are There Historically Unusual Reports? S36

37 Which Problems Persist? S37

38 Do Other Airlines Have Our Problems? S38

39 5 Specific Questions What significant mechanical malfunctions occurred in the last 6 months? How unlikely were they? How hazardous where they? Are the trends changing over time? Which problems should be approached first? S39

40 Relative likelihood Hazard level Time trends Recent increase Stable Threat Matrix Summarized by Traffic Light Icons S40

41 Risk Matrices Combined to Show Trajectory-Time Trends S41

42 Periodic Priority Plot This list shows the four components with the highest priority scores The priority scores indicate the problems were highest probable risk to safety These quantified priority scores help identify potential audit topics, compare component performance between models and form aggregate performance measures. A link identifies the specific parts involved for each component S42

43 CADORS: Civil Aviation Daily Occurrence Report System CADORS data are used to provide timely information concerning operational occurrences within the National Civil Air Transportation System and in the early identification of potential aviation hazards and system deficiencies. Source of Data: NAV CANADA s AOR reports, Transportation Safety Board reports, Airports, Police forces, Public, etc. Users of Data: Transport Canada Civil Aviation and External Aviation Stakeholders. S43

44 CADORS Severity Cone For every fatal aviation accident there are over 450 safety-significant events reported Fatalities Injuries Damage substantial + Damage slight TSBC Class 5 Other (Excludes General Aviation) S44

45 CADORS Top Events CADORS: Top events Engine malfunction, other Other AC operational Landing gear Aircraft Event Bird strike Declared emergency Diversion Aerodrome Conflict potential ATC operation irregularity Alleged CAR Infraction Rate per 100,000 movements Air Traffic System S45

46 CADORStat: Rates Per 100,000 Movements National Trends Geographic regions Aerodrome Airline (DIY) reports Top events Rate per month: number and percentage of benchmark Trends S46

47 CADORStat: Airport XX Top Events Prototype CADORStat Algo Airport 3/21/2005 to 3/21/2006 Analyzed: Monday, March 20, 2006 CADORS data may be incomplete Top 3 CADORS events, past 12 months Event: Declared Emergency Mean per month: % of Statistical norm Latest Report: "XX, a B , departed XX for XX and shortly after takeoff, the flight crew advised ATC of a stabilizer issue and requested to divert to XX. They also requested the CFR equipment be on standby for their landing. The aircraft landed safely at XX at 0413z, TSB XX is looking into the occurrence." (Occurred: 12/25/2005) Event: Bird Strike Mean per month: % of Statistical norm Latest Report: " XX an A320, struck a Snowy Owl during roll-out on Runway 28 at XX. The strike occurred at the 3,000 foot mark on the runway and the bird was ingested into the number 1 engine. XX maintenance confirmed that 2 fan blades and a guide vane were damaged. The damaged parts were replaced." (Occurred12/22/2005) Event: alleged CARs infraction Mean per month: % of Statistical norm Latest Report: " The crew of XX, a Canadair RJ, was instructed to taxi via Taxiway C2 and hold short of Runway XX at XX. They subsequently failed to comply with the instruction and entered Runway XX with traffic on short final. The crew of XX, a B , was instructed to go around at 0.5 mile final. " (Occurred: 12/8/2005) S47

48 Airport XX: Leading Event: Declared Emergency 1. Mean 2. Rate relative to benchmark 3. Trend chart S48

49 Airport XX: Second Leading Event: Bird Strike S49

50 Airline Rates: DIY program for smaller aircraft AlgoPlus has traffic denominators for aerodromes and larger aircraft but not for smaller aircraft. Our AlgoPlus program (DIYStat) for operators with smaller aircraft compares the operators CADORS reported performance with the national norms. Step 1) ed Report gives number of reports Step 2) User enters number of operations Step 3) DIYStat program converts CADORS event counts to rates per 100,000 movements Step 4) DIYStat program shows airline s Relative Risk, how it compares with the Canadian statistical norm and whether the result is statistically significant With these external data smaller operators can learn how their performance compares to the statistical norm S50

51 Statistical Process Control (SPC) and Six Sigma SPC techniques continuously monitor and chart SDR and CADORS trends in order to signal significant changes from predetermined limits. SPC is specially useful in detecting rare occurrences (less than one per month) when it analyzes the time between occurrences. This continuous monitoring promotes earlier identification leading to more rapid resolution of operational difficulties. S51

52 SMS Requires Multiple Supports S52

53 Stages of an Evidence- Based Culture COSMETIC SYMBOLIC SUPPORTIVE An analytic unit, shown on the table of organization, satisfies minimum regulatory requirements, but its bulky annual reports are remote from managers. Analytic staff meet periodic requests from management for non-routine reports, with minimum discussion The quantitative evidence supports decisions after they are made HELPFUL Occasionally quantitative evidence helps decision makers INTRINSIC Data-based evidence is a continuous, essential part of the managerial process S53

54 The Evolution of Evidence Based Culture SMS Increasingly Effective HELPFUL Associated occasionally with decision-making INTRINSIC How we manage SUPPORTIVE Enhances already-made decisions TOKEN Barely satisfies regulatory requirements SYMBOLIC Occasional and remote involvement Epidemic of Trust & Accountability S54

55 SMS Essential Foundations S55

56 JetStats System for SMS 1 Priority List for Model by JASC 5 weeks or 6 months 4 Share Report 2 Source Reports 5 Out Mail Tracking 3 Threat Matrix 6 Knowledge Management 11/10/2005 S56

57 Proactive Analysis of External Databases: Mechanical Malfunctions Trend Analysis Risk assessment: likelihood and severity Prioritizes findings Disseminates information Integrates with quality assessment and assurance Continuous monitoring identifies new problems S57

58 Thank You. AlgoPlus Consulting Limited Suite Spring Garden Road Halifax, NS, Canada B3J 1H6 Voice: AlgoPlus Cell Fax: S58