Monitoring & Control Tim Stevenson Yogesh Wadadekar

Monitoring & Control Tim Stevenson Yogesh Wadadekar

Monitoring & Control M&C is not recognised as an SPDO Domain However the volume of work carried out in 2011 justifies a Concept Design Review M&C is all pervasive, generally low risk, but critical to SKA operations (similarly to Signal Transport & Networks) M&C is key to successful integration and verification of SKA The management of M&C development tasks is TBD

Monitoring and Control: a status report Yogesh Wadadekar National Centre for Radio Astrophysics, India

Overview M&C History within PrepSKA Recent work in M&C domain High level concerns M&C context and scope Candidate design concepts derived from requirements, which in turn are derived from system requirements Gaps, risks, way forward

A brief history of M&C in PrepSKA M&C is a subpackage of System (WP 2.1) In January 2011, NCRA took over the role of lead institution Substantial work has been done in this domain NCRA Indian industry partners Tata Consultancy Services, Embedded Computing Machines, Persistent Systems Limited With continuous involvement of the SPDO And with support from M&C team members of ASKAP, LOFAR, MeerKAT

M&C concept design review Aim: is to confirm that the problem has been thoroughly explored and is well understood. Candidate technologies, technology trends, technology options are reviewed at this point will be held at NCRA, Pune from 8-10 Nov. 2011 draft documents circulated for comment Feedback solicited via the COAR spreadsheet On Friday 21 Oct, at 13:00, we will provide more details on the design concepts that we have explored

Architecture for SKA1 or SKA2? Architecture will be designed keeping extensibility in mind. Although realisation for SKA2 (in hardware and software) will be postponed

M&C responsibilities M&C is responsible for ensuring that all the system Elements, including the stations, signal transport & networking, all central facilities, science data processing and archiving, and environment monitoring equipment operate in concert during telescope operations. How it works: It acquires and processes engineering data from all these systems, provides support for drillable displays to operators, and performs the orchestration of system elements needed to carry out commands. Naturally Interfaces to local M&C, SKA operations and other domains are of great importance

Three tiers of M&C System M&C Central M&C Regional M&C systems M&C systems

Scope of M&C control responsibility

M&C OS layer Setpoints node Data Events / Alarms Commands Data server Commands Validation data Worldview Config DB Events Detection M&C Node Events Distribution Control Logic Data Processing Data acquisition events Plug-in Control algorithms Events reception Continuous Feedback Control Scripts Commands Distribution Data Events / Alarms Setpoints Commands Logging, archiving and prioritised events are not included in the above diagram

Operations support databases Observation Scheduling Schedule Preparation Science Data Interpretation Central M&C Regional M&C Observatory Calendar Configuration & Status DB Engineering Data Archive Context Information DB Regional Data Archive Observatory Operations Operators, Engineers, Scientists Weather stations Resource providers

M&C real-time databases Observatory Calendar maintains the schedule of planned future operations, including the observations schedule, maintenance schedules, anticipated unavailability and anticipated instrument configuration changes (e.g. commissioning and upgrades) Configuration and Status DB holds current (real-time) configuration and health status information about all system entities Engineering Data Archive holds archives of engineering data to support troubleshooting and metadata drilldown for science data quality control. Context Information DB maintains real-time information about the environment, including weather patterns and RFI levels, resource availability and quality information (e.g. current and anticipated future outages).

Common software platform There are considerable data that need to be shared between M&C and other systems, to facilitate the above integrations The observation schedule representation needs to be shared between Science Computing and M&C. Observatory configuration and status information data needs to be shared between Science Computing, M&C and enterprise applications. Meta-data representation needs to be shared between M&C, science computing and the data archive. M&C and other domains involved need to converge upon a common data representation and content for each of these data entities. In addition, they could develop a common software platform for data exchange..

Context and integration interfaces

Hierarchical Semi Autonomy candidate architecture Tiers form a strict hierarchy: the Central M&C node directs the functioning of Regional M&C nodes, which in turn direct the functioning of M&C nodes hierarchy is pre-defined at design time or at system deployment. Each parent node has pre-programmed knowledge of its child nodes. It has pre-programmed intelligence to handle the specific set of alarms generated by each child node, and to pass them up to its parent node, if necessary. This concept has proven to be scalable, robust and is amenable to slight augmentations to eliminate single points of failure and allow plug-n-play of child nodes. Autonomy needed to handle safety and security, control logic, data processing, feedback control, troubleshooting and state management. Almost all existing M&C systems are variations on this basic theme.

Data flows Alarms/events Commands Central M&C Predefined handling of alarms Resource allocation Knowledge of impact of region status on capabilities Metadata augmentation based on alarm severity Regional M&C Knowledge of child types Predefined handling of propagated alarms Resource allocation Knowledge of impact of child status on capabilities Regional M&C M&C M&C M&C M&C M&C Statically defined parent handling + propagation of alarms M&C

Service capability matching Each M&C node is viewed as providing a set of services to its parent node and utilising services from its child nodes. Nodes compose themselves dynamically into a hierarchy using service discovery and commitment concepts. Each node is aware of the capabilities and QoS it can provide, and can update these if it has internal failures. Parent nodes are viewed as service consumers, who can describe the services they need with the desired QoS parameters, and accept commitments from any node that can provide the desired services. Since nodes know the effect of failures on QoS, and they know the target (desired) QoS, they can determine when the targets cannot be met, and can provide metadata augmentations that reflects the precise gaps between target QoS and actual QoS. Futuristic and not currently feasible

Service capability matching Knowledge of impact of child QoS on commitment: basis for metadata augmentation Data flows Alarms/events Commands Possibility of multiple parents service requestors Possibility of concurrent commitments Regional M&C Central M&C Dynamically acquire knowledge of child types & handling of alarms Receive requests for services, issue commitment requests Resource allocation (parent-child bindings) based on commitments Map science needs to region capabilities and QoS Knowledge of impact of QoS gaps on capabilities: basis for metadata augmentation, abort/reschedule decisions Regional M&C Component-to- Component services M&C M&C M&C M&C M&C M&C Plug-n-play: Dynamic identification of parent Request-commitment as basis for allocation handling + propagation of alarms QoS composition: output QoS changes if input QoS changes

Risks related to Management and Organisation of Monitoring & Control Development related to Engineering risks related to Architecture risks related to Design Risks in above categories and associated ideas for mitigation are discussed in the risk register.

Gaps insufficient resource to investigate within WP2 e.g. data exchange interfaces need to exist between the science data path and the M&C system outside the scope of WP2: e.g. plan of operations insufficient detail available at the present time e.g. regulatory requirements, interfaces with other domains..

Many more aspects covered in docs requirements and concept descriptions in areas of safety, security, integrity, alarm handling, fault management, availability, reliability, scalability, metadata generation, user roles (scientist, operator, engineer etc.) and user interfaces, evolvability/commissioning, logging and archiving, realisation technologies (EPICS/PVSS etc.), sensor standards Costs plans for the PEP phase.

Beyond the codr since M&C will have many complicated interfaces with several other domains, a lot of work is needed to define these clearly. A working M&C system is necessary as a test harness for commissioning of receptors. So its development needs to taken up early. We expect that this area will be a significant part of the Indian in-kind contributions in the PEP phase..

For more information Attend the M&C breakout on Friday 21 Oct, starting at 1300 hours Read through the M&C co-dr documentation available at: http://wiki.skatelescope.org/bin/view/monitoringcontrol/mandccodr and provide feedback using the COAR spreadsheet.