In this week’s bulletin, Charlie continues his discussion into backups, looking at devices that are responsible for keeping machinery running smoothly.
Today’s bulletin sounds like a scintillating subject and is guaranteed to send you to sleep, but bear with us, as this is an extremely important topic. Operational Technology (OT) can be found in industrial environments where physical processes are monitored and controlled, such as manufacturing plants, power grids, water treatment facilities, transportation systems, and oil and gas refineries. It includes systems like Supervisory Control and Data Acquisition (SCADA), Programmable Logic Controller (PLC), and IIoT [1] (Industrial Internet of Things) devices that manage machinery, critical infrastructure, and automation processes. I am writing this as part of my own journey to understand backups since PlanB Consulting has joined Databarracks, whose primary service is managing backups on behalf of its clients.
An example telemetry system
I first came across OT when I joined Anglian Water in 1995. In my interview, I was asked about telemetry and how, if I was appointed business continuity manager, I would ensure that the organisation’s telemetry system could be maintained following any major incident. I had never heard of telemetry or even comprehended that it was a thing, but I must have given a good enough answer to get the job.
When I joined the company, I was shown around the control room. What I saw slightly blew my mind, having never thought about managing water and wastewater and controlling it. In the control room, four controllers were looking at multiple screens displaying pictures of the water system for the regional area they were controlling. They could see water flows and valves opening and closing. The system operated mainly automatically, and only when something failed did they get an alarm. They then needed to triage the event to determine whether they could sort it out from the control room or needed to send out a man—and in those days, it was nearly all men—in a van to fix the problem immediately. Alternatively, they could decide that the alarm could wait and be sorted the next day or as part of more routine maintenance.
The vast system worked on a series of radio and cable links. At one time, I was given access to the system so that, technically, I could monitor the alarms from home or in my office. The managers had the same access but could dial into the system, change parameters, and review alarms. These were the days before cyber incidents, and apart from a login password, the system was pretty insecure, but this didn’t really matter as an attack on the system was considered very unlikely.
What does OT consist of?
A modern OT system, at a basic level, consists of a number of elements. For each part of an electromechanical process, there is a PLC, which controls the process. PLCs monitor inputs from sensors, process data based on programmed logic, and control outputs like motors, valves, and actuators to automate industrial processes. They continuously run in a loop, adjusting operations based on real-time feedback to ensure efficiency and precision.
They also send back data on the process to a SCADA system, which controls a number of PLCs [2]. The SCADA system collects data from multiple PLCs, enabling operators to manage operations. It is connected via different networks such as Ethernet/IP or more bespoke networks like Modbus, PROFIBUS, CAN bus, and OPC UA.
The human controlling the system connects to the SCADA system via a Human Machine Interface (HMI), where they, like the Anglian Water controllers, can monitor the network and respond to alarms. Figure 1 shows how SCADA, HMI, and PLCs fit together.
The principles of backup
The principles of backup for PLC and SCADA systems are the same as those for IT systems. If there is a disaster affecting your systems and you lose the primary data, you want to be able to restore it to reconfigure your systems. In the OT environment, this is especially important as downtime of systems can be hugely disruptive. You may have to shut down processes, which take time to restart and can be very expensive in terms of lost production.
There are three elements to backup: the data produced by the process and the configuration of the SCADA and PLCs.
System data backup
A key role of PLCs may be to take data from the machinery or process they are controlling and feed it back to those managing and optimising the process. The data can be used in multiple ways—it helps optimise efficiency, improve predictive maintenance, enhance quality control, ensure regulatory compliance, and reduce downtime.
This data can be used for real-time monitoring, trend analysis, fault detection, energy management, and automation improvements, leading to increased productivity and cost savings. Loss of this data could set back the optimisation of processes by years.
There may also be a compliance element to the data, where you have to show that discharges or emissions monitoring are within agreed limits or that processes are operating at the required quality, such as drinking water standards. Failure to produce this data could lead to large fines or even the plant being shut down.
This data must be backed up in the same way as IT data, ensuring there is no single point of failure where the backup is physically close to the main data processing system and that the data is also immutable, so it cannot be encrypted in a cyber attack. Determining the RPO of the data will be important to determine how much data you can afford to use. The volume of data produced by large industrial processes may be huge, so the backup solution must take this into account.
PLC backup
When you initially set up a new process, such as a generator, there is an ongoing process of fine-tuning its operation to optimise its use. This means the PLC controlling it is periodically updated with new program and configuration files. At the same time, the manufacturer of the equipment may also send out new firmware updates.
It is extremely important that the firmware version is recorded, as restoring the wrong version may lead to compatibility issues. There could also be network configurations, such as network settings, IP addresses, and communication protocols, which ensure seamless reintegration into the control system after restoration.
It is vital that this data is backed up at the PLC level, as years of work could be lost if the PLC is wiped, destroyed, or corrupted. This often has to be done in coordination with the equipment provider and PLC controller.
Speaking to an OT expert yesterday, he mentioned that operators of systems often leave it up to the equipment provider to decide what is backed up, how often, and where the backups are stored. He has come across several cases where the backups and main systems are next to each other. His advice was to have a conversation with your provider and check that their backup process meets your requirements—after an incident, it may be too late. If not, change the configuration or switch to a specialist PLC backup provider.
SCADA backup
For the SCADA system, backups are also essential. In addition to backing up historical data logs, configuration files, and any custom scripts or applications, you may also consider taking complete disk images of your SCADA system, including the operating system, applications, and configurations.
Some SCADA systems are tied to specific software, so there may be licensing issues with cloning or restoring systems.
Conclusion
Like all IT and backup processes, many organisations only fully understand what is backed up (or not) when they experience a disaster and attempt to restore their operations, systems, and data.
Backup of IT systems is reasonably well understood, with organisations specialising in this and accepted protocols and guidance for best practices. My impression is that, in the OT world, this is less well documented and known about.
So, as a business continuity manager, if you have SCADA systems and PLC controllers as part of your processes, perhaps you should go and speak to those who look after them and discover what they do regarding backups.
[1] IIoT (Industrial Internet of Things) refers to the network of interconnected sensors, instruments, devices, and industrial machines that communicate and share data in real time to enhance automation, efficiency, and decision-making in industrial settings.
