Operational technology security: protecting plant without stopping it

Cue 1 The office priority stack inverting into the plant priority stack

AMARA 00:00 Start with the thing security teams get wrong on their first day on a plant.

NADIA 00:06 They arrive with the office order of priorities. Confidentiality first. On a plant it is close to reversed, with safety in front of everything. Almost every argument between the two teams traces back to that one difference.

AMARA 00:20 Spell the plant order out.

NADIA 00:22 Safety of people. Then availability of the process. Then integrity of the data and the commands. Confidentiality last, which does not mean unimportant, it means fourth.

AMARA 00:33 Why does availability rank so high? Everywhere else it is last.

NADIA 00:37 Because a stopped process is not a file waiting patiently for you. Molten metal solidifies in the vessel. A batch spoils. A furnace takes days to relight. A water works that stops has customers with no water this evening.

Cue 2 A patch attempt meeting three blockers in turn: outage, warranty, no vendor

AMARA 00:53 So patch promptly.

NADIA 00:54 Collides with three hard facts. Many controllers cannot be patched without stopping a process that runs for months. Patching outside the vendor's validated list can void support on a machine worth more than the whole security budget. And a lot of installed plant is past vendor support, so no patch exists at any price.

AMARA 01:16 That sounds like an excuse.

NADIA 01:18 It is a constraint, and the difference matters. The honest response is not to pretend the patch will happen next quarter. It is compensating controls. You cannot fix the asset, so you change what is allowed to reach it.

AMARA 01:33 Second instruction. Scan the estate.

NADIA 01:35 On older equipment that is actively dangerous. These controllers have small network stacks built to hold a few connections on a quiet, deterministic network. An ordinary scan can exhaust one and fault it.

Cue 3 A scan reaching an old controller and the controller faulting

AMARA 01:48 And a faulted controller is...

NADIA 01:50 A stopped process. People find this hard to believe until they have done it once. Which is why passive monitoring is the default here: you read the traffic already on the wire and you touch nothing.

AMARA 02:05 Third instruction. Isolate the compromised host.

NADIA 02:07 Isolation is not free on a plant. Cut a control system off from the process it is regulating and you have a process with no control, which can be a worse problem than the intruder you were responding to.

AMARA 02:23 Move to the network. What should it look like?

NADIA 02:26 Layers. The process at the bottom, sensors and actuators. Controllers above them. Control room and historians above those. Production management above that. Business network at the top.

Cue 4 A flat plant network resolving into layers with an industrial demilitarised zone

AMARA 02:37 And in a plant that grew rather than being designed?

NADIA 02:41 It is flat. Everything can talk to everything, and one compromised laptop on the office network is three hops from a controller. The value of the model is not the drawing, it is that it makes the question askable at all: what is allowed to talk to what.

AMARA 03:00 The industrial demilitarised zone. What is it for?

NADIA 03:03 It is the only meeting point, and nothing crosses it directly. A business system that wants production data reads it from a server in that zone which the plant wrote to. The plant never accepts an inbound connection from the office side.

AMARA 03:20 So an office compromise has no path that ends on a controller.

NADIA 03:25 That is the whole design. And it is worth saying plainly, because people install one and then punch a firewall rule through it for a reporting tool, at which point they have the cost of the zone and none of the protection.

Cue 5 A vendor connection drawn straight through every boundary, then constrained

AMARA 03:42 Safety systems. Why keep them separate?

NADIA 03:44 A safety instrumented system takes the process to a safe state when something goes badly wrong. Its own sensors, its own logic, its own final elements. A protection layer that shares components with the thing it protects against is not independent, and the safety case was written on the assumption that it is.

AMARA 04:06 What pushes people to integrate them?

NADIA 04:08 Convenience, always. One screen, one connection, one engineer. Every step in that direction erodes the independence somebody signed for. If integration exists it should be one way and demonstrable: safety reports to control, control never commands safety.

AMARA 04:23 Last one. Where does the boundary actually leak?

NADIA 04:26 Vendor remote access, almost every time. Beautifully drawn zones and one permanent connection straight through all of them, agreed once at commissioning by people focused on getting the line running.

AMARA 04:38 Why is it still there years later?

NADIA 04:41 Because it works and nobody is asked. Shared account, password left with an employee three years ago, no logging, and reach far beyond the one machine it was granted for. Make it requested rather than standing, enabled by the plant, named accounts, recorded, and limited to the asset. That one change moves more risk than any patching programme you will manage to run.

Cue 1 An asset inventory filling in, with the which process stops column added last

AMARA 05:06 First control in every framework is an asset inventory. Why does nobody have one?

NADIA 05:12 Not negligence. The plant has been extended over twenty years by four contractors, a line was moved in from another site, and the engineer who commissioned the oldest cell retired in 2014. Nobody decided not to have one.

AMARA 05:27 What has to be in it?

NADIA 05:29 More than a list of boxes. What it is, what firmware it runs, what it talks to, whether the vendor still supports it, whether a configuration backup exists and when it was last proven. And one more column that matters more than the rest.

AMARA 05:47 Which is?

NADIA 05:48 Which process stops if this stops. That is what turns a spreadsheet into a decision tool, because during an incident the question is never which asset is most interesting. It is which one can be taken away with the least physical consequence.

Cue 2 Repetitive plant traffic forming a baseline, then one new conversation standing out

AMARA 06:04 How do you build it without disturbing anything?

NADIA 06:08 Start from the process rather than the network. Walk the line and write down what each machine needs in order to run. It is slower, and it produces the column that actually matters, which a network discovery tool cannot give you.

AMARA 06:24 And monitoring?

NADIA 06:25 Passive first. Read the traffic already on the wire from a mirror port or a tap and send nothing. On a plant that is unusually productive, because the traffic is repetitive. The same devices exchanging the same messages at the same intervals.

AMARA 06:42 Why does that help?

NADIA 06:43 Because anything new is loud. A device that has never been on the network. Two machines that have never spoken to each other suddenly talking. A programming command outside a maintenance window. An engineering workstation reaching twelve controllers when it has always reached two. In office traffic all of that would be lost in the noise.

Cue 3 The four sectors with the same dull entry point and four different consequences

AMARA 07:06 Take me into an actual incident. Where does it start?

NADIA 07:10 Almost always somewhere unremarkable. A phishing email on the business side, a contractor's laptop, a vendor connection, a remote access service left facing the internet. The entry is dull. What differs between sectors is the consequence.

AMARA 07:24 Give me the four.

NADIA 07:26 Energy has a regulated obligation to keep supplying. Water has a treatment process and a public health duty and cannot simply be paused. Manufacturing has processes that spoil or set if you interrupt them. Automotive stops a line and idles suppliers and plants in other countries within hours.

AMARA 07:45 So containment is not cheap.

NADIA 07:47 Which is why the response is built around reaching a defined safe state rather than around isolation. And the safe state is not always off. Sometimes it is manual operation. Sometimes it is a hold at a defined point in the batch. Sometimes it is a controlled shutdown, which takes time and carries its own risks.

Cue 4 A process moving to a defined safe state rather than being isolated

AMARA 08:09 Manual operation assumes somebody can do it.

NADIA 08:12 That is the sharp edge. A plant that has run on automatic for nine years may have nobody on shift who has driven it by hand. An incident is not the moment to discover that, so it has to be practised while nothing is wrong.

AMARA 08:30 Now the decision. Who stops production?

NADIA 08:32 The question that goes wrong most often, and it is entirely avoidable, because it is about authority rather than technology.

AMARA 08:40 Both sides think they are right.

NADIA 08:43 Both sides are right. A security team that isolates a segment can stop a furnace without knowing it. A plant manager refusing to stop may be protecting a process that cannot be interrupted safely at that moment. You will not resolve that at three in the morning with an intruder in the network.

Cue 5 A controller restore attempt stalling at missing software and licences

AMARA 09:04 So it goes in the plan.

NADIA 09:06 In writing. Who may call a stop, who is consulted, what happens when they cannot be reached, and a standing rule that safety advice from the duty holder is not overridden by a security judgement. Then rehearse it with the operators who would be running the plant by hand.

AMARA 09:26 Recovery. What do plants find out the hard way?

NADIA 09:30 That server backups exist and controller backups often do not. And where they exist they have usually never been restored, so nobody knows whether the file is complete or whether the engineering software that reads it still runs on any machine the site owns.

AMARA 09:47 What is the test?

NADIA 09:49 Pick a controller and ask how long to bring it back from nothing. If the answer contains the words we would have to call the vendor, then that is your recovery plan, and it should be written down as one, with what it costs and how long it takes.

AMARA 10:08 Last thing. What does the law require in this country?

NADIA 10:12 Operators of essential services in energy, transport, water, health and digital infrastructure fall under the network and information systems regulations, enforced by a competent authority per sector. Appropriate and proportionate measures, and notification of incidents with a significant disruptive effect.

AMARA 10:28 And if the incident touches personal data?

NADIA 10:31 Then data protection duties run on their own clock, and if anybody is hurt the health and safety reporting duties apply as they always would. One incident, potentially three reports to three different bodies on three different timescales. Much better learned now than on the day.