One of the best parts of my job is I get to walk around and look over what has been implemented in the way of physical and cyber security. Most of the time I am very impressed by what has been done as more and more companies are realizing what is at stake should their infrastructure be compromised. Whether its intellectual property or malicious activity, the costs of a breach could be significant, even catastrophic if the right circumstances were realized.

Ok, here is where it gets really fun. I was recently performing a Cybersecurity Vulnerability Assessment on an oil refinery.  The main PCS in place was a form of redundant Ethernet. The main communication was broadcast and multicast traffic from all devices in a producer/subscriber configuration. It was one very busy network.

Assessment Findings:

In a significant number of cabinets there was a PLC and some other devices directly connected to one of the redundant legs. 

I looked at the installation and asked the technician walking around with us if they were having any communication issued. The technician looked at his counterpart and sort of smirked and asked why I asked. I said because of the amount of traffic the PCS generated I was suspicious of the PLC tolerating it well. He then revealed that as a matter of fact the system dropped off line every month or so, sometimes more often and required a hard reset to get it back. He asked if I knew what was causing the drop outs. By the way, this was their safety system.

Funny they should ask; I think I do know.  Many PLCs do not like excessive traffic on their Ethernet ports. They simply can’t handle it, they have been known to lock up, corrupt, stop communicating, stop processing IO, etc. The fix is to put a barrier device in place in front of the PLC. This barrier device has to accomplish two tasks, first, limit the traffic to only what is needed by the PLC, and second, rate limit traffic should a broadcast storm occur. However, if you think about it, the way the PCN operates, the PLC is under continuous attack in a constant state of broadcast storm as it did not use the native traffic the PCN used but instead in this case used Modbus/TCP. 

Subsequent Recommendations:

We worked with the local PCN group and developed a very simple yet effective set of options for them to implement. All reducing the traffic to the safety PLCs.

The first option was to place an Industrial switch in the line then putting a barrier device in front of the switch. 

The benefit is simplicity of design and minimal cost impact.

The second option was to place a barrier device in front of each device. 

The benefit here is reducing the single point of failure of either they switch or the firewall.

The company opted to go with the second design are in the process of implementing it at this time. There is no doubt this will reduce or eliminate the communications issues and increase the availability and reliability of their system. 

Key Points Learned:

Network segmentation is extremely important and comes in all shapes and sizes. Sometimes you are talking about major network sections being segmented so improve reliability and distribute communications. Sometimes however you need to focus on the essential operations. Such as the above example., The SIS was losing communication, the reliability and safety of the process was then called into question. While the remediation was rather simple, it took looking at the network architecture design with a view and the knowledge of how PLCs can be fragile when exposed to inappropriate or excessive network traffic.