OT Patch Management Strategy: Seven Best Practices

Windows: The Common Gateway

The most common attack vector for any company is the Microsoft Windows operating system, on which almost all software runs. Employees who have work-issued personal computers benefit from behind-the-scenes patches scheduled by IT to update software and systems to fix bugs or introduce improvements. OT patching is slightly different in that it needs to be scheduled during maintenance-related downtime – but it’s just as important.

Patches are released weekly by Microsoft on its “Patch Tuesdays.” As recently as May 2019, the company was releasing important patches related to warnings of potentially catastrophic “zero-day” exploits attacking Windows vulnerabilities, dubbed “Bluekeep.”

Recent attacks including LockerGoga in March 2019 and NotPetya and WannaCry in 2017 were successful because they took advantage of known vulnerabilities for which Windows had already released patches.

In short, the fixes had been available, but they hadn’t been implemented. The sad truth is, had the victims of those attacks been proactively patching, they would have been in a much better place to protect their assets. There’s no good excuse to avoid patching, especially when the stakes are so high.

Developing a Patch Strategy

Here are some best practices to apply when developing a patch strategy.

1. Start with identifying your vulnerabilities. This includes a thorough inventory of your devices – not just their identities but also their attack surfaces, and not just at a single site but at scale across a regional or global supply chain. Take advantage of tools that will help you understand what the known attack surface looks like.
2. For collecting this inventory, determine if a passive or active approach is best. An active approach can carry some risk – scanning the entire environment introduces traffic onto an OT network that might cause older or legacy devices to go down. So in most cases, it’s best to initially take a more passive approach to identifying what devices are out there and what their attack surface is. You may find both virtualized and physical compute devices running Windows operating systems; consider leveraging technologies such as virtualization that allow you to consolidate compute and operating system surfaces into a single more manageable environment, in turn allowing you to speed up the patching process.
3. Investigate with your vendors to determine if their software has been tested and validated. When Microsoft releases patches, it is your responsibility to determine if those patches have been approved or validated for the software installed on your systems. For example, Rockwell Automation validates Microsoft’s weekly patches on its software and releases notifications that classify them as fully qualified, partially qualified or not qualified.
4. Stage patches and group devices. The qualified patches must now be staged in the industrial control environment where they are needed. But your environment may be running different versions or different vendors’ automation software, so group the devices according to how you would be deploying these qualified patches. Tools like WSUS (Windows Server Update Services) and SCCM (Microsoft System Center Configuration Manager) can be utilized to define separate groupings for windows devices dedicated to the operations environment. This allows you not only to apply specific qualified patches only to the devices that are approved for them, but also gives you the ability to set specific schedules that meet your downtime requirements.
5. Test before you apply. Consider funding a test environment that mimics and runs the production applications. If that is not fiscally feasible, consider creating groups of devices based on type of criticality. If there are low-priority lines or systems that aren’t running continuously, consider patching those first as a test case. Also, note any customizations on applications, as they can have potential impacts of patching on the environment.
6. Schedule your patch deployments. In the OT world, patching isn’t as simple as applying it whenever it’s needed. It has to be coordinated with downtime schedules. Many patches require a reboot on systems that may not have been rebooted in years. So plan ahead to determine how much time you might need and where that window of time will fit in the overall downtime schedule.
7. Apply and perform “hyper-care.” Once the patches have been applied and the device successfully reboots, define test scenarios that should be run during a “hyper-care” period. You should be looking at everything that factors in to the machine being fully available – confirming it is running as normal and applications are functioning properly. Failure to adequately test could result in unplanned downtime.

Of course, without support from plant management, operations management and senior-level management, a patch management strategy can’t be successfully implemented. Help leadership to understand patching is just as important as other preventative maintenance and safety activities that often get prioritized during scheduled downtime. Also important is your organization’s larger cybersecurity strategy, which should address the entire cyberattack continuum to provide planning and protection before, during and after an attack.

A patch management strategy that is relevant, efficient and well-structured will help manufacturers overcome the complexity of OT patch management. By identifying your vulnerabilities, staying up to date on patches released by Windows, and creating a plan to test and deploy patches, you can help protect your company from potential calamity at the hands of hackers and cyberattackers.