The three common bias conditions that consistently contribute to a disconnect between IT and OT can be categorized into philosophy, project, and technology biases.
- Philosophy biases revolve around the overall perspectives on the operational technology function.
- Project biases delve into the physical and mental obstacles that hinder progress.
- Technology biases center on the selection of appropriate tools for OT-defined projects, with the support, budget, and perspective of the OT department.
This post will examine the three biases between IT and OT teams and discuss ways for both teams to overcome them.
The Philosophy Bias
Below are three important points to consider about ‘The Philosophy Bias’. They highlight the fundamental differences in how IT and OT see things and explain why some vital aspects of OT might be overlooked or misunderstood.
Operational Technology Often Goes Unnoticed in Analyst Research
When IT organizations seek guidance and insights, they typically rely on traditional IT research sources. These research firms invest significant time, resources, and effort into studying a wide range of IT tools. However, it’s important to note that these analyses are carried out by IT professionals, primarily for IT purposes within IT environments. As a result, when selecting technology and aligning it with components for an OT environment, there can be a significant disconnect between IT and OT programs.
To put it differently, IT tools don’t always seamlessly integrate into an OT environment, at least not in the way they are initially intended for use in traditional IT settings.
An example of an IT disconnect within OT network systems is an IT security tool that focuses on frequent patching. While it helps protect from vulnerabilities, it requires constant restarts. Applying this tool within the OT environment could be disastrous, especially if it’s a manufacturing or power plant control system. Forced restarts can interrupt critical processes and lead to outages or safety hazards. This is why it’s important to consider your OT environment's operational realities and constraints before making recommendations.
OT Network Systems Aren’t Uniform
In many cases, IT teams rely on outsourced expertise and centralized tools to manage a fleet of similar and nearly identical systems. This approach allows them to efficiently handle hundreds or even thousands of assets using a single toolset or a small, centralized, or offshore team.
However, in the realm of OT, things are different. Although there may be numerous IT-looking assets, they come in various configurations, run different software, have unique modifications, and may have special requirements. This diversity often leads to a situation where a tool selected for a particular generation or profile of operating system may not be suitable for all types of assets in the OT domain. Consequently, any tool choice that only caters to a subset of assets falls short of providing comprehensive coverage.
While System Center Configuration Managers (SCCM) are great for installing and securing all computers within an organization from one location in an IT setting, they are not equipped to handle the needs of over 1,000 Linux or Unix operational assets frequently encountered in operational environments.
In OT Security, it’s Essential to Prioritize the Fundamentals Over Complexity
How often have you come across reports indicating that the operational side of a business lacks crucial elements like perimeter monitoring, SIEM (Security Information and Event Management), or SOC (Security Operations Center) oversight? While these aspects are undeniably crucial for a robust security program, the challenge is that alerting or monitoring often occurs as an after-the-fact response. What’s been overlooked or neglected for years in many OT environments are the fundamental building blocks of security, such as patching, backups, system hardening, and implementing the principle of least privilege.
If you aim to bring about a significant improvement in OT security, it’s imperative to start with these foundational measures.
The Project Bias
Below we walk through the unique challenges of ‘The Project Bias’, arising from the tightly integrated nature of OT with its immovable components. These three insights shed light on why managing OT projects differs significantly from traditional IT endeavors and the complex factors that may lead to overlooked or misunderstood aspects of OT.
Operational Technology is Tied to Immovable Objects
We understand that OT systems often involve outdated hardware and operating systems that are no longer supported, which makes a straightforward upgrade to Windows 10 impossible. These older systems typically run specialized software and communication protocols that are essential for the safe functioning of facilities. If the vendor doesn’t offer an upgrade solution or if the facility lacks the budget and downtime required for software upgrades, testing, documentation, and getting back to regular operations, then upgrading the asset becomes impractical. In some cases, these assets oversee significant portions of an operation. Upgrading a Distributed Control System (DCS) or Supervisory Control and Data Acquisition (SCADA) system demands a substantial amount of time and financial resources, resulting in extended production downtime. When contemplating an OT upgrade or requesting a system upgrade, it’s important to understand that it’s not as straightforward or isolated as simply upgrading a single operating system; there are broader implications involved.
Operational Technology Systems Require OT Services and Support
Securing OT environments presents two challenges and conflicting priorities from 1) IT teams and 2) Original Equipment Manufacturers (OEM).
Aligning Efforts Between IT and OT Teams
OT teams are wary when it comes to unauthorized access and making changes to their critical infrastructure. IT initiatives like software updates or new technology deployments further amplify these concerns.
Bottom line: OT teams need to be comfortable with the idea that anyone in their environment could access or change their assets. The differing perspectives between IT and OT teams underscore how crucial it is to build trust between the groups. Building this trust takes time and requires consistent communication, but it’s critical for the successful deployment and maintenance of security tools in OT.
Navigating OEM Vendor Relationships
Original Equipment Manufacturer (OEM) vendors are another source of influence when it comes to security. Frequently, these vendors hesitate when OT teams want to implement security solutions because they worry about how these changes might affect their support for critical systems. This hesitancy manifests in two ways:
- Vendor reliance: OT teams depend on the OEM for operational support and default to the vendor’s objections, even with necessary security changes.
- Contractual constraints: OEM vendors may use their contracts as a defense to prevent OT teams from deploying security tools that haven’t been tested or approved by the vendor itself.
In both cases, understanding the role of OEM vendors in plant operations poses a significant challenge within OT, an area where IT may lack experience. Before making any OT upgrades, it’s important to understand the existing relationship between your OT team and the OEM vendors.
The IT Budget Should Be Separate from the OT Budget.
Often, CISOs or IT executives hesitate to approve security proposals for OT because they underestimate the sheer number of assets in an OT environment. In larger facilities or global companies, there can be tens of thousands of assets, sometimes even surpassing the number of IT assets. When an OT project requests substantial budgets to enhance plant security, it faces resistance. They may be told to reduce the project scope or phase the deliverables, and the already stretched operational staff may be assigned the deployment and maintenance tasks to cut costs. Unfortunately, this often leads to projects that are never fully implemented or properly maintained. Many OT environments lag for months or even years in basic security practices, and the initial investment required to deploy technology and secure these assets represents a significant upfront cost.
Technology Bias
The three insights below provide a better understanding of the unique difficulties involved in OT security management, explaining why OT projects differ significantly from traditional IT projects and the complex factors that can sometimes cloud or hinder critical aspects of OT security.
IT Management Solutions Assume Relatively Robust Endpoints
The reality is quite different in IT and OT.
In truth, most scan-based IT tools can be invasive and have a history of causing disruptions in the more delicate and proprietary OT systems. To make use of scan-based technology in OT environments, you have to carefully scale down the scan, allocate extra time for OT staff supervision, and limit scanning to offline systems or during planned outages. When you factor in all these conditions, you end up with minimal security coverage from scan-based security tools.
To truly succeed, you need reliable, OT-tested profiling and data collection tools that can maximize asset coverage and automate asset insights while keeping operations safe. In other words, it’s crucial to adapt security measures to the specific challenges and nuances of OT rather than relying on standard IT approaches that may not be suitable in this context.
IT Best Practices Break OT Systems
One common IT practice for system hardening involves having endpoints display a login banner when the system starts up. The idea behind this is to remind users that they are working on a corporate-owned or critical system. However, there’s a challenge in OT because these systems must maintain 100% uptime. Consequently, these assets are often configured for auto-reboot and auto-login to ensure redundancy and continuous monitoring of safety systems. When logon banners are introduced, they disrupt the auto-login process for these vital OT systems.
Let’s illustrate an example of this with a chemical plant’s control system. This control system is designed for automated restarts. The login banner is problematic because it halts the routine update and prevents it from responding to dangerous pressure changes.
This is why most OT environments only implement around 40 to 50 of the top 100 security controls outlined in the CSC 20. Many of these controls are either not applied or can interfere with critical operations. In essence, adapting standard IT security practices to OT can pose significant challenges and may not always be suitable for the unique requirements of OT environments.
Service Level Agreements (SLAs) in OT Are More Demanding Than Those in IT
In typical IT environments, users expect internet and mail or file servers to be readily available when they connect. If there’s an issue, they can usually carry on with their tasks while IT resolves the problem and restores connectivity. These outages or scheduled maintenance windows in IT typically take three to four hours, during which End Users may not have access to the system or service. However, in OT, a reboot or misconfiguration of a switch or communication point can immediately disrupt safe operations.
For many industries, this disruption can result in a loss of product specifications and quality. In more critical cases, it can pose a safety hazard, as there’s no visibility into vital parameters like pressure, flow, temperature, or speed, leading to instant product degradation or even a complete shutdown. Such production interruptions can significantly impact revenue. This problem becomes even more complex in industries where restarting production is not as simple as turning a conveyor belt on or off.
For instance, coal-fired generation units may take 25 to 30 hours to reach full capacity after a shutdown, and in fields like refining and petrochemicals, it can take hours or even days to return to the proper product specifications.
I remember a particular OT security presentation to an operating company. They had recently suffered a severe cyber incident in their corporate network, resulting in substantial damage. During my presentation, the IT team raised numerous concerns about potential security vulnerabilities, which was not surprising, as no security system is foolproof. They assured me they had it under control, but I later found out that this meant they had disconnected internet access for all operational facilities.
I expressed my concern that plant managers might resort to using “sneaker nets” and USB drives to transfer data, updates, and files in and out of the facility. They didn’t believe their plant managers would defy the USB usage policy. However, when we visited the facility later that day, the plant manager’s desk was littered with USB drives. I asked him why he was ignoring the corporate USB policy, and he simply smiled and said, “How much trouble do you think I would be in if the plant stopped producing? I’m pretty sure I’ll get a pass on USB use.”
Bridging the Divide Between IT and OT Teams
Now that we covered the three biases, let’s dive into the solutions that can address the disconnects. Aligning IT and OT teams involves having a better understanding of operational processes and regulations. This includes:
- Introducing OT processes and machinery to IT teams
- Understanding IT’s cybersecurity measures and data management protocols
- Diving into the compliance regulations IT and OT realms
- Conducting bi-weekly or monthly sessions to discuss challenges and action items
After developing a better understanding of one another, you can start developing OT Systems Management (OTSM) policies and procedures that meet the needs of both. If you can’t implement a standard IT practice for specific infrastructure, compensating controls can help you balance security and efficiency.
The next step in bridging the IT/OT gap is finding tools and technology that work for OTSM but can also integrate with your IT systems. This can include:
- Vendor-agnostic endpoint management
- Network protection measures like firewalls and intrusion detection systems
- Tools that incorporate real-time monitoring and predictive maintenance capabilities
Balancing Production and Protection
Protecting your infrastructure and production is critical. Simply imposing IT solutions onto OT can have dire consequences. True success requires educating yourself about the distinctions between IT and OT needs, addressing biases related to philosophy, projects, and technology, and fostering collaboration that strengthens the IT/OT divide. It’s crucial to set realistic expectations that security improvements won’t happen swiftly or without encountering challenges along the way. In this dynamic landscape, the key is perseverance and a commitment to making OT security stronger over time.
1 - https://www.ibm.com/reports/data-breach
