DevOps and Operational Technology: a security perspective

3391525700_187606bd23_z

I have worked in the Operational Technology (OT) environment for years, predominantly in major Oil and Gas companies. And yes, we all know that this space can move quite slowly! Companies traditionally employ a waterfall model while managing projects with rigid stage gates, extensive planning and design phases followed by lengthy implementation or development.

It’s historically been difficult to adopt more agile approaches in such an environment for various reasons. For example, I’ve developed architecture blueprints with a view to refresh industrial control assets for a gas and electricity distribution network provider in the UK on a timeline of 7 years. It felt very much like a construction project to me.  Which is quite different from the software development culture that typically is all about experimenting and failing fast. I’m not sure about you, but I would not like our power grid to fail fast in the name of agility. The difference in culture is justified: we need to prioritise safety and rigour when it comes to industrial control systems, as the impact of a potential mistake can cost more than a few days’ worth of development effort – it can be human life.

The stakes are not as high when we talk about software development.  I’ve spent the past several months in one of the biggest dot-coms in Europe and it was interesting to compare and contrast their agile approach to the more traditional OT space I’ve spent most of my career in. These two worlds can’t be more different.

I arrived to a surprising conclusion though: they are both slow when it comes to security. But  for different reasons.

Agile, and Scrum in particular, is great on paper but it’s quite challenging when it comes to security.

Agile works well when small teams are focused on developing products but I found it quite hard to introduce security culture in such an environment. Security often is just not a priority for them.

Teams mostly focus on what they perceive as a business priority. It is a standard practice there to define OKRs – Objectives and Key Results. The teams are then measured on how well they achieved those. So say if they’ve met 70% of their OKRs, they had a good quarter. Guess what – security always ends up in the other bottom 30% and security-related backlog items get de-prioritised.

DevOps works well for product improvement, but it can be quite bad for security. For instance, when a new API or a new security process is introduced, it has to touch a lot of teams which can be a stakeholder management nightmare in such an environment. A security product has to be shoe horned across multiple DevOps teams, where every team has its own set of OKRs, resulting in natural resistance to collaborate.

In a way, both OT and DevOps move slowly when it comes to security. But what do you do about it?

The answer might lie in setting the tone from the top and making sure that everyone is responsible for security, which I’ve discussed in a series of articles on security culture on this blog and in my book The Psychology of Information Security.

How about running your security team like a DevOps team? When it comes to Agile, minimising the friction for developers is the name of the game: incorporate your security checks in the deployment process, do some automated vulnerability scans, implement continuous control monitoring, describe your security controls in the way developers understand (e.g. user stories) and so on.

Most importantly, gather and analyse data to improve. Where is security failing? Where is it clashing with the business process? What does the business actually need here? Is security helping or impeding? Answering these questions is the first step to understanding where security can add value to the business regardless of the environment: Agile or OT.

Image by Kurt Kurasaki.


Governance Models – Cloud

Your company has decided to adopt Cloud. Or maybe it was among the ones that relied on virtualised environments before it was even a thing? In either case, cloud security has to be managed. How do you go about that?

Before checking out vendor marketing materials in search of the perfect technology solution, let’s step back and think of it from a governance perspective. In an enterprise like yours, there are a number of business functions and departments with various level of autonomy. Do you trust them to manage business process-specific risk or choose to relieve them from this burden by setting security control objectives and standards centrally? Or maybe something in-between?

Centralised model

1

Managing security centrally allows you to uniformly project your security strategy and guiding policy across all departments. This is especially useful when aiming to achieve alignment across business functions. It helps when your customers, products or services are similar across the company, but even if not, centralised governance and clear accountability may reduce duplication of work through streamlining the processes and cost-effective use of people and technology (if organised in a central pool).

If one of the departments is struggling financially or is less profitable, the centralised approach ensures that overall risk is still managed appropriately and security is not neglected.  This point is especially important when considering a security incident (e.g. due to misconfigured access permissions) that may affect the whole company.

Responding to incidents in general may be simplified not only from the reporting perspective, but also by making sure due process is followed with appropriate oversight.

There are, of course, some drawbacks. In the effort to come up with a uniform policy, you may end up in a situation where it loses its appeal. It’s now perceived as too high-level and out of touch with real business unit needs. The buy-in from the business stakeholders, therefore, might be challenging to achieve.

Let’s explore the alternative; the decentralised model.

Decentralised model

2

This approach is best applied when your company’s departments have different customers, varied needs and business models. This situation naturally calls for more granular security requirements preferably set at the business unit level.

In this scenario, every department is empowered to develop their own set of policies and controls. These policies should be aligned with the specific business need relevant to that team. This allows for local adjustments and increased levels of autonomy. For example, upstream and downstream operations of an oil company have vastly different needs due to the nature of activities they are involved in. Drilling and extracting raw materials from the ground is not the same as operating a petrol station, which can feel more like a retail business rather than one dominated by industrial control systems.

Another example might be a company that grew through a series of mergers and acquisitions where acquired companies retained a level of individuality and operate as an enterprise under the umbrella of a parent corporation.

With this degree of decentralisation, resource allocation is no longer managed centrally and, combined with increased buy-in, allows for greater ownership of the security programme.

This model naturally has limitations. These have been highlighted when identifying the benefits of the centralised approach: potential duplication of effort, inconsistent policy framework, challenges while responding to the enterprise-wide incident, etc. But is there a way to combine the best of both worlds? Let’s explore what a hybrid model might look like.

Hybrid model

3

The middle ground can be achieved through establishing a governance body setting goals and objectives for the company overall, and allowing departments to choose the ways to achieve these targets. What are the examples of such centrally defined security outcomes? Maintaining compliance with relevant laws and regulations is an obvious one but this point is more subtle.

The aim here is to make sure security is supporting the business objectives and strategy. Every department in the hybrid model in turn decides how their security efforts contribute to the overall risk reduction and better security posture.

This means setting a baseline of security controls and communicating it to all business units and then gradually rolling out training, updating policies and setting risk, assurance and audit processes to match. While developing this baseline, however, input from various departments should be considered, as it is essential to ensure adoption.

When an overall control framework is developed, departments are asked to come up with a specific set of controls that meet their business requirements and take distinctive business unit characteristics into account. This should be followed up by gap assessment, understanding potential inconsistencies with the baseline framework.

In the context of the Cloud, decentralised and hybrid models might allow different business units to choose different cloud providers based on individual needs and cost-benefit analysis.  They can go further and focus on different solution types such as SaaS over IaaS.

As mentioned above, business units are free to decide on implementation methods of security controls providing they align with the overall policy. Compliance monitoring responsibilities, however, are best shared. Business units can manage the implemented controls but link in with the central function for reporting to agree consistent metrics and remove potential bias. This approach is similar to the Three Lines of Defence employed in many organisations to effectively manage risk. This model suggests that departments themselves own and manage risk in the first instance with security and audit and assurance functions forming second and third lines of defence respectively.

What next?

We’ve looked at three different governance models and discussed their pros and cons in relation to Cloud. Depending on the organisation the choice can be fairly obvious. It might be emerging naturally from the way the company is running its operations. All you need to do is fit in the organisational culture and adopt the approach to cloud governance accordingly.

The point of this article, however, is to encourage you to consider security in the business context. Don’t just select a governance model based on what “sounds good” or what you’ve done in the past. Instead, analyse the company, talk to people, see what works and be ready to adjust the course of action.

If the governance structure chosen is wrong or, worse still, undefined, this can stifle the business instead of enabling it. And believe me, that’s the last thing you want to do.

Be prepared to listen: the decision to choose one of the above models doesn’t have to be final. It can be adjusted as part of the continuous improvement and feedback cycle. It always, however, has to be aligned with business needs.

Summary

Centralised model Decentralised model Hybrid model
A single function responsible for all aspects of a Cloud security: people, process, technology, governance, operations, etc. Strategic direction is set centrally, while all other capabilities are left up to existing teams to define. Strategy, policy, governance and vendors are managed by the Cloud security team; other capabilities remain outside the Cloud security initiative.
Advantages Advantages Advantages
  • Central insight and visibility across entire cloud security initiative
  • High degree of consistency in process execution
  • More streamlined with a single body for accountability
  • Quick results due to reduced dependencies on other teams

 

  • High level of independence amongst departments for decision-making and implementation
  • Easier to obtain stakeholder buy-in
  • Less impact on existing organisation structures and teams
  • Increased adoption due to incremental change
  • High degree of alignment to existing functions
  • High-priority Cloud security capabilities addressed first
  • Maintains centralised management for core Cloud security requirements
  • Allows decentralised decision-making and flexibility for some capabilities

 

Disadvantages Disadvantages Disadvantages
  • Requires dedicated and additional financial support from leadership
  • Makes customisation more time consuming
  • Getting buy-in from all departments is problematic
  • Might be perceived as not relevant and slow in adoption

 

  • Less control to enforce Cloud security requirements
  • Potential duplicate solutions, higher cost, and less effective control operations
  • Delayed results due to conflicting priorities
  • Potential for slower, less coordinated development of required capabilities
  • Lack of insight across non-integrated cloud infrastructure and services
  • Gives up some control of Cloud security capability implementation and operations to existing functions
  • Some organisation change is still required (impacting existing functions)