The challenge of securing a production environment

Are you tasked with securing a production environment? I know how hard this is – I've been there. Production environments are extremely difficult to secure for both technical and business reasons. In this post, we’ll show how to identify your production environment and discuss the major challenges of securing the environment. In the next post, we’ll walk through a lightweight process to help evaluate and communicate the current and future risk of your production cloud environment to your executive team and board.

What is a production environment?

A production environment is directly tied to top line growth of the company either directly or indirectly. If your company offers an application that external customers pay for, where that application runs is the production environment. The production environment also includes your company’s APIs, feeds, or data that customers pay to access. In contrast, your corporate laptops and customer billing data are part of the corporate environment, which we view as related to, but distinct from the production environment.

In the past, production environments consisted of physical datacenters or on-premise servers. Today production environments are increasingly or exclusively within one or more cloud providers. Sometimes the production environment is called “infrastructure” or in the context of security, “infrastructure security.”

What are the major risks in a production environment?

I know from years of defending the largest production environments from persistent adversaries that every environment is different and poses unique challenges. To identify the top risks and successfully secure your company’s production environment, you must understand these core truths:

1. Production environments contain your company’s most “important” data.
2. Production environments change extremely fast, when compared to corporate IT environments.
3. Production environments are controlled by engineering.

Truth 1: Production environments contain all kinds of data that generally are your customers data – your customers are entrusting you with some, or in many cases, all of their data. From a risk perspective, this means adversaries are highly motivated to find a way to access and steal this data from your environment. To date, the security industry has focused most of the spend and effort on security for corporate environments, not production environments. While securing identities and endpoints used to access production environments is an important part of a robust security program, it is not the same as securing the production environment itself.

Truth 2: Production environment changes are driven by the velocity of the product and engineering teams, which are typically driven by the company vision, customer needs, and the sales organization. Typical security products are not built for production environments – they cannot handle the rate of change, cause significant performance problems, drive COGS up (and margins down), or don’t work within the workflows of the engineering and product teams. Any security solution must operate at the speed of the business, and work seamlessly in the production environment.

Finally, Truth 3: Production environments are controlled by engineering teams and not security teams, and typically for good reason. Businesses typically have service level agreements (SLAs) with customers, and customer satisfaction is at risk if the company has unplanned downtime or fails to communicate about availability issues. Security is typically gated by engineering due to past unintended changes by the security team that caused availability issues, or control specific things such as WAFs at the edge, or scanners that are part of the CI/CD pipelines. Any solution you recommend must involve the product and engineering teams and be viable from a business perspective.

In short, the adversary wants to live in your production environment, and you, the person accountable for security, must rely on influence, business cases, and communication - aided by technology and people - to minimize this risk.

As someone with accountability for security of the production environment, what is the first step?

Step one is to understand your company’s current implicit (or explicit) risk tolerance, especially in the wake of the new SEC legislation. If your company doesn’t have explicit documentation, or it needs updating, check out our next post for a step-by-step, lightweight process to help evaluate and communicate the current and future risk of your production cloud environment to your executive team and board.

Do you have your own way of framing what it means to “be secure” in a production environment? Experienced similar challenges? If so, I’d love to learn more, DM or start a discussion!