Introducing runtime container visibility, attribution, and isolation

Today, we’re excited to announce the extension of our cloud-native runtime introspection and isolation capabilities to containers. Our sensor now provides the context of the originating container if runtime activity occurs inside of container technologies such as Docker, containerd, Kubernetes (K8s); on cloud-native services such as ECS, EKS, GKE; or on bare-metal nodes such as EC2 or GCE. The ClearVector sensor is available for both AMD64 and ARM64 and is supported on a variety of operating systems including Bottlerocket and Amazon Linux.

This purpose-built, runtime view of your container activity, enables you to quickly know the originating container, and in many cases the originating identity, of activity in container environments.

The situation - a compromised container

To illustrate, imagine a situation where a developer’s GitHub credentials are stolen and used to backdoor an image that is then used in a production environment. How would you know, and how quickly? What would you do next?

Figure 1 shows the high-level flow of the compromised container scenario:

More specifically, Figure 1 outlines the follow steps:

1. A developer’s GitHub credentials are stolen
2. The adversary uses the credentials to modify a Dockerfile in a GitHub repo the developer has access to
3. CI/CD automatically builds a new image based on the Dockerfile modifications
4. CI/CD automatically pushes the compromised image to ECR
5. K8s uses "latest" tag for images, and spins up new container using the modified image
6. The adversary connects to compromised container using a reverse shell (SSH)
7. ClearVector is used to isolate the image repository

Next, we’ll walk step-by-step through this scenario within the ClearVector product.

The notification and isolation

You receive a notification from ClearVector, as shown in Figure 2:

Immediately, you recognize this isn’t normal for your environment, and you pivot into ClearVector to investigate. Figure 3 shows more detail about the activity - this doesn't look good!

From the ClearVector timeline, you pivot on the container image, skynet-ai:latest, to see all associated activity, and go back in time to the creation date. As shown in Figure 4, you notice a GitHub identity, kat-resistance, was involved in approving changes in GitHub that caused your CI/CD process to build a new image and store the image in ECR. To read more about how ClearVector provides intelligence about GitHub identities and activity in AWS, read our post here.

You know you need more time to investigate, and, as shown in Figure 5, you use ClearVector to isolate the ECR repository the image came from - this prevents new containers from being run based on the compromised image. Additional actions you consider include isolating the GitHub identity, disabling the AWS CI/CD identity, stopping all running instances spawned from the image, or disabling the GitHub account while you investigate further.

The investigation

At this point, and within minutes, you’ve:

1. Been notified proactively by ClearVector about risky activity occurring inside containers in your production environment (in Slack, Teams, or the web app)
2. Isolated and contained what appears to be the source of ongoing persistent access (the skynet-ai image repository)
3. Identified next steps for investigation and containment, including the originating identity or “patient zero” (the kat-resistance GitHub user)

Additional investigative steps you might take with ClearVector include, as shown in Figure 6, pivoting from the timeline via the GitHub workflow link to look at the specific GitHub Action workflow run that performed the activity in AWS, as shown in Figure 7.

You might also deep dive into the activity around the time of the reverse SSH tunnel, to understand the impact. Figure 8 shows the compromised container starting an sshd server (1) and then connecting outbound from the container using a reverse SSH tunnel (2), which the adversary then tunnels through to access the sshd service inside the container (3), which subsequently launches the shell (4). The adversary now has root access, remotely, within the container!

You also notice, that after connecting through the reverse SSH tunnel, the adversary immediately begins to explore the environment to see what services they can access on the control plane via the EC2 execution role. Figure 9 shows the adversary running whoami followed by retrieving the EC2 instance metadata.

Conclusion

In this post we explored how ClearVector proactively surfaces risky activity occurring in containers, and attributes activity to originating identities while providing attribution of the activity to specific containers. Further, ClearVector connects this activity across GitHub, the control plane, and inside of container workloads to provide fast and easy detection and isolation capabilities when credentials are stolen by adversaries.

ClearVector customers can deploy sensors from the Sensors tab - check us out at BSidesLV, Defcon, Blackhat or contact us directly to learn more!