Merge Order Hijacking in AWS Lambda

As serverless computing increasingly becomes popular, third-party vendors are incentivized to provide runtime enhancements to serverless workloads. These enhancements often include telemetry collection, log enrichment, or runtime security enhancements. In this blog, we examine how AWS and third-party vendors deploy these enhancements within AWS Lambda, and how attackers can leverage these enhancements to compromise Lambda functions.

Background: Layers and Extensions

AWS Lambda enables developers to deploy enhancements to Lambda functions through at least two mechanisms: Lambda layers and Lambda extensions. In this section, we’ll briefly summarize each of these mechanisms and provide examples of publicly available solutions. For further information, refer to this previous post about Lambda internals and this follow-up post regarding a Lambda implant.

AWS customers use AWS Lambda to enhance their serverless workloads using a feature called Lambda layers - modular packages attached to Lambda functions that developers typically use to add additional resources and libraries without changing the Lambda function code. Layers must be formatted as ZIP archives and their contents are unzipped into the Lambda execution environment at initialization time.

Lambda extensions are another Lambda feature that enhance Lambda environments by allowing modules to deploy and execute within Lambda execution environments. AWS supports two extension types: internal and external. Internal extensions are packaged as part of the Lambda function code package. External extensions can deploy dynamically via the AWS API or the AWS console and are typically packaged within a Lambda layer. External extensions run as independent processes within the Lambda environment, unlike internal extensions deployed as part of the Lambda function code.

AWS allows extensions to be easily deployed using Lambda layers. Since layers are simple ZIP archives, they can also contain and deploy one or more extensions.

AWS and many third-party vendors release Lambda extensions to provide telemetry, logging, and security monitoring. Third parties are releasing Lambda extensions for telemetry collection, log enrichment, and security solutions. Next, we will look at how these extensions are loaded into a Lambda environment and the significance of load order.

Background: Lambda Layer Merge Order

When the Lambda service invokes and executes a Lambda function, the function’s execution environment is created and configured layers’ files are added or “merged” into the environment. Lambda defines a “merge order” when adding layers to a function. The merge order is defined as the order in which the layer contents are added to a Lambda execution environment. More specifically, for each layer, the Lambda service extracts the layer contents into the /opt directory in a function’s execution environment. Further, Lambda uses the same directory layout used for the layer ZIP archive at creation time (e.g., a layer in which the ZIP archive layout is /mylayer/myfile will be merged into the Lambda environment at /opt/mylayer/myfile).

By design, if a path collision (multiple layers attempt to extract their contents to a specific path) occurs, the last layer in the merge order will “win” and supplant the previously merged layer’s contents.

Now that we have some background on how these technologies work, let’s explore how an attacker could use the Lambda layer merge order to compromise a serverless environment by replacing files loaded earlier in the merge order. First, we will take a detailed look at a Lambda function that uses a Lambda extension. Then, we will show how an attacker can supplant one of the legitimate extensions to implant a Lambda function that uses the AWS Parameters and Secrets Lambda extension.

Lambda Extension Deployment and Execution Flow

For this example, let’s consider a Lambda function that is configured with a Lambda layer. This function makes use of the AWS Parameter and Secrets Lambda extension. Figure 1 illustrates the deployment process of this extension.

• In step 1, the extension is added to the Lambda function by attaching the layer named “AWS-Parameters-and-Secrets-Lambda-Extension" using the UpdateFunctionConfiguration AWS API.
• Next, in step 2, the layer is attached to the Lambda function - all subsequent Lambda initializations will extract the layer contents to the execution environment.
• Finally, in step 3, the extension executable “AWSParametersAndSecretsLambdaExtension” is placed into the /opt/extensions directory, where all extensions are executed from. When Lambda environment initializations occur, any executable file within /opt/extensions is executed as an external Lambda extension.

The AWS Parameter and Secrets Lambda extension is provided by AWS to cache values stored within the AWS Parameter Store and Secrets Manager to improve application performance and reduce cost. Since frequent lookups to these AWS services can be quite expensive, this extension will cache the results of API calls made to them within its process memory space. This extension starts a local HTTP server (by default on port 2773) that serves the cache contents to the Lambda function handler code. When the function needs access to a value stored in AWS Secrets manager, the function sends the request to the AWS Parameter and Secrets Lambda extension, calling the AWS API directly only when necessary. Figure 2 illustrates the execution flow of the Lambda function using the extension to get cached secret values.

• In step 1, the function code sends a request for a secret value to the AWS Parameter and Secrets Lambda extension. If the extension’s cache has the requested secret, the stored value is returned to the requester.
• If the extension’s cache does not contain the requested value, an API call is made to the AWS Secrets Manager service in step 2.
• The requested secret value is then returned to the function in step 3.

Lambda Extension Merge Order Hijacking

Now let's examine a scenario where an attacker wants to intercept all AWS Secrets Manager lookups made to the AWS Parameter and Secrets Lambda extension. By exploiting how the Lambda service merges layers and extensions into the execution environment, an attacker can craft a malicious layer containing a specifically named extension to force a path collision. This path collision would cause the malicious extension to load instead of the legitimate extension – the malicious extension performed a “merge order hijack” attack. This attack has the following impacts:

• Other extensions are effectively “disabled” because this specifically crafted extension is merged over them in the /opt/extensions directory in the Lambda environment.
• These disabled extensions may provide runtime security enhancements or logging - they are now rendered completely ineffective.
• Functionality being provided by legitimate extensions can easily be intercepted, modified, or outright denied.

Detecting these merge order attacks can also be difficult and not straightforward for several reasons:

• When querying the Lambda configuration via the AWS API, the configuration will show the legitimate layer attached as expected; however, the extension will not execute.
• To detect if a Lambda is affected by this style of attack, the contents of each layer would need to be downloaded and inspected using an AWS API such as GetLayerVersion.
• Each layer's contents would have to be compared to detect path collisions, which could be time consuming and may not be considered during an incident response.

Recall that Lambda layers are merged in a specific order and path collisions cause files to overwrite each other. If another layer is added to the Lambda environment and contains the same file path used by another AWS extension, the last layer in the merge order is the only one remaining in the execution environment. Other files and extensions remain in the environment with only the specific files involved in the path collision affected. Figure 3 illustrates this merge order hijack attack.

• Step 1 shows two layers, each containing an extension executable being added to a Lambda function using the UpdateFunctionConfiguration AWS API.
• The legitimate Lambda layer (layer A) and extension (extension A) are added to the execution environment first in merge order 1. An attacker layer (layer B) and extension (extension B) are added next in merge order 2.
• Step 2 shows these layers and extensions present in the Lambda execution environment alongside the Lambda function handler code.
• In step 3, the legitimate extension is extracted to /opt/extensions/AWSParametersAndSecretsLambdaExtension. The attacker extension is extracted next to the same path.
• This path collision results in the attacker extension supplanting the legitimate AWS Parameter and Secrets Lambda extension. The legitimate extension is no longer present in the Lambda execution environment and will not execute.

The attacker extension is now executed by Lambda in place of the AWS extension. Since our function handler code expects to use the local secret cache on port 2773, the attacker extension is now free to listen on this port and man-in-the-middle these lookups. In addition, any other configuration details supplied via environment variables to the legitimate extension are also available to the attacker extension. This enables the attacker extension to easily function in place of the legitimate extension even if a custom configuration is used. The attacker extension now has full control of these secrets, including modifying them or sending them to an attacker-controlled server for later use.

This man-in-the-middle attack is illustrated in Figure 4. The attacker extension is executing in place of the AWS extension, which is no longer running in the Lambda execution environment.

• The Lambda function is invoked and, in step 1, requests an AWS secret from the cache listening on port 2273.
• Instead of communicating with the AWS extension, the function code is sending requests to the attacker extension, which, in step 2, makes the API call to the AWS Secrets Manager service.
• The attacker extension sends a copy of the secret value to a remote server controlled by the attacker, shown in step 3.
• In step 4, the attacker extension returns the secret value to the Lambda function.

People familiar with DLL “search order hijacking” attacks on operating systems may see some similarities. These attacks hijack the flow of execution of legitimate applications based on system load semantics.

Anyone familiar with Lambda service role permissions may wonder what benefit this type of attack has over simply looking up the AWS secret values directly. One reason this is not always feasible is the Secrets Manager API has a different permission set for listing secrets versus getting their values: the attacker would have to guess the name of the secret. In addition, API calls to Secrets Manager may generate logs or alerts when accessed more than expected. By supplanting the original AWS extension, the attacker doesn’t generate an unusual amount of API calls to AWS Secrets Manager.

Possible Mitigations

One might consider this type of attack and think, “I’ll simply avoid untrusted layers in my environment,” which is certainly the correct mindset. However, what about currently trusted layers in your environment? What if a supply chain style attack adds malicious code to a trusted vendor’s layer? As mentioned, when layers are added to a Lambda function, their actual contents are extracted to the Lambda’s file system, which makes detecting malicious activity difficult. Even if another extension could scan for malicious activity, it, too, could be disabled. AWS allows the easy configuration of layer merge orders, but when dealing with third party layers, it is difficult to know the safe or correct merge order to use.

Architecturally, these types of attacks could be mitigated at the Lambda platform level by either returning an error when a merge-time path collision occurs or by guaranteeing a unique path when the layer’s contents are extracted into the Lambda executing environment. Currently it remains up to the Lambda function developer to know the correct merge order. We believe this is challenging to know in most environments.

Conclusion

In this post, we examined how attackers might use Lambda layer merge orders to implant Lambda functions by supplanting other extensions. Lambda extension merge order hijacks can disable or replace legitimate applications and are difficult to detect without inspecting the contents of the associated Lambda layer.

Attack surface will increase as serverless deployments increase – therefore, it remains essential to understand the risk being introduced into serverless environments to effectively defend them.