One of the common use cases for serverless applications is to serve as an orchestration component for cloud applications. In such cases, serverless functions serve as the glue that holds everything together.
As an example, consider an application in which a serverless function is responsible for launching container instance when new data is available for processing, which is done inside the container. Many folks refer to such applications as “hybrid cloud applications”.
Today’s episode deals with container-serverless-hybrid application security.
I recently went over an AWS tutorial titled “Better Together: Amazon ECS and AWS Lambda” ( https://amzn.to/1QJidM0 ).
In this scenario, the system works as follows:
- User uploads a ZIP file to an S3 bucket
- The S3 bucket is configured with an S3 event notification, which triggers a Lambda function upon a file upload
- The Lambda function takes the event data it received from S3 and sends it as a message into an Amazon Simple Queue Service (SQS) queue
- The Lambda function then starts an ECS task that can fetch and process the messages from SQS.
- An ECS task containing a simple shell script which reads messages from SQS, extracts the S3 bucket and key information, downloads and unpacks the .ZIP file from S3 and proceeds with the file processing work
- Once the processing work is done, results are stored back in the S3 bucket
Here’s a rough sketch of the overall process:
Alrighty then… we have a system, that eventually takes user input, and uses it inside a shell script, I wonder what could go wrong?!
Let’s start by looking at the Lambda function, which is invoked when a file is uploaded to the S3 bucket - the Lambda function uses a function called checkS3SuffixWhiteList, which makes sure that only files with the whitelisted extensions will be processed. In our case, only .zip files:
The first weakness is that the file extension sanity check is rather rudimentary, and only checks that the string ends with .zip. This allows an attacker to craft whatever malicious payload he/she wishes (as a filename), as long as the payload ends with “.zip”.
Let’s now look at the shell script, which runs inside the container and does the actual processing:
Note that “key” holds the filename to be processed, it is then parsed to the filename (base) and the extension (ext). Some additional sanity checks are applied to make sure that only filenames with the expected format are processed, however, no real input sanitization is done on the key, before it is used in several shell commands - essentially allowing the attacker to inject remote commands through the filename.
Perhaps the only thing that stands between the attacker and a successful exploit, is the fact that in this case, S3 URL-encodes filenames, and nobody is performing any kind of URL-decoding. However, it’s not unlikely that other systems will decode the filename. In fact, the PureSec research team actually tested a similar application, which decodes the input, and verified that it was indeed possible to inject malicious commands.
A successful exploit in the scenario above, would result in the attacker’s ability to execute arbitrary remote shell commands inside the container image, and in turn perform unauthorized actions such as leakage of all files on the S3 bucket, harness the container image for running rogue tasks, connect to any resource on which the container has access to, and so forth.
Protecting Hybrid Apps
In a previous blog post ( Securing Serverless - Episode 0x06 ) I described the Swiss Cheese Model for security. In the hybrid app scenario above, several weaknesses aligned (almost) perfectly, allowing a successful exploit. By applying application security protection at the serverless (Lambda) layer, the entire chain of events could have been avoided. If this was a web application, any security minded organization would probably deploy a WAF between the web front-end and backend application logic. However, in this case, we are looking at an S3 bucket which serves as the front-end, and a Lambda function as the middleware - a configuration that doesn’t allow a WAF to be deployed (see episode 0x01 of our Securing Serverless blog series) and requires a serverless-specific application security solution such as the PureSec Serverless Security Runtime Environment (SSRE).
It goes without saying, that the PureSec solution would’ve blocked such an attack during the moment the Lambda function would get invoked with such a malicious payload.
One might argue that a sophisticated machine-learning based OS (or container) runtime protection might have also prevented the attack, but there are a few issues with this approach:
- It would be extremely hard to enforce such a protection in an accurate way without generating false positives or suffering from false negatives
- Preventing the attack inside the container increases the risk, as it applies protection at the final step of the kill chain.
To summarize, if you are developing a hybrid serverless-container application, make sure that you perform proper threat modeling to understand the risks involved. If your serverless functions are the middleware between the front-end and the jobs running in containers - make sure that you apply application layer protections already at the serverless layer using a solution that is suitable for serverless.