Here's a short blog post on design-for-failure, serverless scalability, App layer DoS and what happens when you rely on open source 3rd party libraries.
I was approached by someone who runs a sarcastic newsletter, who asked me to try and hack his serverless-based newsletter registration app (and no, it wasn't Jeremy Daly--it was someone far more sarcastic). I believe the promise was that if we'll do some serious damage, he'll give us credit and even deploy PureSec. Needless to say, I never say no to an opportunity to inflict some damage and most importantly - to get access to more serverless security data and knowledge.
In order to speed things up - that person granted us access to the source code, which was hosted on Github, with the promise that - "there's nothing really special about the app, it's running with least-privileged permissions" and "I doubt you will find anything interesting, but give it your best shot!".
App-DoS via Email Addresses
Looking at the source code, the first thing that caught our attention was the fact that the app was using the "SendGrid" service for handling email operations.
Digging a bit more into the SendGrid library, we noticed it was leveraging Python's 'email' package, and specifically using the method email.utils.parseaddr().
Alrighty then - parsing emails are ya? Parsers are always the most interesting component to target,
The parseaddr() method is used to parse a given email address and return an Email object. Sifting through the code, we found out that this function supports all sorts of email formats mentioned in RFC 822 - e.g. <foo bar> email@example.com - looking at the code, we figured it will run inefficiently when presented with long email addresses that contain certain characters with special meaning.
With this information at hand, we created a 100KB long email address in the format:
Sending a single API call with this email address to the "subscribe" API endpoint demonstrated that the serverless function stalled for a few seconds, whereas with a standard email address it would simply reply within milliseconds. Bingo.
We then invoked the newsletter subscribe API endpoint using 1,000 concurrent API calls eventually hitting the account's concurrency limit. Any additional user which then tried to subscribe to the newsletter waited and waited and eventually received a 500 HTTP error from the cloud provider, and was obviously not subscribed.
This is a classic case of application-layer denial of service, where instead of bombarding with huge volumes of traffic, you hog system resources by exploiting inefficient application code. The attack was run from a simple Macbook Pro with a terrible keyboard, connected to a DSL line, and it only took a few seconds to run.
Interestingly, when we stopped sending API calls, it took a few minutes for the system to recover - not something we expected to see, but it did give us time to record a nice screen capture video.
- First and foremost - I think it's crucial to state that this is not a weakness, vulnerability or drawback of serverless platforms. However, it does come to demonstrate that you should pay attention to how you design your serverless applications to withstand attacks, and how you protect them against abuse.
- DoS & financial resource exhaustion in serverless is a real risk which you should be aware of and design your systems accordingly (See Serverless Top 10 section 8: SAS-08)
- It doesn't matter how secure your own code is, if you're bringing in 3rd party libraries, your security can quickly degrade (See Serverless Top 10 section 6: SAS-06).
- When designing serverless applications, pay attention to scalability (more on this in this blog post).
- While this didn’t qualify as a security issue outright, this did demonstrate why API usage plans are important; without those denial of service attacks become trivial to implement, which in a Lambda environment could end up costing the victim tens of cents per month.