In the last few months, I’ve been busy researching serverless security. My sources for this research were mostly blogs, tutorials and conference talks - basically any kind of public material I could get my hands on. One thing that immediately caught my attention, was that the majority of materials dealt with generic app security topics, and only a small fraction of the available data actually touched on serverless-specific concerns or challenges.
Could it be that securing serverless applications is just a variation on a known theme that was already thoroughly charted? Or is serverless security uncharted territory, and perhaps people try to project what they know from traditional application security onto serverless, just because that’s what they know?
We could state the obvious, and say that serverless applications are after all - applications. And as such, application security applies to them just as it applies to any kind of software. Sounds logical enough, right? That’s partly true. Following secure software design can indeed help with designing and building secure applications, of any kind and form. However, each type of application also has its own world of idiosyncrasies. When you secure an IoT device, a web application, or a mainframe application, you use common guidelines and approaches to software security, however, each of these systems also require that you apply domain-specific security controls and follow relevant best practices. The same applies to serverless applications.
Last week, we published the first “Serverless Security Top 10 Most Common Weaknesses” guide. The guide covers what we discovered to be the most common and severe pitfalls related to serverless application security. Since the guide was meant to provide clear and concise explanations, we couldn’t dig deep into some of the topics as we would’ve liked, which is the reason that today we are starting this “Securing Serverless” blog series.
Episode 0x01: There is no “W” in Serverless
Take a moment to get some perspective -
When you develop for serverless architectures, you develop dozens or even hundreds of serverless functions. Each function is programmed to trigger by a unique set of events, and there are dozens of different event types supported by most serverless platforms. These serverless functions, trigger millions of executions per day (I read that Expedia performed 2.3B executions per month, in late 2016), and this whole thing executes on ephemeral compute power, which runs on a public distributed cloud, that you don’t own.
How do you secure such applications? Can you honestly say that it all boils down to applying good old application security best-practices? In this episode, we will take a look at one trivial application security best practice - Use a Web Application Firewall.
First thing’s first, as you probably noticed, there is no “W” in Serverless (well, not always). The “W” in WAF happens to stand for Web, so if your serverless application doesn’t expose a web interface…. I think you see where I’m going with this…
Point #1: WAFs Are Not Ephemeral
Serverless functions are ephemeral. Moreover, they execute in globally distributed data centers that you have no access to, on servers that you don’t own, in networks that you don’t control. One moment they are running, and in the next, they are gone. How do you deploy a WAF in such a challenging environment? The short answer is - You can’t. At least not the (legacy) appliance kind. After all - when you don’t own the infrastructure, you can’t install hardware.
Your next deployment option is to use cloud compute instances in order to run virtual WAF appliances. Sort of beats the point, doesn’t it? You went serverless so you won’t have to deal with servers. You went serverless so you won’t pay for idle time, and you went serverless so that your application will be able to instantly and elastically grow as needed. A virtual appliance just doesn’t meet these requirements.
One more thing - you’ll also need to place your TLS certificate on those virtual appliances if you want your WAF to inspect encrypted traffic (HTTPS). Does that make you feel safe? I doubt it.
Next on our list - Cloud WAFs. After all, that’s as serverless as WAFs can get - at least from the consumer’s point of view - no hardware to manage.
I’ll come out as a moron if I say anything bad about cloud WAFs, given I just spent 5+ years developing them and preaching about how great they are. They truly are. However, the two main drawbacks when attempting to protect serverless applications using a cloud WAF would be:
- Serverless applications are in many cases not web applications. They might expose some web interfaces to the world, but that doesn’t make them a web application
- They perform their input inspection in the cloud - that’s fine and cool if you have to inspect inbound web traffic, but less appealing if you have to inspect data from other sources, in which case, you will have to route the data for inspection out of your serverless environment, inspect it in the cloud, and route it back to the application. This will obviously inflict a severe performance penalty, and might pose challenges with regards to some data security regulations
Point #2: Serverless Events Are Like a Box of Chocolates
We already concluded in point #1, that not all serverless applications expose a web interface. Even those that do, will oftentimes contain serverless functions that will get invoked by different types of events. For example:
- NoSQL database events
- Stream processing events
- Code repository change events
- Message queue events
- Cloud storage events
- HTTP API calls
- Email events
- SMS messages
- IoT telemetry events (e.g. MQTT or WebSockets)
Great - but you knew that when you read previous materials on serverless applications, right? Have you stopped to think what does this even mean? for starters - your application might be exposing entry points that your WAF is not monitoring. Duh! Why would a WAF inspect SMS messages? After all, there is no “W” in SMS, right?
Ok then, so we have a bunch of new entry points that WAFs were never designed to inspect. So What?!
Let’s say, that for some miraculous reason you managed to route all inputs through a WAF - then what? What kind of input is your serverless function actually consuming through these entry points? How are messages formatted? Can a WAF inspect such events? The short answer is No. WAFs were designed to analyze HTTP traffic. As such, they are terrific in parsing HTTP transactions - extracting and inspecting headers, query parameters, POST body parameters, URLs, and even multipart/form-data messages.
To make the point more clear, here’s a sample AWS SNS message (taken from the AWS documentation page):
A few things pop to mind when looking at the message:
- It’s JSON. Some WAFs handle JSON, and some don’t (I hope yours does btw. It’s time to check with your vendor).
- Which fields should your WAF inspect? Which can be controlled by an attacker? Do you inspect all of them and pay the price in performance? Or do you stick to only a few fields, and risk false negatives?
- Some fields are encoded - could be base64, could be binary, or it could be whatever your developer chose to brew at home. I haven’t seen a WAF that handles this encoding/decoding uncertainty with grace.
WAFs are terrific for providing protection against web application layer attacks - for your web applications. However, as the saying goes - “You don’t bring a knife to a gunfight” - if you want to protect serverless functions, you will have to look elsewhere.
Point #3: The Innocent Bystander
WAFs are deployed “in-line”, as a reverse proxy. This positions them between the end user (or the attacker), and the web application. Such positioning is what gives WAFs the ability to inspect HTTP traffic in real-time and to provide protection against application layer attacks.
In essence, WAFs are like a sieve or some kind of filter. They go through all HTTP traffic, and remove the unwanted or potentially hazardous requests. 20 years after being introduced, it’s safe to say that most WAFs can now detect the majority of injection-based attack payloads.
WAFs use a wide range of techniques to spot malicious payloads. These techniques may use signatures such as regular expressions, or they can use some sort of an anomaly scoring mechanism, some use automated learning processes, in an attempt to build a policy of expected input formats, and some use lexical analysis of input. But the one thing all WAFs have in common, is that they rely on input inspection.
Is that bad? Not necessarily. After all, security is all about adding more and more layers of protection, however, it is certainly not enough.
There are several types of attacks, where input inspection that is done on a single medium or entry-point, will not be enough in order to detect and block the attack. When the target application logic spans across multiple entry points or input types and requires several steps to accomplish - input inspection will not help. But isn’t that exactly the case in many serverless applications?
Serverless applications use several functions which act as “nano-services”. Each function interacts with other functions or cloud services via different event triggers. Inspecting a single entry point, or a single “step” will most likely fail to detect any attack that is slightly more sophisticated.
In other words - while WAFs provide great input filtering capabilities, they lack the ability to look beyond transactional input. They lack the ability to monitor application behavior, and detect that something bad is going on, as a result of a sequence of actions or interactions with other entities.
WAFs are like those bouncers standing outside a club or a bar. They are quite capable of detecting unwanted folks, which look like trouble-makers, however, once a person is inside the club - he or she can pretty much do whatever they want.
To make a long story short - given the unique composition, distributed nature and logic of many serverless applications, even if WAFs (or any kind of input filter) were able to analyze input going into the application, they would still have limited visibility to attacks - visibility only from the outside.
But What Does This Have To Do With Securing Serverless?
Well, that’s sort of my point exactly. WAFs are a terrific security solution for web applications. However, If you want to secure a serverless application, you will have to look for a suitable solution. The protection you are seeking needs to be relevant for serverless architectures, and should have the following characteristics:
- Event-data inspection: Ability to detect application layer attacks by inspecting function event trigger data, in all types and formats. Inspection of HTTP traffic alone is not enough and only accounts for one of dozens of application entry points.
- Local protection: Provide protection as close as possible to the serverless function without compromising performance or violating data security regulations. After all, if your serverless function runs for 15ms, you can’t afford sending data out to the cloud for inspection - the round trip time would inflict an unbearable performance penalty with disproportional magnitude
- Serverless real-time behavioral protection: serverless applications tend to contain multiple functions, each serving as a nano-service, consuming different types of inputs, and orchestrated by event triggers. If you want to be able to detect malicious attacks in such environments, you can’t rely solely on input filtering. Your solution must also be able to perform deep inspection of application behavior.
- Ephemerality: the same reasons that got you to use serverless architectures, will apply to the security choices you need to make. Using servers and maintaining their infrastructure in order to apply security doesn’t make sense in this case. Whatever solution you choose should be able to scale together with your serverless application, and it shouldn’t require server provisioning and maintenance.
Stay tuned for the next episode in this series….and in the meantime, take a look at https://www.puresec.io for more information on how to secure serverless applications.