Brief:
Serverless functions, like AWS Lambda, are transient, temporary and do not have traditional defenses like WAF/IPS/IDS/RASP. Moreover, these functions are not always invoked through API gateway over HTTP(S), where one can provide defenses like WAF to filter for malicious payloads. Lambda functions consume “events” coming from various sources like S3, DynamoDB, SQS etc. (as shown in the below figure). Hence, there are no frontline defenses for these functions. These functions must be protected from "inside" rather than having defenses from outside. They should have self-defense abilities embedded in the function itself. In this blog, we are going to cover this type of self-defending techniques for Lambda functions.Problem & Challenges:
Lambda functions have two important aspects – "event consumption" and "response to specific events". It is imperative to build strong event processing and filtering capabilities before the event stream (payload) hits the actual function. Hence, if any malicious payload is being injected to the stream then it gets isolated and disconnected from actual lambda function. This is the first and foremost defense one needs to provide to the lambda function, from inside without trusting any outside defenses.
Secondly, in cases where an attack is successful and the functions are compromised, it may lead to sensitive data disclosure through "outgoing stream". These outgoing responses and streams must be protected and blocked from "inside". A "post exploitation" scenario would include permission controls, invoking other channels like pushing data out on a network, spinning another process to advantage etc., providing controls for “post exploitation” scenario would be not fruitful and can be bypassed in some cases. One should be focused on securing the event stream rather than waiting for the exploit to hit the function. We have seen many defensive controls come into play during exploitation phase and they can be bypassed in one or the other way. Hence, it is always advisable to protect the function from inside, for vulnerabilities, rather than taking care of exploit scenarios.
Secondly, in cases where an attack is successful and the functions are compromised, it may lead to sensitive data disclosure through "outgoing stream". These outgoing responses and streams must be protected and blocked from "inside". A "post exploitation" scenario would include permission controls, invoking other channels like pushing data out on a network, spinning another process to advantage etc., providing controls for “post exploitation” scenario would be not fruitful and can be bypassed in some cases. One should be focused on securing the event stream rather than waiting for the exploit to hit the function. We have seen many defensive controls come into play during exploitation phase and they can be bypassed in one or the other way. Hence, it is always advisable to protect the function from inside, for vulnerabilities, rather than taking care of exploit scenarios.
Solution:
One can achieve protection for the event stream as well as outgoing stream information leakage by wrapping lambda function without changing the actual code. It is very simple - one can use a regex to build effective rules to guard against known vulnerabilities. The same wrapping code can be used to enhance and build many other validations like data type, length, characters etc. Also, as an extra defense one can block all possible patterns of malicious payloads which are well known for years.Introducing 'protectLambda':
As part of the 'lambdaScanner' toolkit (for python), there is a small utility called 'protectLambda'. It helps in building a solid defense against vulnerabilities, by leveraging regex, in a simple and effective way. The working of the 'protectLambda' function is shown in the below figure. 'protectLambda' sits before the actual lambda function and keeps a watch on both incoming and outgoing streams. One can have a predefined set of rules in place and based on those rules, any attack coming to the lambda function would get blocked.
Let’s look at its implementation. One can leverage this utility or build a similar component for lambda functions. This same idea can be applied to other lambda technology stacks as well.
Vulnerable Lambda Function and Command Injection:
Here is an invoice processing system, which is taking event from SQS in the below format (let’s store it in the "event.txt" file). This SQS message is coming through the end user through various different applications as described in previous blogs.{
"Records": [
{
"body": "invoice-98790",
"receiptHandle": "MessageReceiptHandle",
"md5OfBody": "7b270e59b47ff90a553787216d55d91d",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
"eventSource": "aws:sqs",
"awsRegion": "us-east-2",
"messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
"attributes": {
"ApproximateFirstReceiveTimestamp": "1523232000001",
"SenderId": "123456789012",
"ApproximateReceiveCount": "1",
"SentTimestamp": "1523232000000"
},
"messageAttributes": {}
}
]
}
We can invoke this function by using the 'scanLambda' component from the 'lambdaScanner' toolkit or by any other means.
bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt
==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================
(+)Configuring Invoking ...
(-) Loading event from file ...
(-) Request Id ==> 53fd2b95-b331-11e8-866c-07f43ff67f8a
(-) Response ==>
(-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"
(-) Log ==>START RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a Version: $LATEST
END RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a
REPORT RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a Duration: 74.64 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
Following line is the response from the function: -
(-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"
We get "Debug" info, which is just for education/dummy purpose. The content from debug information (in this case, we got ASCII) suggests that some kind of command like "file" which is used to fetch type of the file-content is run.
Now, let’s inject the below message which is vulnerable to command injection. (It was covered in detail in the past blogs)
{
"Records": [
{
"body": "invoice-98790|ls -la",
"receiptHandle": "MessageReceiptHandle",
"md5OfBody": "7b270e59b47ff90a553787216d55d91d",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
"eventSource": "aws:sqs",
"awsRegion": "us-east-2",
"messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
"attributes": {
"ApproximateFirstReceiveTimestamp": "1523232000001",
"SenderId": "123456789012",
"ApproximateReceiveCount": "1",
"SentTimestamp": "1523232000000"
},
"messageAttributes": {}
}
]
}
In the above message, we injected the following: -
"body": "invoice-98790|ls -la",
We have piped the invoice filename with "ls –la" command. Let’s invoke the function (using the 'scanLambda' component): -
bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt
==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================
(+)Configuring Invoking ...
(-) Loading event from file ...
(-) Request Id ==> 6e578940-b331-11e8-bf9a-836686cf50ac
(-) Response ==>
(-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x 4 root root 118 Sep 8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep 8 05:23 ..\n-rwxr-xr-x 1 root root 14 Sep 8 06:33 in_protect.cfg\n-rw-r--r-- 1 root root 354 Sep 8 06:26 lambda_function.py\n-rw-rw-r-- 1 root root 9 Sep 8 06:25 out_protect.cfg\ndrwxr-xr-x 2 root root 33 Aug 21 16:37 protectLambda\ndrwxrwxr-x 2 root root 36 Sep 8 05:55 tmp\n]"
(-) Log ==>START RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac Version: $LATEST
END RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac
REPORT RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac Duration: 68.29 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
Hence, the command is being executed and we can see the following output: -
(-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x 4 root root 118 Sep 8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep 8 05:23 ..\n-rwxr-xr-x 1 root root 14 Sep 8 06:33 in_protect.cfg\n-rw-r--r-- 1 root root 354 Sep 8 06:26 lambda_function.py\n-rw-rw-r-- 1 root root 9 Sep 8 06:25 out_protect.cfg\ndrwxr-xr-x 2 root root 33 Aug 21 16:37 protectLambda\ndrwxrwxr-x 2 root root 36 Sep 8 05:55 tmp\n]"
Deploying 'protectLambda' against Command Injection:
Here, we have a potential command injection. Now, let’s secure it with 'protectLambda' utility. To achieve it, include 'protectLambda' as part of your project as shown below or as part of your package zip for AWS. There is no need to change the code as such.
Once it has been added, you can go ahead and add two lines to the lambda function as shown below: -
--------
from protectLambda.protect import protect
@protect
def lambda_handler(event, context):
…
…
The added lines can be seen in the lambda function on AWS: -
Now, let’s add a rule for all incoming event streams. For example: - we can have this simple rule - .*[|;<>&%*#].* - which blocks a list of malicious characters listed between square brackets[ ]. For a successful command injection, one needs to leverage one of these characters. One can add any rule that fits the criteria to the "in_protect.txt" file.
Once loaded, it starts blocking malicious characters from the event stream. One can enter rules for specific to parameters as well. For example: - the below rule will search specifically in the "body" parameter which is sent in the message. In our case also, we are more interested in this parameter.
".?body.*:.*".*[|;&<>*].?"
Once the rule is entered, we try to invoke the function with the same malicious payload as before: -
bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt
==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================
(+)Configuring Invoking ...
(-) Loading event from file ...
(-) Request Id ==> 1affd2d8-b332-11e8-861c-89d1677e2525
(-) Response ==>
(-) "Security Violation..."
(-) Log ==>START RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525 Version: $LATEST
Input Rule violation for .*[|;<>&%*#].*
END RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525
REPORT RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525 Duration: 0.98 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 21 MB
Bingo! As we can see, the event is blocked and could not touch the actual lambda function; it is being rejected before it gets into the function. We end up getting the following message: -
(-) "Security Violation..."
Also, if we look in the CloudWatch logs, we can see the following line: -
Input Rule violation for .*[|;<>&%*#].*
It is important to have proper logging mechanisms in place. One can go ahead and setup a trigger with rules such that this message is sent, to the configured email address/phone number, immediately through email or SMS. One can quickly take action against IP address or setup more monitoring on the function if needed.
This is the way we can achieve "defense from inside". We can apply multiple rules as well and protect the function from various possible attack vectors.
Deploying 'protectLambda' for Outgoing Stream (Information Leakage):
As we leveraged the 'protectLambda' function to protect against incoming payloads, it can be used to protect the outgoing stream using the "out_protect.txt" file as shown below. It is equally important to protect the information going out through error messages, debug information or post exploitation harvesting.
Here, we just added a rule to block if "debug" information is sent in the outgoing stream. After adding the rule, we invoke the function as we did earlier: -
bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt
==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================
(+)Configuring Invoking ...
(-) Loading event from file ...
(-) Request Id ==> 8b9175cd-b333-11e8-ab42-e9c542af285d
(-) Response ==>
(-) "Security Violation..."
(-) Log ==>START RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Version: $LATEST
Output Rule violation for .*debug.*
END RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d
REPORT RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Duration: 45.06 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 21 MB
Bingo! We can see the following message: -
(-) "Security Violation..."
If we go and check the CloudWatch logs then we can find the following line as well: -
Output Rule violation for .*debug.*
Hence, this way the outgoing stream is also protected by the utility.
Here, we just added a rule to block if "debug" information is sent in the outgoing stream. After adding the rule, we invoke the function as we did earlier: -
bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt
==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================
(+)Configuring Invoking ...
(-) Loading event from file ...
(-) Request Id ==> 8b9175cd-b333-11e8-ab42-e9c542af285d
(-) Response ==>
(-) "Security Violation..."
(-) Log ==>START RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Version: $LATEST
Output Rule violation for .*debug.*
END RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d
REPORT RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Duration: 45.06 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 21 MB
Bingo! We can see the following message: -
(-) "Security Violation..."
If we go and check the CloudWatch logs then we can find the following line as well: -
Output Rule violation for .*debug.*
Hence, this way the outgoing stream is also protected by the utility.
Conclusion:
The nature of lambda functions is such that various sets of payloads can be injected by manipulating the event streams. As it is not necessary that the lambda functions would always be triggered by an API gateway though HTTP(s), it would not be possible to deploy traditional defense solutions for lambda functions. Thus, the use case scenario of the lambda functions makes it imperative to use strong defense techniques from "inside" for incoming as well as outgoing streams. As the first line of defense, 'protectLambda' can guard both incoming as well as outgoing streams through a set of predefined rules. This is also a practical solution from its model perspective, as having a protection where every request makes a call over a network and gets validated at a central place would break the whole idea or model of serverless functions. Thus, defense techniques integrated with the lambda function itself along with proper logging mechanisms seems to be a correct approach. This way, we can achieve lambda function's runtime protection by providing self-defense from inside, independent from any other system/application/program/services.Article by Amish Shah and Shreeraj Shah