Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time)

In the last blog, we discussed about how traditional defense solutions are not sufficient for protection of lambda functions and how 'protectlambda', a small utility from the 'lambdaScanner' toolkit, can be used to guard incoming as well as outgoing stream through a set of predefined rules. Please refer to http://blog.blueinfy.com/2018/09/runtime-lambda-protection-self-defense.html (here) for more details. 

In this blog post, we are going to build over the 'protectLambda' function to set up a strong real time monitoring, analytics and alerts capability using Amazon CloudWatch. Amazon CloudWatch, a component of AWS that collects monitoring and operational data in form of logs, metrics, graphs and events, provides a unified view for AWS resources and applications running on AWS infrastructure. Moreover, CloudWatch allows to customize and set filters, triggers, alarms to troubleshoot issues and take automated actions. As shown in the below figure, all these functionalities can be leveraged to set up real time analysis of the lambda functions.


As discussed in the last blog post, we can introduce the 'protectLambda' utility for lambda functions, define a set of rules for the event stream and the outgoing stream and then monitor and track the logs for rule violations using CloudWatch. A customized dashboard can be created to monitor logs based on filters, correlate logs and metrics, view data in graphs and set alerts with CloudWatch alarms. This dashboard can then be leveraged for real time monitoring of the attacks on lambda functions and for checking how these attacks are blocked by the 'protectLambda' utility. Alarms and notifications can be configured to inform the team of attack on lambda functions and violations of pre-defined rules.

Below is an example for creating a dashboard by defining basic filters, graphs and alarms: -

1.    Setting up Metric Filters on lambda function logs

We can define filters, using the Logs Metric Filter, for specific log groups. In this case, we are creating a filter for the records logged from the 'protectLambda' function (the function added to protect the incoming and outgoing stream of the lambda functions through a set of predefined rules). The 'protectLambda' function blocks the event stream and outgoing response according to a defined regex and logs a "Violation" entry as and when it blocks something. Thus, we define a filter for the word "Violation" which will keep tracking for the word "violation" in this group of logs. This metrics can later be plotted on a graph.

 

2.    Build Dashboard and Graphs

The metrics from the above filter can be plotted on a graph as shown below. The graph show the number of instances of violations defended by the 'protectLambda' utility.



3.    Setup Alarms, Alerts and Notification

We can create attack notification alarms for the created metric filters as shown below. In this case, we have created an alarm for the filter 'protectLambda' which looks for the word "violation" in the selected log group.


We can define a rule where an alarm notification will be triggered as soon as we get one instance of attack in a specific window of 5 minutes.

 

Thus, we can see that as soon as a lambda function is attacked and 'protectLambda' defends an attack on the function we get a notification on our dashboard.

 

We can also configure to receive these notifications in email or on the CloudWatch mobile application as shown below.

(email notification and alert)


(mobile app)
 
Looking at the inbuilt capabilities of CloudWatch, there is no need to set up and create a dashboard on a different domain. The provided functionalities can be easily leveraged to monitor the lambda functions, analyse logs in real time and in turn perform the required actions depending on the analysis.

Runtime Lambda Protection – "Self Defense" from Inside

Brief:

Serverless functions, like AWS Lambda, are transient, temporary and do not have traditional defenses like WAF/IPS/IDS/RASP. Moreover, these functions are not always invoked through API gateway over HTTP(S), where one can provide defenses like WAF to filter for malicious payloads. Lambda functions consume “events” coming from various sources like S3, DynamoDB, SQS etc. (as shown in the below figure). Hence, there are no frontline defenses for these functions. These functions must be protected from "inside" rather than having defenses from outside. They should have self-defense abilities embedded in the function itself.  In this blog, we are going to cover this type of self-defending techniques for Lambda functions.

Problem & Challenges:

Lambda functions have two important aspects – "event consumption" and "response to specific events". It is imperative to build strong event processing and filtering capabilities before the event stream (payload) hits the actual function. Hence, if any malicious payload is being injected to the stream then it gets isolated and disconnected from actual lambda function. This is the first and foremost defense one needs to provide to the lambda function, from inside without trusting any outside defenses.

Secondly, in cases where an attack is successful and the functions are compromised, it may lead to sensitive data disclosure through "outgoing stream". These outgoing responses and streams must be protected and blocked from "inside". A "post exploitation" scenario would include permission controls, invoking other channels like pushing data out on a network, spinning another process to advantage etc., providing controls for “post exploitation” scenario would be not fruitful and can be bypassed in some cases. One should be focused on securing the event stream rather than waiting for the exploit to hit the function. We have seen many defensive controls come into play during exploitation phase and they can be bypassed in one or the other way. Hence, it is always advisable to protect the function from inside, for vulnerabilities, rather than taking care of exploit scenarios.

 

Solution:

One can achieve protection for the event stream as well as outgoing stream information leakage by wrapping lambda function without changing the actual code. It is very simple - one can use a regex to build effective rules to guard against known vulnerabilities. The same wrapping code can be used to enhance and build many other validations like data type, length, characters etc. Also, as an extra defense one can block all possible patterns of malicious payloads which are well known for years.

Introducing 'protectLambda':

As part of the 'lambdaScanner' toolkit (for python), there is a small utility called 'protectLambda'. It helps in building a solid defense against vulnerabilities, by leveraging regex, in a simple and effective way. The working of the 'protectLambda' function is shown in the below figure. 'protectLambda' sits before the actual lambda function and keeps a watch on both incoming and outgoing streams. One can have a predefined set of rules in place and based on those rules, any attack coming to the lambda function would get blocked.


 


Let’s look at its implementation. One can leverage this utility or build a similar component for lambda functions. This same idea can be applied to other lambda technology stacks as well.

Vulnerable Lambda Function and Command Injection:

Here is an invoice processing system, which is taking event from SQS in the below format (let’s store it in the "event.txt" file). This SQS message is coming through the end user through various different applications as described in previous blogs.

{
  "Records": [
    {
      "body": "invoice-98790",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We can invoke this function by using the 'scanLambda' component from the 'lambdaScanner' toolkit or by any other means.

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 53fd2b95-b331-11e8-866c-07f43ff67f8a
    (-) Response ==>
    (-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"
    (-) Log ==>START RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a Version: $LATEST
END RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a
REPORT RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a    Duration: 74.64 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB    


Following line is the response from the function: -

(-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"

We get "Debug" info, which is just for education/dummy purpose. The content from debug information (in this case, we got ASCII) suggests that some kind of command like "file" which is used to fetch type of the file-content is run.

Now, let’s inject the below message which is vulnerable to command injection. (It was covered in detail in the past blogs)

{
  "Records": [
    {
      "body": "invoice-98790|ls -la",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


In the above message, we injected the following: -

"body": "invoice-98790|ls -la",


We have piped the invoice filename with "ls –la" command. Let’s invoke the function (using the 'scanLambda' component): -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 6e578940-b331-11e8-bf9a-836686cf50ac
    (-) Response ==>
    (-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x  4 root root  118 Sep  8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep  8 05:23 ..\n-rwxr-xr-x  1 root root   14 Sep  8 06:33 in_protect.cfg\n-rw-r--r--  1 root root  354 Sep  8 06:26 lambda_function.py\n-rw-rw-r--  1 root root    9 Sep  8 06:25 out_protect.cfg\ndrwxr-xr-x  2 root root   33 Aug 21 16:37 protectLambda\ndrwxrwxr-x  2 root root   36 Sep  8 05:55 tmp\n]"
    (-) Log ==>START RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac Version: $LATEST
END RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac
REPORT RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac    Duration: 68.29 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB


Hence, the command is being executed and we can see the following output: -

(-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x  4 root root  118 Sep  8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep  8 05:23 ..\n-rwxr-xr-x  1 root root   14 Sep  8 06:33 in_protect.cfg\n-rw-r--r--  1 root root  354 Sep  8 06:26 lambda_function.py\n-rw-rw-r--  1 root root    9 Sep  8 06:25 out_protect.cfg\ndrwxr-xr-x  2 root root   33 Aug 21 16:37 protectLambda\ndrwxrwxr-x  2 root root   36 Sep  8 05:55 tmp\n]"


Deploying 'protectLambda' against Command Injection:

Here, we have a potential command injection. Now, let’s secure it with 'protectLambda' utility. To achieve it, include 'protectLambda' as part of your project as shown below or as part of your package zip for AWS. There is no need to change the code as such.

 

Once it has been added, you can go ahead and add two lines to the lambda function as shown below: -

--------
from protectLambda.protect import protect


@protect
def lambda_handler(event, context):



The added lines can be seen in the lambda function on AWS: -




Now, let’s add a rule for all incoming event streams. For example: - we can have this simple rule - .*[|;<>&%*#].* - which blocks a list of malicious characters listed between square brackets[ ]. For a successful command injection, one needs to leverage one of these characters. One can add any rule that fits the criteria to the "in_protect.txt" file.




Once loaded, it starts blocking malicious characters from the event stream. One can enter rules for specific to parameters as well. For example: - the below rule will search specifically in the "body" parameter which is sent in the message. In our case also, we are more interested in this parameter.

".?body.*:.*".*[|;&<>*].?"

Once the rule is entered, we try to invoke the function with the same malicious payload as before: -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 1affd2d8-b332-11e8-861c-89d1677e2525
    (-) Response ==>
    (-) "Security Violation..."
    (-) Log ==>START RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525 Version: $LATEST
Input Rule violation for  .*[|;<>&%*#].*
END RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525
REPORT RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525    Duration: 0.98 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 21 MB    


Bingo! As we can see, the event is blocked and could not touch the actual lambda function; it is being rejected before it gets into the function. We end up getting the following message: -

(-) "Security Violation..."

Also, if we look in the CloudWatch logs, we can see the following line: -

Input Rule violation for  .*[|;<>&%*#].*


It is important to have proper logging mechanisms in place. One can go ahead and setup a trigger with rules such that this message is sent, to the configured email address/phone number, immediately through email or SMS. One can quickly take action against IP address or setup more monitoring on the function if needed.

This is the way we can achieve "defense from inside". We can apply multiple rules as well and protect the function from various possible attack vectors.

Deploying 'protectLambda' for Outgoing Stream (Information Leakage):

As we leveraged the 'protectLambda' function to protect against incoming payloads, it can be used to protect the outgoing stream using the "out_protect.txt" file as shown below. It is equally important to protect the information going out through error messages, debug information or post exploitation harvesting.

 

Here, we just added a rule to block if "debug" information is sent in the outgoing stream. After adding the rule, we invoke the function as we did earlier: -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 8b9175cd-b333-11e8-ab42-e9c542af285d
    (-) Response ==>
    (-) "Security Violation..."
    (-) Log ==>START RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Version: $LATEST
Output Rule violation for  .*debug.*
END RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d
REPORT RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d    Duration: 45.06 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 21 MB    


Bingo! We can see the following message: -

(-) "Security Violation..."


If we go and check the CloudWatch logs then we can find the following line as well: -

Output Rule violation for  .*debug.*


Hence, this way the outgoing stream is also protected by the utility.

Conclusion:

The nature of lambda functions is such that various sets of payloads can be injected by manipulating the event streams. As it is not necessary that the lambda functions would always be triggered by an API gateway though HTTP(s), it would not be possible to deploy traditional defense solutions for lambda functions. Thus, the use case scenario of the lambda functions makes it imperative to use strong defense techniques from "inside" for incoming as well as outgoing streams. As the first line of defense, 'protectLambda' can guard both incoming as well as outgoing streams through a set of predefined rules. This is also a practical solution from its model perspective, as having a protection where every request makes a call over a network and gets validated at a central place would break the whole idea or model of serverless functions. Thus, defense techniques integrated with the lambda function itself along with proper logging mechanisms seems to be a correct approach. This way, we can achieve lambda function's runtime protection by providing self-defense from inside, independent from any other system/application/program/services.

Article by Amish Shah and Shreeraj Shah

Leveraging tunnelLambda with pentesting tools for serverless function testing

One of the challenges for lambda function testing is to incorporating and integrating traditional effective tools like netcat, burp proxy or sqlmap. These tools runs on HTTP(S) pipes while lambda function's events can be coming from any where without HTTP pipe. Hence, one can leverage tool like tunnelLambda while performing pentesting. It is part of our scanLambda toolkit (Here - http://blog.blueinfy.com/p/blog-page.html).

It is very simple as shown below. It establishes tunnel between your target lambda function and your specific tool on localhost/port.



'tunnelLambda' helps in establishing a tunnel from your shell to a targeted lambda function. It helps in sending HTTP traffic to the selected port, which will automatically tunnel to the test function. Hence, now we can use some standard HTTP tools like netcat, sqlmap, Burp or ZAP to test the lambda function.


Once it is set, the script will listen on the target port for both GET and POST requests as shown below: -



When you make a GET request it will serve a simple HTML page which can be used to interact with the lambda function as shown in the below figure. We can just open the page in a browser, put the event stream and click on "Send" button. It will show the output once it is invoked.



Also, now we can use tools like netcat, burp, sqlmap or any other tool to make a POST request directly. Here is our HTTP request,

$ cat sqltest.txt
POST / HTTP/1.1
Host: localhost:8888
Content-Length: 17

{"name":"john"}


We can push it to netcat,

$ cat sqltest.txt | nc localhost 8888
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.6.4
Date: Fri, 07 Sep 2018 03:04:41 GMT
Content-type: application/json

{"id": "1239873"}


We can start running sqlmap as well.

$ python sqlmap.py -r ../sqltest.txt
        ___
       __H__
 ___ ___[.]_____ ___ ___  {1.2.9.6#dev}
|_ -| . [)]     | .'| . |
|___|_  [']_|_|_|__,|  _|
      |_|V          |_|   http://sqlmap.org

[!] legal disclaimer: Usage of sqlmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program

[*] starting at 08:37:00

[08:37:00] [INFO] parsing HTTP request from '../sqltest.txt'
JSON data found in POST data. Do you want to process it? [Y/n/q] Y
[08:37:05] [INFO] testing connection to the target URL
[08:37:07] [INFO] testing if the target URL content is stable
[08:37:09] [INFO] target URL content is stable
[08:37:09] [INFO] testing if (custom) POST parameter 'JSON name' is dynamic




We need to configure the details in Burp repeater as shown below:

 

Once it is set, we can make the call as shown below:

 

Next, we can simply send the request to intruder and run attacks as shown below:



Hence, this allows us to quickly leverage all popular tools against lambda functions.

Article by Hemil Shah


lambdaScanner - Scan and Secure serverless lambda functions

'lambdaScanner' is a toolkit which has a combination of scripts for performing penetration testing of lambda functions. The scripts available in the toolkit help assessing the lambda functions from a security standpoint. It helps the tester to discover vulnerabilities in deployment as well as code. It aids in checking vulnerabilities like improper permissions, SQL injections, command executions etc. to name a few. This is not an automated scanner, but a toolkit that helps pen-testers to perform the testing of functions, so it needs to be used wisely by crafting customized requests and payloads. The lambda functions are invoked through various events encompassing AWS like S3, DynamoDB, SQS etc. so the scripts in the toolkit are very helpful in evaluating functions as well as directly testing with various sets of payloads. All these scripts are written in python by using boto3 APIs. The toolkit also has a package called 'lambdaProtect' which can be integrated with an existing lambda function to guard both incoming event stream as well as outgoing response.

This toolkit is "in progress/prototype" and would be enhanced with time by an addition of various functionalities.

Here is a diagram, which describes 'lambdaScanner': -

 

For more detail please visit - here

URL - http://blog.blueinfy.com/p/blog-page.html