In the last two blog posts, we covered an approach of pentesting lambda functions, using DAST and SAST methodologies, by footprinting, enumerating, scanning and tracing lambda functions to discover and verify security vulnerabilities. In this blog post, we will leverage instrumentation/IAST (Interactive Application Security Testing) approach to analyse and test lambda functions. This approach involves analysing the application behaviour at run time and then using DAST for inducing attacks. Also, a combination of DAST, SAST and IAST can be used to get a comprehensive view of the application, even at run time, and then discover and confirm vulnerabilities.
Below is a figure outlining the architecture of an application using lambda functions with sensors added to the code before performing a pentest of the application/lambda functions: -
“AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. With X-Ray, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. X-Ray provides an end-to-end view of requests as they travel through your application, and shows a map of your application’s underlying components. You can use X-Ray to analyze both applications in development and in production, from simple three-tier applications to complex microservices applications consisting of thousands of services.“
This helps in getting a map, but from a pentesting perspective we are more interested in putting hooks and sensors into the lambda code. We can do this using X-Ray SDK (https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html).
We can also patch all inherent components, which are supported by X-Ray. As shown in the above figure, we can create a recorder and use a decorator provided by the library with adding a line, before the lambda function, starting with “@”. Once this line is added we can start inserting sensors for fetching runtime information. Let’s add some annotation and metadata: -
In the above code, we are defining a sub-segment with an annotation and then using the defined annotation as our sensor. Hence, we can start recording the runtime values before or after interesting the API calls. In the above case, we are recording the "command" variable before it goes to its sub-process. All these interactions will get recorded at runtime by X-Ray services in the logs.
Now, after invoking the function, when we go and see the X-Ray logs (as shown in the below figure) we see the execution time for each function is logged and we see that the function is not calling any other AWS component like S3, DynamoDB, SQS etc.: -
In the below figure, we see the "lambda" sub-segment values in the logs according to the defined annotations and sub-segments: -
We can also see full metadata with the event we pushed in the record as shown below: -
In this way, we can inject a payload and see our injected payload go through to initiate a successful command injection after the function is invoked (a detailed process of this was explained in the last blog post).
The code provided in the package can be modified and added to the lambda function as shown below: -
We get its @record and it will print the log to CloudWatch using a simple "print" function.
We can see the following log in our CloudWatch console: -
We get a full dump of all the variables with line numbers. Also, we can see how values are changing in runtime line-by-line execution. Hence, it can have much more valuable information for the analysis needed to detect a vulnerability.
Apart for third-party tools that are available we can also define our own decorator function, to record events as well as output, in python, which we can leverage for instrumentation as shown below: -
We define a function "instrument", which we can add before our actual lambda function in our python code as shown below: -
At the point of execution, we can see these entries in our CloudWatch logs: -
In this way, we can add multiple "print" functions in between the code and use something like "locals()" to dump runtime variables as well.
Article by Amish Shah & Shreeraj Shah
Below is a figure outlining the architecture of an application using lambda functions with sensors added to the code before performing a pentest of the application/lambda functions: -
Leveraging AWS X-Ray SDK/APIs:
X-Ray is a service provided by AWS to analyse the integration of lambda functions and perform debugging across lambda functions and other points of interest by request tracing, exception collection and profiling capabilities. It is described as following on AWS (https://aws.amazon.com/xray/): -
“AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. With X-Ray, you can understand how your application and its underlying services are performing to identify and troubleshoot the root cause of performance issues and errors. X-Ray provides an end-to-end view of requests as they travel through your application, and shows a map of your application’s underlying components. You can use X-Ray to analyze both applications in development and in production, from simple three-tier applications to complex microservices applications consisting of thousands of services.“
This helps in getting a map, but from a pentesting perspective we are more interested in putting hooks and sensors into the lambda code. We can do this using X-Ray SDK (https://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html).
X-Ray Sensors using Python:
Let’s try to do it using a python SDK. We can integrate X-Ray into our lambda code base as shown below: -
We can also patch all inherent components, which are supported by X-Ray. As shown in the above figure, we can create a recorder and use a decorator provided by the library with adding a line, before the lambda function, starting with “@”. Once this line is added we can start inserting sensors for fetching runtime information. Let’s add some annotation and metadata: -
In the above code, we are defining a sub-segment with an annotation and then using the defined annotation as our sensor. Hence, we can start recording the runtime values before or after interesting the API calls. In the above case, we are recording the "command" variable before it goes to its sub-process. All these interactions will get recorded at runtime by X-Ray services in the logs.
Now, after invoking the function, when we go and see the X-Ray logs (as shown in the below figure) we see the execution time for each function is logged and we see that the function is not calling any other AWS component like S3, DynamoDB, SQS etc.: -
In the below figure, we see the "lambda" sub-segment values in the logs according to the defined annotations and sub-segments: -
We can also see full metadata with the event we pushed in the record as shown below: -
In this way, we can inject a payload and see our injected payload go through to initiate a successful command injection after the function is invoked (a detailed process of this was explained in the last blog post).
Instrumentation without AWS X-Ray:
Apart from AWS X-Ray, we can also use other available tools for debugging purpose. For example, let’s use the following package (https://github.com/mihneadb/python-execution-trace) to get a runtime variable dump along with the executed code.
The code provided in the package can be modified and added to the lambda function as shown below: -
We get its @record and it will print the log to CloudWatch using a simple "print" function.
We can see the following log in our CloudWatch console: -
We get a full dump of all the variables with line numbers. Also, we can see how values are changing in runtime line-by-line execution. Hence, it can have much more valuable information for the analysis needed to detect a vulnerability.
Apart for third-party tools that are available we can also define our own decorator function, to record events as well as output, in python, which we can leverage for instrumentation as shown below: -
We define a function "instrument", which we can add before our actual lambda function in our python code as shown below: -
At the point of execution, we can see these entries in our CloudWatch logs: -
In this way, we can add multiple "print" functions in between the code and use something like "locals()" to dump runtime variables as well.
360 Degree – DAST, SAST and IAST
We can implement the following strategy to get a 360 degree view of the lambda function.
- We can scan the code to identify some interesting API calls which can directly be mapped to vulnerabilities like command injection, SQL injections etc. and monitor those specific calls. (SAST)
- We can inject sensors by using AWS X-Ray, at interesting places, before or after the call and record the values going to the API. (IAST)
- We can then invoke the lambda function and fuzz the event stream with customized payloads. (DAST)
- All the events will be recorded in the logs as per the injected sensors and we will have the data to be analysed. By analysing this data, as well as the output of the invoked functions we will be able to detect and confirm a vulnerability. Moreover, this information would also aid in crafting exploits for the detected vulnerability.
Conclusion:
A most common use case scenario of lambda functions involves asynchronous functions, where there is no output coming back to the tester. This makes vulnerability confirmation difficult for the tester. Thus, IAST/Instrumentation is a very interesting and more or so an imperative approach to pentesting lambda functions. There are multiple tools and approaches that can be deployed for instrumentation. Moreover, a combination of all the three approaches, DAST, SAST and IAST, would save a lot of effort and would help in detecting a vulnerability with much more precision. It would help avoid unnecessary guesswork and focus on identifying pointers, doing a run time analysis, crafting customized payloads with correct test cases and analysing the logs to detect loopholes in the application from a security standpoint.Article by Amish Shah & Shreeraj Shah