Tuesday, February 12, 2019

Deep Lambda Inspection by age-old LD_PRELOAD Trickery

Lambda function uses AWS Fargate as an underlying platform and it is being leveraged by Firecracker. Firecracker is a Micro-VM, which is used to run lambda functions. As shown in the below figure, it runs in a guest OS and inherits the environment before executing the lambda code. At this time, one can leverage age-old preload trick.

Leveraging LD_PRELOAD:

Lambda function runs under a Linux based environment and it is possible to leverage preloading mechanism as shown in the above figure. If we look at “LD_PRELOAD” usage in the man page –

“A list of additional, user-specified, ELF shared objects to be loaded before all others.  The items of the list can be separated by spaces or colons, and there is no support for escaping either separator.  This can be used to selectively override functions in other shared objects.”

Hence, LD_PRELOAD is an environmental variable that contains paths to shared libraries and objects; which the loader will load before any other shared library including the C runtime library. This is the way preloading would work within lambda function as well.

It allows us to load and override various functions, which are important, and we would like to review these functions from a security perspective. For example:

1.    Underlying OS calls – we can load/override all remote command execution calls, made by lambda function. Hence, when the lambda function makes these calls, we can monitor them to discover vulnerabilities.

2.    Underlying file operations – we can load/override all file operations and monitor them at runtime. We can see if function is making any calls which are vulnerable or the file path it is using. This would aid in identifying path traversal and other file system related vulnerabilities.

3.    Underlying network operations – we can load/override networking functions and observe how lambda function is making calls to external entities. By monitoring third party outgoing calls, vulnerabilities like SSRF or any other network related abuses can be identified.

Let’s look at a simple implementation to observe networking calls. Through the below snippet, we are injecting code to capture socket, connect and send calls in the shared object. We dump the content into “/tmp/lammon.txt” file and can retrieve this content via function or write it to CloudWatch logs.

// Overriding socket call…
int socket(int domain, int type, int protocol)
    int (*new_socket)(int domain, int type, int protocol);
    new_socket = dlsym(RTLD_NEXT, "socket");
    FILE *logfile = fopen("/tmp/lammon.txt", "a+");
    fprintf(logfile, "socket function called Process %d:%d:%d:%d\n",
getpid(), domain, type, protocol);
    return new_socket(domain, type, protocol);  


// Overriding connect call…
int connect(int sockfd, const struct sockaddr *addr, socklen_t addrlen)
    int (*new_connect) (int sockfd, const struct sockaddr *addr,
socklen_t addrlen);
    new_connect = dlsym(RTLD_NEXT, "connect");
    FILE *logfile = fopen("/tmp/lammon.txt", "a+");
    fprintf(logfile, "connect function called Process %d:%d:%d:%s\n",
getpid(), sockfd, (int)addr->sa_family, addr->sa_data);
    return new_connect(sockfd, addr, addrlen);

//Capturing send call…
ssize_t send(int sockfd, const void *buf, size_t len, int flags)
    ssize_t (*new_send)(int sockfd, const void *buf, size_t len, int flags);
    new_send = dlsym(RTLD_NEXT, "send");
    FILE *logfile = fopen("/tmp/lammon.txt", "a+");
    fprintf(logfile, "send function called Process %d:%d:%s\n",
getpid(), sockfd, (buf == NULL ? " " : (char *)buf));
    return new_send(sockfd, buf, len, flags);      

We compile the code and build a shared object. Now, we can put this into lambda layers so it can be used into any lambda functions in a shared way as shown below: -

 Now, let’s add to the lambda function as shown below: -


Once it is set, all functions will be intercepted by our code and actual outgoing sniffing can be seen as shown in the below figure: -

As shown above, we were able to sniff or see actual calls being made to other hosts by lambda function.

Integrating into assessment methodology:

This type of object can help in assessing lambda functions for vulnerabilities. We can load the shared object, which can gather critical information while lambda function, is invoked. It is getting logged into file or CloudWatch logs. We can invoke functions with different payloads and monitor the behaviour. It allows us to identify various sets of vulnerabilities. It ends up being a similar approach to IAST along with DAST by leveraging this simple trick.

Can it help in protection?

Yes, one can build shared object in such a way that when anything objectionable is found at runtime then it can stop execution of that call.


“LD_PRELOAD” trick seems interesting for lambda function assessment. One can leverage it by adding a small code and loading it before the start of execution. It can intercept all calls and log important information. This information can help in validating and identifying vulnerabilities like – remote command execution, SSRF, third party calls, path traversal, information leakage and other architecture layer issues. This approach can be expanded to defend a lambda function as well. One can disallow all remote execution calls, block out going traffic, disallow file system access etc. We would like to focus more into vulnerability detection while performing pen-testing or automated scanning. This module can be integrated to any other methodology by common logs with timestamps.

Article by Amish Shah & Shreeraj Shah