Like many engineering organizations, our software stack relies heavily on the Go programming language. Given the widespread adoption of easyjson in the Go ecosystem, as highlighted in our previous article, The Russian Open Source Project That We Can’t Live Without, it was no surprise that easyjson became deeply integrated into our codebase; it was everywhere.
Once we learned about the potential risks associated with easyjson, we decided to phase it out. Here’s what we did to achieve that, and how you can do it yourself.
1. We confirmed that our code depended on easyjson
First, we searched for easyjson in the dependencies of all our services. Depending on how much code you’re searching, there are a few options you might consider when conducting your search:
- Use the search interface in a source code repository like GitHub or GitLab.
- Use an Integrated Development Environment (IDE) or text editor to examine your code on your local machine.
- Use grep to explore directories locally, for example:
grep --recursive --line-number --word-regexp "easyjson" <directory to search>
2. We investigated why easyjson was in our codebase
Once we identified which applications had easyjson as a dependency, we investigated why easyjson was a dependency. The context of how, why, and where easyjson was in use told us which code had to change to remove the dependency. That’s not always easy, but what I’ve outlined below is how our team approached this problem.
First, we used the built-in tools from Golang to search for easyjson locally. We’ll use go-openapi/spec as an example for this post, although our goal is not to criticize this project. We chose the easyjson package because it is a common dependency for people working on web frameworks. As you’ll see, easyjson is not an easy dependency to remove (pun intended!).
To follow along, first clone the repository: git clone https://github.com/go-openapi/spec
Then, change into the spec directory. We’re using commit cebbc5a in this example.
We’ll start with the mod graph tool:
$ go mod graph | grep easyjson
github.com/go-openapi/spec github.com/mailru/easyjson@v0.9.0
github.com/go-openapi/jsonpointer@v0.21.1 github.com/mailru/easyjson@v0.9.0
github.com/go-openapi/jsonreference@v0.21.0 github.com/mailru/easyjson@v0.7.7
github.com/go-openapi/swag@v0.23.1 github.com/mailru/easyjson@v0.9.0
github.com/mailru/easyjson@v0.9.0 github.com/josharian/intern@v1.0.0
In this output, the left column shows packages that have a dependency on easyjson.
The first four lines show that go-openapi/spec, go-openapi/jsonpointer, go-openapi/jsonreference, and go-openapi/swag all rely on easyjson.
To find whether this is a direct or indirect dependency, we can inspect the go.mod files for each of the modules.
As it happens, easyjson is a direct dependency of go-openapi/swag. We can visit go-openapi/swag tag v0.23.1 and see easyjson in the go.mod file:
Next, we need to find the package from easyjson that go-openapi/swag imports, so that we can locate where easyjson is.
Fortunately, we can continue to work with the go-openapi/spec package to gather this bit of intel. The list tool shows the package dependencies of go-openapi/spec, and, while searching for easyjson, we find:
$ go list all | grep easyjson
github.com/mailru/easyjson/buffer
github.com/mailru/easyjson/jlexer
github.com/mailru/easyjson/jwriter
The results show us that the buffer, jlexer, and jwriter packages are in use.
Then, the mod why tool shows us the shortest path in the import graph from the main module of go-openapi/spec to each of these packages:
$ go mod why $(go list all | grep easyjson)
# github.com/mailru/easyjson/buffer
github.com/go-openapi/spec
github.com/go-openapi/swag
github.com/mailru/easyjson/jwriter
github.com/mailru/easyjson/buffer
# github.com/mailru/easyjson/jlexer
github.com/go-openapi/spec
github.com/go-openapi/swag
github.com/mailru/easyjson/jlexer
# github.com/mailru/easyjson/jwriter
github.com/go-openapi/spec
github.com/go-openapi/swag
github.com/mailru/easyjson/jwriter
This output reveals that go-openapi/spec indirectly depends on easyjson via go-openapi/swag. As one example, the easyjson/jlexer and easyjson/jwriter packages are called in the go-openapi/swag/json package.
3. We devised a remediation solution
Once we identified where easyjson existed, we worked to figure out if easyjson was actually used, whether indirectly, or imported by something in the dependency that our application doesn’t execute. Here’s how you can ascertain this information for yourself and come up with a remediation plan:
In the previous go-openapi/spec example, we found that go-openapi/swag imports easyjson. So, the question we need to answer is: Does go-openapi/spec use the functions in go-openapi/swag that use easyjson?
Let’s start by finding where go-openapi/spec imports go-openapi/swag. Searching for the import on the local copy of go-openapi/spec turns up 24 files where go-openapi/swag is imported:
$ grep '"github.com/go-openapi/swag"' *.go
contact_info.go: "github.com/go-openapi/swag"
header_test.go: "github.com/go-openapi/swag"
header.go: "github.com/go-openapi/swag"
helpers_spec_test.go: "github.com/go-openapi/swag"
helpers_test.go: "github.com/go-openapi/swag"
info.go: "github.com/go-openapi/swag"
items_test.go: "github.com/go-openapi/swag"
items.go: "github.com/go-openapi/swag"
license.go: "github.com/go-openapi/swag"
operation.go: "github.com/go-openapi/swag"
parameter.go: "github.com/go-openapi/swag"
parameters_test.go: "github.com/go-openapi/swag"
path_item.go: "github.com/go-openapi/swag"
paths.go: "github.com/go-openapi/swag"
resolver.go: "github.com/go-openapi/swag"
response.go: "github.com/go-openapi/swag"
responses.go: "github.com/go-openapi/swag"
schema_loader.go: "github.com/go-openapi/swag"
schema_test.go: "github.com/go-openapi/swag"
schema.go: "github.com/go-openapi/swag"
security_scheme.go: "github.com/go-openapi/swag"
swagger.go: "github.com/go-openapi/swag"
tag.go: "github.com/go-openapi/swag"
validations_test.go: "github.com/go-openapi/swag"
Now, we need to investigate some of these files to see how go-openapi/swag is used and whether it calls any easyjson functions. For example, examining resolver.go on Line 48 reveals a call to swag.DynamicJSONToStruct():
Following the chain into go-openapi/swag, we find that swag.DynamicJSONToStruct() calls WriteJSON() and ReadJSON() on Lines 86 and 90, which in turn use the MarshalEasyJSON and UnmarshalEasyJSON interfaces and the jwriter.Writer function from easyjson.
In the function DynamicJSONToStruct(), you can see both WriteJSON & ReadJSON are called.
In the example above, it is clear that go-openapi/spec package indirectly uses easyjson for both JSON marshaling and unmarshaling. In this case, our options to reduce reliance on easyjson from the go-openapi/spec package are more limited.
Option One: Do it yourself and maintain the code yourself
- Maintain your own
go-openapi/swagand refactor it to use a different JSON marshal/unmarshal solution. - Maintain your own
go-openapi/spec, modifying the dependencies to include your modifiedgo-openapi/swag.
Option Two: Request that the maintainers remove easyjson and wait
- Open an issue with the
go-openapi/swagproject and request that the maintainers replace easyjson. This may not be a priority for the project, especially since many maintainers work for free or with limited sponsorship. However, making the request is the first step. - If we’re lucky, one of the project maintainers will take on the work and remove the easyjson dependency.
Option Three: Maintain your own version of spec while using the updated version of the officially maintained swag
- If you’ve pursued the second option and the maintainers can’t do the work, but they’re open to receiving a pull request from their community, there are two additional options:
- Wait for
go-openapi/specto update to the latest version ofgo-openapi/swag(without easyjson). - Maintain your organization’s own version of
go-openapi/specpointing it to the new version ofgo-openapi/swagwhere easyjson was refactored out. Fortunately, forgo-openapi/swagspecifically, work has already begun to remove the requirement foreasyjson, see https://github.com/go-openapi/swag/issues/68
$ go mod why -m github.com/mailru/easyjson
# github.com/mailru/easyjson
istio.io/istio/istioctl/pkg/cli
k8s.io/cli-runtime/pkg/resource
k8s.io/kube-openapi/pkg/spec3
github.com/go-openapi/swag
github.com/mailru/easyjson/jlexer
Now, easyjson is four layers into the dependency tree, once again pulled in via go-openapi/swag. In the best case, you now have to wait for go-openapi/swag, k8s.io/kube-openapi/, k8s.io/cli-runtime/, and istio.io/istio/istioctl to release an update before your application no longer depends on <code>easyjson</code>. In the worst case, you’d have to fork and maintain this entire chain of dependencies yourself, in addition to your actual application code.
The final situation is also the luckiest. It may be the case that easyjson is included for a specific purpose, such as testing, which can be refactored to remove the dependency from the main package. For example, google/pprof, a profiling visualizer, was able to do exactly this and reduce its indirect dependency footprint by moving some code to a separate module.
This is a lot of work. How bad is easyjson really?
As a JSON serialization module, easyjson works at the boundary of your system, a privileged location from which it is easy to cause major disruptions. For example, because they touch so much data, JSON serializers could:
- Exfiltrate sensitive data
- Allow remote code execution
- Break payload integrity by inserting or removing data
Software supply chain attacks are often difficult to spot, even if you’re closely watching the code base.
The potential threat from easyjson goes beyond just looking at vulnerabilities. Yes, easyjson is a highly performant choice if your application needs to serialize JSON in GoLang, particularly over the encoding/json module from the standard library, and, no, there aren’t any known vulnerabilities associated with easyjson as of today. But, as we noted in our threat report:
“easyjson is maintained by “[a] group of developers from VK, an entity with leadership that is under active U.S. and E.U. sanctions and has connections to Russian security services.”
This kind of threat has not traditionally been considered by most risk management plans for software that relies on open source dependencies. But they should. Most software seems to have a blind spot for this kind of threat, and we at Hunted Labs believe that must change.
Figuring out your next steps
Is it bad to use easyjson? The answer to this question depends on how your organization evaluates risk to its systems. However, to fully evaluate the risks requires identifying all threats in your environment, not just vulnerabilities, but also software maintained by sanctioned groups or foreign adversaries, as well as software that is poorly maintained. Once the complete risk picture is available, you can decide whether to invest the time and resources to remove easyjson.
Regardless of your decision, you now have a much clearer understanding of how much of your software is at risk.
How to be notified about the next easyjson
Everyone should use an appropriate lens of risk management to evaluate software dependencies consumed from outside of your organization. Unfortunately, easyjson is not the only potential risk in the open source ecosystem. There are many geopolitical and business factors that maintainers (see xkcd 2347) and poor security hygiene represent, all posing serious risks throughout the open source ecosystem.
If you’d like to learn about other potential software risks, you can request a demo and sign up for our mailing list.
Stay Ahead of Emerging Cyber Threats
Enter your email below to receive exclusive reports and expert analysis.
In submitting this form, you agree to receive information from Hunted Labs related to our products, events, and special offers. You can unsubscribe from such messages at any time. We never sell your data, and we value your privacy choices. Please see our Privacy Policy for information.
