Easy Does It:

The Russian Open Source Project That We Can’t Live Without

Hunted Labs Discovers Suspicious Origins of Ubiquitous Open Source Package

Written by:

Hunted Labs

Share:

Background

Hunted Labs has discovered an open source software package that appears to be completely owned, maintained, and controlled by developers based in Moscow who work for one of Russia’s largest internet services conglomerates, VK Group (VK). Also known as Mail.ru, VK is controlled by Russian state-owned entities, and a member of its leadership is subject to U.S. and E.U. sanctions.

While using our platform Entercept™ to help a customer identify foreign ownership, control, and/or influence in their software, we found a suspicious component known as easyjson. This component is used across U.S. Government systems, Fortune 500 enterprises, and serves as the cornerstone of Cloud Native Computing Foundation projects that underpin the entire cloud-native ecosystem.

At Hunted Labs, we believe knowing who is behind the keyboard writing the code that powers your organization’s software is vital to securing your business. In this blog, we’ll explain how developers inside VK fully control easyjson, a vital piece of open source. We will also tell you who VK is, why you should care, how we found easyjson, and what you can do about it. 

Before we dig into the methodology behind our research, here is a summary of everything you need to know:

Key Takeaways

  • What is easyjson?

Easyjson is a Go package designed to optimize JSON serialization and deserialization processes by generating Go code for JSON encoding and decoding. Widely adopted across cloud-native ecosystems, it is a critical dependency for numerous open source and enterprise projects. This includes high-performance JSON handling in distributed systems, real-time data serialization for financial and analytics platforms, and optimization of cloud-native applications.

  • Who maintains easyjson?

A group of developers from VK, an entity with leadership that is under active U.S. and E.U. sanctions and has connections to Russian security services.

  • Who is impacted?

Cornerstones of the modern software supply chain and cloud-native tools have dependencies on easyjson, and all applications that pull in these dependencies could potentially be impacted, including, but not limited to:

  • Helm
  • Istio
  • Kubernetes
  • What makes this extremely dangerous?

Although easyjson is a code serializer implemented in the Go language and is more secure than many similar packages, any compromise of a serializer is extremely dangerous because they are:

  • Invisible
  • Deeply Integrated
  • Hard to Remove
  • Trusted by Default
  • How could this be weaponized or exploited?

Russia doesn’t need to attack directly. By influencing state-sponsored hackers to embed a seemingly innocuous OSS project deep in the American tech stack, they can wait, watch, and pull strings when it counts. A well-placed backdoor or subtle bug could become the digital equivalent of a sleeper cell—with impact spanning from the Pentagon to your iPhone. Below are the top ways this package could be exploited: 

  • A supply chain backdoor (mass compromise)
  • A remote code execution (RCE) via deserialization
  • Espionage & data exfiltration
  • A kill switch activation

Behind the Code: Who is VK?

VK is one of the largest technology companies in Russia by user numbers. More than 95% of the Russian internet audience uses VK services, which enable people to keep in touch, play video games, master new skills, listen to music, watch and create video content, discover and buy goods and services, and fulfill a wide range of other needs. Through Vkontakte, its popular social media platform, VK is also known to share user data and information with Russian security services, and is a State-owned company via Gazprom Media. VK is used for censoring political opposition and participating in government surveillance activities on behalf of the state.

VK has consistently played a role in information warfare in the Ukraine conflict. VK has been under fire in the past for censoring content on its various platforms, with the most recent example showing them complying with requests from the Kremlin to censor content concerning Russia’s invasion of Ukraine. VK’s role seems clear: Whatever information the Kremlin considers a threat, VK must act to remove it from the site immediately and entirely. 

So this raises the obvious question: Why did we build our entire software ecosystem and our applications on top of Russian code of dubious origin without proper vetting and due diligence? Better yet, why did we allow so many critical CNCF projects to be run on software that could contain potential sleeper cell code? When Hunted Labs investigated, we found that Russian contributors maintained the repository, accounting for over 85% of all commits. That code now acts as an umbilical to our cloud-native ecosystem.

How We Discovered a Critical Open Source Project Under Russian Control

A few months ago, we conducted a security analysis to determine which open source components currently leveraged in enterprise software on behalf of the U.S. Government were under foreign ownership, control, or influence. Our product, Entercept, detected a package called easyjson, which was automatically flagged because the project appeared to be owned and controlled by Moscow-based software developers. 

At the time, we saw this as being pretty standard. Open source is an extremely vast ecosystem, and we expect to see heavy participation from countries such as Russia and China in a myriad of open source projects. But, months later, on a late evening, while reviewing our customer analysis, we asked ourselves some very simple questions as a research team:

  1. Who are these maintainers? Who do they work for? Is this organization or its affiliated individuals under any current U.S. sanctions?
  2. Where else could we find this dependency?
  3. How widely used is easyjson? How dependent are we on easyjson?

 

We had no idea we were opening a huge can of worms. 

STEP 01

Dependency Hunting

We actually began this analysis backward, reverse-engineering the process by starting with what is usually step three. Our threat research team analyzed over 2,500 images and source code repositories with Entercept for critical software dependencies.

Engineers often add an untold number of open source dependencies to their projects when building a new product. Free open source code lets developers go faster, but, as we pointed out earlier, it often slips past traditional security checks because it is hidden and implicitly trusted. 

Entercept surfaced one dependency, named easyjson, with questionable origins and major potential impact if exploited, based on the massive footprint of contributions coming from Moscow. We then scanned various important open source projects using the Blast Radius component of our platform to visually explore various projects’ interconnectedness to easyjson. That’s when we started to see alerts light up our screen. We found extremely popular projects repeatedly impacted by easyjson.

Initially, we thought it was a false positive. It was hard to believe something with this kind of potential risk could be so widely used. So, to confirm our findings, we leveraged Entercept’s threat search feature to automatically identify every single piece of software from a specific large sample of various open source projects and enterprise software to identify each instance of easyjson. In this case, we found thousands of projects using easyjson as both direct and indirect dependencies. 

The results of our analysis were startling, both in terms of the quantity and the quality of projects that had easyjson as a dependency. A lot of the software containing easyjson serves as the foundation of cloud-native platforms and underpins our modern software supply chain, including Helm, Istio, Kubernetes.The updated list of OSS projects leveraging easyjson can be found in our full report. These are projects everyone relies on every single day to do their jobs, whether they work at a Fortune 500 company or the Department of Defense.

If anything happened to easyjson, we would be in a world of hurt.

STEP 02

Determining Compromised Package Ownership

We then dug deeper, looking into who is responsible for maintaining and shipping this package to all of the critical open source projects that power and control everyone’s cloud environments. It turns out that it is individuals working for VK, which Entercept quickly surfaced.

In the screenshot of its GitHub repository (see below), you can spot the easyjson package in the mailru repository.  It didn’t take long to figure out that this was an organization with a history of cooperation with the Russian state. While we can’t confirm that these contributors are malicious actors at the moment, we know they work for a highly controversial organization in Russia.

Below, we provide a screengrab from Entercept, detailing results around contributors and easyjson ownership.

STEP 03

Determining the Impact The Pervasive Use of Easyjson Across the Cloud-Native Landscape

As previously mentioned, critical open source projects need to work for cloud-native deployments to flow flawlessly day in and day out. These platforms include Helm, Istio, Kubernetes, and the list goes on. Our cloud applications simply can’t live without easyjson. A comprehensive analysis is available in our full report. Any degradation to the performance of this package would be detrimental to every cloud system.

What We Do Now

Hunted Labs has provided exhaustive evidence regarding this advisory to the U.S. government and relevant stakeholders. So, what’s next? 

The widespread use of easyjson makes finding a solution challenging. However, we cannot continue to blindly rely on this package due to the state of the current threats to our increasingly fragile software supply chain.

We must collectively work together among the open source community, industry, and government to leverage safer alternatives that perform the same or similar functionality in our code. That is the only way to eliminate the risk currently associated with the practice of automatically ingesting code from groups affiliated with hostile or potentially hostile nations or even sanctioned entities tied to these nation states. 

To be clear, even if sanctions were removed and Russia suddenly became an ally, it wouldn’t remove the risk associated with easyjson or similar components, given the country’s history of persistent and offensive cyber activity against the West. 

Mitigation options include forking and maintaining easyjson yourself, moving to an alternative JSON serialization tool, and/or making a better version of easyjson as a community with a group of maintainers that is more transparent and where ownership is spread across a multitude of stakeholders with various backgrounds. As always, our suggestion box is open! 

Oh, and we haven’t even talked about China…yet.

#HuntOrBeHunted

Sources:

Share

The Hunting Ground

Hayden Smith

The following is a story about the recent XZ Utils security breach and how things came about. Formore context on the

Our Blog

Request A Demo

Fill out the form below so we can arrange a product demo for you.

    Request A Demo

    Fill out the form below so we can arrange a product demo for you.

    Thank You

    We have received your submission.