Background

Hunted Labs™ has discovered an open source software package that appears to be completely owned, maintained, and controlled by developers based in Moscow who work for one of Russia’s largest internet services conglomerates, VK Group (VK). Also known as Mail.ru, VK is controlled by Russian state-owned entities, and a member of its leadership is subject to U.S. and E.U. sanctions.

While using our platform Entercept™ to help a customer identify foreign ownership, control, and/or influence in their software, we found a suspicious component known as easyjson. This component is used across U.S. Government systems, Fortune 500 enterprises, and serves as the cornerstone of Cloud Native Computing Foundation projects that underpin the entire cloud-native ecosystem.

At Hunted Labs, we believe knowing who is behind the keyboard writing the code that powers your organization’s software is vital to securing your business. In this blog, we’ll explain how developers inside VK fully control easyjson, a vital piece of open source. We will also tell you who VK is, why you should care, how we found easyjson, and what you can do about it.

Before we dig into the methodology behind our research, here is a summary of everything you need to know:

Key Takeaways

• What is easyjson?

Easyjson is a Go package designed to optimize JSON serialization and deserialization processes by generating Go code for JSON encoding and decoding. Widely adopted across cloud-native ecosystems, it is a critical dependency for numerous open source and enterprise projects. This includes high-performance JSON handling in distributed systems, real-time data serialization for financial and analytics platforms, and optimization of cloud-native applications.

• Who maintains easyjson?

A group of developers from VK, an entity with leadership that is under active U.S. and E.U. sanctions and has connections to Russian security services.

• Who is impacted?

Cornerstones of the modern software supply chain and cloud-native tools have dependencies on easyjson, and all applications that pull in these dependencies could potentially be impacted, including, but not limited to:

• Helm
• Istio
• Kubernetes

• What makes this extremely dangerous?

Although easyjson is a code serializer implemented in the Go language and is more secure than many similar packages, any compromise of a serializer is extremely dangerous because they are:

• Invisible
• Deeply Integrated
• Hard to Remove
• Trusted by Default

• How could this be weaponized or exploited?

Russia doesn’t need to attack directly. By influencing state-sponsored hackers to embed a seemingly innocuous OSS project deep in the American tech stack, they can wait, watch, and pull strings when it counts. A well-placed backdoor or subtle bug could become the digital equivalent of a sleeper cell—with impact spanning from the Pentagon to your iPhone. Below are the top ways this package could be exploited:

• A supply chain backdoor (mass compromise)
• A remote code execution (RCE) via deserialization
• Espionage & data exfiltration
• A kill switch activation

Behind the Code: Who is VK?

VK is one of the largest technology companies in Russia by user numbers. More than 95% of the Russian internet audience uses VK services, which enable people to keep in touch, play video games, master new skills, listen to music, watch and create video content, discover and buy goods and services, and fulfill a wide range of other needs. Through Vkontakte, its popular social media platform, VK is also known to share user data and information with Russian security services, and is a State-owned company via Gazprom Media. VK is used for censoring political opposition and participating in government surveillance activities on behalf of the state.

VK has consistently played a role in information warfare in the Ukraine conflict. VK has been under fire in the past for censoring content on its various platforms, with the most recent example showing them complying with requests from the Kremlin to censor content concerning Russia’s invasion of Ukraine. VK’s role seems clear: Whatever information the Kremlin considers a threat, VK must act to remove it from the site immediately and entirely.

So this raises the obvious question: Why did we build our entire software ecosystem and our applications on top of Russian code of dubious origin without proper vetting and due diligence? Better yet, why did we allow so many critical CNCF projects to be run on software that could contain potential sleeper cell code? When Hunted Labs investigated, we found that Russian contributors maintained the repository, accounting for over 85% of all commits. That code now acts as an umbilical to our cloud-native ecosystem.

How We Discovered a Critical Open Source Project Under Russian Control

A few months ago, we conducted a security analysis to determine which open source components currently leveraged in enterprise software on behalf of the U.S. Government were under foreign ownership, control, or influence. Our product, Entercept, detected a package called easyjson, which was automatically flagged because the project appeared to be owned and controlled by Moscow-based software developers.

At the time, we saw this as being pretty standard. Open source is an extremely vast ecosystem, and we expect to see heavy participation from countries such as Russia and China in a myriad of open source projects. But, months later, on a late evening, while reviewing our customer analysis, we asked ourselves some very simple questions as a research team:

1. Who are these maintainers? Who do they work for? Is this organization or its affiliated individuals under any current U.S. sanctions?
2. Where else could we find this dependency?
3. How widely used is easyjson? How dependent are we on easyjson?

We had no idea we were opening a huge can of worms.

‍

What We Do Now

Hunted Labs has provided exhaustive evidence regarding this advisory to the U.S. government and relevant stakeholders. So, what’s next?

The widespread use of easyjson makes finding a solution challenging. However, we cannot continue to blindly rely on this package due to the state of the current threats to our increasingly fragile software supply chain.

We must collectively work together among the open source community, industry, and government to leverage safer alternatives that perform the same or similar functionality in our code. That is the only way to eliminate the risk currently associated with the practice of automatically ingesting code from groups affiliated with hostile or potentially hostile nations or even sanctioned entities tied to these nation states.

To be clear, even if sanctions were removed and Russia suddenly became an ally, it wouldn’t remove the risk associated with easyjson or similar components, given the country’s history of persistent and offensive cyber activity against the West.

Mitigation options include forking and maintaining easyjson yourself, moving to an alternative JSON serialization tool, and/or making a better version of easyjson as a community with a group of maintainers that is more transparent and where ownership is spread across a multitude of stakeholders with various backgrounds. As always, our suggestion box is open!

Oh, and we haven’t even talked about China…yet.

#HuntOrBeHunted

Sources:

https://www.wired.com/story/vk-russia-democracy/
https://www.wsj.com/world/russia/putin-aide-russia-propaganda-us-election-3b3ef3d2
https://gizmodo.com/vk-vkontakte-russia-censorship-ukraine-invasion-1850680416
https://foreignpolicy.com/2020/10/26/russia-internet-freedom-kremlin-tech/
https://foreignpolicy.com/2022/08/22/information-warfare-in-russias-war-in-ukraine/ https://dgap.org/en/research/publications/key-player-russias-cybersphere
https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L:2022:080:FULL&from=EN
https://vk.company/en/investors/info/
https://sanctionssearch.ofac.treas.gov/Details.aspx?id=34596
https://www.reuters.com/article/russia-vk-idCNL8N2SO3IY/
https://www.article19.org/data/files/medialibrary/38696/case_studies_R02_A5_WEB.pdf https://www.theverge.com/2022/2/26/22951307/us-sanctions-russia-vk-ceo-vladimir-kiriyenko https://www.theguardian.com/world/2013/apr/18/russian-internet-social-media-network
https://www.themoscowtimes.com/2021/12/13/russias-vk-appoints-son-of-kremlin-insider-as-new-ceo-a75802

‍