How half-a-second of suspicious activity led an engineer to prevent a massive cyberattack

“I accidentally encountered a security issue while running some performance testing,” software engineer Andres Freund wrote on the Mastodon social media site. That chance discovery prevented one of the longest and most sophisticated hacking operations from progressing: it had attempted to illegitimately access millions of devices around the world.

The message led to a link where Freund explained how he had encountered “a bunch of strange symptoms” when he was updating a program. He was struck by the fact that the action used more of his processor’s capacity and — above all — that it took half-a-second longer to access than usual. This half-a-second made him suspicious, leading him to discover two years’ worth of nefarious activity, which was supposedly done by a state intelligence agency.

“It’s very unlikely that it was the work of amateurs. There were no immediate rewards,” says Lukasz Olejnik, an independent cybersecurity researcher. “The time spent on this deceptive operation, the sophistication of the backdoor system and its code all point towards an organization or agency that can afford such a project. It’s much more likely that it was done by paying salaries [to experts].”

The attack was a so-called “supply chain attack,” which affects the software that supports the most well-known and common programs. In this case, the target was a file compression tool used on Linux, a free and open-source operating system. Said tool is used in millions of machines. The goal of the attack was similar to creating a backdoor with a special code (which only they had) to access any building in the world that had that entrance.

This system — known as XZ Utils — is maintained thanks to volunteer developers, who spend hours maintaining and updating different programs. A little over two years ago, the attacker began collaborating with the programmer who was in charge of updating this software. This person — in charge of updating the system and responding (by email) to requests for tweaks from other developers — was overwhelmed. Hence, part of the attack consisted of pure social engineering: convincing him to leave part of his tasks to the person behind an account calling itself “Jia Tan.”

Over time, if the attacker gained the trust of the person responsible for that code, he would be able to implement his own malicious code. And, if it hadn’t ultimately been detected, this software would have been deployed on millions of servers, gaining privileged access to users’ data. It’s unclear whether the intent was to use the software to break into one or more specific machines, or to stage a massive attack.

While the code and method require extraordinary computer skills, control of these programs often depends on stressed-out developers. In a message thread, the manager admits to not getting to everything: “I haven’t lost interest but my ability to care has been fairly limited, mostly due to long-term mental health issues but also due to some other things. Recently I’ve worked off-list a bit with Jia Tan on XZ Utils… perhaps he will have a bigger role in the future, we’ll see. It’s also good to keep in mind that this is an unpaid hobby project.” After repeatedly being asked about the recent incident by reporters, he posted the following statement: “I won’t reply for now because first I need to understand the situation thoroughly enough.”

“There are a lot of people burned out in [the field of] software, both open-source and commercial. In this case, [being caught off guard] can be useful [to attackers], but it’s not a decisive factor,” says Olejnik. “[The recent event] is strong proof that even niche or obscure, semi-orphaned software can be a matter of national and international security. It’s a hidden cost of the software. On the other hand, no one can blame the maintainer of XZ, [because] there isn’t a wide range of developers for this type of software,” he adds.

It’s likely that other fake accounts pressured the manager to hand over his work to Jia Tan. The case reveals both a success and a hole in the community that maintains a good part of the code of our entire digital infrastructure. The hole is that finding the weak link is relatively easy, while the success is that the code is easily available and accessible, so that someone such as Andres Freund can detect the trap and become famous.

Freund believes that, this time, they got lucky: “It’s not that I think I didn’t do anything — I did it. What I mean is that we had an irrational amount of luck and we can’t just rely on something like that from now on,” he wrote on Mastodon. The recent attack is special due to a combination of factors, but the free, open-source software blocks — on which the internet is based — have been attacked on other occasions, also by alleged intelligence agencies. It’s likely that there are other similar attacks underway, or being planned. When it comes to closed-source software, there have also been various high-profile cases.

A famous X account (formerly Twitter) dedicated to malicious code has made a viral meme thanking Freund. “The xz backdoor was initially caught by a software engineer at Microsoft. He noticed 500ms lag and thought something was suspicious. This is the Silver Back Gorilla of nerds. The internet final boss.”

Another meme shows how, in this case, the world’s essential software was “suspiciously maintained by one state-sponsored actor during office hours.” The original drawing on which this meme is based is the work of cartoonist Randall Munroe. Beneath it, he makes a wry note about the reality of this situation: “A project some random person in Nebraska has been thanklessly maintaining since 2003.”

Sign up for our weekly newsletter to get more English-language news coverage from EL PAÍS USA Edition