Poison Code

Published by Andy Farnell



People who just want to help

We should always be suspicious of "people who just want to help". It's the "just" that gives it away. Cautionary stories of con artists from the Twilight Zone, Tales Of The Unexpected, and Inside Number Nine follow a familiar, sinister script:

It's a dark night when a kindly stranger appears… After they sidle up, insinuate themselves into your life, become indispensable, that's when the mask slips. That's when the knives come out.

Always too good to be true, it's a lucky coincidence that a job recruiter "just" happens to be on the same bus the day after you're fired. Or single Mr Perfect, an old friend of the deceased husband "just" happens to be at the funeral to meet the rich widow now heir to a fortune.

To the outsider it's obvious that these people have been spying on you for months, combing the news and obituaries. It's hard to see through charm. It's hard to imagine such calculated malevolence. And they always come when we are vulnerable.

But these "helpers" have been turning up in rather a lot of Open Source software projects lately. And we want to warn developers about them.


What is code?

First let's talk about code. For many of us code is fun. It's a rewarding hobby to solve difficult problems using computers. Code can be an end in itself.

Instead, let's see that "code is power". Many authors like Richard Stallman or Lawrence Lessig in his Code and Other Laws of Cyberspace spell out how computer code is the new legal scripture. Code undergirds all aspects of modern life from shopping, banking, and education, to how our governments are run. Who controls the code controls the world.

But who does write the code? Turns out almost all of it is written by individuals or very small groups. People like you. The world now runs on Free Open Source Software (FOSS). It runs our cars. It runs our space satellites. Our phones. The medical equipment in our hospitals. The servers that run social media and mainstream news. Everything runs on FOSS code. A significant lie about code

Few people know the giant lie that sits at the heart of our digital society, one so big and bold as to feel shocking once you hear the truth said plainly out loud.

Most of us believe that giant corporations like Microsoft, Google and Apple create the code that runs the world. We believe they employ thousands of the smartest people to ensure the spotless quality of proprietary code. We hear all about their singular innovation. About the enormous salaries their top coders earn. We believe their offerings are robust and secure.

The myth that Big-Tech are the originators of anything much is easy to see through with minimal research. Apple built their operating system on an open system called BSD. Windows is now almost entirely Linux. As is Android and Azure.

In reality these companies have not attained status through hard work and innovation, but by acquiring other smaller companies. That's how modern business works.

They've also been running the largest industrial espionage operation in history. Because they've been reading your emails, and those of every competing business for decades. There isn't a single business in the Western hemisphere who BigTech have not had a total heads-up on their R&D, recruiting, internal development, marketing strategy, trade secrets and financial affairs. Even the NSA and GCHQ now prefer to just buy civilian intelligence from them.

Big corporations do not write code. They take other people's code and act as a directing mind. They process and mix other people's code, shaping it into recognisable products with colourful branded packaging.


Vulnerable coders

When you imagine the Penguin book publishing company, do you see a headquarters at which there is a giant penguin in front of a giant writing machine? No! Still, people imagine big tech companies like Willie Wonka's Chocolate Factory, with Oompa Loompas running around winding the handles on machinery which create products ex nihilo, from thin air.

The reality is mundane. Code simply comes from millions of individual authors sitting in their homes, in rooms as ordinary as the one you're in now, with the kids jumping around while they try to do some work.

Like big supermarkets take food from small farmers, big tech businesses harvest this everyday code, package it, brand and market it with their logo, and sell it back to the people who made it.

Now you may be wondering, how can this happen? How can they "harvest" it? Well they just take it. There are no fences around the coder's farms. You see, what makes FOSS free and open is the license that controls it.

Coders who want to share their work with the world must deal with copyright and plagiarism and all the bad sides of human nature that mean someone can come along, steal their work and put their own name on it - something that should feel familiar to any professional writers or academics. So they gain a bit of legal protection from licences like the GPL (GNU Public License) which ensure other coders can freely swap code. A weakness of this license is that it also allows for-profit corporations to take and use their code.


Code is social

Before we feel too much doom, it's worth remembering here that most people are kind and trustworthy. Trust is the basis of civilised society and high trust-societies are happier and have a higher GDP. The whole FOSS ecosystem is built on assumptions of trust and common desires to help build a better society through code.

But computer programs can grow big, and take a long time to write. Unlike a blog post or even a novel, most serious programs are beyond the ability of a single author. Some projects take decades to complete. It's more like creating an encyclopedia. Many hands and minds are needed.

So an international free and open source community has sprung up over many decades to allow sharing of ideas, coordination and quality control. FOSS is invariably a group effort. Code is collected together in openly accessible "repositories".


Why the people must control code

So the answer to our earlier question; "who controls code"; is we, the people do.

There are several reasons it's highly desirable that code remains democratic and under public control with unrestricted opportunity for individuals to learn, examine, modify, share and contribute.

Most importantly, if only giant corporations wrote code we have no reason to trust them any more than a random member of the public. Indeed we have less, since they are motivated by profit. Commercial code tends toward tasteless McNuggets, the lowest common denominator that people will stomach.

Further, experience shows that, at least in peace-time, giant monolithic efforts, whether undertaken by corporations or governments, rarely succeed. The task of supplying the world with code in an ever changing landscape dwarfs even the Apollo and Manhattan projects. It is more akin to farming the food necessary to feed the whole planet.

We also find that mono-cultures are catastrophic. They are brittle and prone to "class" failures, such as happened when the entire world's banana crops were almost eradicated by a single pest. All of the biological diversity and resistance had been bred out as the same variety were grown everywhere. Lack of diversity and independent innovation will surely kill digital technology.

That is why computer software is one of the most democratic institutions in history. And yet that makes it, and all its participants, very vulnerable too.

Coders often work alone, unsupported, unpaid, at labours of love. They must build tenuous trust relations, be clever diplomats and team leaders, and be articulate in stating their needs and aims.

And yet the psychological stereotype of the lone coder is insular, autistic, and socially awkward. Being a free software coder makes huge demands. We need others, but are often forced to put up barriers and isolate for fear of corporate gangs stealing (or "forking" and "monetising") our creations without attribution or remuneration.


Wannabe gig economy

Surely though, if "code runs the world" all the young programmers must be the wealthiest people alive? Absolutely not. The myth of the rich tech worker is a fabrication of the media. A lucky few in Silicon Valley earn well. The rest have no job security and hop from gig to gig in an unforgiving economy. Most tech workers are treated like rubbish. Millions have recently been laid-off to boost the profits of BigTech companies.

Your smartphone is probably made by underage workers enslaved in Chinese factories. Likewise the software that runs it, or your home assistant or your Ring doorbell may well be the labour of a coder in India who could not afford the meal you are eating. Tech is a monument to inequality.

In contrast the BigTech companies who take and sell other people's work are the richest in industrial history. The combined wealth of tech giants makes them the third superpower - after the US and China, "BigTech" occupies third place above Russia and Europe. Never in the history of Capitalism has there been such disparity between the goods people create and the power accrued by those who take and resell them.

A few high profile coders are disproportionately rewarded as in any Pareto distribution. Most coders live in a state of "wannabeism". Like aspiring pop stars or YouTube influencers. Each hopes to one day "make it". To be discovered, recognised and rewarded! Their work makes a few others rich, as they struggle in obscurity.


Why this matters

So far we have a picture of a world that runs on code, code which comes from millions of vulnerable and exploited people who sit on the margins of society while a few massive corporations take credit for it.

But code is volatile, precarious and delicate stuff. A single missing comma in some code can transform a program that does one thing under the control of its master into a totally different program that operates for a stranger. We call this "hacking".

As s
ecurity expert Ross Anderson once explained to his class who asked, "What's the worse thing that can happen?" - turning a self driving car into a weapon to mow down crowds of people is probably not the worst… not if we can instead take control of a nuclear power station or drop an airliner out of the sky.

Anyone who considers these threats exaggerated is an fool who should stop reading immediately and go and put their head back into the sand. Cybersecurity is one of the biggest issues of our time. It is also one of the least understood, commonly dismissed and underfunded areas.


The supply chain

Most people think of hackers as breaking-in through computer networks to take control of a single device. That still happens, but to understand why it's an outdated trope and how code gets hacked to affect a whole country, let's go back to 1978 in Israel. Palestinians at that time took up economic terrorism against Israel by injecting mercury into oranges destined for export.

All it took was a hypodermic syringe and some waste chemicals to almost completely destroy the Jaffa fruit company and one of Israel's most important exports. At the time the poisoned fruit would have been hard to detect - whereas today farm goods are analysed and x-rayed. Of course one doesn't need to poison every fruit, just one or two will do.

We call this the "supply chain". In practice there are many safeguards in place. When you buy medicines from the pharmacy you'll see various seals and anti-tamper strips on the container, there to make sure what you take is what the manufacturer put in the bottle.

But supply chain attacks are becoming more common. Increasingly the tactic is to attack the manufacturer, or to try to get as close to the source as possible. Why hack one app on one phone when by hacking the app-store a million people can be tricked into downloading the poisoned program?


Supply chain hacking

Here's how the bad hackers operate.

Find a vulnerable coder who works on a critical component or program that is used by a lot of other programs. We call this a dependency.

It is best if this coder is isolated and overworked.

Approach the coder offering to help. A good ruse is to spot a bug in their code, and then volunteer to fix it. In FOSS talk we call that "submitting a patch" often via something called a "pull request".

The coder may be grateful. Nonetheless they will typically not trust the volunteer contribution on face value. The coder should review the bug-fix. Most programmers will quickly spot a malicious attempt to hack the program, revert the sabotaged code, lock out the interloper, publicly report the incident, and never allow them near their code again.

But that's not how it always works.

Clever hackers will not poison the code at the first opportunity. They will build trust. Infiltrate the project. They will make many valuable contributions. They will flatter the lead coders and make their own offerings indispensable - much the way a good spy becomes ensconced in an organisation or the way BigTech insinuates itself into our lives, not by threats but by being useful. By "just" helping.

Later, after subtle, well hidden malicious changes have been made, other coders who rely on the dependency - and eventually one of the BigTech corporations that feed on FOSS - will incorporate the poisoned code into a product. The product will be distributed to millions or billions of computers. This is what happened with the xz compression hack on ssh (secure shell). Only the hack was caught at the last moment by a vigilant hero Andres Freund.

Sometimes the bad hackers work in packs. Not just one contributor, but many will join the code project as "useful volunteers". They may join slowly over many years. These "sleepers" will eventually gang up on the original project owners and try to take control of the project, just like the hostile takeover of a company by a group of hidden majority shareholders. Frequently they will use more subtle bullying tactics, gaslighting, social pressure, guilt-tripping and creating "codes of conduct" that are then leveraged against the original owners who may have willingly accepted them as good, progressive measures.

It's a pattern that's becoming more common, and recently happened to a developer I know from my days in Pure Data; Hans-Christoph Steiner, a developer on the F-Droid project noted irregular activities and bullying behaviour apparently aimed at compromising that project.


Resisting "help"

Of course this craft is as old as the hills. In the 1960s our agencies like GCHQ, MI5 and MI6 were stacked to the rafters with the likes of Cairncross, Philby, Burgess, Blunt and Maclean. Likely, little has changed today. Indeed, that's how the game works.

Instead of hoping to exclude all infiltrators one needs to plan as if already compromised. Good defensive hacking is about anticipating treachery and minimising its effects when the inevitable happens.

It involves a strange mixture of compartmentalisation and transparency, checks and balances, independent random audits, random pairing and horizontal accountability. There are ways to preserve some semblance of integrity in groups - at least up to a Byzantine fault threshold and having 51% of the team corrupted. The trick is to avoid any single entity accumulating too much control. That is the very opposite of what is happening in tech today.

During the past years corporations like Microsoft have been taking over critical public code infrastructure. A tool that coders use to collaborate and communicate, called git was acquired by the company (in the form of Github). It's since become a centre for millions of coders. Such a concentration of talent in one place, under the control of one organisation is dangerous for the rest of us. Similarly, Google fund many open source projects and run "summer of code" hackathons. What we must remain mindful of is that charitable and benevolent as these corporations present themselves, they are not disinterested or helping out of the pure goodness of their hearts.


Protection versus security

The difference between security and protection seems subtle, but it is clear. It is the difference between giving a man a fish and teaching a man to fish. To be secure is to be autonomous, free, and confident in one's ability to defend. To be protected is to rely on the strength or goodwill of another. That strength may be leveraged against you.

In our cybersecurity teaching at Boudica we aim not to cultivate dependency. To offer security one must set free what you love. By contrast, protection is a power that one retains over a subject.

Protection rackets commonly arise in the cyber world. Protection racketeers play both sides. For example; anti-virus companies are notoriously linked with virus writers. Companies that offer help removing people from personal-information databases notoriously work as "people finders" populating the same databases. This duplicity has been well documented by cyber-reporter Brian Krebs. It is something even the Mozilla foundation got mired in.

This trust issue is sometimes called a principal agent problem. Who watches the watchers?

Our concern in this analysis is as follows;

As the world wakes up to the vulnerability of software supply chains the freedom and security of software development comes under a double attack.

The European Union has recently been drafting legislation to make developers more accountable for bugs in their code. This seems like a good thing - until we remember that most of the actual developers are dirt poor individuals. Of course they are already at risk from hostile actors.

At this point, when BigTech giants offer to "take developers under their wing", to offer protection, our alarm-bells should sound about their supposed benevolence.

Now we see a new threat. It serves BigTech companies rather well if exposed, independent developers working on critical dependencies were to fall victim to social engineering, bullying and take-over attacks. Even the perception of that would be useful if Microsoft and Google were to suddenly appear to be stepping up to "corporate social responsibility".

GitHub is already more than a mere repository. It is a a social media site and a career development platform much like Linked-In which Microsoft already owns. It is a "community" and network which offers unique free tools like AI coding assistance (CoPilot) trained using the code existing Github users have already located with Microsoft (albeit without their permission).

Microsoft are thus becoming a "Code Bank". Like a financial bank uses your money to invest, and pays interest, a code-bank can train AI on your code, and have preferential access to it for its own uses. As a bank can control reputation, access to money and opportunity, so a code bank gate-keeps careers. Github accounts are increasingly used as credentials for online portfolios, a place for coders to keep their curriculum vitae and showreel.

Comments on Bruce Schneier's site and on YCombinator's Hacker News gave away a whiff of what is in the air. "It would have been so easy", remarked one commenter, "for Microsoft to have warned the victim of this hack".

There you have it. It is not hard to see how Github is positioned to become a protection racket for coders. Further enclosure of free software into walled gardens will soon be on offer. Of course giant agro-businesses are only putting up fences around farmers' fields for our own good, to make sure no wolves get in through the gate.


Recommendations for diversifying help

To avoid the death of independent code and world dominanated by already too-powerful tech monopolies we are going to need more help.

The antidote to malevolent influence is more, different sources of advice and support. Some of this can come from governments. Money and help for independent developers to set up alternatives to Github and to have access to free, safe AI are needed.

Government money also implies questions around the role of security organisations like the NCSC and GCHQ. Does national security not include a careful heads-up for developers working on key dependencies who may not be so socially and politically savvy? Of course many would say that the biggest enemy is government agencies - because they will abuse power for illegal surveillance. Let's hope that they are not also "just wanting to help".

The role of larger power blocs like the European Union is also important. Continental regulation can help build decentralised networks that afford coders security through loose association rather than patriarchal protection. Perhaps we should remember that the European Union came about initially from a need to coordinate agriculture as well as to prevent wars over resources.

European Law can ensure that companies who take and use the code of its citizens pay proper remuneration and take proper precautions to ensure its fitness for public consumption. When I find grubs in my apples, it is the supermarket I hold responsible, not the farmer.

At the same time more must be done to ensure that small farmers have access to markets and can safely sell produce directly. In many ways the purity of code is related to how many hands it has passed through en route to the consumer. Organic code with simple provenance is usually superior to highly processed fare.

Security is increasingly used as a dishonest excuse to exclude independent coders from selling or just freely sharing their wares. A balance must be struck to stop pedlars of poisoned or very unhealthy code. Those who run and benefit from large marketplaces must pay the way here.

Development safety standards and good practices must be democratically determined but also maintained outside of interested power centres. This is where non-profit organisations and societies shine. Chartered engineering institutions like the Institute of Electrical and Electronics Engineers (IEEE) and British Computer Society (BCS), and rights groups like the Free Software Foundation (FSF) can usefully recognise the danger of allowing BigTech commerce to become code protection rackets, and step up to help individual coders in their craft.

Outside of safety critical work like avionics and medical devices the distinction of "Professional" is not that helpful. Unlike architects and surgeons who can usefully be regulated in their practice for safety reasons, coders cannot. No more than writers can be "professionalised" and granted a licence to author books.

Professional software engineering still has a long way to go politically to find the correct balance of regulating processes and oversight of critical infrastructure while allowing independent freedom and innovation.

Currently, this creates a grey area where organisations that could help, like software engineering institutions and standards bodies (CCITA, CCCI, ISO etc), see the problem as outside their scope of paying professional members and industry interests. They are weak on "Public interest" cybersecurity.

The role of specific organisations such as the Debian Project is also important. Some of the best practical standards have come from these sources. However, they are also subject to capture as the systemd debacle clearly illustrated, and there are problems when these projects lie too close to profitable interests; such as the Mozilla Foundation's proximity to Google.

Governments, corporations and non-profits are not the only players. A strong civic society where debate and free intelligence exchange may happen is a healthy necessity. The rise of walled-garden social media has made ideological censorship rife on "platforms" and eroded the open culture that fired the early waves of tech innovation. Decentralised social media with diverse ownership and proper privacy remains a necessity.


Digital diet

Five years ago I wrote the book "Digital Vegan" with its metaphor of code as food. It remains an eclectic work with a only a few readers. But the message, that we should be selective in what we consume in the digital world too, seems more relevant with each day. I gave rather little attention to the idea that deliberate poisoning would become such an issue. I focused on the problem that BigTech companies like Google and Apple had become "McDonald's of code", displacing local restaurants and cafes.

Now I see that the metaphor extends to the supply chain, and to the growers of our food as much as the shops and kitchens that determine what we put on our table. Farmers have always been a class we take for granted. It is time we all did more for them.

Posted on Apr 19, 2024.