Three months ago, a clean-cut former Boy Scout by the name of Ross Ulbricht pleaded for clemency in front of a Manhattan Federal Court.

Ulbricht is a Silicon Valley start-up kid who loves his mum and plays the djembe, but he’s also the Dread Pirate Roberts, founder of the anonymous online market Silk Road, an enterprise that began in libertarian idealism and ended in a mess of six contracted murders but no actual deaths, Mormon drug runners, crooked cops and doublecrosses.  

More than 100,000 users traded more than $200 million in drugs and contraband on Silk Road until the FBI arrested Ulbricht in October 2013. In his pre-sentencing letter, Ulbricht wrote that “Silk Road was supposed to be about giving people the freedom to make their own choices, to pursue their own happiness, however they individually saw fit. What it turned into was, in part, a convenient way for people to satisfy their drug addictions.”

At his sentencing, he told the court: “I never wanted that to happen. I wish I could go back and convince myself to take a different path.”

The judge handed down five sentences: five years, 15 years, 20 years, and two for life, with no chance of parole. “No drug dealer from the Bronx has ever made this argument to the court,” she told Ulbricht. “It’s a privileged argument and it’s an argument made by one of the privileged.”

Within days of Ulbricht’s arrest, Silk Road 2 had popped up. But even before that newer, “better” Silk Roads were appearing – Atlantis opened with all the trappings of a normal internet start-up, right down to a twee, clean-cut YouTube video featuring a dude named Charlie who just needed a blunt.

Silk Road 2 and its fellow hydras exist on the “deep web” – communications and sites not indexed by search engines and run using anonymising browsers and software, the most well known of which is The Onion Router, or Tor.

The controversy over the deep web and the corner of it known as the “dark web” – a hidden, encrypted layer of the internet where Silk Road and others operate – shines a light on the ongoing battle for privacy at the dawn of the internet age. At a time when we assume that everything can and should be known, the story of Tor, how it came to be and how it continues, contains more than dread pirates, drugs and child porn. It is also a story of divided governments, whistleblowers and liberty.

“Onion routing”, which lets people communicate and host websites anonymously online, was created by the very institution many of its users are trying to avoid. In the 1990s, researchers at the United States Naval Research Laboratory and later the Defence Advanced Research Projects Agency (DARPA) developed the network to protect the intelligence community’s communications.

The non-profit organisation that maintains Tor for public use, which formed in 2006, remains close to branches of the US government. By 2012-13, a mixture of governmental bodies were providing $US1.8 million of the organisation’s $US1.9 million grants budget. But while the US departments of defence and state provide the support to keep Tor running, the CIA, FBI and NSA attack Tor, proclaiming it an enemy and its users criminals.

Tor works by using multiple layers of encryption to protect data, and passing it on a randomly generated path through a network of 6000 volunteer-run relays. Your encrypted communications enter the Tor network through a “guard node”, which unwraps the first layer of encryption and passes the data on to “bridge nodes”. Each bridge node knows your data’s previous stop in the network, but not the stop before. After bouncing around the network your data is sent on to an “exit node”, which then unwraps the final layer of the onion of encryption.

Two million people use Tor every day. NSA whistleblower Edward Snowden used it to contact the press about the agency’s PRISM surveillance program. Domestic violence prevention services use Tor to make sure their clients’ internet activities and locations are shielded from their abusers. Dissidents in the Arab Spring movements and human rights workers throughout the world have relied on Tor to fight back against governments who would persecute them if they could intercept their communications; Iranians are some of the most active Tor users.

Tor not only shields browsers, it can also shield the location of sites. Its “hidden services” use nodes on the network to route browsers to sites anonymously, obscuring both the IP address of the person looking for the site and the site’s location from the server.

Encryption is not yet illegal in Australia or the US. Like any tool, it’s what you use it for that counts. While Tor is employed by child pornographers, for example, most of the illegal activity on the network is in the form of drug sales. Drug dealers and their customers on the deep web argue that buying drugs online is a better and safer alternative to buying on the streets, providing higher quality control and the ability to educate users. Silk Road even had an advice forum.

The data coming through a Tor exit node could be from a persecuted Iranian dissident or a Texan running a drug empire. The thousand people running exit nodes throughout the world are not breaking the law, but they are still at risk of being raided and investigated by police for the data passing through their node. They believe supplying exit nodes is a moral imperative, and assume the risk in the name of privacy. Over the years, there have been hundreds of raids on their homes and businesses. In contrast, companies running public web servers that store images of child porn, or the providers of email services used to send it, are largely viewed by the law as neutral.

US government security agencies are not alone in their apparent fear of the tools that give citizens the potential to communicate privately on the internet. As part of a counterterrorism act, China is now pushing for all foreign tech companies to allow the government surveillance access. In January this year, Britain’s PM David Cameron said that all online communications should be able to be read by the government if needed. And things aren’t looking good for internet privacy in Australia, either.

In April, the Australian government passed the Defence Trade Controls Act, under which even teaching of cryptography in universities could be classified as a criminal offence.

Peter Eckersley, chief computer scientist at the Electronic Frontier Foundation, describes Australia’s new law as “incredibly bizarre”. The EFF, an international non-profit digital rights organisation, litigated against and defeated American laws similar to the DTCA in the 1990s. When the law comes into effect next year, he says, it will mean that the development of networks such as Tor would be illegal.

“Until now, Tor has been most essential to citizens of countries like Egypt, Iran and Syria, where wildly intrusive internet surveillance and censorship has been the norm,” Eckersley says. “But alarmingly, Australia now seems destined to join that list. With the metadata bill and the censorship copyright bill that just passed, the Abbott government is taking an explicitly anti-internet political position. They see the internet as a dangerous thing.”

Tor’s administrators welcome attempts to crack the network, as they have to keep ahead of legions of hackers, not to mention the NSA. Knowing about breaches makes the network stronger, not weaker. But someday another mechanism for anonymous browsing and communications online will need to replace it, and Tor as it exists now still needs more exit nodes.

A month after Ulbricht’s sentencing, Kilton Public Library in the tiny American town of Lebanon, New Hampshire, population of 13,000, announced it was installing Tor on its computers. It will soon become the first public library in the world to host a Tor node.

In light of the need for more nodes, and especially exit nodes, the decision by the Kilton library to join the fray is significant. Public and commercial institutions don’t need to explain to the authorities that the traffic leaving their nodes is not personal. Hosting exit nodes in public libraries brings the deep web up into the most symbolically public and open of spaces. 

The group behind the push, the Library Freedom Project, see Lebanon’s library as a test case. “Libraries are our most democratic public spaces,” they wrote on the day they announced their partnership. More than 60 national librarians’ associations, including Australia’s and America’s, are signatories to the International Federation of Library Associations’ code of ethics. The code’s opening clause rejects “censorship in all its forms”, and another protects the privacy of all library users.

The story of The Onion Router has layers within layers. If libraries are to be the protectors and champions of Tor, it’s ironic that Ross Ulbricht was arrested in a down-at-heel public library branch in San Francisco, with the customer service page of Silk Road still open on his laptop. If it hadn’t been for a fight between two homeless people in the library – who in fact were undercover FBI agents – he might never have looked away from his desk long enough for another agent to grab his laptop. 

The Kilton library in Lebanon is much more idyllic than the embattled libraries of Silicon Valley. It has a community garden, solar panels and a spillover of doctors from nearby Dartmouth College. And soon, it will have The Onion Router, too.

The battle for privacy online is creating strange new bedfellows. It pits government agencies against each other, provides cover for the best and the worst of us, and makes both drug kingpins and Joe Blow whose credit card is hacked lament the need for better encryption online.