At what ought to have been the height of preparations for the 2016 census, its head, Duncan Young, sent his colleagues in the Bureau of Statistics executive a crisis memo.
"As most of you are aware, the census program has alerted its steering committee and board that it has assessed its status as RED," he wrote in February.
More News Videos
- Video duration
- 00:58
Census heads front the media
- Video duration
- 01:33
Sick journalist detained for appearing high
- Video duration
- 01:24
Protesters served with eviction notices
- Video duration
- 00:59
Denis Ryan- the moral cop
- Video duration
- 00:26
Baby found unconscious in bath
- Video duration
- 01:36
Was the census attack Australian?
- Video duration
- 02:14
Not a political decision: Treasurer
- Video duration
- 01:08
Ausgrid sale vetoed
Census heads front the media
The census website is back online after a two-day outage but the Australian Bureau of Statistics' top brass were not taking questions. Vision courtesy ABC News 24
He put the word "RED" in capital letters.
"This means that we have assessed that the program will not be able to deliver on the current scope, timetable and/or budget. This status is as a consequence of both budget reductions since program commencement and program delays during 2014."
Young set out five options to shrink the census or save money, each "not taken lightly". One was to ask far fewer questions, another was to reach fewer people.
At the same time the Bureau's newly installed chief, David Kalisch, was war-gaming an even grander solution. Dubbed "Project Archer" after Keith Archer, the Australian Statistician who introduced computers to the ABS in the 1960s, it would make the 2016 census go away, freeing up $200 million to $400 million to upgrade the Bureau's aging computer systems, some of which ran code that was 30 years old.
His predecessor Brian Pink had left at the start of 2014, warning in his final annual report that the Bureau had barely enough cash to "keep the lights on".
When Pink arrived in 2007 the Bureau received $302 million in a non-census year. Seven years of growing expenses and relentless "efficiency dividends" later, it received scarcely any more, $312 million.
Pink had responded by axing or indefinitely postponing some of Australia's most loved surveys. One was the national time-use survey which records how Australians spend every 15 minutes. It hasn't been updated since 2006, before the arrival of the iPhone.
Kalisch was interviewed for the $705,030 job at the start of 2014 but wasn't appointed until December, leaving the Bureau without a leader during a year in which it was meant to be fine-tuning Australia's first predominantly digital census.
Tony Abbott and his treasurer Joe Hockey dithered because they didn't like the look of the candidate the selection committee had first recommended. Project Archer would deliver the census once every 10 years instead of five, directing the savings to buy new computers.
To sell the idea, Kalisch had to diss the census. "There is a lot of, perhaps, misinformation about the value of census," he told a Senate hearing. "There is a sense in the community that a lot of the information is derived from the census, which is just not true."
The primary purpose of the census is indeed prosaic â it is to count the number of people aged 18 and over in order to determine the shape of electorates. It's why politicians are particularly keen on keeping it, and why cancelling it was always a big ask. The government said no, and gave the Bureau an extra $235 million over five years in order to upgrade its computer system.
But the census itself had to be cut-price, costing more like $200 million than the previous $300 million to $400 million. Work on the questions as good as stopped.
Every five years there's an additional special-interest question. It's incredibly valuable real estate, fought over in the same way as the payload on a mission to space.
In 2001 as Peter Costello was gearing up to replace John Howard as Australia's prime minister, he gave a speech about the "spirit of the volunteer" in an attempt to humanise himself.
Who's responsible for the census hack?
There are a few different theories as to who is behind the disruption on census night. Fairfax David Wroe explains the possibilities.
As treasurer, he instructed the ABS to make the 2006 special question about volunteering, even though it already collected more detailed statistics on volunteering in another survey.
The Bureau reluctantly complied, and then when the Rudd government left it short of funds in the lead up to the 2011 census, left it in because it didn't have the money to devote to framing another question. By the lead up to this census it was short of money again and desperately short of time. It left in the ill-defined question for the third consecutive census.
Directed to actually conduct the census, and keen to extract some value from it, Kalisch and his team revived an idea categorically ruled out by his predecessor. Pink had said no to retaining names.
"It wasn't going to happen. I can tell you that," Pink said this week.
"I always used to say to my people: you can't kill the goose that lays the golden egg, and the golden egg is the census. In my view, you only need 20 per cent of Australians who are concerned about security and you put the census at risk."
When given the option of having their names and forms retained and stored in the archives for release a century later instead of being destroyed after processing, 39 per cent of Australians had said no. The immediate use of their names might have alarmed them more.
Names had always been retained for a short time in order to eliminate duplicates and establish the relationship between household members, but destroyed after checks, usually well before 18 months.
On October 19, Kalisch convened a meeting of his executive group. It agreed to conduct a privacy impact assessment into the permanent retention of names as well as exact addresses, which had also previously been destroyed after checking.
Whereas in Pink's day the privacy impact assessment had been conducted externally, and had savaged the proposal, this one would be conducted in-house "consistent with our practice with data integration projects and leveraging the experience and knowledge we have built since 2005".
Its publication along with a half-hearted endorsement from focus groups conducted by Colmar Brunton Social Research would be timed "to quickly follow" the release of the regular Trust in ABS survey which always produced impressive results.
Appearing before the Senate economics committee two days later, Kalisch said nothing about the plan to retain names and addresses. He made a short statement to "update the committee about our census preparations". Things were "on track" and momentum was building. A few months earlier he he had told the committee things were coming along "beautifully".
On December 8 the executive group considered the proposal in more detail. The ABS had published a statement of intent on its website on November 17, unreported in the mainstream press, and received just three responses, all negative, from what it termed "concerned private citizens".
The internal privacy impact assessment had given it a tick. The Bureau had sent a minute to the office of the assistant minister to the treasurer Alex Hawke, appointed a few weeks earlier by the new Prime Minister Malcolm Turnbull. A report prepared for the meeting said it had been "noted".
The Bureau wanted to build a reputation as Australia's "premier integrator of government data". If it couldn't retain names and addresses, potential users might see it as "unnecessarily constraining itself and therefore constraining whole-of-government data integration".
"There are many administrative datasets that are likely to have considerable statistical value," the report said. "In addition to the personal income tax data which has already been used in data integration projects, future data integration projects could include the use of welfare payments data, Centrelink unemployment benefits data, Medicare and Pharmaceutical Benefits Scheme data, Australian Immunisation Register, the electoral roll, and other nationally important datasets."
The report envisioned no limit on what the ABS could link and charge for, so long as the names and addresses themselves were kept within the ABS. Information from the census on ethnic or religious backgrounds could be linked to information from the immunisation register to work out what type of families on what types of incomes were the least likely to immunise.
Criminal records could be linked to census records, if permission were given, to see what sort of Australians were convicted of what sort of crimes.
Until that point the Bureau had mainly relied on "bronze" linkage â the rough linking of files using identifiers other than names and addresses. A move to "gold" linkage using names and addresses would get "maximum value for what is already one of the most valuable statistical assets the ABS holds".
On the Friday before Christmas, the Bureau released an eight-paragraph statement deceptively titled ABS response to Privacy Impact Assessment. Once more unreported in the mainstream media, it said the Bureau would retain the names and addresses collected in the census "to provide a richer and dynamic statistical picture of Australia through the combination of census data with other survey and administrative data".
In April, with the census imminent, after reports in Crikey and the Australian Financial Review, Kalisch backed down somewhat. Names would be kept for only four years, but the really useful linkage keys derived from them would still be kept indefinitely.
One of Kalisch's predecessors, Bill McLennan described what was planned as "without doubt, the most significant invasion of privacy ever perpetrated on Australians by the ABS".
"I am appalled that the ABS can think it can use the threat of prosecution to make me provide data that allows the ABS to set up what is, in effect, a statistical Australian Card," he wrote.
As it moved to counter declarations by Crikey reporter Bernard Keane and high-profile politicians including Nick Xenophon that they would either not complete the census or not provide their names, the Bureau emphasised the $180 per day fines. They applied for each day the forms weren't complete, without limit. Although the Bureau was also careful to point out that they applied only after September 23, the main message received was that the forms had to be completed on census night itself, August 9.
By 7.30pm, as millions of Australians tried to get online at once amid what may have been denial of service attacks, the system crashed and was taken down. It had been built by IBM for $9.6 million and load-tested by Revolution IT for $469,000. ABS robots, set up to automatically respond to tweets, encouraged Australians to continue to try to log on.
Kalisch had said just the day before the Bureau was "ready" with the best security features for which "you could ever ask".
The new minister, Michael McCormack, in the job for mere weeks, at first couldn't get through to Kalisch. McCormack had been appointed after an embarrassing interlude in which there seemed to be no minister responsible. Hawke's position had been abolished and neither treasurer Scott Morrison nor financial services minister Kelly O'Dwyer had been given the job. A fortnight after being appointed small business minister McCormack was told it was his.
Before the website went back online late Thursday Prime Minister Malcolm Turnbull promised an inquiry after which he said "heads would roll".
45 comments
New User? Sign up