The most frightening thing about the Centrelink malware debacle is the verve with which the government embraced it.
Malware is software designed to do damage under the cover of providing a service.
More National News Videos
- Video duration
- 01:57
Government's robo debt recovery
Government's robo debt recovery
The machine relied on by the government to recover welfare overpayments has actually misapplied the law. The Age's Economics Editor Peter Martin explains.
Up Next
How the Stoccos were captured
- Video duration
- 02:19
- Video duration
- 02:19
How the Stoccos were captured
How the Stoccos were captured
This is how Channel Ten reported the capture of Gino and Mark Stocco on October 28th 2015 after eight years on the run.
Up Next
Largest haul of illegal abalone in 20 years
- Video duration
- 00:30
- Video duration
- 00:30
Largest haul of illegal abalone in 20 ...
Largest haul of illegal abalone in 20 years
The Department of Primary Industries, Fisheries and the NSW Police have seized kilograms of illegal shark fin and abalone.
Up Next
Turnbull: Melbourne's 'concerning vulnerability'
- Video duration
- 01:52
- Video duration
- 01:52
Turnbull: Melbourne's 'concerning ...
Turnbull: Melbourne's 'concerning vulnerability'
Prime Minister calls for access to busy pedestrian areas to be tightened up in Melbourne, suggesting more bollards could help prevent future incidents like the Bourke Street rampage.
Up Next
Australia's Day off
- Video duration
- 01:41
- Video duration
- 01:41
Australia's Day off
Australia's Day off
With the national holiday on a Thursday this year, how many people are taking a sickie today for a long weekend?
Up Next
Two dead after plane crashes into Swan River
- Video duration
- 00:32
- Video duration
- 00:32
Two dead after plane crashes into Swan ...
Two dead after plane crashes into Swan River
Two people have died after a plane crashed into Perth's Swan River in front of thousands of people gathered for the Australia Day Skyworks display. Vision courtesy Seven News.
Up Next
What's Australia Day like for an Aboriginal community?
- Video duration
- 03:39
- Video duration
- 03:39
What's Australia Day like for an ...
What's Australia Day like for an Aboriginal community?
Aboriginal people lived on the land now known as Lake Tyers, also known as Bung Yarnda, for at least 50,000 years before the campaign to dispossess and eradicate them began. Here, like many other aboriginal communities, January 26 is nothing to celebrate.
Up Next
What makes a great teacher?
- Video duration
- 02:02
- Video duration
- 02:02
What makes a great teacher?
What makes a great teacher?
Everyone has met one, but what does it really take to provide the perfect learning environment? We asked an expert.
Government's robo debt recovery
The machine relied on by the government to recover welfare overpayments has actually misapplied the law. The Age's Economics Editor Peter Martin explains.
Unveiling the automated Centrelink debt recovery system mid-year treasurer Scott Morrison and social services minister Christian Porter promised more "accurate and appropriate income testing". They were going to work with the prime minister's Digital Transformation Office to "cut red tape and ensure that mistakes are minimised".
It's theoretically possible for machines to do complex things better than humans. These days chess-playing programs do it (at least they do it to me) but those programs are exquisitely designed and have goals that are properly specified.
Related Content
What Morrison and Porter promised was an automated system that would issue Centrelink debt notices "better" than human beings.
Humans did the job extremely well. A former Centrelink worker with 30 years experience says they would "look at start dates for employment that customers had declared, see if it was the same for the employer [using Tax Office records] and roughly work out if it lined up."
"If it looked as if a person had possibly been overpaid they would write to the customer and ask them to call and tease out where the discrepancy was, and ask for proof, if it was still available, in the form of things such as payslips. If the customer didn't have them and it looked like there was a possibility of an overpayment, they would write to the employer to ask for the information. If evidence was collected that the customer had not declared the income correctly and a debt existed, then the debt calculator would raise the debt in accordance with the legislation and the customer would be written to."
What's important in this description is the humans charged with applying the law didn't issue debt notices unless they had evidence that a debt existed. To do so without evidence would be to break the law.
But a wrongly-programmed computer need have no such scruples. Even better, its decisions can be presented as objective, hard to overturn. Data scientist Cathy O'Neil outlines scores of examples in her new book Weapons of Math Destruction, from the systems used by credit rating agencies in the lead up to the global financial crisis, to systems that automatically select teachers for the sack on the basis of secret algorithms that grade performance, to systems that deny people job interviews on the basis of proxies for mental health, even though that's illegal.
They are used because they are quick rather than accurate. As an expert told O'Neil, the primary purpose of a workplace hiring system is "not to find the best employee, but to exclude as many people as possible as cheaply as possible".
By necessity, they do it unfairly. People who are wise to the systems will mention the right words in job applications to get to the top of the pile. As she says, they are usually not from racial and ethnic minorities. Here, it's the persistent and well-resourced people who get the better of Centrelink. They are unlikely to be the hardest up.
Many of the automated systems are malicious, created to do harm in the guise of providing a service. The formula used by Centrelink produces consistently false estimates of debts by dividing by 26 the annual wages employers report paying in order to overestimate income received during the smaller number of fortnights claimants get benefits.
Humans didn’t issue debt notices unless they had evidence a debt existed. To do so without evidence would be to break the law.
The formulas used by for-profit colleges in the US to target internet advertising zero in on single mothers of colour who are poor enough to earn the colleges' valuable subsidies and ill-informed enough not to twig to debt. The formulas O'Neil herself worked on in financial markets presented securities as safe that weren't.
O'Neil says that to be a "weapon of math destruction" a formula has to be used en masse (as the Centrelink formula will be), it has to be difficult to question (as the Centrelink formula will be for many people) and it has to cause damage (as the Centrelink formula is doing).
That isn't to say that credit risk and Centrelink and other software can't be designed to do the job better than humans. It's a worthy aim, one Morrison and Porter apparently thought they had achieved.
The man Turnbull hired to prevent such stuff-ups describes what happened as as "cataclysmic". Paul Shetler left the prime minister's Digital Transformation Office in November as the Centrelink debt collection program gathered pace.
185 comments
Comment are now closed