Mainframe TITSUP totals Oz tax tech, again

In a speech at the National Press Club yesterday, Australian Taxation Office commissioner Chris Jordan apologised for the organisation’s repeated IT outages, while the ATO’s online portals hit the canvas again.

The ATO says it addressed the five-hour TITSUP (total inability to support usual performance) by rebooting its mainframe (something that has Australia’s tech Twitterati scratching their heads), and has told the Australian Broadcasting Corporation “things could have been worse”.

An ATO statement said there were “intermittent issues ... affecting our mainframe and impacting on our services to the community. This was caused by applications running incorrectly.

“We took controlled action to reboot our mainframe and resolve this issue. We then brought our services back online methodically to ensure system availability and stability.”

The office has launched a post-incident review into what happened, which acting COO Frances Cawthra says should “ensure it doesn’t happen again”.

Or, at least, try to prevent it happening again. Cawthra also said: “These are very large, very complex systems and in today’s world it’s just not a guarantee we can give, but we do have every confidence we are well on our way to a very good tax time”.

IT has been a persistent pest for the ATO, with major outages in December 2016 and February 2017 eventually traced to HPE 3PAR SANs, installed in 2015 to replace the previous EMC storage network.

A review of what went wrong, published in June, found the SANs were configured for speed rather than resilience, and that the SAN recovery software was on the same network it was protecting.

The February outage was a simple matter of cable failure, and because while replacing cables someone managed to dislodge some of the SAN’s line cards.

And in April, the site got ankle-tapped again, and responded by taking a dive. At the time, the ATO told us it wasn’t related to the Easter weekend project to finish fixing the SAN (but didn’t explain what had happened). ®