nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

In the red corner: Malware-breeding AI. And in the blue corner: The AI trying to stop it

Behind the scenes of infosec's cat-and-mouse game

By Katyanna Quach, 2 Aug 2017

Feature The magic AI wand has been waved over language translation, and voice and image recognition, and now: computer security.

Antivirus makers want you to believe they are adding artificial intelligence to their products: software that has learned how to catch malware on a device. There are two potential problems with that. Either it's marketing hype and not really AI – or it's true, in which case don't forget that such systems can still be hoodwinked.

It's relatively easy to trick machine-learning models – especially in image recognition. Change a few pixels here and there, and an image of a bus can be warped so that the machine thinks it’s an ostrich. Now take that thought and extend it to so-called next-gen antivirus.

Enter Endgame, a cyber-security biz based in Virginia, USA, which you may recall popped up at DEF CON this year. It has effectively pitted two machine-learning systems against each other: one trained to detect malware in downloaded files, and the other is trained to customize malware so it slips past the aforementioned detector. The aim is to craft software that can manipulate malware into potentially undetectable samples, and then use those variants to improve machine-learning-based scanners, creating a constantly improving antivirus system.

The key thing is recognizing that software classifiers – from image recognition to antivirus – can suck, and that you have to do something about it.

“Machine learning is not a one-stop shop solution for security,” said Hyrum Anderson, principal data scientist and researcher at Endgame. He and his colleagues have teamed up with researchers from the University of Virginia to create this aforementioned cat and mouse game that breeds better and better malware and learns from it.

“When I tell people what I’m trying to do, it raises eyebrows,” Anderson told The Register. “People ask me, ‘You’re trying to do what now?’ But let me explain.”

Generating adversarial examples

A lot of data is required to train machine learning models. It took ImageNet – which contains tens of millions of pictures split into thousands of categories – to boost image recognition models to the performance possible today.

The goal of the antivirus game is to generate adversarial samples to harden future machine learning models against increasingly stealthy malware.

To understand how this works, imagine a software agent learning to play the game Breakout, Anderson says. The classic arcade game is simple. An agent controls a paddle, moving it left or right to hit a ball bouncing back and forth from a brick wall. Every time the ball strikes a brick, it disappears and the agent scores a point. To win the game, the brick wall has to be cleared and the agent has to continuously bat the ball and prevent it from falling to the bottom of the screen.

Endgame’s malware game is somewhat similar, but instead of a ball the bot is dealing with malicious Windows executables. The aim of the game is to fudge the file, changing bytes here and there, in a way so that it hoodwinks an antivirus engine into thinking the harmful file is safe. The poisonous file slips through – like the ball carving a path through the brick wall in Breakout – and the bot gets a point.

It does this by manipulating the contents, and changing the bytes in the malware, but the resulting data must still be executable and fulfill its purpose after it passes through the AV engine. In other words, the malware-generating agent can't output a corrupted executable that slips past the scanner but, due to deformities introduced in the binary to evade detection, it crashes or doesn't work properly when run.

The virus-cooking bot is rewarded for getting working malicious files past the antivirus engine, so over time it learns the best sequence of moves for changing a malicious files in a way that it still functions and yet tricks the AV engine into thinking the file is friendly.

“It’s a much more difficult challenge than tricking image recognition models. The file still has to be able to perform the same function and have the same format. We’re trying to mimic what a real adversary could do if they didn’t have the source code,” says Anderson.

It’s a method of brute force. The agent and the AV engine are trained on 100,000 input malware seeds – after training, 200 malware files are given to the agent to tamper with. These samples were then fed into the AV engine and about 16 per cent of evil files dodged the scanner, we're told. That seems low, but imagine crafting a strain of spyware that is downloaded and run a million times: that turns into 160,000 potentially infected systems to your control. Not bad.

After the antivirus engine model was updated and retrained using those 200 computer-customized files, and it was given another fresh 200 samples churned from the virus-tweaking agent, the evasion rate dropped to half as the scanner got wise to the agent's tricks.

AI is still immature

Although this method is capable of beefing up security, it's still pretty crude, says David Evans, a professor of computer science at the University of Virginia interested in adversarial machine learning. Evans was not directly involved with the research.

Evans said the issue of security in AI “is essential and pressing.”

“We are deploying machine learning models in systems that have serious consequences for humans, and this trend is set to accelerate," he told The Register. "It is easy to be overconfident in these systems based on the impressive results they obtain in laboratory testing, but we need to understand how robust the systems are when adversaries try to fool them in deployment, and we need to learn methods for making AI systems that are more robust and trustworthy.

“Deep learning, in particular, is an area where the experimental results have gotten way ahead of our understanding of how things work, and this especially matters in contexts where an adversary may be motivated to make them misbehave.”

The researchers from Endgame and the University of Virginia are hoping that by integrating the malware-generating system into OpenAI’s Gym platform, more developers will help sniff out more adversarial examples to improve machine-learning virus classifiers.

Although Evans believes that Endgame's research is important, using such a method to beef up security “reflects the immaturity” of AI and infosec. “It’s mostly experimental and the effectiveness of defenses is mostly judged against particular known attacks, but doesn’t say much about whether it can work against newly discovered attacks," he said.

“Moving forward, we need more work on testing machine learning systems, reasoning about their robustness, and developing general methods for hardening classifiers that are not limited to defending against particular attacks. More broadly, we need ways to measure and build trustworthiness in AI systems.”

The research has been summarized as a paper, here if you want to check it out in more detail, or see the upstart's code on Github. ®

The Register - Independent news and views for the tech community. Part of Situation Publishing