Artificial intelligence firm DeepMind has tackled games like Go and Starcraft, but now it is turning its attention to more sober affairs: how to solve school-level maths problems.
Researchers at the company tasked an AI with teaching itself to solve arithmetic, algebra and probability problems, among others. It didn’t do a very good job: when the neural network was tested on a maths exam taken by 16-year-olds in the UK, it got just 14 out of 40 questions correct, or the equivalent of an E grade.
There were also strange quirks in the AI’s ability. For example, it could successfully add up 1+1+1+1+1+1 to make 6, but failed when an extra 1 was added. On the other hand, it gave the correct answer for longer sequences and much bigger numbers.
Other oddities included the ability to correctly answer 68 to the question “calculate 17×4.”, but when the full stop was removed, the answer came out at 69.
Puzzling behaviour
The DeepMind researchers concede they don’t have a good explanation for this behaviour. “At the moment, learning systems like neural networks are quite bad at doing ‘algebraic reasoning’,” says David Saxton, one of the team behind the work.
Despite this, it is still worth trying to teach a machine to solve maths problems, says Marcus du Sautoy, a mathematician at the University of Oxford.
“There are already algorithms out there to do these problems much faster, much better than machine-learning algorithms, but that’s not the point,” says du Sautoy. “They are setting themselves a different target – we want to start from nothing, by being told whether you got that one wrong, that one right, whether it can build up how to do this itself. Which is fascinating.”
An AI capable of solving advanced mathematics problems could put him out of a job, says du Sautoy. “That’s my fear. It may not take too much for an AI to get maturity in this world, whereas a maturity in the musical or visual or language world might be much harder for it. So I do think my subject is vulnerable.”
However, he takes some comfort that machine learning’s general weakness in remaining coherent over a long form – such as a novel, rather than a poem – will keep mathematicians safe for now. Creating mathematical proofs, rather than solving maths problems for 16-year-olds, will be difficult for machines, he says.
Noel Sharkey at the University of Sheffield, UK, says the research is more about finding the limits of machine-learning techniques, rather than promoting advancements in mathematics.
The interesting thing, he says, will be to see how the neural networks can adapt to challenges outside of those they were trained on. “The big question is to ask how well they can generalise to novel examples that were not in the training set. This has the potential to demonstrate formal limits to what this type of learning is capable of.”
Saxton says training a neural network on maths problems could help provide AI with reasoning skills for other applications.
“Humans are good at maths, but they are using general reasoning skills that current artificial learning systems don’t possess,” he says. “If we can develop models that are good at solving these problems, then these models would likely be using general skills that would be good at solving other hard problems in AI as well.”
He hopes the work could make a small contribution towards more general mathematical AIs that could tackle things such as proving theorems.
The DeepMind team has published its data set of maths questions, and encouraged people to train their own AI.
For more such insights, log into www.international-maths-challenge.com.
*Credit for article given to Adam Vaughan*