Headlines like “Robots can now read better than humans, putting millions of jobs at risk” and “Robots are better at reading than humans” have flooded the internet recently, leaving many of us wondering: well, are they better at reading, going to take away my job or even take over the world? And if so, what would that mean for me or for my child?
The Artificial Intelligence (AI) at the center of this frenzied discussion are Microsoft- and Alibaba-built computer models, both of which recently beat human test-takers on the Stanford University Question Answering Database (SQuAD), with scores of 82.65 for Microsoft and 82.44 for Alibaba (humans came in at 82.30).
The test is comprised of more than 10,000 question-answer pairs. Human and machine test-takers respond to questions based on information provided in short paragraphs sourced from over 500 Wikipedia articles.
Only a year ago, Microsoft’s previous model, SLQA+, scored a 79.199. While computer models are now able to analyze data more effectively and efficiently than human participants, are they better than humans at reading? According to Yoav Goldberg, interviewed by the Verge’s James Vincent, the answer is: no. It is important to keep in mind, states Goldberg, who specializes in natural language processing, that the SQuAD does not correspond to conventional understanding of a reading comprehension test.
“Reading comprehension” conjures up visions of analyzing poems about blue curtains with metaphorical intent, alliterations, allegories, similes, and words like “serendipitous,” which inevitably show up on the vocabulary sections. In short, we tend to associate reading comprehension with reading and understanding. It’s the latter component of our operative definition—understanding— that SQuAD does not measure.
As Goldberg states, even elementary school administered reading comprehension tests are more difficult than the SQuAD because they contain questions such as, ‘Why did x do this?’ Although computer models are now able to effectively outperform humans in data analysis, to answer, “Why did x do this?” one must first understand what x is, and AI does not (yet) have that understanding.
Microsoft and Alibaba have made strides in AI development, allowing for faster and more effective data extraction, but will computer models ever understand language the way humans do? It is not yet clear.
In the meantime, when you find yourself in a siege-like situation, defending against an onslaught of whys led by a curious, exuberant child, take comfort. Your ability to answer those questions is what separates you from the machine.