Can a robot pass a university entrance exam?

Earlier this year, we discussed whether robots were better at reading than humans

In that blog post, we learned that, according to Bar Ilan University lecturer Yoav Goldberg, it is mistaken to think of computers as “reading” in the sense associated with reading comprehension tests (such as the ones included on the SAT, ACT, and SAT Subject Test in Literature).

Robots search and optimize, which allows them to scan vast amounts of information for valuable data, but they lack understanding.

Our ability to understand and interpret information separates us from the machine (for now).

But do you need understanding in order to gain entrance to a competitive post-secondary institution?

Perhaps not.

In a TED talk given last year, researcher Noriko Arai introduced the Todai Robot.

Todai’s task? Pass the entrance exam to Japan’s most prestigious post-secondary institution, the University of Tokyo.

University of Tokyo’s entrance criteria is intense.

Students must score at least an 85 per cent on the following:

  • Mathematics
  • Two Natural Sciences
  • Two Social Sciences
  • Japanese
  • English

After passing the first stage, applicants may move on to phase two: the written exam.

Students must complete a written exam in the following:

  • Mathematics
  • Two Natural Sciences OR Two Social Sciences
  • Japanese
  • English

So, how did Todai do?

Arai states that Todai scored “among the top one per cent in the second stage written exam in mathematics” and surprisingly, wrote a better 600-word essay than most students.

Todai wrote its essay on a prompt similar to this one:

Discuss the rise and fall of the maritime trade in East and Southeast Asia in the 17th century.

Todai compiled information from various textbooks and Wikipedia, effectively mashing the pieces together into a cohesive essay.

Without understanding a thing, Todai outperformed most students.

So, did Todai enter the University of Tokyo?

No. Although Todai was in the top 20 per cent of all exam-takers in this particular format of the exam and was able to gain entrance to more than 60 per cent of universities in Japan, Todai didn’t make it into the University of Tokyo.

Todai made crucial mistakes on questions that required understanding (The question Arai uses as an example in her TED talk is from an English reading comprehension test.)

You would think that Arai is pleased that her robot performed so well.

On the contrary, the researcher is alarmed.

How is it, Arai says, that an unintelligent machine was able to outperform so many students?

Arai states that we have been operating under the mistaken, and perhaps dangerous, assumption that “everyone can learn and learn well” if they have access to free resources and information on the web.

But as it turns out,  the percentage of those of us who can read well may be “much less than expected,” Arai warns.

In the TED talk, the researcher shares a junior-high level test question to illustrate her point.

The question contained the following:

Buddhism spread to…, Christianity to…and Oceania, and Islam to…

and asked students to fill in this blank:

______ has spread to Oceania.

given the following options:

1. Hinduism

2. Christianity

3. Islam

4. Buddhism 

The answer to the fill-in-the-blank is obviously “Christianity,” and Todai answered as such.

But a third of junior high school students got the question wrong.

This alarming finding lead Arai to an even more alarming conclusion.

Humans understand meaning, which is something that artificial intelligence (AI) cannot do as of yet. But most students spend their time memorizing, or as Arai puts it, “packing knowledge without understanding the meaning of the knowledge.”

But packing knowledge without understanding is equivalent to what Todai does when searching and optimizing pieces of information, and as we have learned, Todai can search and optimize better than humans can.

AI outperforms us in memorization. Maybe we all (students and educators) ought to focus on meaning.

VIDEO: Future Career Trends for the 21st Century | Ivy Global Seminar

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.