Tuesday, 22 May 2018

AI Reading comprehension scores

In January 2018 there was a great many blog posts about AI surpassing human reading comprehension scores.
Humans comprehension of a piece of text is 82.304% (Human Performance Stanford University, Rajpurkar et al. '16)

As of March 19 2019, AI is now hitting scores of 83.877, by QANet (ensemble) Google Brain & CMU.

Plotting the scores on a graph:
 

Extrapolating the scores, we should see an AI hit 100% sometime in 2020.
Spreadsheet I created for the graph above: AI-Comprehension-score.xlsx

Original data source: https://rajpurkar.github.io/SQuAD-explorer/