After ChatGPT—an online program that generates essays on demand—arrived on the scene a few months ago, it inspired predictions that teachers and writers are all doomed. Here was a program that could, and would, replace all of us.
Students at all levels would inevitably turn to ChatGPT for their schoolwork. Cheating would be rampant. “AI has basically ruined homework,” tweeted a business professor at Wharton. “Education will never the be same,” moaned an article in RealClearEducation. A technologist named Peter Wang went even further: “College as we know it will cease to exist.”
We’ve seen such hype before. In the early 2000s, MOOCs (Massive Open Online Courses) arrived, and very quickly, they were promoted the same way.
Thomas L. Friedman, for example, gushed, “Nothing has more potential to unlock a billion more brains to solve the world’s biggest problems. And nothing has more potential to enable us to reimagine higher education than the massive open online course, or MOOC, platforms that are being developed by the likes of Stanford and the Massachusetts Institute of Technology and companies like Coursera and Udacity.”
But MOOCs flopped. Hardly anyone completed the free courses, and those who did complete the course often in the end failed. Sebastian Thrun, Stanford professor and founder of Udacity, called MOOCS “a lousy product.” Today, one never hears about them. Cryptocurrency seems to be heading the same way.
So, what about ChatGPT? Is this program yet another example of tech utopianism? Will it really change education completely? To test this hypothesis, I logged on to ChatGPT, created an account, and gave the program several prompts to find out what it could do, and what it could not do.
Let me begin with the positive. I asked ChatGPT “tell me about the iguana. 1000 words,” and I quickly got a halfway decent essay that rehearses a bunch of facts about this lizard. Not terrible, but no organization, although the writing was grammatically correct.
Here’s the ending: “In conclusion, iguanas are a fascinating group of lizards that are known for their distinctive appearance, ability to swim, and adaptability to a variety of habitats. They are popular pets but require a lot of space and a special diet. They are also threatened by habitat loss and hunting, and some species are considered endangered. They have unique behavior, and their body structure is adapted to their habitat and lifestyle.”
Note that the essay seems to start over again in order to fulfill the word requirement. So let’s be generous, and give this essay a “B-.”
But once I asked more specific questions requiring more specialized knowledge, and which required evidence to back up the argument, ChatGPT utterly and totally failed.
The first prompt was “Tell me about the relationship between Shakespeare’s The Taming of the Shrew and the Homily on Matrimony.” The latter text is a sermon about marriage that is really a warning against domestic violence.
While the Homilist tells the bride to obey her husband, the Homilist also condemns toxic masculinity, and tells the husband that under no circumstances is he allowed to beat his wife or his servants. So there’s an obvious connection between this sermon and the play.
Yet according to ChatGPT, “While there may be some similarities in the ideas presented in the two works, they are not directly related. The Homily on Matrimony is a religious text and The Taming of the Shrew is a work of fiction.” Well, yes, the two belong to different genres. But the point is to see how one part of the culture influences the other. That ChatGPT cannot do.
Even worse, the ChatGPT essay is devoid of quotations from Shakespeare’s play. We get the assertion, “The Taming of the Shrew is a comedy that satirizes the patriarchal society of Shakespeare’s time,” but no examples of how the comedy satirizes Shakespeare’s patriarchal society. So, this essay would get an “F” for that reason alone.
I tried another prompt. Shakespeare wrote A Midsummer Night’s Dream in 1596 in the middle of the Crisis of the 1590s, a time when severe weather ruined crops across Europe, resulting in death, starvation, class warfare, and rebellions. The fairy queen, Titania, directly alludes to the Crisis when she talks about how “The ox hath therefore stretched his yoke in vain, / The plowman lost his sweat, and the green corn / Hath rotted ere his youth attained a beard.” But when I asked ChatGPT for an essay on MND and the Crisis, I got this nonsense:
“The play reflects the crisis of the 1590s in several ways. The play’s portrayal of the struggle for power among the fairy rulers and the Duke of Athens mirrors the political struggles of the time. The play also reflects the religious conflicts of the time, as the fairy world is depicted as a place where the traditional religious beliefs and practices are turned upside down. The play also reflects the rise of the new social class, as the mechanicals, a group of working-class characters, are depicted as being at odds with the aristocracy.”
Notice, again, the lack of quotations. Really, the lack of any evidence of any kind. Instead, the essay is a tissue of assertions without anything to back them up. Another “F.”
A friend gave ChatGPT this prompt, “Homer is obsessed by body parts. Discuss,” which yielded this result: “I am not aware of Homer being obsessed with body parts. Homer is a fictional character from ancient Greek literature, specifically from the epic poem the “Illiad” and the “Odyssey” written by Homer. These works do not depict Homer as being obsessed with body parts. If you are referring to another Homer, please provide more context.”
Not only wrong, but nonsensical (Homer is a fictional character in the poems written by Homer?). Yet one more “F.”
Clearly, ChatGPT cannot do literary analysis, so I decided to change direction and ask about history: “Examine the relationship between the English and American Revolutions. Give specific examples.” The resulting “essay” was laughably bad because it lacked a basic sense of chronology: “One major similarity between the two revolutions is the influence of Enlightenment ideas on their respective ideologies.”
The Enlightenment occurred in the eighteenth century; the English Revolution, on the other hand, occurred in the seventeenth century. Even more ridiculous: “the English Revolution was heavily influenced by the works of philosophers like John Stuart Mill, who wrote about the importance of democracy and individual rights.” Mill lived from 1806-1873, the nineteenth century, so it would be quite a trick for his writing to have influenced John Milton (1608-1674) or Oliver Cromwell (1599-1658). If an essay could earn a grade lower than F, this one would get it.
There’s no doubt that ChatGPT can generate a mediocre essay on a general topic, such as the iguana. There’s also no doubt that the prose is, truth be told, on average a lot better than what I often get in my classes.
But ChatGPT is (at the moment, at least) incapable of any sort of higher analysis. It cannot supply evidence from a text to support its assertions, and it certainly cannot check basic chronology.
More importantly, there is no intelligence in artificial intelligence. ChatGPT cannot think. There are no insights in these essays, no moments when the essay shows something surprising in the text. The program cannot take two seemingly unrelated data points and connect them. Or in literary terms, connect text and context.
Ask it to write on something the least bit unconventional, and the program is graveled. All ChatGPT can do is survey what’s already available on the internet and rearrange it. But ask ChatGPT to go beyond what’s on Wikipedia, and it flops. Badly.
Perhaps the program will learn to think creatively, but if that happens, it will be long in the future. At present, will ChatGPT be the revolutionary technology that “disrupts” education and replaces teachers the way cars replaced the buggy? Will it be the end of homework? Will it make authors, reporters, and writers of all kinds obsolete?
The answer is “no.”
Peter C. Herman is professor of English literature at San Diego State University. He has published on Shakespeare, Milton and the literature of terrorism, and has published essays in Salon, Inside Higher Ed, as well as Times of San Diego. His most recent book is “Unspeakable: Literature and Terrorism from the Gunpowder Plot to 9/11” (Routledge, 2020).