Can ChatGPT write high quality essays?
You’ve probably heard and read a lot about ChatGPT over the last few days. Microsoft even took the stage (literally) and announced their ChatGPT integration into Bing. I’ve had the opportunity to play around with ChatGPT 3.0 and the Bing version, and I asked it to do my students’ homework because I wanted to see whether ChatGPT was able to write a quality literary essay.
If you want to know what a literary essay is, read this article on the topic.
Asking the right questions
AI software can do a lot these days, but it all comes down to asking the right question and giving it the correct instructions. This is particularly important for Bing, as Microsoft has limited the number of prompts it can handle per conversation.
When working in ChatGPT, I wanted to see how much it could do without me altering the question, and I also wanted to see how much of the previous work it would use when I changed the question.
First question: write an essay
My first question was quite simple: “Write a literary essay on the relevance of the flowers in Flowers for Algernon.” This prompt was relatively easy, but it does require some digging around on the internet as the book is quite old.
The result: an adequately structured 281-word essay with well-structured paragraphs and clear argumentation.
ChatGPT gave the correct answer to the question and provided arguments that actually fit its thesis quite well.
What does Grammarly say?
I became interested in the language level and used Grammarly to get some metrics on the essay. There are some basic metrics that are interesting to analyse regarding language level and complexity: word length, sentence length, and vocabulary.
Grammarly rated the readability of this essay as “above average”, with an average word length of 4.8 characters and an average sentence length of 23.4 words per sentence. This article you’re reading now, in comparison, has an average of 4.7 characters per word and 17.3 words per sentence. Vocabulary wasn’t anything to write home about, as the number of unique words and the number of rare words were above average.
Enhancing the essay
I found 281 words short, so I wanted to know what it would do if I asked it to write a 500-word essay. I posed the same question but included the 500-word requirement. The 500-word version included some extra explanations in the body paragraphs (more Level 2 sentences), and it added a short summary of the story to the introduction.
Our Grammarly metrics changed slightly: the essay still had an average of 4.8 characters per word, but the average sentence length decreased slightly to 22.6 words per sentence.
Analysing the essay as an English teacher
One of the things that stands out in both ChatGPT essays is the fact that it’s able to write long, complex sentences. That’s quite impressive, as one of the ways GPT-detective.nl was able to recognise [Dutch article] that something came from ChatGPT was by analysing sentence length, complexity, and the lack of descriptive language.
Reading it as an English teacher, I found that the essay follows the perfect structure of an essay. The introduction is concise, to the point, and follows the funnel shape, while the paragraphs have a good topic sentence and include nice linkers. The essay did lack examples from the book to support the claims made, and after adjusting my prompt to include those as well, one of the examples given didn’t really fit the argument properly.
How ChatGPT gave itself away
One thing that really stood out made me spot that this is a ChatGPT essay: it repeats the same constructions. This is something that I’ve seen in other uses of ChatGPT as well. When I asked it to formulate 50 passive sentences, it came up with the same basic constructions in the same grammatical tense. That happens throughout this essay too. Instead of varying sentence constructions, it repeatedly sticks to the same construction. This is what GPT-detective.nl means with ‘descriptive language’, as good writers can vary their language and thus say the same thing but in a different way.
Conclusion: can ChatGPT write good essays?
The answer to this question is: yes, it can write proper essays. Would I recommend writing your essays in ChatGPT? Obviously not. I do see that there’s something here and that ChatGPT can be used for writers in several ways. It can create blueprints or do research for articles you’re writing and even do some research for literary analyses.
There’s a lot to discover regarding AI and Bing, and there are still caveats that we need to encounter. Would you use ChatGPT for essay writing?
