ChatGPT in exams: does ABCD questions well, makes up scientific resources elsewhere

Already University of Bath looked at the issue of using artificial intelligence in the studio. Today, students often use it to write essays, so it was interesting to see what it can actually do in different types of tasks. So the question was whether it could be used by students to answer questions in tests. ChatGPT proved to be a very good system for ABCD multiple choice questions. The researchers themselves didn’t expect the AI ​​to do as well as it did. Her success rate was very close to 100%. Unfortunately, it was not so good in more complex questions. If critical thinking is to be engaged, this is a problem.

When the system was asked to write an essay on why it is important to understand the timing of exercise in relation to the current satiety of overweight people, the response was insufficient. The text itself was not written badly, but the scientists did not like, for example, the fact that he often repeated the question itself in the text. Personally, I don’t think this is so bad, because the writer draws attention to what his essay is actually trying to answer, but it is true that it does not have to be repeated verbatim. But some parts of the paragraphs were too simplistic, presenting conclusions but without evidence or a more detailed description of “why”. But the worst part was citing sources. Authors existed, as well as magazines where articles could be published. But there were no listed scientific articles that AI presented, and thus no page numbers where the alleged article was supposed to be located. The AI ​​just made them up.

ChatGPT continues to show that it cannot produce factually correct essays based on information, however it is usually good at answering simple clear questions (especially when given a choice of answers) and is also very good at overcoming creative blocks and creating a basic structure of texts . It can very well help students to “get started”. Further addition of texts and factual correctness is a task primarily for humans and not AI.