Interview: How ChatGPT is changing teaching and assessments

Gerd Kortemeyer, Manuel Sudau: ChatGPT is raising a lot of questions for teaching staff right now. What are they?
Gerd Kortemeyer: Teaching staff are suddenly being confronted with machine-​generated texts they can’t necessarily distinguish from human-​written texts. Simple writing exercises like short essays and summaries are basically obsolete now. We’re being confronted with questions like: “How do you block ChatGPT?” and “How can I recognise AI-​generated text?”. I completely understand these concerns. The discussion reminds me of when calculators came into use. They also initially prompted concern and defensiveness. Then people started asking what this new tool would mean for maths instruction. Like with calculators, keeping AI out of teaching entirely is not an option.

What’s the origin of the scepticism about ChatGPT?
Manuel Sudau: The defensive stance is likely due to the fact that many people haven’t yet had the time to engage with these tools themselves. As an educational developer, I hope we will confront the issue with curiosity and explore the possibilities and opportunities arising from it. We could be self-​critical and ask ourselves whether, based on our own experiences, all our forms of assessment still make sense and whether we’re even really testing what we want to assess.  

What is new about ChatGPT?
Kortemeyer: The topic isn’t new. ETH itself helped develop the technologies behind it. What’s new, however, is that an application like this is publicly accessible. Also new is the volume of data with which the model was trained. AI research projects rarely approach the volumes used in a commercial product like this.

What can ChatGPT do and what are its weaknesses?
Kortemeyer: GPT-​3, the predecessor of the current GPT-​4, demonstrably had difficulties with basic symbolic and numerical arithmetic operations. The model couldn’t do reliable calculations. GPT-​4 is somewhat better, but still hasn’t reached the level of an average student at ETH. But note that despite these weaknesses, I was able to show in an experiment that GPT-​3 would just about manage to pass a typical physics course at a US university, including coursework, tests and examinations, with about half the possible points.

Sudau: What we’re talking about here are the strengths and weaknesses of a status quo, of an already outdated version of ChatGPT. But what does the future hold? You can’t stop progress. We should consider how we’re going to handle language models and other AI-​based tools that are much, much better. AI applications will soon be ubiquitous in almost all areas of our lives.

Most questions seem to revolve around assessments at the moment. What’s your advice for teaching staff?
Kortemeyer: AI is a tool. Just like, for example, spell checkers, calculators, literature search engines and statistics programs are tools. As such, it could be integrated into assessments. One approach could be to have students evaluate AI answers or outputs. This is a skill that will be in greater demand in the future.

So assessments would more often be open book, or open AI?
Kortemeyer: Of course not only open book. There will always be basic tasks students have to be able to complete without external help, whether human or AI. One solution could be two-​part or multi-​part assessments in which students first solve tasks without access to AI and other tools and then additional ones with access to these tools. One part covering the indispensable basics, the others realistic work methods. We already offered technical solutions and support for such multi-​stage online assessments at ETH before ChatGPT took off.

How does ChatGPT benefit teaching?
Kortemeyer: That remains to be seen. But if AI relieves students of some kinds of work in exercises and assessments, that may open up space for new things as well. When students still had to solve their assignments without calculators, they needed a lot of time for pure calculation. Afterwards, this time was suddenly freed up and could be used for actually solving the problems. We can likewise ask ourselves today whether integrating AI allows us to make some things more challenging – and also what the human skills are that we actually want to assess.

Can you give an example?
Kortemeyer: Creativity. Language models may sometimes give the appearance of creativity, but they are not creative. They have no vision. Maybe other models will have this capacity some day, but language models don’t. But ETH engineers must be creative. They also need the ability to think critically. Students should be asking themselves whether a result is really true. Does it make sense?

Besides in assessments, where else does ChatGPT have consequences for ETH teaching staff?
Sudau: In curriculum development, for example. We have to think about what we want to teach in the future. What are the basics and what interdisciplinary skills do ETH graduates need? Ultimately, they will have to use and work with these tools in their future professional lives, as well as be able to adapt to the unknown challenges of the job market. We have to teach the necessary skills for this and practise them with our students.

Kortemeyer: And that’s already happening. For example, we’re already discussing with lecturers in the Department of Computer Science what consequences ChatGPT could have for courses on the Python programming language. What do students need to learn in order to test AI-​generated Python code? These are exactly the things we should be thinking about now to be prepared to integrate the tool into teaching later.

Do we actually know how students are using ChatGPT right now?
Sudau: We know relatively little at the moment. A small, unrepresentative survey of students in the D-​BAUG and D-​USYS by my colleague Christian Sailer and myself showed that Bachelor’s students are the least likely and doctoral students the most likely to regard ChatGPT as useful. In other words, the further along in their studies or the more experienced they are, the more useful they consider it. That tells me we absolutely need to empower students to use tools like this – and to use them in a critical and considered way.

What other tools besides ChatGPT should be on our radars?
Kortemeyer: The next thing for us will probably be Google’s language model Bard. Nvidia is also working on new developments. Microsoft’s “Copilot” will probably become a hot topic very soon, too. It will be integrated into all Office tools – AI in Word and PowerPoint. AI is unlikely to disappear. We can’t stick our heads in the sand when it comes to our teaching and assessments.