The ETH chatbot Ethel also corrects exercises

A chatbot that only reproduces reliable information from the course context, knows the level of its counterpart, adjusts answers accordingly and can even refer to specific passages in a lecture recording - this is still a dream of the future.

However, Gerd Kortemeyer, a specialist in AI in teaching at the ETH AI Centre and the Rectorate, has such a vision in mind when he works on the current chatbots for specific courses.

In his project called Ethel, he is developing digital assistants that can take over routine tasks: Answering students' questions about courses, giving feedback on homework or exercises and even helping with the grading of exams or homeworks.

Chatbots rely on course materials

Such bots are already in operation on six courses. At the moment, they are based on the commercially available language model GPT-4. When students ask the chatbots questions, they rely on the respective course documents for the answers. According to Kortemeyer, this is in line with a student wish that emerged from a large survey on the use of AI among ETH students. And it is already working surprisingly well.

In contrast to ChatGPT, the reference data - i.e. the course documents - are stored locally on ETH servers. In addition, ETH has a contractual assurance that this data will not be used by the operator of GPT-4 for training purposes.

Kortemeyer creates such bots together with the IT services for interested ETH lecturers; they can contact him directly for this purpose.

Feedback on exercise sheets

The custom bots have also already proved their worth as "mentors": In physics exercises students can submit their handwritten exercise sheets as part of a pilot study and receive qualitative feedback from the bot. For example, an indication of where the student has made a mistake. In this example, the sample solutions serve as a reference for the language model.

Of course, these answers are not always correct, says Kortemeyer. But the students are aware of this. "Nevertheless, the majority of them rate the bots as useful because they also learn from dealing with the bots' mistakes."

Kortemeyer is currently researching whether chatbots are also suitable as grading assistants. In principle, they can also make a grading suggestion for exercise sheets and exams based on the sample solutions and the grading rubric. In other words, a score suggestion paired with an indication of the reliability of the information. Kortemeyer is currently testing how accurately the system works - or rather, how well the system can estimate its accuracy. He is using a data set from a large thermodynamics exam and carrying out measurements.

A language model of our own?

As part of the Swiss AI Initiative of ETH Zurich and EPFL, there is a plan to develop a new foundation model for teaching. Such a model would be more transparent and trustworthy because it would be clear what it is being trained on. The initiative has been promised computing time from the Supercomputing Centre (CSCS) for this purpose. However, training it would require an immense corpus of teaching materials.

As a preliminary stage, Kortemeyer is planning to fine-tune a "pre-trained" open source model such as Llama (from Meta) or Mistral with his own course materials. To do this, he is dependent on receiving even more course materials from lecturers.

He hopes that such a model will speak the language of technical universities even more precisely, i.e. that it will be able to express itself in the technical language of mathematics, natural sciences and engineering appropriate to the level of the students. He plans to start fine-tuning it in the summer, with the aim of using the new model as the new "brain" for the course chatbots in the spring semester of 2025.