A transparent multilingual talent

Since the end of 2024, the German AI language model Teuken-7B has been available to anyone interested from science and industry. Stefan Kesselheim and Andreas Herten from the Jülich Supercomputing Centre helped to develop Teuken-7B. Here they explain what distinguishes the multilingual open-source model from commercial products such as ChatGPT.

What’s so special about the Teuken-7B large language model?

Kesselheim: It is made in Germany, freely accessible, and anyone can download it. It was trained not only in English, but in all 24 official languages of the EU.

Herten: Teuken-7B is the result of the OpenGPT-X project, which was launched three years ago to counter commercial language models, primarily from North America, with something transparent and open source. There was no venture capital investor and no commercial interests. The project was funded by the Federal Ministry for Economic Affairs and Climate Action. We trained the model on supercomputers in Jülich and Dresden – using public infrastructure.

Why is it important that this field is not left exclusively to financially strong companies?

Kesselheim: Companies often keep the source code of AI and the training data under lock and key. Their actions and intentions are not always transparent. With such a revolutionary technology that is changing the way humanity passes on and processes information, it is dangerous to be dependent on commercial interests.

Herten: Companies want to use language models to operate chatbots and software agents, for example, in order to make money. As scientists, we are interested in the benefits to society. That’s why we’re more open to other research questions and ideas that emerge from the development of the technology.

How did you contribute to the development of Teuken-7B?

Herten: We divided Jülich’s tasks between two teams. My team ensured that the hardware was used efficiently during the training of the language model. We selected suitable methods and parameters before analysing and adapting them. The insights we obtained gave us important indications about what to look out for when procuring the new Jülich exascale computer JUPITER.

Kesselheim: There are many variables when training a large language model. These must be set in advance to control how the model is trained. My team helped to find the best settings. This allows us to achieve our goal with as few training steps as possible.

An important aspect in the development of AI models is their consumption of resources. How energy-efficient is Teuken-7B?

Kesselheim: We have found a good approach to improving the energy efficiency of multilingual language models. Teuken-7B breaks down the words of different languages into smaller units, called tokens, in a particularly clever way. A token is more than a letter; it is more like a syllable. For German, you would choose slightly different tokens than for Finnish, for example. Our tokenizer divides the mix of European languages in such a way that the language model performs fewer calculations to arrive at a result.

There are now many European projects similar to OpenGPT-X, some of which Jülich is involved in. Does this make sense?

Kesselheim: AI and language models are disruptive technologies, perhaps comparable to the internet. Their huge importance means that many people want to be part of their development. Many national and international initiatives have been set up. This is a good thing, because a lot of expertise is needed to create a globally competitive ecosystem in Europe consisting of research institutions, start-ups, and companies. The challenge lies in coordinating these initiatives in the best possible way.

So, Teuken-7B is by no means the end of language model development....

Kesselheim: Exactly. Since training our model, a lot of new insights have been obtained into training methods and the best data. Moreover, the model is small: 7B stands for 7 billion parameters. DeepSeek’s largest model has 650B and is more powerful, but it also consumes more computing resources. With Teuken-7B, however, we have shown that Germany can play a role in the development of language models. We have learned an enormous amount through this project, which is important because language models are paving the way for the development of AI strategies and methods in various fields of research.

Can you give an example?

Kesselheim: Atmospheric researchers at Jülich use AI methods that originated in language models for their work. The researchers have adapted these methods to process weather and climate data. The same applies to AI models in energy and brain research. Language models provide valuable insights for the development of such AI models – especially if you develop them yourself. It is not sufficient to simply use the language models of commercial providers.

The questions were posed by Frank Frick.