Choosing an open-source language model for German: How to test correctly

Which open-source language model can speak best is German not reliably answered by a ranking – but by a test with your own tasks. Compile 30 to 50 real examples, determine how you can recognize a good answer, and compare two to four models such as Qwen, Llama, Mistral, and gpt-oss under the same conditions.

Public leaderboards usually measure English tasks, often academically. It does not say whether a model politely formulates a German customer letter or reads deadlines from a notice correctly. This article shows a simple test method.

The usual candidates

All four can be used via providers such as the IONOS AI Model Hub from German data centers or locally, for example with Ollama. Ollama explains what Ollama is in simple terms.

Step 1: Collect real tasks

Take examples from everyday life, not invented test questions:

• 10 Emails to Classify or Reply to
• 10 Questions for Your Documents with Known Correct Answers
• 10 documents from which data is to be read (amounts, deadlines, names)
• 5 texts to summarize
• some borderline cases: dialect, technical terms, abbreviations, typos

Step 2: Set criteria

Step 3: Compare fairly

1. Same statement and examples for all models.
2. Same settings (e.g. low "temperature" for factual tasks).
3. Have answers evaluated covertly – the evaluator does not know which model answered.
4. Summarize results in a table, award points for each criterion.

Step 4: Include costs and operations

The best model is not always the largest. If a medium-sized model solves 95% of the tasks correctly and a large one 97%, the medium-sized model can be clearly ahead economically. If you run your own business, it also counts which hardware the model runs on – see Hardware for your own AI.

Typical findings from such tests

• Small models are often surprisingly good at classifying and reading out, weaker for long, free texts.
• Linguistic subtleties such as polite forms differ noticeably between models.
• Good instructions get more out of any model than a model change.

We carry out such comparative tests with your data and recommend a model from our overview of open source language models – suitable for the task, budget and data protection.

Counselling and accompanied implementation is eligible for funding: Through INQA coaching, the federal government covers 80% of the consulting costs of up to €1,200 per day for up to 12 days, i.e. up to €11,520 subsidy. In Mecklenburg-Western Pomerania and the other new federal states, BAFA consulting funding is also eligible at 80% until 31.12.2026 (50% in the old federal states). Funding is provided for consulting, not software. The funding check shows which program fits.

Frequently Asked Questions

Which open source model does German best?

This depends on the task and changes with each new model generation. Common candidates are Qwen, Llama, Mistral and gpt-oss. A test with your own examples is more meaningful than any ranking.

How many test examples do I need?

For a reliable decision, 30 to 50 real examples from your everyday life are usually enough, including some borderline cases.

Are small models suitable for German?

For clearly defined tasks such as classifying or reading data, often yes. For long, free texts, larger models usually provide more natural German.

Can I test the models without buying hardware?

Yes, for example via the IONOS AI Model Hub, which offers several open models from German data centers and bills according to consumption.

Would you like to find the right model for your tasks? Get in touch via the contact page.