How Well Do AI Models Know Welsh (Cymraeg)?
Welsh is the strongest Celtic language in our benchmark, averaging 85.1% across nine models. Scores run from 69.9% to 91.5%, with a four-way tie at the top, yet chatbots still miss soft and nasal mutations when checking learner sentences.
When you paste a Welsh sentence into an AI chatbot, it often gets the gist right and sounds fluent. Yet mutations after prepositions, numbers and particles still trip models up without warning.
We ran Teanga Bench on nine AI models on 15 and 16 November 2026 to see how reliably they handle Welsh. Each model answered 280 Welsh items three times across five tasks.
The short answer
Welsh scores highest of the six Celtic languages on the Blas suite, averaging 85.1% across nine models. Claude Fable 5.2 leads at 91.5%, tied within 95% intervals with Gemini 4 Hydrogen (89.4%), GPT-6 Astra (89.2%) and Gemini 4 Argon (88.2%). Qwen3.8 Flash scores 69.9%.
What each score measures
The Welsh slice has 280 items across five tests, each asked three times:
- Lesson exercises (60 items): multiple-choice and fill-the-blank drills from the Blas Welsh course, scored against accepted answers.
- Word forms (60 items): producing noun plurals, prepositional pronouns, numbers and literary verb forms against attested references.
- Translation (60 items): translating 20 sentences into English, 20 out of English and 20 legal or administrative texts, scored with character overlap (chrF++).
- Round-trip translation (40 items): translating Welsh into English and back again, then checking closeness to the original Welsh sentence.
- Tutor accuracy (60 items): finding a single error in a learner sentence, fixing only that span and keeping everything else unchanged.
Every score carries a 95% interval showing sample uncertainty. Models with overlapping intervals are statistically tied.
Welsh scores across all nine models
Overall Welsh score, 95% interval and per-test scores (9-model averages: 95.9% lesson exercises, 89.6% word forms, 81.3% tutor accuracy, 80.5% round-trip, 78.3% translation):
| Model | Overall | 95% range | Lessons | Word forms | Translation | Round-trip | Tutor |
|---|---|---|---|---|---|---|---|
| Claude Fable 5.2 | 91.5% | 89.4% to 93.4% | 98.9% | 97.8% | 83.8% | 83.6% | 93.3% |
| Gemini 4 Hydrogen | 89.4% | 87.1% to 91.6% | 100.0% | 95.0% | 79.1% | 83.6% | 89.4% |
| GPT-6 Astra | 89.2% | 87.0% to 91.2% | 96.7% | 100.0% | 80.8% | 79.6% | 88.9% |
| Gemini 4 Argon | 88.2% | 85.7% to 90.5% | 99.4% | 91.1% | 79.8% | 82.4% | 88.3% |
| DeepSeek V4.1 Flash 1017 | 86.7% | 84.1% to 89.1% | 96.1% | 96.1% | 76.3% | 81.0% | 83.9% |
| Kimi K3.1 Pro | 85.0% | 82.4% to 87.6% | 95.6% | 92.2% | 78.9% | 76.9% | 81.7% |
| GPT-6 Luna | 83.5% | 80.6% to 86.2% | 94.4% | 90.0% | 77.9% | 79.5% | 75.6% |
| Claude Haiku 5.5 | 82.7% | 79.8% to 85.5% | 91.7% | 90.0% | 77.1% | 83.0% | 71.7% |
| Qwen3.8 Flash | 69.9% | 66.4% to 73.4% | 90.6% | 53.9% | 70.7% | 75.3% | 58.9% |
Gemini 4 Hydrogen hit 100.0% on Welsh lesson exercises and GPT-6 Astra hit 100.0% on Welsh word forms. Even so, tutor accuracy drops to 58.9% on the weakest model.
Reading Welsh vs writing Welsh
Welsh has the smallest translation direction gap of the three core languages, averaging 77.4% into English vs 73.3% out of English (4.1 points):
| Model | Into English (20) | Out of English (20) | Gap (pts) | Legal / admin (20) |
|---|---|---|---|---|
| Claude Fable 5.2 | 81.5% (71.9% to 89.8%) | 78.9% (70.9% to 86.4%) | +2.6 | 90.9% (85.9% to 95.3%) |
| Gemini 4 Hydrogen | 76.5% (67.7% to 84.5%) | 74.2% (65.6% to 81.7%) | +2.4 | 86.7% (80.0% to 92.7%) |
| GPT-6 Astra | 80.5% (70.9% to 88.9%) | 74.4% (66.7% to 81.5%) | +6.2 | 87.5% (81.9% to 92.6%) |
| Gemini 4 Argon | 78.4% (69.2% to 86.3%) | 73.0% (64.3% to 80.7%) | +5.4 | 88.0% (82.2% to 93.1%) |
| DeepSeek V4.1 Flash 1017 | 76.3% (67.5% to 84.2%) | 70.5% (61.9% to 78.4%) | +5.8 | 82.2% (75.8% to 87.9%) |
| Kimi K3.1 Pro | 78.0% (67.8% to 86.8%) | 74.1% (66.0% to 81.6%) | +3.9 | 84.5% (78.7% to 89.9%) |
| GPT-6 Luna | 77.0% (67.0% to 86.2%) | 74.3% (65.4% to 82.0%) | +2.7 | 82.3% (76.6% to 87.8%) |
| Claude Haiku 5.5 | 76.8% (67.2% to 85.4%) | 72.8% (63.8% to 81.0%) | +4.0 | 81.9% (75.1% to 88.2%) |
| Qwen3.8 Flash | 71.8% (62.5% to 80.3%) | 67.3% (61.7% to 72.6%) | +4.5 | 73.0% (66.8% to 79.0%) |
All nine models score slightly higher into English than out of English, though every model's 95% intervals overlap between directions. Public legal and administrative text scores highest (73.0% to 90.9%).
Concrete patterns from public sample items
Public sample items and published failure logs reveal three recurring Welsh errors:
- Missed soft and nasal mutations: On
Cafodd ei anghofio gan pawb.(should mutate togan bawb), GPT-6 Luna said there was no error, even writinggan bawbin its explanation while leavinggan pawbunfixed. It also missed nasal mutation inam bum blynedd(needsam bum mlynedd). - Rewriting focus clauses instead of fixing the verb: On
y llyfr mae ar y bwrdd(needsy llyfr sydd ar y bwrdd), Gemini 4 Hydrogen and Kimi K3.1 Pro rewrote the whole phrase to VSO order (mae'r llyfr ar y bwrdd) instead of correctingmaetosydd. - Over-correcting imperatives and missing typos: GPT-6 Astra flagged clean
Pleidlais, felly, ar welliant 12.as needingPleidleisiais. OnGGeiriadur termau ... a Owain, Claude Fable 5.2 fixeda Owaintoac Owainand skipped the doubled letterGGeiriadur.
What this means when you learn Welsh
Welsh is the safest Celtic language for chatbot study, provided you check mutations and clause edits:
- blog.ai_models_welsh_language_benchmark.l1
- Watch for silent rewrites: When you ask an AI tutor to check a focus sentence or relative clause, verify that it fixed your mistake rather than rewriting your word order.
- Cross-check corrections before memorising them: Even top models score 88.3% to 93.3% on Welsh tutor accuracy, and lower models drop to 58.9%. Verify edits with the Welsh grammar checker or main checker.
Honest limits of this test
Keep these benchmark limits in mind when comparing models:
- No human check on gold answers: Every item carries
human_checked: false. Scores measure agreement with the Blas course, dictionaries and parallel corpora. - Public Welsh translation text: Welsh translation and round-trip sentences come from public corpora, so models may have seen some items during pretraining.
- Overlapping top intervals: Claude Fable 5.2, Gemini 4 Hydrogen, GPT-6 Astra and Gemini 4 Argon all have overlapping 95% intervals on Welsh and cannot be cleanly ranked apart.
- Synthetic tutor errors and reruns: Most tutor errors are course distractors inserted into sentences. Four models hit early 2,048-token cutoffs and were rerun at 16,000 tokens.
See the full data
Every run file, score and public sample is open:
- Browse the Blas leaderboard, language breakdown and findings page.
- Read the benchmark methodology for scoring rules, intervals and truncation reruns.
- Inspect the code and 20 public samples per test in the Teanga Bench repository.
What to do next
We will update the Welsh scores as new models are released. For daily study, pair AI reading practice with our reference tables and Welsh course so mutation triggers stick without guesswork.
Editorial notes
- Review methodology
- Every number in this post comes from the published Teanga Bench Blas-suite run files (research.blasapp.com, results as of 16 November 2026). All concrete examples come solely from public sample files in samples/blas/ and published model pages; no private test items are quoted. No human reviewer checked the gold answers (human_checked is false) or this post.
Ready to make this stick?
blas. is the language app for adults coming back to Welsh. Treigladau, grammar, conversation — all with spaced repetition so you actually remember it.
Keep reading
How Well Do AI Models Know Celtic Languages?
We tested nine AI models on Irish, Welsh, Scottish Gaelic, Manx, Breton and Cornish. See the scores and what they mean if you learn with AI.
Welsh Mutations Chart: Soft, Nasal & Aspirate
Complete Welsh mutation chart (treigladau): soft, nasal, and aspirate triggers, full sound change tables, and spoken examples with interactive drills.
Welsh Grammar for Beginners: Bod, Mutations, and Word Order First
A practical roadmap for Welsh grammar. Bod (to be), soft mutation, word order, gender, and plurals in the order that actually matters.
