How Well Do AI Models Know Welsh (Cymraeg)?

Welsh is the strongest Celtic language in our benchmark, averaging 85.1% across nine models. Scores run from 69.9% to 91.5%, with a four-way tie at the top, yet chatbots still miss soft and nasal mutations when checking learner sentences.

Cymraeg2026-11-17·6 min read

When you paste a Welsh sentence into an AI chatbot, it often gets the gist right and sounds fluent. Yet mutations after prepositions, numbers and particles still trip models up without warning.

We ran Teanga Bench on nine AI models on 15 and 16 November 2026 to see how reliably they handle Welsh. Each model answered 280 Welsh items three times across five tasks.

The short answer

Welsh scores highest of the six Celtic languages on the Blas suite, averaging 85.1% across nine models. Claude Fable 5.2 leads at 91.5%, tied within 95% intervals with Gemini 4 Hydrogen (89.4%), GPT-6 Astra (89.2%) and Gemini 4 Argon (88.2%). Qwen3.8 Flash scores 69.9%.

What each score measures

The Welsh slice has 280 items across five tests, each asked three times:

  • Lesson exercises (60 items): multiple-choice and fill-the-blank drills from the Blas Welsh course, scored against accepted answers.
  • Word forms (60 items): producing noun plurals, prepositional pronouns, numbers and literary verb forms against attested references.
  • Translation (60 items): translating 20 sentences into English, 20 out of English and 20 legal or administrative texts, scored with character overlap (chrF++).
  • Round-trip translation (40 items): translating Welsh into English and back again, then checking closeness to the original Welsh sentence.
  • Tutor accuracy (60 items): finding a single error in a learner sentence, fixing only that span and keeping everything else unchanged.

Every score carries a 95% interval showing sample uncertainty. Models with overlapping intervals are statistically tied.

Welsh scores across all nine models

Overall Welsh score, 95% interval and per-test scores (9-model averages: 95.9% lesson exercises, 89.6% word forms, 81.3% tutor accuracy, 80.5% round-trip, 78.3% translation):

ModelOverall95% rangeLessonsWord formsTranslationRound-tripTutor
Claude Fable 5.291.5%89.4% to 93.4%98.9%97.8%83.8%83.6%93.3%
Gemini 4 Hydrogen89.4%87.1% to 91.6%100.0%95.0%79.1%83.6%89.4%
GPT-6 Astra89.2%87.0% to 91.2%96.7%100.0%80.8%79.6%88.9%
Gemini 4 Argon88.2%85.7% to 90.5%99.4%91.1%79.8%82.4%88.3%
DeepSeek V4.1 Flash 101786.7%84.1% to 89.1%96.1%96.1%76.3%81.0%83.9%
Kimi K3.1 Pro85.0%82.4% to 87.6%95.6%92.2%78.9%76.9%81.7%
GPT-6 Luna83.5%80.6% to 86.2%94.4%90.0%77.9%79.5%75.6%
Claude Haiku 5.582.7%79.8% to 85.5%91.7%90.0%77.1%83.0%71.7%
Qwen3.8 Flash69.9%66.4% to 73.4%90.6%53.9%70.7%75.3%58.9%

Gemini 4 Hydrogen hit 100.0% on Welsh lesson exercises and GPT-6 Astra hit 100.0% on Welsh word forms. Even so, tutor accuracy drops to 58.9% on the weakest model.

Reading Welsh vs writing Welsh

Welsh has the smallest translation direction gap of the three core languages, averaging 77.4% into English vs 73.3% out of English (4.1 points):

ModelInto English (20)Out of English (20)Gap (pts)Legal / admin (20)
Claude Fable 5.281.5% (71.9% to 89.8%)78.9% (70.9% to 86.4%)+2.690.9% (85.9% to 95.3%)
Gemini 4 Hydrogen76.5% (67.7% to 84.5%)74.2% (65.6% to 81.7%)+2.486.7% (80.0% to 92.7%)
GPT-6 Astra80.5% (70.9% to 88.9%)74.4% (66.7% to 81.5%)+6.287.5% (81.9% to 92.6%)
Gemini 4 Argon78.4% (69.2% to 86.3%)73.0% (64.3% to 80.7%)+5.488.0% (82.2% to 93.1%)
DeepSeek V4.1 Flash 101776.3% (67.5% to 84.2%)70.5% (61.9% to 78.4%)+5.882.2% (75.8% to 87.9%)
Kimi K3.1 Pro78.0% (67.8% to 86.8%)74.1% (66.0% to 81.6%)+3.984.5% (78.7% to 89.9%)
GPT-6 Luna77.0% (67.0% to 86.2%)74.3% (65.4% to 82.0%)+2.782.3% (76.6% to 87.8%)
Claude Haiku 5.576.8% (67.2% to 85.4%)72.8% (63.8% to 81.0%)+4.081.9% (75.1% to 88.2%)
Qwen3.8 Flash71.8% (62.5% to 80.3%)67.3% (61.7% to 72.6%)+4.573.0% (66.8% to 79.0%)

All nine models score slightly higher into English than out of English, though every model's 95% intervals overlap between directions. Public legal and administrative text scores highest (73.0% to 90.9%).

Concrete patterns from public sample items

Public sample items and published failure logs reveal three recurring Welsh errors:

  • Missed soft and nasal mutations: On Cafodd ei anghofio gan pawb. (should mutate to gan bawb), GPT-6 Luna said there was no error, even writing gan bawb in its explanation while leaving gan pawb unfixed. It also missed nasal mutation in am bum blynedd (needs am bum mlynedd).
  • Rewriting focus clauses instead of fixing the verb: On y llyfr mae ar y bwrdd (needs y llyfr sydd ar y bwrdd), Gemini 4 Hydrogen and Kimi K3.1 Pro rewrote the whole phrase to VSO order (mae'r llyfr ar y bwrdd) instead of correcting mae to sydd.
  • Over-correcting imperatives and missing typos: GPT-6 Astra flagged clean Pleidlais, felly, ar welliant 12. as needing Pleidleisiais. On GGeiriadur termau ... a Owain, Claude Fable 5.2 fixed a Owain to ac Owain and skipped the doubled letter GGeiriadur.

What this means when you learn Welsh

Welsh is the safest Celtic language for chatbot study, provided you check mutations and clause edits:

  • blog.ai_models_welsh_language_benchmark.l1
  • Watch for silent rewrites: When you ask an AI tutor to check a focus sentence or relative clause, verify that it fixed your mistake rather than rewriting your word order.
  • Cross-check corrections before memorising them: Even top models score 88.3% to 93.3% on Welsh tutor accuracy, and lower models drop to 58.9%. Verify edits with the Welsh grammar checker or main checker.

Honest limits of this test

Keep these benchmark limits in mind when comparing models:

  • No human check on gold answers: Every item carries human_checked: false. Scores measure agreement with the Blas course, dictionaries and parallel corpora.
  • Public Welsh translation text: Welsh translation and round-trip sentences come from public corpora, so models may have seen some items during pretraining.
  • Overlapping top intervals: Claude Fable 5.2, Gemini 4 Hydrogen, GPT-6 Astra and Gemini 4 Argon all have overlapping 95% intervals on Welsh and cannot be cleanly ranked apart.
  • Synthetic tutor errors and reruns: Most tutor errors are course distractors inserted into sentences. Four models hit early 2,048-token cutoffs and were rerun at 16,000 tokens.

See the full data

Every run file, score and public sample is open:

What to do next

We will update the Welsh scores as new models are released. For daily study, pair AI reading practice with our reference tables and Welsh course so mutation triggers stick without guesswork.

Editorial notes

Review methodology
Every number in this post comes from the published Teanga Bench Blas-suite run files (research.blasapp.com, results as of 16 November 2026). All concrete examples come solely from public sample files in samples/blas/ and published model pages; no private test items are quoted. No human reviewer checked the gold answers (human_checked is false) or this post.

Ready to make this stick?

blas. is the language app for adults coming back to Welsh. Treigladau, grammar, conversation — all with spaced repetition so you actually remember it.

Keep reading