How Well Do AI Models Know Irish (Gaeilge)?

Moderately well, with clear blind spots. Across nine AI models tested on 280 Irish items, scores run from 69.1% to 89.8%, with an 82.2% average. The top four models are tied, and models translate Irish into English better than English into Irish.

Gaeilge2026-11-17·6 min read

When you paste an Irish sentence into a chatbot and ask for a check or translation, it usually sounds sure of itself. That confidence can hide subtle errors in mutations, verb forms or tense.

We ran Teanga Bench on nine AI models on 15 and 16 November 2026 to measure how often they get Irish right. Every model answered 280 Irish items three times across five tasks.

The short answer

On Irish, the nine models average 82.2%. Claude Fable 5.2 scores highest at 89.8%, and Qwen3.8 Flash scores lowest at 69.1%. Four models tie at the top because their 95% intervals overlap: Claude Fable 5.2, Gemini 4 Argon, Gemini 4 Hydrogen and GPT-6 Astra.

What each score measures

The Irish slice of the Blas suite has 280 items across five tests, each asked three times:

  • Lesson exercises (60 items): multiple-choice and fill-the-blank items from the Blas Irish course, marked against accepted answers.
  • Word forms (60 items): giving one inflected form, such as a verb tense, plural or prepositional pronoun, marked against accepted forms.
  • Translation (60 items): translating 20 sentences into English, 20 out of English and 20 legal or administrative texts, scored by character overlap (chrF++) against references.
  • Round-trip translation (40 items): translating an Irish sentence into English and back again, then scoring how close the result stays to the source.
  • Tutor accuracy (60 items): spotting a single mistake in a learner sentence, fixing it and leaving the rest intact. All three rule checks must pass on the same item.

Every score carries a 95% interval showing how far results could shift on another sample of questions. When two intervals overlap, neither model clearly leads.

Irish scores across all nine models

Overall Irish score, 95% interval and per-test scores (averages across models: 92.3% lesson exercises, 85.8% word forms, 79.3% tutor accuracy, 76.9% translation, 76.7% round-trip):

ModelOverall95% rangeLessonsWord formsTranslationRound-tripTutor
Claude Fable 5.289.8%87.5% to 92.0%97.2%96.7%82.0%82.0%91.1%
Gemini 4 Argon87.3%84.5% to 89.8%95.0%88.3%81.5%82.8%88.9%
Gemini 4 Hydrogen87.3%84.5% to 89.9%93.3%92.2%81.4%82.1%87.2%
GPT-6 Astra86.6%83.8% to 89.3%93.3%91.7%80.8%77.7%89.4%
Kimi K3.1 Pro83.5%80.5% to 86.4%93.3%90.6%78.0%78.6%77.2%
DeepSeek V4.1 Flash 101781.3%78.1% to 84.2%92.2%87.2%76.9%74.1%76.1%
GPT-6 Luna81.1%77.7% to 84.3%90.0%87.2%76.9%78.2%73.3%
Claude Haiku 5.573.6%70.0% to 77.0%87.2%75.6%69.7%68.9%66.7%
Qwen3.8 Flash69.1%65.4% to 72.6%88.9%62.8%64.9%65.5%63.3%

Gemini 4 Argon and Gemini 4 Hydrogen both score 87.3% overall on Irish. Models drop fastest between closed course drills (92.3% average) and open generation or error correction.

Reading Irish vs writing Irish

Models understand Irish better than they write it. Across all nine models, translating into English averages 84.9%, compared with 75.2% out of English (a 9.7-point gap):

ModelInto English (20)Out of English (20)Gap (pts)Legal / admin (20)
Claude Fable 5.287.7% (82.0% to 92.8%)78.3% (68.3% to 87.6%)+9.480.0% (72.4% to 87.3%)
Gemini 4 Argon82.0% (74.9% to 88.8%)82.4% (73.4% to 89.9%)-0.479.9% (72.1% to 87.4%)
Gemini 4 Hydrogen85.9% (78.3% to 92.6%)80.4% (70.1% to 89.4%)+5.578.0% (70.5% to 85.6%)
GPT-6 Astra86.2% (77.9% to 93.5%)79.3% (69.1% to 88.3%)+6.977.0% (69.1% to 84.4%)
Kimi K3.1 Pro86.5% (80.0% to 92.3%)76.8% (67.4% to 85.5%)+9.770.6% (63.0% to 77.9%)
DeepSeek V4.1 Flash 101786.2% (78.9% to 92.4%)76.5% (66.9% to 85.6%)+9.767.9% (60.2% to 75.5%)
GPT-6 Luna86.0% (78.9% to 92.7%)75.7% (65.2% to 85.7%)+10.368.8% (62.2% to 75.4%)
Claude Haiku 5.581.3% (74.3% to 87.9%)67.8% (57.3% to 78.1%)+13.560.1% (52.9% to 66.7%)
Qwen3.8 Flash82.0% (74.7% to 89.0%)59.3% (48.5% to 70.4%)+22.753.4% (47.5% to 59.9%)

Eight of nine models score higher into English than out of English. Only Gemini 4 Argon is flat (82.0% vs 82.4%), while Qwen3.8 Flash drops by 22.7 points with non-overlapping intervals.

Concrete patterns from public sample items

Looking at the public benchmark samples and published failure logs shows three ways models slip on Irish:

  • blog.ai_models_irish_gaelic_benchmark.pat_1
  • Over-correcting valid Irish sentences: On clean sample Tá sé tábhachtach nach sárófaí a gcearta daonna., Claude Haiku 5.5, Gemini 4 Argon and Kimi K3.1 Pro all flagged sárófaí as wrong and rewrote the mood or tense.
  • Mutation rules and cleft emphasis drift: Public tutor items also test eclipsis vs lenition after ní (Ní ndúnann sé an doras. vs Ní dhúnann) and feminine adjective lenition (cathair mhór). On round-trip Ba sa bhliain 1976 a cuireadh pionós an bháis ar ceal., DeepSeek V4.1 Flash 1017 dropped the copula cleft and returned Cuireadh deireadh le pionós an bháis i 1976. (41.7%).

What this means when you learn Irish

Use AI models where they are strong, and verify the forms they tend to miss:

  • Trust reading help more than Irish prose: Models average 84.9% translating Irish into English. When asking a model to write Irish for you, treat the output as a rough draft.
  • blog.ai_models_irish_gaelic_benchmark.l2
  • Double-check chatbot corrections: Models average 79.3% on Irish tutor accuracy and sometimes invent errors on clean sentences. Cross-check sentences in the Irish grammar checker or main checker.

Honest limits of this test

Keep these caveats in mind before reading too much into small score gaps:

  • No human check on gold answers: Every item has human_checked: false. Scores measure agreement with the Blas course, dictionaries and parallel corpora, not expert-reviewed gold text.
  • Overlapping 95% intervals: With 280 Irish items, intervals span roughly 4.5 to 7.2 points. We cannot separate Claude Fable 5.2, Gemini 4 Argon, Gemini 4 Hydrogen and GPT-6 Astra on Irish.
  • Synthetic tutor errors and reruns: Most tutor mistakes are course distractors placed into sentences, not real learner writing. Four models also hit 2,048-token cutoffs on early passes and were rerun at 16,000 tokens.
  • Irish translation text is private: Unlike Welsh and Scottish Gaelic, Irish translation items use private Blas sentences, so contamination risk is lower there.

See the full data

Every run file, score and public sample is open:

What to do next

We will add new models to the Irish leaderboard as they launch. Until models close the gap between passive reading and active Irish grammar, pair chatbot practice with structured reference tables and daily Irish lessons.

Editorial notes

Review methodology
Every number in this post comes from the published Teanga Bench Blas-suite run files (research.blasapp.com, results as of 16 November 2026). All concrete examples come solely from public sample files in samples/blas/ and published model pages; no private test items are quoted. No human reviewer checked the gold answers (human_checked is false) or this post.

Ready to make this stick?

blas. is the language app for adults coming back to Irish. Séimhiú, urú, grammar, conversation — all with spaced repetition so you actually remember it.

Keep reading