How Well Do AI Models Know Celtic Languages?
Fairly well, but not well enough to trust without a check. In our first round, nine AI models scored between 61.6% and 86.1% on tests built from a language course. Manx was the hardest language, and models translate into English better than out of it.
Many learners now ask a chatbot for help with Irish, Welsh or another of the six Celtic languages. How often is the answer right? We built an open test to find out. We ran it on nine AI models on 15 and 16 November 2026.
The test is called Teanga Bench. Teanga is the Irish word for language. It is not BlasBench, our separate speech-recognition benchmark for Irish. Teanga Bench tests how well AI models do the work of a language tutor.
The short answer
The best model scored 86.1% overall and the weakest scored 61.6%. The top three models are tied, so we cannot say which of them is best. On average, Welsh was the easiest language and Manx was the hardest.
What we tested
The Blas suite has five tests and 1,500 items. Every item was asked three times. The items come from the Blas course, dictionaries and published parallel texts. No language model wrote any of the answers.
- Lesson exercises: questions from a language course, such as multiple choice and fill the gap.
- Word forms: give one form of a word, such as a verb tense or a plural.
- Translation: translate a sentence into or out of the language.
- Round-trip translation: translate a sentence into another language and back again, then compare it with the original.
- Tutor accuracy: find the mistake in a sentence, fix it, and change nothing else. This test covers Irish, Welsh and Scottish Gaelic only.
Fixed rules mark the scores, such as a list of accepted answers or how closely a translation matches a reference. Two judge models mark the tutor's explanations, and we report those marks separately.
Each score comes with a 95% range. It shows how far the score could move by chance, because we asked only a sample of questions. When two ranges overlap, the models are not clearly different.
What we found
Overall score on the Blas suite, with the 95% range, for each model:
| Model | Overall score | 95% range |
|---|---|---|
| Claude Fable 5.2 | 86.1% | 84.9% to 87.3% |
| Gemini 4 Hydrogen | 84.4% | 83.1% to 85.6% |
| GPT-6 Astra | 83.6% | 82.3% to 84.9% |
| Gemini 4 Argon | 81.9% | 80.5% to 83.3% |
| DeepSeek V4.1 Flash 1017 | 79.2% | 77.7% to 80.7% |
| Kimi K3.1 Pro | 78.4% | 76.9% to 79.9% |
| GPT-6 Luna | 75.6% | 73.9% to 77.2% |
| Claude Haiku 5.5 | 67.1% | 65.3% to 68.9% |
| Qwen3.8 Flash | 61.6% | 59.8% to 63.4% |
- The top three are tied. Claude Fable 5.2 scores highest, but its range overlaps those of Gemini 4 Hydrogen and GPT-6 Astra.
- Manx is hardest and Welsh is easiest. This is the average across all nine models.
- Models translate into English better than out of English. All nine do, and the gap is clear for seven of them.
- Fixing a mistake is the weak step for a tutor. On average, models score 77.7% at fixing the error. They score 84.7% at leaving the rest of the sentence alone.
- A correct answer can cost about 51 times more from one model than from another. The cost per 1,000 correct answers runs from $0.22 for GPT-6 Luna to $11.49 for Gemini 4 Hydrogen. That is the price through OpenRouter on the test days, not the price of a chatbot subscription.
We run a second suite, Tonada, which tests English tutoring for learners whose first language is Portuguese, Spanish, Vietnamese or Japanese. Each of the nine models scored higher there than on the Blas suite. The two suites use different tests, so this is a rough comparison, not an exact one.
What this means if you learn with AI
These are cautious points. Our tests use short items that can be checked. They do not show how a model does in a long chat.
- Treat AI as a helper, not the last word. Even the best model scored 86.1%, and the weakest scored 61.6%. Check anything that matters against a dictionary, a course or a speaker.
- Be most careful with Manx. It was the hardest language for the models.
- Be more careful when the model writes the language than when it reads it. Translation into English was better than out of English. Treat text a model writes for you as a draft.
- Do not assume a correction worked. Models were weakest at fixing the error in a learner's sentence, so check that the mistake is really gone.
- The model you pick matters. Higher-scoring models were also more consistent when asked the same thing three times.
The limits
This is a first round, and it has clear limits:
- No person reviewed the items. The answers come from sources, and some will be wrong. A score measures agreement with the course content and the sources, not with a standard that experts approved.
- The mistakes in the tutor test are not real learner mistakes. Most were made by putting a wrong option from a course exercise into a sentence.
- Only the Irish translation sentences are private text. The other languages use public sentences that the models may have seen before.
- The first runs cut off long answers from four models. We asked those questions again with a higher limit, and the methodology page records this.
See everything
The results and the method are public:
- The results site shows every model on every test, with findings on its findings page.
- The methodology page explains how the items are built and marked.
- The GitHub repository holds the code, the scoring rules and 20 sample items per test. It is open under the Apache-2.0 licence. The full item sets stay private, so models cannot be trained on them.
What happens next
We will test new models as they come out. The results sites update automatically, so check back there for the latest scores.
Editorial notes
- Review methodology
- Every number in this post is copied from the published Teanga Bench results (research.blasapp.com and research.tonadaapp.com, as of 16 November 2026). The authoring AI agent and the orchestrating AI session checked each number against the published results files. No person has reviewed the post or the benchmark items, and no outside review is claimed. The post contains no sentences in a Celtic language.
Ready to make this stick?
blas. is the language app for adults coming back to Irish or Welsh. Grammar, vocabulary, mutations, conversation — all with spaced repetition so you actually remember it.
Keep reading
The 6 Celtic Languages: Goidelic & Brythonic Tree
Compare all 6 living Celtic languages side-by-side: Goidelic (Irish, Scottish Gaelic, Manx) vs Brythonic (Welsh, Breton, Cornish) family tree, map & audio.
Building a Writing Assistant for the Celtic Languages
How the blas. browser extension reads what you are typing, why Irish and Welsh need their own rules, what runs on your device, and where the checker is still wrong.
blas. App Review 2026: What It Does and Doesn't Do
An honest review of blas., the Celtic language app for Irish, Welsh, Scottish Gaelic, Breton, Cornish, and Manx. What it covers, what it doesn't, who it's built for, and whether it's worth your time.
