The Best LLM Doesn’t Exist: How to Benchmark Multilingual AI (TAUS Vancouver 2026)

Key takeaways 

Every few weeks another leaderboard crowns a new “best” LLM, and almost as often someone asks me whether it is time to switch or which one they should observe. At the TAUS Conference in Vancouver (14–16 October 2026), I hosted a panel titled “The Best LLM Doesn’t Exist: How to Benchmark Multilingual AI” that tried to answer that question head-on. Is there a best LLM for translation? In a word, no – and the more interesting discussion, which took up most of the session, was about what buyers need to do instead. 

Joining me on stage were Alon Lavie (Carnegie Mellon University), Marina Pantcheva (RWS), and Amir Kamran (TAUS). Between them, they brought long experience in evaluation research, a language services view of what clients are asking for, and the perspective of a team building evaluation tools. 

I opened the session with a quote from Yinghon Lan, formerly of Netflix and now at OpenAI, that frames the problem better than I could: 

“In generative AI products, evaluation is not the scoreboard after the game. It is the mechanism that decides what game the team is actually playing. That statement is not an exaggeration – whatever your evaluation rewards becomes the product spec in practice. For example, if your image-generation eval scores prompt adherence but never aesthetic coherence, you will ship a model that follows instructions and looks wrong. The evaluation system operationalizes the project objectives – design it carelessly and you optimize for the wrong thing without knowing it.” 

The point, as the rest of the session made clear, is that finding the right LLM is work, and it is work no leaderboard can do for you. 

Why Don’t LLM Leaderboards Help Translation Buyers? 

Leaderboards generally fail buyers for multilingual requirements for four reasons:  

Evaluate the Harness, Not the Model 

The clearest consensus of the session was that you are much better off evaluating the “harness” (the prompts, context, tools and workflow an LLM runs inside) than the model itself, because that tells you how it is likely to behave in practice rather than in the abstract. Leaderboards may be useful for building a shortlist, but the best model for a given organization or task often does not rank particularly high on any of them, simply because it happens to have the capabilities a specific use case needs. 

Cost belongs in the same frame. The old speed-quality-cost triangle still applies even though costs are far lower than they were, but comparing input and output token prices tells you surprisingly little. A harness adds retrieval, instructions, and other steps around each request, so costs in practice can vary dramatically from the list price, and the only meaningful number is what a translation costs inside your actual workflow. 

The panel also agreed that traditional segment-based evaluation is inadequate for LLMs. Because these models rely on longer text and can draw on context, sending out a single sentence, getting a sentence back, and scoring it no longer produces a meaningful result. Evaluation needs to move to longer chunks, or to segments evaluated in context. 

Some rankings are evolving in a more useful direction. Intento’s AI leaderboard, for example, has moved away from a static snapshot toward a tool that lets people enter what they value and generates recommendations based on their particular requirements. That is a much more productive way to think about rankings, because it starts from the buyer’s needs rather than from a single score. 

How Does LLM Translation Quality Change for Long-Tail Languages? 

It changes dramatically, and it depends on where in the long tail you are. For tier one languages – such as English, French, German, Spanish, Chinese, Japanese, and Russian – it almost does not matter which major model you use, because all of them have reasonable capabilities. Beyond those languages, results do not generalize, and once you get past roughly the top 20 to 30 languages, the training data often simply does not exist, so models either make things up or break down. 

Regional leaderboards may help in some cases. The SEA-HELM leaderboard, from AI Singapore’s SEA-LION project, focuses on Southeast Asian languages, for example. Even so, the advice for long-tail languages comes down to three words: test, test, and test. 

Can LLMs Judge Translation Quality? 

Partially. The panel saw a real role for “LLM-as-a-judge” (using one model to assess another’s output), but with humans as the final arbiters. Nobody expected automatic metrics such as COMET or MetricX to be particularly useful on their own, and Multidimensional Quality Metrics (MQM) review by people came up as the arbiter of record, even though you do not want humans doing all the work. 

The panel was generally bullish on LLM-as-a-judge, which often comes very close to human judgment, and notably skeptical of any system that reduces quality to a single number (a striking position, given how much of the field was built on such metrics). The rub is bias. If the same LLM that generated a translation also judges it, the evaluation becomes circular: The model is likely to miss its own mistakes, because the output matches what its training tells it to expect. Using a separate LLM as the judge is one way to reduce that risk. 

There is a further wrinkle. It has been shown in most cases that LLMs rate their own translations the highest, but in our own experiments we have found one consistent exception: LLMs often rate Google Translate output above their own. Although we cannot state this with certainty, the likely explanation is that so much Google-translated text exists online that it statistically matches what the models learned to treat as good translation. 

The session’s one real disagreement was over open benchmarks, the ones anyone can run to confirm a score. In theory they keep everyone honest. In practice, they are also the benchmarks most likely to leak and be overfit to, although there is a reasonable case that they remain worth having despite those problems. 

Worth knowing: CSA Research put a deliberately naive version of this idea to the test by asking GPT-4o to act as a translation evaluation agent, checking the claim that it was a PhD-level expert in every subject. For translation evaluation, it was essentially useless (“Is GPT-5 Truly a PhD-Level Translation Analyst?”). 

How Should Enterprises Build Their Own LLM Evaluation? 

The panel did not lay out a step-by-step process, so what follows is my own extrapolation from the discussion. The short version: Define your requirements, derive a metric from them, build a benchmark, and re-evaluate regularly. This is where the opening quote applies: Whatever your evaluation rewards effectively becomes your specification, so the requirements have to come first. 

As an aside, LLMs are finally forcing enterprises to do the requirements work they should have been doing all along – the work translators always asked for and seldom got. 

For procurement, it makes sense to ask vendors how they will incorporate your requirements into their evaluation protocol, whether they have ongoing monitoring, and how it works: LLM self-judgment, quality estimation (QE), or human review. You need to know which, so you can tell whether the evaluation is adequate. In addition, budget evaluation separately from throughput, because it is an expert, value-added task rather than overhead. 

What Else Stood Out at TAUS Vancouver 2026? 

If anything surprised me, it was the level of sophistication in both the approaches and the questions. I do not think it existed six months ago, and this is an area maturing far more rapidly than I expected. More broadly, AI seems to have moved from a purely disruptive force to something people are learning to channel productively. 

Concern about the role of human expertise came up constantly, although nobody has yet found a financially viable model to reward it, because experts keep getting pushed into the “per-word trap” and treated as a commodity. Long-tail languages were another thread: The two highest-scoring “sandbox pitches” both addressed them, and the winner focused on developing language resources for African languages, reflecting a perception of real growth opportunities where major model developers supply little. 

On the speech side, a demonstration from the Karlsruhe Institute of Technology (KIT) showed meeting interpretation that can separate speakers and generate punctuation, problems models have traditionally struggled with, and it is starting to look usable.  

Where LLM Evaluation Is Heading 

Within 12 to 24 months, I expect AI to be taken for granted as a productivity aid, with many of today’s holdouts giving in, apart from cases such as compartmentalized defense work where using a model simply is not possible. We will likely see more sophisticated testing and more benchmarks that focus on language and cultural knowledge. Even then, cultural benchmarks will need to get past simple recall: Asking a model in French for the date of Bastille Day tells you only that the fact is somewhere in its training data, not that it understands French culture, so the better benchmarks will ask applied questions that test whether a model can use what it knows. 

The one thing to remember is that there is no substitute for doing your own evaluation. Test, test, and test. 

Related Research 

Frequently Asked Questions 

Is there a best LLM for translation? No. Panelists at TAUS Vancouver 2026 agreed that no single large language model is best for translation. The right choice depends on your content; target languages; and the balance of speed, cost, and quality that you need, so the model that fits a specific use case often ranks low on public leaderboards. 

Why are LLM leaderboards unreliable for multilingual AI? Many multilingual leaderboards measure reasoning rather than language ability, since a model may solve a French problem in English and translate back. Benchmark data can also leak into training, inflating scores and leaderboards rarely weigh cost or latency. They are useful for building a shortlist, not for making a final choice. 

What does it mean to evaluate the harness instead of the model? The harness is the prompts, context, tools, and workflow an LLM runs inside. Evaluating the full harness shows how the model will behave, and what it will cost, in your real workflow. Bare-model scores and token prices often differ substantially from performance and cost in practice. 

Can an LLM judge translation quality? Partially. LLM-as-a-judge can come close to human judgment, but humans need to remain final arbiters, for example through MQM review. Use a different LLM as judge than the one that produced the translation, because models tend to miss their own errors and rate their own output highly. 

How should an enterprise evaluate LLMs for translation? Start by defining requirements: content type, purpose, and audience. Turn those into a metric, use the metric to build a benchmark, and compare models or vendors against it. Evaluate longer chunks or segments in context rather than isolated sentences, and budget for evaluation as a separate expert task. 

How often should translation LLMs be re-evaluated? Re-evaluate whenever there is a major model update or a technical change such as a new tokenizer, since either can change output without warning. LLM evaluation is an ongoing process rather than a one-time selection. 

Do LLMs work well for low-resource languages? It varies widely. Major languages such as French, German, Spanish, Chinese, and Japanese perform reasonably on most models, but results do not generalize beyond them. Past roughly the top 20 to 30 languages, training data is scarce and quality can break down, so testing on your own content is essential.