The fact that an LLM tops English leaderboards says little about its ability to summarize in Bengali. That is the central finding of a zero-shot benchmark of eight closed-source and open-weight models on size-matched English and Bengali XL-Sum test sets. The study combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment. The result is a more realistic, but also more uncomfortable, picture of multilingual summarization.

The first crack concerns the top scores. GPT-5.6 Luna leads several English lexical metrics, but it does not keep the same advantage in Bengali: there, LLaMA 3.3 70B posts the highest ROUGE and BLEU scores. The picture becomes even more complicated with factuality, where Gemini 2.5 Flash records the best observed Bengali result. Three models effectively wear three different crowns depending on language and metric. It is a clear signal that no single ranking is enough to choose a model in a multilingual context.

For teams managing on-prem or self-hosted deployments, the open-weight detail is not neutral. LLaMA 3.3 70B, being an open model, is a natural candidate to run on owned hardware, while Gemini 2.5 Flash's factuality lead points toward an external service. The second-order consequence is that the benchmark does not only separate better models from worse ones: it also separates evaluation criteria. A team building a Bengali summarization pipeline may face a choice between an open model with better lexical overlap and a closed model with stronger factuality. That is not a technical preference; it is a trade-off between control, data sovereignty, and factual reliability.

The third level concerns evaluation infrastructure. Low-resource languages often suffer from limited reference datasets. Reference-free factuality and LLM-as-a-Judge attempt to reduce that dependence, but they introduce a new variable: judges are also LLMs, with their own biases. The finding that different evaluation methods produce different rankings indicates that model selection cannot be delegated to a single indicator. For teams doing fine-tuning or quantization of open models on local languages, continuous evaluation on the real domain is necessary, not an imported benchmark.

In this scenario, closed-source vendors have an incentive to defend English benchmarks, where certain leads are easier to communicate. The open-weight ecosystem, by contrast, gains from results like LLaMA 3.3 70B's, because a lexical win in Bengali becomes a concrete argument for self-hosting. But the factuality gap is a reminder that running a model in-house does not by itself guarantee more reliable summaries. For those evaluating on-prem deployments, there are trade-offs between model control and factual quality that require specific analytical frameworks; AI-RADAR's /llm-onpremise section offers tools to weigh these constraints.

The Bengali case is a structural reminder: an English leaderboard is a partial map. Without evaluation frameworks that hold together lexical quality, semantics, and factuality, every model choice in a low-resource language remains a bet.