An often overlooked aspect in the development of large language models (LLMs) is the efficiency of the tokenizer, especially in multilingual contexts.

Anthropic's approach

A user pointed out that Anthropic has never publicly released the tokenizers for its models, making it impossible to analyze their performance. This contrasts with other leading companies in the sector such as Google, OpenAI and Meta, which have made the tokenizers of their Gemma, GPT and Llama models available respectively.

Implications for the community

The lack of transparency on Anthropic's tokenizers makes it more difficult for the scientific community and developers to fully understand and compare the performance of Claude models compared to open-source alternatives. For those evaluating on-premise deployments, there are trade-offs related to model transparency. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.