An Evolution for Cohere Transcribe: Diarization and Timestamps

Cohere Transcribe has established itself as one of the most performant open-source speech-to-text models on the market, often favorably compared even to proprietary solutions. Its ability to convert speech into text with high accuracy has made it a valuable tool for developers and businesses. However, one of its main limitations lay in the absence of features crucial for many professional contexts: diarization, which is the identification of different speakers in a conversation, and the assignment of precise timestamps to specific text segments.

This gap, despite the presence of dedicated tokens in the original model's tokenizer suggesting its potential, represented an obstacle to adoption in scenarios requiring detailed analysis of voice interactions. The open-source community, once again, demonstrated its responsiveness, leading to an important innovation that significantly extends Cohere Transcribe's capabilities.

Technical Details and Fine-tuning Performance

A recent fine-tuning project has addressed these limitations, successfully integrating diarization and timestamps into the Cohere Transcribe model. The process enabled the activation of latent functionalities, leveraging existing tokens in the tokenizer to train the model to recognize and mark speaker changes and temporal points within the transcription. The generated output now follows an easily parsable standard, including temporal markers and speaker identifiers.

The accuracy metrics for timestamps are notable: the model achieves an average precision of 0.097 seconds, with 90% of timestamps falling within 0.006 seconds of the vocal event. Regarding diarization, the model can distinguish up to 4 speakers per 30 seconds of audio. Furthermore, the use of a dedicated script, diarize_long.py, extends this capability, accurately identifying up to 32 people in broader conversational contexts. This fine-tuned version has been made freely available on Hugging Face, making it accessible to a wide audience of developers and businesses.

Implications for On-Premise Deployments and Data Sovereignty

The introduction of these features in an open-source model like Cohere Transcribe has significant implications for organizations prioritizing on-premise or hybrid deployments. The ability to run an advanced speech-to-text model, complete with diarization and timestamps, within one's own infrastructure offers unprecedented control over data. This is particularly relevant for sectors such as finance, healthcare, or public administration, where data sovereignty, regulatory compliance (e.g., GDPR), and security are non-negotiable requirements.

Adopting self-hosted solutions reduces reliance on third-party cloud services, eliminating risks associated with the transit and storage of sensitive data outside the corporate perimeter. Moreover, the open-source nature of the model allows for greater flexibility in customization and integration with existing technology stacks, potentially optimizing long-term TCO. For those evaluating on-premise deployments, AI-RADAR offers analytical frameworks on /llm-onpremise to assess trade-offs between costs, performance, and control.

Future Prospects and the Role of the Community

This fine-tuning of Cohere Transcribe is a prime example of the added value that the open-source community can bring to artificial intelligence development. Enhancements like diarization and timestamps not only make models more versatile but also expand their scope of application in critical enterprise contexts. The free availability of these innovations democratizes access to advanced technologies, allowing more organizations to experiment with and implement robust AI solutions without prohibitive financial burdens.

As the LLM and AI model ecosystem continues to evolve, the trend towards more open and customizable solutions strengthens. Companies investing in on-premise inference and training infrastructure can benefit from these developments, building resilient AI pipelines that comply with their specific needs while maintaining full control over their digital assets.