A year ago, Hayk Grigorian began pre-training language models solely on 19th-century London data. Now the TimeCapsuleLLM project has reached a milestone: a 40-billion-token dataset (160 GB) of English texts from 1800 to 1875, sourced from England and the United States. On a 5-billion-token sample, Grigorian has already trained from scratch a 500-million-parameter evaluation model, then fine-tuned it on synthetic question-answer pairs derived from the same historical texts.

The result, available on Hugging Face and GitHub, is an LLM that can answer questions about historical figures, places, and events of the era—performing especially well on London-related content. The model is still rough (it was built as a proof of concept), but the author sees promising signs for a future full-scale training with a 2-billion-parameter model. One concrete example: the plum pudding recipe generated by the current version is, in Grigorian’s words, “insane”—one hopes the larger model won’t suggest stirring with your feet.

Beyond the historical curiosity, TimeCapsuleLLM signals a broader trend relevant to on-premise deployment. Training a language model from scratch on a specialized corpus can be done entirely on owned hardware with modest-sized models, without sending data to cloud providers. This means retaining full control over the entire pipeline: from source selection to production, including fine-tuning for specific tasks. For cultural institutions, archives, or organizations handling sensitive textual assets, the prospect is tangible: a custom LLM trained on local servers, able to interrogate documents without a single word leaving the corporate perimeter.

The project also demonstrates technical feasibility: a 500-million-parameter model can be trained in reasonable time on a single consumer GPU, such as an NVIDIA RTX 3090 or 4090 with 24 GB of VRAM, and the 40-billion-token dataset is manageable with inexpensive storage. The total cost of ownership (TCO) of such an operation, excluding labor for data curation, runs to a few thousand euros of hardware—an order of magnitude less than renting equivalent cloud instances for extended training runs. Scaling to a 2-billion-parameter model requires more memory and time, but remains within reach for multi-GPU workstations or servers with unified memory.

The initiative fits a broader current: the creation of “artisanal” LLMs trained on niche corpora, from legal documents to medieval manuscripts. For those evaluating the trade-off between control, cost, and quality, AI-RADAR provides analytical frameworks at /llm-onpremise to navigate deployment choices. TimeCapsuleLLM reminds us that data sovereignty is not a mirage but a practical reality—even for a solo developer working with texts from two centuries ago.