On-premise LLMs for the Italian SMB

Note: this is practical guidance, not legal advice. For a specific compliance decision, involve your DPO or a qualified data-protection lawyer.

Step 1 — The decision test: do you actually need it?

On-premise makes sense if you answer yes to at least one of these. If all are no, start with a compliant EU cloud API.

• Do you process personal, health, legal, financial or trade-secret data that legally or contractually cannot leave your control?
• Would sending your documents to a US-based AI provider trigger an international-transfer problem your clients or sector rules won't accept?
• Is your query volume high and steady (thousands per day) — the regime where per-token API bills overtake hardware?
• Do you need guaranteed availability, offline operation, or full auditability of the model and its data?

The deeper cloud-vs-local trade-off — including the hybrid pattern most companies actually land on — is in our on-premise vs cloud guide.

Step 2 — The compliance picture: GDPR + EU AI Act

Two rulebooks apply to an Italian company using AI. GDPR governs the personal data flowing through the model: on-premise doesn't grant compliance, but by keeping data inside your walls it removes the hardest questions — international transfers, third-party processors, and where exactly your customers' data is being read. That's why sensitive-data SMBs reach for it: it shrinks the compliance surface rather than expanding it.

The EU AI Act adds a risk-based layer, phasing in through 2026–2027. Most SMB uses (drafting, summarizing, internal search, a support assistant) are minimal or limited risk — your main duty is transparency: tell people when they're talking to an AI. You climb into high-risk obligations only in specific domains (recruitment, credit scoring, and similar). Running the model locally doesn't change your risk tier, but it does make the required record-keeping and auditability dramatically easier, because everything the system did happened on hardware you control. The full breakdown and a checklist are in our EU AI Act guide.

Step 3 — The three realistic budget tiers

These are hardware figures. Add setup and upkeep — a pilot can be internal-IT time; a production system wants a few days of specialist setup plus ongoing maintenance. The picks behind each tier are in our GPU guide, and the full ownership cost in the cost guide.

Step 4 — Build vs API: where they cross

Ignoring compliance for a moment and looking at cost alone: a cloud API is cheaper until your volume is high enough that the hardware pays for itself. The crossover depends on your real usage, but the shape is simple — a €3k pilot box amortizes against API spend within roughly a year at a few hundred thousand tokens a day of steady use, and much faster at higher volume. Below that, the cloud wins on cost; above it, local wins and you keep the compliance benefits for free.

The honest decision rule: let compliance decide whether you go on-premise, and let volume decide when the economics also agree. When both point the same way — sensitive data and steady volume — the choice makes itself.

Step 5 — Where Italian SMBs get real value first

• Internal knowledge search over contracts, manuals and procedures (RAG over your own documents) — high value, low risk.
• Drafting and rewriting quotes, emails and reports in Italian — a strong 27B handles this locally with no data leaving.
• A support assistant grounded in your product docs — deflects repetitive questions without exposing customer data.
• Structured extraction from invoices, PDFs and forms — a workhorse task where a small local model shines.

The build pattern for the first two is our RAG chatbot tutorial; the enterprise architecture is in private ChatGPT for business.