Private AI vs Cloud AI: What Mid-Market IT Leaders Need to Know

Your legal team signed off on the data processing agreement. Checked the box. Then someone actually read the retention terms and asked the question nobody in the room could answer: how long does the provider keep our inputs, and where?

The answer is in the documentation, and it is more precise than most comparison articles admit. OpenAI writes, verbatim: "By default, abuse monitoring logs are generated for all API feature usage and retained for up to 30 days, unless longer retention is required by law, or is reasonably necessary to protect our services or any third party from harm" (OpenAI, Your data, read 2026-08-13). Thirty days of abuse logs is not a scandal. It is a default you need to know about before client files go into it.

What changed in 2026 — and it argues against us

The standard opening for private AI is: your data leaves the EU. That sentence was true for a long time. It is no longer true as a blanket statement, and a vendor who does not tell you that is selling you an out-of-date fear.

The same documentation page lists the available regions as the United States and Europe (EEA + Switzerland). The condition sits right next to it, verbatim: "To use data residency with any region other than the United States, you must be approved for abuse monitoring controls, and execute a Modified Retention amendment" (ibid., read 2026-08-13). Get the approval, sign the amendment, and you have European data residency from an American provider.

That is a real improvement and it narrows the gap. The honest consequence for this comparison: data residency on its own is no longer a reason to run your own infrastructure. If your only requirement is that data rests inside the EEA, a provider's EU region is the faster and cheaper route, and you should take it.

Three questions an EU region does not answer

Who decides retention? A region is a storage location. The 30-day default remains the provider's default, and it changes when they change it. The amendment that moves it is a contract with a party who has their own reasons to keep logs. That is not an accusation — it is a question of authority, and the answer is: not you.

What happens between two tenants? That is the section below, because it is the one comparison articles routinely skip.

What does it cost in eighteen months? A price per token is not a cost structure. It is a function of your usage, and it rises exactly when the system finally gets used — that is, when the project succeeds. This is the least pleasant property of usage-based pricing: it taxes adoption, and you notice only after the department has grown fond of the tool.

Owned hardware behaves the other way round. A GPU server at Hetzner in Falkenstein costs what it costs: Hetzner's own announcement puts the GEX44 at "€ 184.00 a month and a one-time setup fee of € 79.00" (Hetzner, 04.03.2024, read 2026-08-13) — NVIDIA RTX 4000 SFF Ada with 20 GB, Intel Core i5-13500, 64 GB RAM. Larger configurations cost a multiple of that, and we have not verified current configurator pricing here; treat the number as an order of magnitude, not a quote. What matters is not the amount. It is that the amount is the same in January as in June.

The multi-tenant problem most comparisons skip

Cloud and private AI share a problem vendor comparisons stay quiet about, because it is awkward for both sides: contamination between tenants.

If HR, finance and legal query the same knowledge base, what stops a personnel question from surfacing a contract draft from another store? With a cloud service you depend entirely on the provider's separation architecture. You cannot inspect it and you cannot show it to an auditor.

So we build the separation structurally rather than as a rule: a dedicated PostgreSQL schema per client, vector search scoped so it never reaches past the schema, API keys stored as SHA-256 hashes, no cross-schema query in the data model. The difference is that it is inspectable — you put the schema in front of your data protection officer. A contractual assurance is a different object from a separation you can draw.

For anyone bound by professional secrecy this stops being a nicety. German tax advisers are the clearest case: § 62a StBerG imposes four cumulative conditions, and a data processing agreement satisfies none of them on its own — access only "soweit dies für die Inanspruchnahme der Dienstleistung erforderlich ist" (para. 1); a duty to select the service provider carefully and to end the arrangement "unverzüglich" if the conditions lapse (para. 2); a contract in Textform obliging the provider to secrecy "unter Belehrung über die strafrechtlichen Folgen einer Pflichtverletzung" (para. 3); and, for mandate-specific services, the client's consent (para. 5) without that consent switching off paras. 2 and 3 (para. 6) (§ 62a StBerG, gesetze-im-internet.de).

The notable thing about that provision: it dates from 9 November 2017 and has not been amended since. It does not mention artificial intelligence once — and never needed to. Its test was always whether a third party gains access to someone else's secrets, and whether that party is under an obligation. A language model is exactly that third party.

Which setup fits you

A cloud provider's EU region is the right call when you are validating a use case, when no professionally privileged data is involved, and when usage stays modest. It is fast, it is good, and it is now defensible under GDPR.

Dedicated or self-run infrastructure earns its keep when three things coincide: you process client, personnel or financial records; several departments or clients share one system; and usage grows to the point where a token price becomes a running unknown. For EU clients we run on Hetzner in Germany; for clients outside the EU, on the infrastructure their own law requires — a blanket statement about where data sits is precisely the promise nobody can keep later.

There is a middle path that gets recommended rarely, because it earns nobody a margin: a European-hosted managed model. You keep per-token pricing, the data stays in Europe, and you operate no GPUs. For many companies that is the sensible answer for the first twelve months.

If you did not recognise your own situation above, you probably need neither. That is a permissible outcome, and we would rather say it now.

---

See it running on your own documents — indexed in Germany, answered in your language, at a cost that fits a budget line. Book 30 minutes, no slides.