Private AI vs Cloud AI for the Mid-Market

Your legal team signed off on the data processing agreement. Checked the box. Then someone actually read the retention terms and asked the question nobody in the room could answer: how long does the provider keep our inputs, and where?

The answer is in the documentation, and it is more precise than most comparison articles admit. OpenAI writes, verbatim: "By default, abuse monitoring logs are generated for all API feature usage and retained for up to 30 days, unless longer retention is required by law, or is reasonably necessary to protect our services or any third party from harm" (OpenAI, Your data, read 2026-08-13). Thirty days of abuse logs is not a scandal. It is a default you need to know about before client files go into it.

Is private AI better than cloud AI?

Private AI is not better across the board, because a cloud provider's EU region is enough while you validate a use case and process no professionally protected data. Dedicated or self-run infrastructure pays off once client, personnel or financial records, several departments in one system and growing usage coincide, and that is where AILoopwise builds most systems: inside your own systems and accounts, agreed per project.

What changed in 2026 — and it argues against us

The standard opening for private AI is: your data leaves the EU. That sentence was true for a long time. It is no longer true as a blanket statement, and a vendor who does not tell you that is selling you an out-of-date fear.

The same documentation page lists the available regions as the United States and Europe (EEA + Switzerland). The condition sits right next to it, verbatim: "To use data residency with any region other than the United States, you must be approved for abuse monitoring controls, and execute a Modified Retention amendment" (ibid., read 2026-08-13). Get the approval, sign the amendment, and you have European data residency from an American provider.

That is a real improvement and it narrows the gap. The honest consequence for this comparison: data residency on its own is no longer a reason to run your own infrastructure. If your only requirement is that data rests inside the EEA, a provider's EU region is the faster and cheaper route, and you should take it.

Three questions an EU region does not answer

Who decides retention? A region is a storage location. The 30-day default remains the provider's default, and it changes when they change it. The amendment that moves it is a contract with a party who has their own reasons to keep logs. That is not an accusation — it is a question of authority, and the answer is: not you.

Where does the separation between departments live? That is the section below, because it is the one comparison articles routinely skip.

What does it cost in eighteen months? A price per token is not a cost structure. It is a function of your usage, and it rises exactly when the system finally gets used — that is, when the project succeeds. This is the least pleasant property of usage-based pricing: it taxes adoption, and you notice only after the department has grown fond of the tool.

Owned hardware behaves the other way round. A GPU server at Hetzner in Falkenstein costs what it costs: Hetzner's own announcement puts the GEX44 at "€ 184.00 a month and a one-time setup fee of € 79.00" (Hetzner, 04.03.2024, read 2026-08-13) — NVIDIA RTX 4000 SFF Ada with 20 GB, Intel Core i5-13500, 64 GB RAM. Larger configurations cost a multiple of that, and we have not verified current configurator pricing here; treat the number as an order of magnitude, not a quote. What matters is not the amount. It is that the amount is the same in January as in June.

Hosted product or a system in your own infrastructure

"Cloud or private" is how comparisons frame it. The choice a mid-sized company actually makes is a different one: subscribe to a hosted AI product, or have a system built into the tools and infrastructure you already run. Both can be GDPR-compliant. They differ in where the separation between departments, clients and vendors lives.

If HR, finance and legal query the same knowledge base, what stops a personnel question from surfacing a contract draft from another store? In a hosted product, the answer is the vendor's design: you read its documentation, its certifications and its contract, and you trust the part you cannot see. That is a legitimate choice, and for many teams the right one.

In a system built into your own infrastructure, the separation sits where you already administer things — your accounts, your access rights, your logs — so your data protection officer checks it with the same tools as the rest of your IT. AILoopwise works this way: we agree with you, per project, where each part of the system runs and who can access the data — typically inside your own systems and accounts. One step stays contractual even then. When a project uses Claude via Anthropic's API, the model call runs at Anthropic, not guaranteed inside the EU, and that step is governed by contract rather than by your own infrastructure.

For anyone bound by professional secrecy this stops being a nicety. German tax advisers are the clearest case: § 62a StBerG imposes four cumulative conditions, and a fifth once the provider works abroad, and a data processing agreement satisfies none of them on its own — access only "soweit dies für die Inanspruchnahme der Dienstleistung erforderlich ist" (para. 1); a duty to select the service provider carefully and to end the arrangement "unverzüglich" if the conditions lapse (para. 2); a contract in Textform obliging the provider to secrecy "unter Belehrung über die strafrechtlichen Folgen einer Pflichtverletzung" (para. 3); and, for mandate-specific services, the client's consent (para. 5) without that consent switching off paras. 2 and 3 (para. 6); and for services provided abroad, protection of secrets comparable to Germany's (para. 4) (§ 62a StBerG, gesetze-im-internet.de).

The notable thing about that provision: it dates from 9 November 2017 and has not been amended since. It does not mention artificial intelligence once — and never needed to. Its test was always whether a third party gains access to someone else's secrets, and whether that party is under an obligation. A language model is exactly that third party.

Frequently Asked Questions

Is private AI really better than cloud AI?

Not across the board. A cloud provider's EU region is enough for validation without professionally protected data; dedicated or self-run infrastructure pays off when protected data, several departments in one system and growing usage coincide.

How long does the OpenAI API keep inputs?

According to OpenAI, abuse-monitoring logs are kept for up to 30 days by default unless the law requires longer. European data residency requires approval and a Modified Retention amendment.

Is an EU region enough for tax advisers and other professional-secrecy holders?

A region sets the storage location, not the duties under § 62a StBerG: necessity, careful selection, a secrecy undertaking in text form, comparable protection for services provided abroad, and client consent for mandate-specific services.

What does self-run AI infrastructure cost?

A fixed monthly price instead of a per-token price. Hetzner listed its GEX44 GPU server at €184 a month plus €79 once (https://www.hetzner.com/pressroom/new-gpu-server/); larger configurations cost a multiple, so the figure is an order of magnitude.

Which setup fits you

A cloud provider's EU region is the right call when you are validating a use case, when no professionally privileged data is involved, and when usage stays modest. It is fast, it is good, and it is now defensible under GDPR.

Dedicated or self-run infrastructure earns its keep when three things coincide: you process client, personnel or financial records; several departments or clients share one system; and usage grows to the point where a token price becomes a running unknown. We agree with you, per project, where each part of the system runs and who can access the data — typically inside your own systems and accounts. With Claude, the model call itself runs at Anthropic — a blanket statement about where data sits is precisely the promise nobody can keep later.

There is a middle path that gets recommended rarely, because it earns nobody a margin: a European-hosted managed model. You keep per-token pricing, the data stays in Europe, and you operate no GPUs. For many companies that is the sensible answer for the first twelve months.

If you did not recognise your own situation above, you probably need neither. That is a permissible outcome, and we would rather say it now.

---

See it running on your own documents — answered in your language, at a cost that fits a budget line. Book 30 minutes, no slides.

Claude and Anthropic are trademarks of Anthropic, PBC. AILoopwise is an independent AI implementation company; the use of these names does not imply endorsement by Anthropic.