Can you run an LLM without sending data to OpenAI or Anthropic?

Yes. Open-weight models run on your own hardware or inside your private cloud, so prompts and documents never leave your boundary. The tradeoff is that you take on model hosting, scaling, and evaluation yourself, and the very best frontier models are still hosted-only.

What is a restricted model policy?

An organizational rule, common in government and regulated industry, that limits which AI models may be used and where data may be sent. In practice it often means no third-party model APIs, approved open-weight models only, and all inference inside a controlled environment.

Are open-weight models good enough for production?

For most enterprise tasks (extraction, classification, summarization, retrieval-grounded question answering) yes. The gap to frontier hosted models is small on constrained, well-evaluated tasks and largest on open-ended reasoning. Scope the task and evaluate before committing.

What is the hardest part of running LLMs in an air-gapped environment?

Everything that normally lives in someone else's cloud: model serving and autoscaling, a vector database, an evaluation harness, and update paths for model weights. All of it reproduced inside the boundary, and all of it subject to the same audit requirements as the rest of the system.

Can You Run an LLM Without Sending Data to a Third Party?

Quick answer: Yes. Open-weight models run entirely inside your own infrastructure, so no prompt or document ever reaches a third party. You trade the convenience of a hosted API for owning model serving, a self-hosted vector store, and an offline evaluation harness. For scoped enterprise tasks, well-evaluated open-weight models are usually good enough, and for regulated work they are often the only option.

Most AI tutorials assume you can call a hosted model API. A large share of regulated-industry work starts from the opposite assumption: data cannot leave the environment, the list of permitted models is short, and every component has to be auditable. Building useful AI under those constraints is very doable. It just moves the work around.

What does “restricted model policy” mean in practice?

It usually comes down to three rules. No calls to third-party model providers, only approved (typically open-weight) models, and all inference inside a controlled or disconnected environment. Everything you would normally offload to a vendor’s cloud now lives inside the boundary and inherits the same compliance requirements as the rest of the client’s stack.

The components you now own

Component	Hosted-API world	Inside your boundary
The model	Vendor endpoint	Open-weight model on your GPUs
Serving and scaling	Vendor’s problem	Your inference server, your capacity planning
Retrieval store	Managed vector DB	Self-hosted vector database
Evaluation	Ad hoc, online	Offline harness with a fixed dataset
Model updates	Automatic	Deliberate, reviewed weight updates

None of this is exotic. It is just yours now, and each piece has to satisfy the same audit and change-control process as any other production system.

Are open-weight models good enough?

For the tasks most enterprises actually need (pulling structured fields out of documents, classifying and routing, summarizing, answering questions grounded in retrieved context) current open-weight models are strong. The gap to frontier hosted models shows up on open-ended reasoning and long-horizon planning, and it shrinks fast when the task is narrow and well-defined.

The move is to scope the task tightly, build an evaluation set from real examples, and measure. Do not assume you need the biggest model. Prove what the task requires. The ingestion pipeline and retrieval quality often matter more than model choice.

Retrieval keeps data in the boundary

Retrieval-augmented generation is a natural fit for restricted environments, because it keeps the knowledge base on your side. Documents are ingested, chunked, embedded, and stored locally. At query time, relevant chunks are pulled from your own vector store and passed to your own model. Nothing about the sensitive content transits an external service.

The engineering discipline is the same as anywhere: embed on ingest, key records for replay, track retrieval hit rate. The added constraint is that every dependency has to be self-hostable.

Evaluation has to work offline

You cannot lean on live traffic and vendor dashboards. Build a golden dataset of representative inputs with known-good outputs, score against it automatically (exact match, structured-field accuracy, or a local model as judge), and run it on every model or prompt change. In a controlled environment this harness is not optional. It is how you demonstrate the system still behaves after an update. More in how to evaluate a RAG system.

Agent security matters more here, not less

If the system uses agents and tools, the isolation story has to be airtight: sandboxed tool execution, default-deny network egress, and no agent inheriting privileges it should not have. In a disconnected environment a mistake here does not leak to the internet, but it can still cross classification or tenancy boundaries inside the enclave. This is the subject of ongoing research. See multi-agent orchestration in production.

The takeaway

Running LLMs without a third party is a solved problem for scoped enterprise tasks. You give up a hosted API and take on serving, storage, and evaluation. All inside your boundary, all auditable. For regulated work that is not a downside. It is the requirement.

If you need AI that runs entirely inside a controlled environment, that is Eximo’s core context.