When a law firm evaluates a cloud AI tool, the first question is usually whether the vendor trains on the data, and under what subprocessing conditions. That matters, but the sharper question is where the data goes.
Depending on how it leaves the network, different regulations and laws apply to it wherever it lands. That’s a data sovereignty problem, and it’s getting harder to ignore as cloud AI tools get easier to spin up.
The fix is straightforward, and it’s not new: a private LLM platform running the model inside an environment the firm controls. Client data, privileged material, and internal matter files never leave the network.
You may have heard “private” and “sovereign” used interchangeably. They’re not the same.
Regulators in financial services and healthcare have started drawing a clear line between them, and the space between them is where firms get caught. A model can run on infrastructure a firm calls private and still sit under a jurisdiction that can reach the data.
The American Bar Association issued Formal Opinion 512 in July 2024, mapping existing duties onto AI use: competence (Rule 1.1), confidentiality (Rule 1.6), and supervision (Rules 5.1 and 5.3). The confidentiality piece is the one sovereignty turns on. A lawyer has to know where client information goes when it’s submitted to a tool—which servers hold it, which company operates them, and which legal system can demand it.
Where Client Data Goes, and Who Can Reach It
Model Rule 1.6(a) prohibits revealing information relating to a client’s representation without consent, and that doesn’t require anyone to intend a disclosure. Pasting a privileged memo into a cloud tool is a transmission to an outside party, whatever the vendor’s terms say about training. Once sensitive information leaves the network, that’s the reality regardless of how secure the connection is.
The third-party doctrine holds that voluntarily sharing information with an outside party can weaken or waive privilege, depending on the jurisdiction. Courts haven’t settled how AI vendor access fits, and “we used a reputable vendor” is not a position anyone wants to argue in front of a judge.
Keeping data in-network is its own architectural subject. Sovereignty is the layer on top.
Sovereignty Is a Jurisdiction Question
Sovereignty covers more than keeping data off shared servers. The harder question is which government can reach it, and under what law.
A private instance running in Frankfurt answers to different rules than the same instance in Virginia. For matters with cross-border exposure—EU client data under the GDPR, regulated industries, government work—the physical location of the computer (or subprocessor, which isn’t always the same machine) matters as much as the isolation around it.
Location alone isn’t enough. Under the U.S. CLOUD Act, an American provider can be compelled to produce data it controls even when the servers sit overseas. “We host in Europe” doesn’t place the data beyond U.S. reach if the company operating it is American. Sovereignty has two inputs: where the data rests, and who controls the operator.
On-premises hardware settles both, because the firm can name the building and the law that governs it. Most other arrangements settle one and leave the other open.
Going “Private”
“Private” gets used loosely in vendor decks. It’s worth being specific about what each arrangement actually controls.
On-Premises Hardware
With on-premises hardware, the model runs on servers the firm owns, inside its own space. Data physically can’t leave, and no outside operator sits in the loop. There are upfront costs, but firms that move away from cloud tools often find they’re no longer paying recurring fees to run sensitive data through infrastructure they don’t control.
Private Cloud
There’s also the private cloud option: a dedicated, network-isolated environment inside a cloud provider, for firms that don’t maintain their own server space. Your data stays separated from other tenants, which handles the confidentiality side. Sovereignty is only partial—the cloud operator still runs the hardware, and that operator answers to its home jurisdiction. A dedicated tenancy on a U.S. hyperscaler is still a U.S.-reachable environment.
Vendor-Hosted Private Instances
Then there are vendor-hosted private instances. A legal AI vendor runs an isolated deployment for the firm. Confidentiality rides on the contract and the architecture. Sovereignty rides on where the vendor runs it and who the vendor answers to—worth getting in writing rather than assuming.
No single model wins outright. The useful distinction is that “private” describes isolation while sovereignty describes reach, and a firm needs to know which one a given product is actually selling. For the hardware and retrieval details under each option, see what a private LLM for law firms actually requires.
Questions That Surface the Gaps
Most of this comes out in a handful of direct questions, asked before signing anything.
Where does inference run, by country?
The query gets processed somewhere physical. Get the country, the region, and whether it can shift to another location under load.
Who is the legal operator of the environment?
The entity that controls the machine is the entity a court or government can compel. Know who that is and what jurisdiction they answer to.
Where are logs and outputs stored, and for how long?
Retention stretches the exposure window. Query logs can hold the same privileged content the queries did.
Can the firm get audit logs?
Supervision under Rules 5.1 and 5.3 needs records of what was submitted, what was retrieved, and what came back. If the vendor can’t produce them, that’s a gap you’ll own.
What happens to the data on termination?
Deletion timelines and return procedures belong in the contract, not in a support ticket two years later.
If the answers come back as contract language where you asked about architecture, note it.
“We contractually restrict access” is a promise. “The model runs on your server, in your building” is a fact.
Keeping data in a known jurisdiction does nothing for whether the output is right. A 2024 Stanford RegLab study found that legal-specific AI tools returned incorrect information between 17% and 33% of the time. That rate comes from how the models work. Where you run them changes none of it. Grounding the model in the firm’s own vetted documents through retrieval brings it down. Human review closes the rest.
The duty doesn’t move with the hardware. Mata v. Avianca involved a cloud tool, and the lesson holds for a private one: the lawyer who signs the filing answers for what’s in it, whatever produced the draft.
What This Looks Like in Practice
Firms tend to start with work that’s high-volume and lower-judgment, where keeping the underlying material inside a known boundary is the whole point.
Matter Research and Precedent Lookup
Matter research and precedent lookup across the firm’s own briefs and memos, with citations, without routing the query through an outside service.
First-Pass Contract Review
First-pass contract review against the firm’s standard positions, flagging deviations for a lawyer to read before anything reaches the client.
Deposition and Hearing Preparation
Deposition and hearing prep over transcripts and exhibits from related matters—source material that would be risky to send anywhere outside the firm.
Litigation Support
Document-heavy litigation support, where a production set can run to hundreds of thousands of files that shouldn’t touch external infrastructure.
The Governance Piece
A private LLM without a governing policy is just a confidential way to make mistakes. The policy has to name which tools are approved for what, who can authorize a new use, how output gets verified before it reaches a client or a court, and how staff are trained on all of it.
Sovereignty makes that policy shorter. When the data handling is settled by where the model runs, the policy can focus on use standards, verification, and supervision instead of re-arguing where client data goes for every tool on the list.
The firms handling this well aren’t waiting for the rules to harden. They’re making defensible calls now and writing down the reasoning, so they can show their work when a client or a regulator asks where the data went.
Conclusion
As AI adoption accelerates across the legal industry, law firms are under growing pressure to balance innovation with their professional obligations around confidentiality, supervision, and client trust. Deploying a private LLM inside a controlled environment gives firms a way to use advanced AI tools without surrendering visibility into where sensitive information goes or who can reach it.
The distinction between privacy and sovereignty matters. A system can be isolated from other users and still remain subject to outside jurisdictional reach. Firms evaluating legal AI platforms need to understand both the technical architecture and the legal authority behind the infrastructure handling their data.
Private LLM deployments do not remove the need for human review, governance, or oversight. What they do provide is a defensible foundation for firms that want to adopt AI while keeping privileged information inside known boundaries. For firms handling regulated, confidential, or cross-border matters, that control is quickly becoming less of a preference and more of a requirement.