Open-Weights vs Closed AI: Enterprise Decision Framework

The Debate That Misses the Point

On July 27, Anthropic CEO Dario Amodei published a position paper clarifying the company's stance on open-weights models. The headline takeaway: Anthropic does not advocate for banning open-weights models. The company supports three specific measures instead: restricting chip sales to authoritarian states, cracking down on industrial-scale distillation, and requiring mandatory safety testing for all sufficiently capable models regardless of whether they are open or closed.

The internet did what the internet does. It polarized. Within hours, the discourse split into two camps: open-weights absolutists declaring victory and closed-model skeptics warning of ungovernable risks. Both sides missed the actual question enterprise buyers are asking right now.

That question is not "are open-weights models good or bad?" It is: "Given my regulatory obligations, my data sensitivity, my risk tolerance, and my procurement requirements, which model deployment approach can I actually defend in an audit?"

This framework answers that question. It maps the open-weights versus closed-model decision onto the evaluation criteria that ISO 42001, the NIST AI RMF, and the EU AI Act actually require. No policy hot takes. Just the checklist you will need when procurement asks for evidence.

The Six Dimensions of Model Evaluation

Most enterprise AI procurement processes still evaluate models on two axes: benchmark scores and API pricing. This is dangerously incomplete. A model that scores 92% on MMLU and costs $0.15 per million tokens can still create an unremediable compliance exposure if you cannot prove where its training data came from or demonstrate that its guardrails survive after fine-tuning.

Here are the six dimensions every enterprise should evaluate before committing to an AI model deployment strategy.

1. Model Provenance and Supply Chain Risk

ISO 42001 Clause 6.1.2 requires organisations to assess risks associated with the AI system's supply chain. For closed models accessed via API, you have a contractual chain: the vendor publishes terms of service, a data processing agreement, and (for enterprise tiers) a security whitepaper. You can point an auditor at these documents.

Open-weights models invert this. You have the weights. You do not necessarily have provenance documentation. Key questions to ask:

• Training data composition: Can the model provider document what datasets were used? If training included copyrighted material or personal data scraped without consent, your organisation inherits that liability when you deploy.
• Distillation source: Amodei's paper specifically calls out industrial-scale distillation as a national security concern. For enterprise buyers, the practical question is simpler: was this model trained by distilling outputs from a closed model whose terms of service explicitly prohibit that? If yes, you are deploying a model built on a terms-of-service violation.
• Geographic supply chain: Where was the model trained? What chips were used? If the model originates from a jurisdiction subject to export controls or sanctions, your legal team needs to know before procurement signs off.

The NIST AI RMF Governance function (GOVERN 1.2) explicitly requires documenting the provenance of training data and understanding the context of model development. This is not optional for frameworks-aligned organisations.

2. Safety Testing and Guardrail Verifiability

This is the dimension where Amodei's argument is strongest and most directly relevant to enterprise risk management. His position paper cites a UK AI Security Institute report that states the core problem plainly: once open-weight models are released, "safeguards can be removed, and copies can be downloaded, redistributed, and run on private systems beyond monitoring."

For enterprise buyers, the practical implication is this: if you deploy an open-weights model, you become responsible for verifying that the safeguards work. A closed-model vendor provides API-level guardrails, content filtering, and usage monitoring. You can audit those. With open weights, you must implement and maintain your own guardrail infrastructure, and you must be able to demonstrate in an audit that it is effective.

ISO 42001 Clause 8.5 requires organisations to implement controls for AI systems based on risk assessment results. If your risk assessment identifies misuse, harmful output, or bias as material risks (and it will), you need controls. Closed models come with vendor-maintained controls. Open models require you to build them.

The key questions:

• What safety testing has been conducted on this model, and are the results published?
• If the model is open-weight, what guardrail infrastructure will you deploy alongside it, and can you demonstrate its effectiveness?
• If a vulnerability is discovered post-deployment, what is your patching path? Closed-model vendors can update the API endpoint. Open-weights models require you to redeploy.

3. Data Sovereignty and Deployment Control

This is the dimension where open-weights models genuinely win, at least on paper. If you deploy a model on your own infrastructure, you control where the data goes. For organisations subject to Australian Privacy Principle 8 (cross-border disclosure), the EU GDPR's international transfer restrictions, or APRA CPS 230's operational risk management requirements, on-premise deployment of an open-weights model can simplify the regulatory narrative considerably.

Closed-model APIs typically route data through the vendor's infrastructure. Even with enterprise agreements that promise data isolation and regional processing, you are trusting the vendor's architecture. Auditors increasingly ask for evidence, not assurances.

The pragmatic position: if your use case involves sensitive personal data, regulated financial information, or government-classified material, the data sovereignty advantages of open-weights deployment are real and defensible. But they come with the safety infrastructure costs described in Dimension 2. You are trading one compliance burden for another.

4. Regulatory Compliance Mapping

Different regulations impose different burdens depending on your deployment model. Map your specific obligations before choosing:

• EU AI Act: The Act classifies AI systems by risk category, not by deployment architecture. Whether you use open or closed models, high-risk classification triggers the same conformity assessment requirements. However, the Act's transparency obligations (Article 52) are easier to satisfy with closed-model vendors who provide documentation packages. Open-weights models require you to generate that documentation yourself.
• ISO 42001: The standard is deployment-agnostic. It asks that you have an AI management system. What changes between open and closed is the scope of what you manage. With closed models, the vendor handles portions of Annex A controls. With open models, you own the full stack.
• APRA CPS 230 (Australia): This operational risk standard requires that material service providers be managed through a service provider management policy. A closed-model API provider is clearly a material service provider. An open-weights model you self-host reduces the number of material service providers but increases your internal operational risk surface.

None of these frameworks say "thou shalt not use open-weights models." They say "thou shalt be able to demonstrate you understand and control the risks." The deployment choice affects how you demonstrate that.

5. Total Cost of Ownership

The API pricing comparison is a trap. Open-weights models appear cheaper because there is no per-token charge, but running inference at enterprise scale requires GPU infrastructure, model serving software, monitoring, and personnel who understand all of the above. That $0.00 per token comes with a monthly cloud bill and a hiring requirement.

Closed-model APIs appear more expensive per call but bundle infrastructure, monitoring, safety filtering, and uptime guarantees into the price. The real TCO comparison depends on scale and use case:

• At low volume (hundreds to low thousands of inferences per day), closed-model APIs are almost always cheaper once you account for the human cost of managing infrastructure.
• At high sustained volume (millions of inferences per day), open-weights self-hosting can break even or win if you already have GPU capacity and MLOps talent.
• The hidden cost nobody budgets for is the audit evidence generation. If you self-host an open-weights model, you need to produce documentation that a closed-model vendor would provide as part of their enterprise agreement. That documentation takes person-weeks to create.

6. Vendor Lock-in and Exit Strategy

Open-weights advocates correctly identify lock-in as a closed-model risk. If you build your application against a specific vendor's API, migrating to another provider requires re-engineering integration layers, prompt engineering, and output parsing. This is real switching cost.

But open-weights models have their own lock-in dynamic. If you fine-tune an open model on your proprietary data and build custom guardrails around it, that model becomes a unique asset. Migrating to a different base model means redoing the fine-tuning, revalidating the guardrails, and potentially retraining downstream systems that depend on specific model behaviors.

The healthier approach is to treat both deployment paths as requiring an explicit exit strategy. Document your migration path before you commit to either. ISO 42001 Clause 10.1 recommends continual improvement planning. That planning should include what happens when you need to switch models.

Putting the Framework to Work

Here is how to apply these six dimensions in a procurement decision:

Step 1: Score each dimension. For your specific use case and regulatory environment, rate each dimension as favoring open-weights, favoring closed, or neutral. Do not average the scores. A single dimension (say, data sovereignty for a healthcare deployment) can be dispositive regardless of what the other five say.

Step 2: Identify your non-negotiables. If your data cannot leave Australian jurisdiction under APP 8, open-weights self-hosted within Australia may be your only option. If you cannot staff an MLOps team capable of maintaining guardrail infrastructure, closed-model APIs provide safety controls you cannot reproduce internally. Non-negotiables override scores.

Step 3: Document your rationale. ISO 42001 auditors are not looking for a specific answer. They are looking for evidence that you asked the right questions and documented the reasoning. A well-documented decision to use a closed model with identified risks and mitigations passes audit. An undocumented decision to use an open-weights model because "open is better" does not.

Step 4: Build your evidence pack. For whatever model you choose, assemble: training data provenance documentation (or evidence of why it is unavailable), safety testing results, guardrail architecture and effectiveness metrics, data flow diagrams showing where data goes, and your exit strategy. If the vendor cannot provide a document, note that gap and explain your compensating control.

Step 5: Schedule the review. Model evaluation is not a one-time procurement gate. It is an ongoing governance obligation. ISO 42001 Clause 9.1 requires monitoring, measurement, analysis, and evaluation. Set a calendar reminder for six months out. In six months, the model landscape will have shifted, new safety research will be published, and your risk assessment will need updating.

What the Policy Debate Misses

Amodei's position paper is worth reading in full. It is measured and specific in ways that the subsequent internet discourse is not. His three proposed measures (chip restrictions, anti-distillation enforcement, mandatory safety testing) are narrower and more actionable than the "ban everything" strawman attributed to Anthropic's critics.

But the paper is written for policymakers, not procurement teams. It does not tell you how to fill out your AI system inventory for ISO 42001. It does not help your legal team evaluate whether a model's training data provenance creates downstream liability under Australian privacy law.

That is the gap this framework fills. The policy debate will continue. Your procurement deadline will not wait for a consensus. Use these six dimensions to make a decision you can defend in an audit, document it properly, and schedule the review. That is what governance actually looks like.

For more on enterprise AI procurement evidence, see our AI Vendor DPA Checklist and analysis of why buyers are ghosting AI products. If you need help building an evidence pack for your next AI procurement decision, get in touch or see a sample report.

Written by David Swan, reviewed and fact-checked against primary regulatory sources. AI-assisted but human-directed.