For much of the generative AI boom, enterprise technology strategy has revolved around one question: Which model should we use?
GPT, Claude, Gemini, Llama and an expanding field of specialist models are compared on reasoning ability, context windows, benchmark performance and cost.
But as organizations move from AI experiments to production systems, that question is becoming less important.
The more consequential architectural question is: What surrounds the model?
An AI model can reason, generate text and interpret information. But it cannot, by itself, reliably operate an enterprise process. For that, organizations need an additional architectural layer responsible for context, memory, tools, permissions, workflow, monitoring and recovery.
That layer is increasingly described as the agent harness.
IBM’s explanation of models and harnesses highlights an important distinction between model inference and the orchestration required to turn model intelligence into useful agent behavior.
For enterprise architects, that distinction has significant consequences.
The model is becoming a component, not the architecture
Consider a conventional AI application, where a user submits a prompt. The application sends it to a language model and the model generates an answer.
The architecture is comparatively simple: User → Application → Model → Response
An enterprise AI agent is different.
Suppose an organization wants an AI agent that can investigate an invoice discrepancy.
The agent may need to:
The language model performs only part of this process.
The surrounding system determines what the model knows, what it can do and what happens when something goes wrong.
That surrounding system is the harness.
What belongs in an enterprise agent harness?
There is no single universal definition, but architecturally an enterprise-grade harness can be thought of as several cooperating capabilities.
Context management
The harness determines what information should be supplied to the model for a particular task.
That might include customer records, documents, policies, previous conversations, application state or dynamically retrieved enterprise knowledge.
This makes context engineering an architectural concern rather than merely a prompting technique.
Memory
Agents may need information that persists beyond a single inference request.
But enterprise memory cannot simply mean “remember everything.”
Architects must decide:
- what should be remembered;
- where it should be stored;
- how long it should survive;
- who can access it;
- whether it contains regulated information;
- how it can be corrected or deleted.
Memory therefore intersects with information architecture, security and records management.
Tool orchestration
Useful agents need to interact with systems. Those tools might include APIs, databases, search services, SaaS applications, workflow engines or emerging interfaces such as MCP. The harness determines which tools are available and when they should be invoked. This creates a new integration layer in the enterprise architecture.
Identity and authorization
An agent capable of calling enterprise systems requires an identity. More importantly, the architecture must determine whose authority the agent is exercising. If an employee asks an agent to retrieve salary information, approve an invoice or modify a customer account, the model cannot be allowed to decide whether the operation is permitted. Authorization must remain deterministic and enforceable outside the model.
Workflow and state
Many enterprise tasks cannot be completed in one model invocation. Agents may need to execute sequences such as:
reason → retrieve → evaluate → call tool → inspect result → reason again → act
The harness maintains that state and controls the loop.
Error recovery
Enterprise systems fail. APIs time out. Documents are missing. Permissions change. Models misunderstand instructions. Tools return unexpected results. Production agent architecture therefore needs explicit handling for retries, fallbacks, alternative tools, human escalation and safe termination.
A clever model without recovery mechanisms is still a fragile system.
Observability
Traditional applications generate logs and metrics. Agentic systems need something richer. Architects may need visibility into:
- prompts and context;
- model selection;
- tool calls;
- intermediate decisions;
- token consumption;
- latency;
- failures;
- policy violations;
- confidence and evaluation results.
The objective is not simply application monitoring. It is decision-path observability.
The architectural shift: intelligence becomes replaceable
This leads to one of the more interesting consequences for enterprise architecture. The model does not necessarily have to be permanent. A well-designed harness could potentially route different tasks to different models.
For example:
Enterprise Agent
│
Agent Harness
│
┌───────────────┼───────────────┐
│ │ │
Memory Tools Policies
│ │ │
└───────────────┼───────────────┘
│
Model Router
/ | \
Model A Model B Model CA relatively inexpensive model might classify requests and a stronger reasoning model might handle complex analysis. A specialist model could process code or images and a private model might handle particularly sensitive information.
The architecture stops treating “the AI” as a single model. Instead, models become interchangeable computational resources behind a stable orchestration layer.
That is a much more familiar enterprise architecture pattern.
Don’t benchmark the model. Benchmark the system.
This distinction also changes how organizations should evaluate AI.
A model can achieve excellent benchmark scores while the application built around it performs poorly.
Consider an agent that uses a highly capable model but:
- retrieves irrelevant documents;
- repeatedly calls expensive tools;
- loses state between steps;
- has excessive latency;
- cannot recover from API failures;
- consumes enormous numbers of tokens;
- or regularly requires human intervention.
The model may be excellent but the system is not. Production performance depends heavily on orchestration capabilities such as memory, tool routing and error recovery rather than model capability alone. Enterprise AI therefore needs system-level metrics.
Instead of asking only: How accurate is the model? -organizations should increasingly measure:
- Cost per successfully completed task
- Time per completed task
- Tool-call success rate
- Human escalation rate
- Recovery rate
- Policy violation rate
- End-to-end task accuracy
Those measurements tell architects considerably more about production viability than a model benchmark alone.
The harness may become the strategic asset
There is another important implication: Models are evolving extremely quickly.
Today’s preferred model may not be the preferred model twelve months from now and an enterprise architecture tightly coupled to one model provider therefore creates unnecessary architectural risk. The organization’s more durable investment may instead be its agent infrastructure:
- its connectors;
- its semantic layer;
- its knowledge architecture;
- its identity model;
- its policies;
- its observability;
- its evaluation framework;
- its workflow definitions;
- and its integration with enterprise systems.
Models can then evolve underneath that architecture and this resembles previous shifts in enterprise computing. Applications became less dependent on particular servers and services became less dependent on particular infrastructure. Cloud-native applications became less dependent on individual machines.
Agentic architecture may produce a similar abstraction: enterprise capabilities become less dependent on individual AI models.
Enterprise architects should design the harness first
This suggests a practical change in how organizations approach agentic AI projects.
Rather than beginning architecture discussions with: Which LLM should we standardize on? – start with: What capabilities must an AI agent have to operate safely inside our enterprise?
That conversation naturally leads to architecture.
- What identity does an agent have?
- What can it access?
- How does it discover tools?
- Where does its context originate?
- How is enterprise knowledge represented?
- What actions require human approval?
- How are decisions audited?
- How does an agent recover when a tool fails?
- Can models be substituted without redesigning the application?
- How do we measure the cost of completing a business outcome?
Those questions are much harder than choosing an LLM and they are also where enterprise architecture creates value.
From model-centric AI to system-centric AI
The first phase of generative AI encouraged organizations to think about models. The next phase will force them to think about systems. Models provide intelligence whereas harnesses turn that intelligence into controlled behavior and in an enterprise, controlled behavior is ultimately what matters.
The winning architecture may therefore not be the organization with access to the most powerful model but the organization that builds the best environment around whatever model it chooses.
For enterprise architects, that means the center of gravity is moving away from model selection and toward orchestration, identity, integration, context, governance and observability. The model may be the brain. But the harness is increasingly becoming the enterprise architecture around it.