A composite architecture for automating high-stakes decisions in domains where data is sparse, conditions shift, and a 5% error is unaffordable.
By Michał Staniszewski, Engineering Manager at Addepto, part of KMS Technology
Most published guidance on applying machine learning to business decisions quietly assumes abundance: enough historical examples to train on, enough volume that individual errors average out, enough stability that last year's patterns still hold. Recommendation engines and ad targeting were built in exactly that environment. High volume absorbed the risk: if a model made a mistake, a user simply scrolled past.
The current wave of enthusiasm extends the same premise to LLMs: if a big enough model solves ad targeting, surely it can solve pricing, underwriting and routing too. It can help, but only as one layer of the system, parsing language and intent. It cannot be the system.
A large class of enterprise decisions looks nothing like that. Consider pricing a piece of bespoke industrial machinery, underwriting a niche commercial liability policy, or quoting a specialized logistics movement that requires permits and non-standard equipment. In these domains:
Transaction volume is sparse. Historical operational data is measured in thousands of records, not petabytes.
External sensitivity is extreme. Operating conditions shift dynamically based on regulatory changes, material costs, labor availability and route access.
Failure is costly. A 5% miscalculation on a $20 consumer transaction is a rounding error, while the same 5% on a $250,000 contract removes the margin, creates legal exposure, and damages a client relationship that took years to build.
Forcing broad, unconstrained models, LLMs included, into these low-data regimes can lead to hallucinated estimates, severe model drift during market shifts, and dangerous operational breakdowns.
The core thesis: Winning in specialized B2B markets requires replacing brute-force algorithms with a Domain-Informed, Composite AI Architecture, a "polyglot" system governed by strict operational KPIs, explicit human-in-the-loop risk boundaries, and deep industry feature engineering.
Why the monolithic approach breaks
The instinct is to build one capable model and give it the whole problem. In a sparse-data environment, that produces a system that cannot reliably distinguish a genuinely unusual case from a case it simply has not seen often enough. Both look the same from inside the model. Both produce an answer. Neither comes with a trustworthy signal about which situation you are in.
That matters because the two require opposite responses. An unusual-but-understood case may require an appropriate risk adjustment. An unfamiliar case should stop and go to a human. A monolithic model collapses that distinction and returns a confident number either way.
Adding a large language model on top does not solve this. It relocates it. LLMs are probabilistic text-generation systems, effective at interpreting messy input and poor at arithmetic and repeatable business logic. Handing one a margin calculation introduces variance into precisely the step that needs to be deterministic.
The decision is one problem, not three
Teams tend to treat pricing as an isolated calculation. In volatile, low-data domains, a price is the output of an operational risk assessment. Three variables have to be resolved together:
Operational feasibility: can the organization physically execute this request, given asset availability, capacity schedules, regulatory constraints, and access bottlenecks?
Risk and cost forecasting: what is the probability of delay, material cost movement, or labor drag on this configuration?
Margin valuation: given the above, what price protects profitability while remaining competitive?
These are coupled. A change in scheduling changes the risk profile, which changes the defensible price. Building three separate tools means each optimizes locally and none of them sees the interaction, and the blind spots live exactly there. It also multiplies the data problem: three independent models each need enough history to train on, at a point where you barely have enough for one. Treating it as a single coordinated decision process lets domain rules fill the gaps between sparse data points rather than having three components each extrapolate in isolation.
Coordinating the decision, though, is not the same as putting it in one model, which would recreate the very problem this architecture is meant to solve. Natural language models handle interaction. Predictive models identify patterns. Deterministic rules enforce business constraints. Human experts take over when the system reaches the limits of its knowledge.
Four pillars of the architecture
1. Separate natural language from business logic
The language model interprets; it does not decide. The conversational and orchestration layer, built with generative AI and frameworks such as LangGraph, takes unstructured input (a customer email, a specification sheet, a booking request in whatever format it arrived) and does four things: identifies intent, extracts relevant parameters, requests whatever is missing, and emits a structured object.
Once those parameters are validated, they cross into the decision core. There, classical machine-learning models estimate likely outcomes and deterministic rule engines apply pricing formulas, eligibility conditions, operational constraints, and compliance requirements. The LLM makes the system usable; it does not set the price. The predictive models estimate what is likely; they do not override hard business rules.
The practical payoff is decoupling. Pricing logic, risk limits, and operational policy can all be updated without retraining the conversational layer. The interface can be improved without putting financial calculations at risk.
2. Turn tacit expertise into predictive variables
Operational systems routinely capture hundreds or thousands of fields. Most carry little predictive value; some reflect temporary circumstances, inconsistent data entry, or processes that no longer exist. Feeding all of them into a model in a low-data regime encourages overfitting: the system starts treating accidental correlations as business rules.
The root cause is usually upstream. Organizations log everything available and defer decisions about structure and purpose. The result is volume without clarity: a data lake that gradually becomes a landfill of duplicated, ambiguous, weakly governed information.
Effective feature engineering therefore starts from a business question rather than a dataset. Define what the model is meant to predict (a delay, a loss, a capacity constraint, a pricing deviation) and then work through historical cases with the people who actually made those calls: dispatchers, underwriters, pricing specialists, operations leaders. In many operational environments, a relatively small group of factors explains a disproportionate share of delays, costs, and margin variance.
What surfaces in those sessions is generally not visible in the raw data. A senior dispatcher may know that a particular combination of route, asset type, unloading requirement, and delivery window carries serious delay risk, even though no single one of those variables is informative on its own. Feature-engineering workshops turn that tacit knowledge into variables the model can evaluate consistently.
Encoding that knowledge as explicit features helps compensate for limited historical data and supports the development of smaller, more interpretable models. In limited-data environments, these models are often more accurate and easier to govern than broad neural networks trained on every available field.
3. Define where automation must stop
Even a well-built model will meet cases it cannot assess confidently: severe weather, unusual permits, unprecedented asset configurations, incomplete documentation. A system designed only to produce answers will still generate one. A system designed to manage risk will recognize when it should stop. Confidence thresholds create that boundary.
For standard requests with complete data, familiar parameters, and sufficiently high model confidence, the workflow proceeds automatically: verify feasibility, calculate a price, apply business rules, and respond within seconds. Requests that trigger anomaly indicators, violate predefined constraints, or fall below the required confidence threshold follow a different path. Instead of guessing, the system pauses the decision and prepares the case for review.
The detail that determines whether this works is what the human receives. Handing a coordinator the raw original request wastes the work the system already did. The escalation interface should present the extracted parameters, preliminary calculations, comparable historical cases, missing information, and the specific reason for escalation, so the expert spends their attention on the genuinely unusual part of the case.
The frequently used 90/10 model captures the intended balance: automate the majority of repeatable work while routing the small but consequential exception set to qualified people. The exact ratio will vary by process, but the principle remains the same: the escalation path is a designed feature with its own interface requirements, not an error handler bolted on at the end. Human intervention is one of the mechanisms through which the system controls liability.
4. Learn from the outcome of every decision
Operational markets do not remain stable. Fuel surcharges change, tariffs are revised, labor costs rise, capacity becomes constrained, customer behavior shifts. A model that performed well six months ago may gradually begin underestimating costs or overlooking new sources of delay.
So the architecture cannot end when a quote is issued or a job is scheduled. When each job is completed, the system records the realized outcome: actual turnaround time, resource consumption, handling costs, delays, surcharges, exceptions, and final margin. These results are compared with the original prediction and returned to a governed MLOps pipeline. The feedback loop shows where the model is drifting: estimates becoming less reliable on a particular route, a new regulatory requirement creating delays, an old pricing variable that no longer reflects current market conditions.
Not every change should trigger automatic retraining. Teams still need validation, approval, version control, and performance monitoring. However, continuously collecting outcome data allows recalibration to become a controlled operational process rather than an expensive rescue project launched after margins have already deteriorated.
What the layering buys you
Generative AI makes complex workflows accessible. Predictive models estimate outcomes. Deterministic rules protect business logic. Confidence thresholds contain exceptional risk. Continuous feedback keeps the system aligned with changing conditions.
The result is not one model pretending to know everything. It is a coordinated decision architecture in which each component is responsible for the task it can perform reliably, and where you can point to which component produced which part of an answer when someone asks why the number is what it is. That auditability is frequently the thing that determines whether the system is allowed into production at all in regulated or high-liability domains.
Delivery governance: hiring and managing for business outcomes, not AI hype
Executing this composite strategy demands a different playbook for AI procurement. Some implementation partners are pitching "LLM-first": wrap a generic language model around your enterprise data and call it solved. It isn't, and Pillar 1 is the reason why: an LLM handling arithmetic directly reintroduces the exact hallucination risk this architecture is built to eliminate.
Building production-ready decision engines takes an engineering partner obsessed with program delivery, risk mitigation, and metrics you can actually measure. These checks are not a fifth architectural pillar. They are how you evaluate whether a partner will actually build the four pillars above. Program managers should hold potential implementation teams to three non-negotiable delivery checks:
Metric commitment. A qualified partner refuses to start development until success is translated into measurable thresholds, set before architecture design, not after launch. Illustrative targets include sub-30-second turnaround on standard requests (Pillar 3), pricing variance held within 2–3% of senior expert human baselines, and a 90/10 automation-to-escalation split (Pillar 3). The right numbers will vary by process.
Stack composition. Avoid vendors that treat generative AI as a universal solution. A qualified partner commits, in writing, to the same three components as Pillar 1: predictive ML for forecasting and risk scoring, a deterministic rule engine for compliance and safety boundaries, and GenAI strictly as the natural-language interface, never as the calculator.
Discovery process. An experienced partner runs the same structured, expert-led discovery sprints described in Pillar 2, prioritizing signal quality over benchmark scores, rather than asking you to trust a model's off-the-shelf accuracy on your data.
The constraint is a design input, not a blocker
Lacking petabytes of "Big Data" is not a barrier to enterprise automation. It is simply a design constraint that dictates a more disciplined architecture. The same pattern transfers across sectors with the same shape: high value, low volume, high consequence.
Custom industrial manufacturing: deterministic physics-calculation engines combined with GenAI specification parsers to generate binding quotes for bespoke machinery without decades of structured CAD history.
Commercial risk underwriting: identifying the most consequential operational risk variables to automate standard policy underwriting while automatically escalating complex, high-liability risks to expert underwriters.
Specialized B2B logistics and supply chain: continuous MLOps feedback loops that adjust service-level agreements and asset deployment pricing in real time as market fees and access constraints fluctuate.
Enterprise market leadership in volatile, high-value sectors will not belong to the organizations that hoard the largest datasets or deploy the biggest language models. It will belong to the organizations that build pragmatic, composite decision engines: systems that respect human domain expertise, enforce strict risk boundaries, measure performance by clear business KPIs, and continuously adapt to real-world execution.
Michał Staniszewski is an Engineering Manager at Addepto, part of KMS Technology.