# When to Train an AI Model and When to Write Code

There's a rush to use AI for almost every problem at the moment. That enthusiasm can make it easy to reach for a model before asking whether a straightforward rule would solve the problem more reliably. Sometimes a lookup, a conditional or a decision table is all the application needs & in most cases its faster and doesn't add a cost.

For example, say a user changes an AI-generated value. Your application records the original answer and the replacement. After a few months, you have thousands of these pairs. It sounds like an obvious opportunity, use the corrections to train the model so it produces the preferred answers next time. But what if the model was correct? Enterprise applications regularly change values for reasons that have little to do with interpreting the input. An internal convention, a legacy integration or a commercial agreement can require a result that wouldn't make sense to someone outside the business. Before turning those changes into training data, you need to understand what they represent. An interpretation error and an intentional business transformation can produce exactly the same before and after record. They call for different engineering decisions.

## A correct answer can still need changing

Imagine a support application receiving this message:

> Ive been charged twice for my subscription. Can you arrange a refund?

The model classifies the request as `Billing`. Thats a reasonable interpretation of the message.

However, the application connects to an older case management system. For one business unit, requests of this kind must be stored under `AccountServices`. The reasons are historical, a reporting arrangement, an existing workflow and an integration nobody is replacing this quarter.

So the application applies a rule:

```json
{
  "id": "legacy-billing-category",
  "version": 3,
  "property": "Category",
  "equals": "Billing",
  "replaceWith": "AccountServices"
}
```

The rule is selected within the relevant business unit context. Other business units may keep `Billing` unchanged. The models answer accurately describes the request. The replacement satisfies the receiving systems requirements. Both values have a purpose, but they answer different questions. This distinction becomes even clearer if you can expose separate concepts in your design, an interpreted category of `Billing` and an operational category of `AccountServices`. A legacy contract may force the final payload to use a single `Category` field, but your internal records can still preserve both meanings. Training the model to return `AccountServices` directly would teach it an organisational exception. That may be technically possible. It doesn't automatically make it a useful investment.

## Decide what you want the model to learn

When I talk about training here, I mean fine tuning an existing model on task specific examples. Putting examples into a prompt doesn't update the model's weights, and recording a user edit doesn't automatically train anything. Microsoft's fine tuning guidance makes that distinction and describes fine tuning as adapting a pre-trained model through additional task specific training.

For the support application, a useful learning objective might be recognising billing requests across varied language:

> You've taken the payment twice.

> There are two identical transactions on my statement.

> I was billed again after renewing yesterday.

These messages don't all use the word "refund", and some may need more context before the category is clear. Teaching the model to interpret that variety could improve the task. The instruction "replace `Billing` with `AccountServices` for this business unit" is already explicit. Once the context and input value are known, a lookup or conditional can perform it directly.

| Requirement | Sensible starting point |
| --- | --- |
| Recognise a concept across varied natural language | Prompt instructions and examples; consider fine tuning if evaluation justifies it |
| Replace one known value with another | Deterministic mapping |
| Apply a tenant specific exception | Contextual rule or decision table |
| Answer using changing product information | Retrieve the current information |
| Enforce an approval threshold | Deterministic validation |
| Improve a model's recurring interpretation failures | Reviewed examples and evaluation, followed by a training experiment if warranted |

The difficulty of describing an exception in ordinary language isnt the deciding factor. Some business rules are complicated. If the business can specify their conditions and outcomes precisely, they can still be executed deterministically.

## Put the transformation after interpretation

A useful architecture makes the stages visible. The model interprets the input, the application validates the result, and the appropriate policy determines the operational value.

![](https://cdn.hashnode.com/uploads/covers/67c36038c69a4b7143c5fc49/ea017002-a392-49ae-88e7-0afeae05f88c.png align="center")

"Usable" needs to be defined for the application. Valid JSON alone doesn't establish that the interpretation is supported by the input. Likewise, a model's self reported confidence shouldn't become an unquestioned approval mechanism. The rules can live in code, configuration, a schema or a decision table. A generic transformation mechanism can apply the same operation to any permitted property. You don't need a bespoke implementation for every field. For example, this small .NET component applies exact string replacements to one property:

```csharp
public sealed record ReplacementRule(
    string Id,
    int Version,
    string Property,
    string ExpectedValue,
    string ReplacementValue);

public sealed record AppliedRule(
    string Id,
    int Version,
    string Before,
    string After);

public sealed record TransformationResult(
    string OriginalValue,
    string EffectiveValue,
    IReadOnlyList<AppliedRule> AppliedRules);

public sealed class ValueTransformer(
    IReadOnlyList<ReplacementRule> rules)
{
    public TransformationResult Apply(string property, string value)
    {
        var effectiveValue = value;
        var appliedRules = new List<AppliedRule>();
        foreach (var rule in rules)
        {
            if (!string.Equals(rule.Property, property,
                    StringComparison.Ordinal) ||
                !string.Equals(rule.ExpectedValue, effectiveValue,
                    StringComparison.Ordinal))
            {
                continue;
            }

            appliedRules.Add(new AppliedRule(
                rule.Id,
                rule.Version,
                effectiveValue,
                rule.ReplacementValue));

            effectiveValue = rule.ReplacementValue;
        }

        return new TransformationResult(
            value, effectiveValue, appliedRules);
    }
}
```

The caller supplies an ordered, applicable rule set for the current business context. This example deliberately allows chaining, a later rule sees the value produced by an earlier rule. A different system may require first match behaviour. Choose that behaviour explicitly, because overlapping mappings can otherwise produce surprising results. With a single rule mapping `Billing` to `AccountServices`, the result retains the models original answer, the effective value and the rule version responsible for the change. The component doesn't need to know which model produced the value. You would still validate configuration, select the correct context and validate the final value against the destination contract. Deterministic execution makes behaviour reproducible for the same inputs and rule set; it doesn't make an incorrect rule correct.

## Before and after values need an explanation

Suppose your feedback store contains this pair:

```json
{
  "originalValue": "Billing",
  "finalValue": "AccountServices"
}
```

That tells you what changed. It doesn't tell you whether the model misunderstood the message, a rule applied a business convention, or a user made a one off override. If you already capture both values, the useful next step is to record the cause where possible:

```json
{
  "originalValue": "Billing",
  "finalValue": "AccountServices",
  "changeReason": "BusinessMapping",
  "ruleId": "legacy-billing-category",
  "ruleVersion": 3,
  "businessUnit": "UnitA"
}
```

An interpretation correction needs different evidence, the original input or an authorised reference to it, the reviewed answer, and enough explanation to establish why that answer is right. Protect sensitive information when assembling that dataset.

![](https://cdn.hashnode.com/uploads/covers/67c36038c69a4b7143c5fc49/3ac6775d-0565-499e-80a7-c787ea06ab9f.png align="center")

The same feedback store can support model improvement and rule maintenance. Those activities need different selections of data.

## Training on business overrides can create the wrong behaviour

If you train on final values indiscriminately, you risk asking the model to optimise for the receiving system's historical conventions while evaluating it as an interpreter of the original request. In the example, the model could learn to associate duplicate payments with `AccountServices`. That might improve agreement with stored records for Unit A while reducing the usefulness of its semantic classification elsewhere. The problem becomes more awkward when the business changes its mapping. Next month, Unit A might move to a new case management system and accept `Billing` directly. Removing a configuration rule is straightforward. Changing behaviour encoded in model weights requires further work and evaluation, simply removing the old rule won't necessarily undo what the model learned. If the mapping remains after training, you can also end up maintaining the same requirement in two places. The model may apply it, the application may apply it, and an incident investigation has to establish which stage produced the final result. Keeping explicit transformations in the application gives those changes an owner, an effective version and a directly testable outcome.

## Where fine tuning earns its place

There are still good reasons to train a model. A support classifier might repeatedly confuse refund requests with cancellation requests despite clear instructions and representative prompt examples. Specialists might use abbreviations or indirect language that the baseline model handles poorly. At sufficient volume, a smaller specialised model may also be worth evaluating for cost and latency. Microsoft lists task specific accuracy, reduced prompt overhead and smaller model specialisation among potential fine tuning benefits. It also makes clear that improvements aren't guaranteed and should be checked against a held out baseline. I would start by turning reviewed interpretation failures into an evaluation set. Define the task precisely, improve the prompt and field descriptions, then check what still fails. If a stable pattern remains and representative training examples are available, fine tuning becomes a concrete experiment. Keep training and final test data separate. Near duplicate messages or cases from the same conversation can make results look better than they are if they appear on both sides of that split. Evaluate interpretation and business transformation separately as well. A final value score can hide an interpretation error if a broad mapping happens to turn several different model outputs into the same result. Conversely, a correct interpretation can look wrong if it's compared directly with a legacy operational label. Useful measures might include category precision and recall, unsupported predictions, review rates and cost per successfully processed request. Rule checks should separately establish that the expected mappings apply, that unrelated values remain unchanged, and that business contexts stay isolated.

## Can a model learn the whole decision?

Yes. A model can be trained to produce operational answers directly, including organisation specific conventions. That can be reasonable when you're deliberately learning a contextual decision process that is difficult to specify, and its measured error rate is acceptable for the task. However, training doesn't give a conditional the same execution guarantee as an explicit application check. If a requirement must hold whenever its conditions are met, keep that enforcement in the application even if the model usually anticipates it. For an exact substitution you already implement reliably, ask what training would improve. If it doesn't remove meaningful cost, improve interpretation or deliver another measured benefit, it adds maintenance without solving a new problem.

## Give each change a clear home

When reviewing a changed answer, ask whether the model misunderstood the input or whether the business intentionally requires a different result. The first can become evidence for improving the model. The second belongs in your policy implementation. Some changes remain ambiguous until someone reviews the context. Enterprise software has plenty of conventions that a capable model couldn't infer from the input alone. Keep those conventions explicit, and use training where learning from examples offers a measurable advantage.

[Microsoft Learn: Fine-tuning in Microsoft Foundry](https://learn.microsoft.com/en-us/azure/foundry/fine-tuning/overview)