Smart Returns Resolver

01 — The problem

Entitlement is a policy question, not a language question.

A customer writes in to say a car seat arrived with a dented box. A model can read that sentence far better than a regular expression can. What it must not do is decide what the customer is owed.

Refunds, replacements and safety escalations are commercial policy. They are written down, they change without warning, and they have to be auditable afterwards — none of which describes a model’s judgement. So the model reads, and a hand-written 36-cell table decides: six issue types against six product categories, each cell naming the entitlement, the SLA, whether it always escalates, and whether it carries a stop-use warning.

Rubric score

88.5%

69 of 78 binary dimensions across 15 cases

Policy cells

36

6 issue types × 6 product categories

Pipeline failures

0

every run schema-valid, no errors

Median latency

9.2s

three sequential model calls

The classifier votes. The orchestrator does not read the vote.

The classification schema includes a needs_human field, and the model fills it in on every request. The escalation gate never looks at it. Escalation happens when the matched policy cell says it always escalates, or when no cell matched at all, or when the model’s own confidence falls below 0.7 — three deterministic conditions, none of which is the model’s opinion about whether a human is needed.

On the committed evaluation that disagreement showed up four times in fifteen cases, in both directions. Three times the model asked for a human and the engine resolved the case anyway. Once, on a dented car-seat box, the model rated it standard urgency and said no human was needed — and the table escalated it regardless, because damaged goods in the infant-safety-critical category are always escalated and carry a stop-use warning. That single case is the reason the design exists.

Where it scores worst is where it should.

Fifteen cases across four tiers, scored on a binary rubric — 78 dimensions in total, 69 passed. The tier it does best on is the one where refusing is the only right answer. The tier it does worst on is the adversarial set, which is the point of having one.

What it deliberately does not do

• It does not let the model decide entitlement. The table is declared so that a missing cell is a compile error rather than a runtime surprise. If the issue type or the product category is unknown, no cell matches and the case goes to a human by construction.

• It does not treat a code-switched message as bilingual. A message mixing English and Arabic is classified as mixed and then answered in English, because the responder branches only on a clean Arabic classification. That is a real gap and the repository documents it.

• Arabic quality is measured, and it is not good. A separate judge scored three Arabic cases on dialect, register, naturalness and clarity: 7 of 12. Published rather than dropped.