10. Model Risk, Agentic Risk Management, and the Observability Gap

The answer travels through three layers

---

🧭 What One Decision Has to Answer

Every credit decision has to answer one question: what did the system decide, and why? A loan officer might remember a case, but five AI agents working in sequence cannot; the records need to explain each decision even years later to someone who was not involved.

Our platform decides whether to offer credit and on what terms. A coordinator directs the work and writes the final explanation. Four specialist agents assess credit, check risk, review Sharia compliance, and simulate the outcome. Sharia review is a useful example: each ruling must cite its basis.

Back to top . Next: Applying Model Risk Guidance to Agents →

---

📋 Applying Model Risk Guidance to Agents

We start with SR 26-2, the model-risk guidance updated by the Federal Reserve, OCC, and FDIC in April 2026. It guides how we review the scoring models in this workflow. However, it says generative and agentic AI are outside its scope. We therefore apply broader risk-management principles to the handoffs between agents too.

What Counts as a Model

SR 26-2 calls a method a model when it uses statistical, economic, or financial theory to estimate something from data; for example, a credit-scoring method estimates an applicant’s credit risk. A fixed cutoff, such as declining applications below an income threshold, does not estimate risk from data. The guidance also excludes generative and agentic AI. So it does not treat an agent that drafts a decision letter as a model, even when that agent uses one behind the scenes. We also use the NIST AI Risk Management Framework and ISO 42001 for broader governance principles, including how agents hand work to each other.

So the platform sorted into three kinds of component

Before a scoring model moves into testing, its model card should explain its purpose, training data, performance, and limits. Ours also records how independently it may act, its risk level, and whether a domain specialist reviewed it. For a model that affects a Sharia decision, our standard requires sign-off from a qualified Sharia Board member. Other fields need to name their own reviewer. I would not release a scoring model without that record.

Four Checks Before a Model Goes Live

Before release, we check four things. The European Commission’s current timeline says rules for the high-risk systems in Annex III apply from 2 December 2027. The list includes AI systems that evaluate creditworthiness.

• Impact assessment: an approver records the assessment before release.
• Model card: records the approver’s name and signature, not a team mailbox.
• Difficult cases: scenario tests check how the model handles unusual or difficult situations.
• Fairness: approval rates across groups stay within the allowed ratio, and an approver runs the latest audit within seven days.

There is an operator bypass for the fairness check. Even so, I would not describe this process as an unconditional gate; I would want promotion reviewers to see that exception.

Back to top . Next: Agentic Risk Management →

---

🕸️ Agentic Risk Management: Six Controls in Our Lending Workflow

We identified six risks in our lending workflow: an agent using a tool it should not, agents disagreeing, an explanation contradicting a fixed decision rule, an agent changing its own permissions, an applicant repeatedly avoiding human review, and a Sharia case with no strong precedent. SR 26-2, the NIST AI Risk Management Framework, and ISO 42001 offer broader governance principles. We applied those principles to agent handoffs. Some controls block an action; others record a concern or send a case to a person.

The six risks at a glance

01 · An Agent Reaching Beyond Its Call List

An agent that can use any tool or contact any other agent has broad access. For example, a faulty simulation agent might ask the credit scorer for a decision without Sharia or human review. We give each agent a list of allowed calls, and a shared check blocks calls outside that list.

02 · An Agent Contradicting Another

A misleading request could make agents reach opposite conclusions. For example, our Sharia agent might accept a case that the credit agent rejects. We count these disagreements for each customer. A third disagreement within 24 hours triggers a warning and, when audit logging is available, an audit record. MITRE ATLAS helps describe broader AI threats; it is not the source of our threshold.

03 · A Narrative Contradicting the Gate

A fixed rule accepts or rejects the application. A large language model (LLM) writes the explanation a person reads. If the rule rejects an application but the explanation sounds positive, a reader may trust the wording. We compare the two and record any mismatch. If the application would otherwise pass, the workflow can mark it “pending review.” That label alone does not prove a person reviewed it. Article 7, Building Real Human-in-the-Loop with LangGraph Checkpointing, explains how a checkpoint waits for a human response. The OWASP 2025 LLM Top 10 calls misleading AI output “misinformation.” This check catches a mismatch; it cannot guarantee every explanation is correct.

04 · An Agent Raising Its Own Autonomy

Each agent has one of three permission levels: full, supervised, or restricted. The level sets what it can do without human approval. A “supervised” label means little if the agent has no extra check before acting. Our policy check enforces each level while the agent runs, and the agent cannot change its own level. Two different approvers must agree to a change and record why. The change takes effect without restarting the service.

05 · An Applicant Gaming the Review Line

An applicant could learn the score that sends an application to human review, then keep each request just below it. We count requests that come close to any review threshold. More than three near-misses from one customer in a day sends the next request to a person, whatever its score. This only works if the workflow waits for that person to respond.

06 · A Sharia Case With a Weak Precedent

For Sharia decisions, the ruling the system finds matters. If the closest ruling’s similarity score is below 0.75, the system marks the case for human review; 0.75 is the default threshold. Every five minutes, a separate monitor compares the latest 100 Sharia decisions with a stored baseline. It flags approval rates more than three standard deviations above that baseline, a large departure from normal variation. The monitor waits for at least 1,000 past decisions before making the comparison. Other regulated fields need their own review criteria.

Back to top . Next: How These Controls Apply Beyond Banking →

---

🌐 How These Controls Apply Beyond Banking

Insurance: An insurer might split claims intake, coverage review, and payment across agents; require an authorized coverage decision before payment.

Healthcare: A team might use an agent to summarize prior-authorization records against a clinical policy; weak evidence or a mismatch should send the case to a clinician, with the policy version retained.

Logistics: A team might let a routing agent rebook a delayed shipment, but only through approved systems; keep a reason for the change and a way to pause that agent alone.

These are illustrative examples, not shared thresholds; each field needs its own rules, reviewers, and escalation path.

Back to top . Next: When a Control Fires →

---

🚨 When a Control Fires

Our policy gives P1 incidents a one-hour target for notifying the Central Bank of the UAE (CBUAE). An alert is only the start: a person must assess the incident, decide what to report, record the notification, and track follow-up. We also need to stop one agent without taking unrelated agents offline.

Map Each Tool to the Response Path

Tool type	What it helps with	What to check
Alerting tools	Send alerts to the response team	Who owns the incident, records the regulator decision, and tracks follow-up?
Model monitoring	Tracks model performance and changes over time	Which agent events does it track, and who responds?
AI call tracing	Shows which model calls happened and their context	How does an alert connect to an owner, regulator notice, and case closure?
Risk and compliance case tools	Help track controls and incidents	Do their features and integrations fit the team’s process and record-keeping needs?

A tool may detect a problem without tracking who handles it or what happens next. This table compares tool types, not specific products; features depend on the product, plan, setup, and integrations. First check what your current tools cover. Then decide whether configuration, an integration, or custom work can fill any remaining need.

How We Handle Incidents Here

In our platform, staff can open an incident, set its severity, record regulator notifications, investigate and close it, and review active cases. Every five minutes, a monitor checks whether staff recorded a CBUAE notification for each P1 incident. If they have not done so after 55 minutes, it creates an overdue incident. Staff can also suspend one agent and record why without stopping the whole platform.

These are choices for our workflow, not proof that another product cannot do the same. Before building anything, check how your tools handle ownership, regulator notices, records, and follow-up. Then compare setup changes, integrations, and custom work against those needs.

Back to top . Next: The Record That Wasn’t There →

---

🔦 The Record That Wasn’t There

We opened the Langfuse dashboard to show an examiner the model calls behind one credit decision. There were no records of those calls. The system was not saving the history we needed.

The service settings looked right and its health check showed green, but model calls never opened a trace and the system raised no error. The workflow, audit events, and decision record existed; the call history did not, so we added the missing tracing step.

Two More Places to Check

We also expected audit records from the Sharia compliance and risk-report steps, but neither step created one. And although Qdrant had searchable collections, no data pipeline was filling them at the time.

Six Records Behind One Decision

When we configure it, Langfuse records prompts, responses, timing, and links between model calls. We run its tracing interface locally, but a model provider may still run elsewhere; we set that provider separately. One shared ID links the calls for a decision, so we do not have to add linking code at every step.

One completed decision, six linked records

Together, these records show the final decision, the policy in force, what the agents said, and which models they used. The model-call history gives detail about each step; the audit record preserves the final decision and who made it. A delivery queue sends that record to Redpanda, an event-streaming service. It marks the record sent after Redpanda confirms receipt and retries if sending fails. If the service stops after receipt but before marking it sent, it may send the same record again. Consumers should use a stable record ID to handle duplicates safely. Article 6, Compliance Infrastructure: Audit Trails, Policy-as-Code, and the Append-Only Principle, explains the audit record in more detail.

None of this is free, and the trace layer adds storage and review work. An examiner will not accept “we have great logs.” I have no clean latency figure to quote, and I would rather say that than invent one.

Back to top . Next: Key Lessons Learnt →

---

---