Tuning, metrics and model governance

A monitoring system that was well designed at launch drifts out of shape within months: products change, customers change, criminals adapt, and thresholds set on guesswork either flood the queue or let activity pass unseen. Tuning, metrics and model governance are how a firm shows that its monitoring still fits its risks. They are also where supervisors look for evidence that decisions to raise a threshold or retire a rule were made on data, not to save staff time.

Measuring a monitoring program

Metrics describe both efficiency and effectiveness. Terms are used loosely in the industry, so a firm should define each one in its procedures.

Note that "false-positive rate" in AML practice usually means the share of alerts closed as not suspicious, which differs from the statistical definition. Recall is the hardest metric because it requires knowing what was missed. Useful reference sets include suspicious activity found outside monitoring (internal referrals, law-enforcement requests, fraud complaints, QA findings) and the results of below-the-line testing.

The Wolfsberg Group's Statement on Effective Monitoring for Suspicious Activity (Part I, July 2024) argues that programs should be judged by the usefulness of their outputs to authorities rather than by alert and report volumes, and recommends analyzing false negatives, using feedback from financial intelligence units on report quality and, where feedback is scarce, proxies such as reporting in priority areas.

A query that produces the basic productivity table per scenario:

-- Productivity of each scenario over one quarter
SELECT scenario_id,
       COUNT(*) AS alerts,
       SUM(CASE WHEN outcome IN ('ESCALATED', 'REPORTED') THEN 1 ELSE 0 END) AS escalated,
       SUM(CASE WHEN outcome = 'REPORTED' THEN 1 ELSE 0 END) AS reported,
       ROUND(SUM(CASE WHEN outcome = 'REPORTED' THEN 1 ELSE 0 END) * 1.0
             / COUNT(*), 3) AS conversion_rate,
       ROUND(AVG(minutes_spent) / 60.0, 1) AS avg_hours_per_alert
FROM alerts
WHERE created_at >= DATE '2026-07-01' AND created_at < DATE '2026-10-01'
GROUP BY scenario_id
ORDER BY conversion_rate ASC;

The query counts alerts per scenario in the quarter, counts those escalated (an alert later reported was also escalated, so both outcomes are included) and those reported, and divides to get a conversion rate; multiplying by 1.0 avoids integer division in engines that truncate. Sorting by conversion puts the least productive scenarios first. On a test set, a structuring scenario showed 900 alerts and 2 reports (0.002), while a mixer-exposure scenario showed 120 alerts and 15 reports (0.125). The structuring scenario is a candidate for review, but low conversion alone does not justify switching it off: it may cover a priority risk, and its alerts may support other cases. That judgment, and the evidence for it, must be recorded.

Above-the-line and below-the-line testing

Above-the-line (ATL) testing samples alerts just above a threshold and asks whether they are productive. If alerts in the lowest band above the threshold almost never lead to escalation, the threshold may be raised.

Below-the-line (BTL) testing temporarily lowers the threshold in a test environment, samples the events that would have alerted just below it, and has investigators review them as if they were real alerts. If the sample contains activity that would have been escalated, the threshold is too high.

A worked example. A rapid in-and-out rule fires on deposits of USD 5,000 or more. For BTL testing, the team runs it at USD 4,000 for three months of history, which yields 300 extra events. Investigators review a random sample of 60. If none would have been escalated, the team keeps the threshold and records the result. If five would have been escalated, about 8% of the sample, the threshold is lowered or the scenario is redesigned, for example by adding a profile condition, and the extra workload is planned.

Good practice for both tests: define the bands and sample sizes before looking at results, use investigators who do not know which events are test events where possible, retain the samples and decisions, and have the change approved by someone other than the person who proposed it. Wolfsberg notes that minor threshold reductions are unlikely to improve effectiveness by themselves; redesigning the scenario or adding context often does more.

Coverage assessment

Tuning a rule says nothing about risks that no rule addresses. A coverage assessment maps each typology from the enterprise-wide risk assessment and from sources such as the FATF Red Flag Indicators to the controls that detect it.

Gaps are recorded with an owner, a deadline and interim measures, such as a manual monthly review. New products, new chains and new typologies should trigger a fresh assessment before launch.

Model validation and governance

Rules, scoring models and analytics feeds are all models in the governance sense: they turn data into decisions using assumptions. A governance framework typically includes:

• a model inventory listing every scenario and model, its purpose, owner, parameters and data sources;
• documentation of design, segmentation, threshold rationale and known limitations;
• change control, so no parameter changes without testing, approval and a record;
• independent validation covering conceptual soundness (does the logic fit the typology?), data integrity (is the input complete and correct?) and outcomes (does it detect what it should?);
• periodic review, at least annually and after material changes.

Wolfsberg's Part II statement, Transitioning to Innovation (August 2025), argues that financial crime models should have governance proportionate to their risk, weighing the risk of an imperfect new model against the risk of keeping an ineffective old one, and favors testing against historical data over long parallel runs.

Machine learning and explainability

Machine-learning models appear in three roles: supervised models that score alerts by their likelihood of being productive, unsupervised models that find anomalies or unusual peer behavior, and graph models that find networks. FATF's report Opportunities and Challenges of New Technologies for AML/CFT (July 2021) notes that such tools can improve the speed and quality of monitoring, while firms remain responsible for understanding them and supervisors need to be able to understand how they work.

Specific risks need controls:

• Label bias. Supervised models learn from past decisions, so they reproduce past blind spots. Recall testing against independent reference sets is essential.
• Drift. Behavior and typologies change; performance must be monitored and models retrained on a schedule.
• Explainability. Each alert needs reasons an investigator can use ("deposit 18 times expected monthly volume; shared device with two flagged accounts"), and the model as a whole needs a description a validator and supervisor can follow.
• Suppression. A model used to auto-close low-score alerts is a decision-maker; its thresholds need the same ATL and BTL testing as rules.

Hybrid designs, with rules for known typologies and models for prioritization and discovery, are common and easier to defend.

Data quality and backlog management

Every metric above is wrong if the data are wrong. Regular checks should confirm that all transfers are mapped to customers, that prices exist for every asset, that analytics calls succeeded, and that alert volumes have not dropped suddenly because a feed failed. Reconciling the monitoring system's transaction count with the ledger is a simple and powerful control.

A backlog forms when alerts arrive faster than they are closed. Responses should be risk-based: prioritize high-severity alerts and those where funds are still on the platform, add trained capacity, find the root cause (a new rule, a data error, a growth spike), and report the position to senior management. Mass-closing old alerts without review, or raising thresholds simply to cut volume, are failures that supervisors and enforcement actions repeatedly cite. Any bulk disposition needs analysis, approval and a record.

Exam focus: definitions of false-positive, escalation, conversion and recall rates, and why recall needs a reference set; Wolfsberg's July 2024 emphasis on useful outputs over volumes; reading a scenario productivity table and why low conversion alone does not justify retiring a rule; above-the-line and below-the-line testing, with a worked sample; coverage assessment and recording gaps; model inventory, change control and independent validation; machine-learning risks (label bias, drift, explainability, suppression); data quality checks and risk-based backlog management.