Artificial intelligence in finance: fraud detection, credit and the risk of black-box decisions

The most damaging mistake firms make with AI in finance is to treat it as a performance upgrade rather than a governance problem. A model catches more fraud, a chatbot clears more calls, an underwriting engine makes faster decisions, and the organisation assumes the rest is detail. In regulated financial services, that assumption fails quickly. The central question never goes away: who is responsible when the system is wrong, and can the firm prove that it exercised proper control.

AI systems differ from traditional software in one crucial respect. Their behaviour can change without obvious code changes, as data shifts, thresholds are adjusted, or models are updated. That makes accountability harder, not easier, and raises the bar for oversight.

Fraud detection: effective, but easy to over-trust

Fraud detection is where AI has delivered some of its clearest gains. Payment fraud and account takeover generate huge volumes of data, with behavioural patterns that machine learning models can identify faster and more flexibly than static rules. Used well, these systems reduce losses and spot emerging threats.

The problem is not that they fail outright. It is that success on one metric can hide harm elsewhere. A fraud model tuned aggressively may block legitimate transactions, disrupt customers’ lives, and generate large volumes of complaints and compensation. Those effects often fall hardest on vulnerable customers. The operational risk is that firms optimise for fraud losses while underestimating conduct and reputational risk.

There is also an arms race dynamic. Criminals adapt to whatever signals are being used, which means detection is never finished. Governance therefore needs continuous monitoring, challenger models and clear escalation routes, not a one-off validation exercise.

Anti-money laundering: pattern recognition meets legal duty

Anti-money laundering monitoring looks like a natural home for AI, but the constraints are different. The goal is not only to identify suspicious behaviour, but to do so in a way that can be documented, justified and defended to supervisors and law enforcement.

Machine learning can help prioritise alerts and reduce noise, but it can also introduce opacity. If a system produces a risk score without a clear, explainable rationale, compliance teams may struggle to show why a case was escalated or closed. That tension is acute in a regime built around evidential standards and audit trails.

There is also a bias risk. Historical suspicious activity reports reflect human judgement, regulatory focus and institutional incentives. Models trained on that history can reproduce those patterns, even when they are uneven or unfair. In this domain, documentation, feature governance and clearly defined human decision points are not optional extras.

Credit scoring: where the black box matters most

Credit decisions shape access to housing, mobility and basic financial resilience. AI promises more granular risk assessment and the possibility of widening access by using alternative data. It also raises the sharpest concerns about fairness and explainability.

Simpler scorecards have an unglamorous advantage. They are easier to explain and easier to challenge. More complex models can outperform on average while still producing pockets of harm, particularly for groups underrepresented in the training data. They can also learn proxy variables that correlate with protected characteristics, even when those characteristics are excluded by design.

In consumer credit, “the model said no” is not an acceptable answer. Firms need to explain the main drivers of decisions in plain language, provide meaningful routes for review, and demonstrate that models are stable over time and tested for disparate impact.

Customer service automation: efficiency with conduct risk attached

Generative AI is increasingly used to summarise calls, draft responses and handle routine queries. When deployed carefully, it reduces waiting times and lowers costs. When deployed carelessly, it can give confident wrong answers, invent policy, or mishandle vulnerability and complaints.

The risk rises sharply as systems move from drafting to deciding, or from assisting agents to taking actions on accounts. Autonomy increases speed, but it also increases the blast radius of mistakes. In a regulated environment, those mistakes can quickly become misconduct or consumer duty failures. Capability boundaries therefore become a core control: what the system can do, what it must not do, and how it behaves when it is uncertain.

Trading and markets: the danger of correlated behaviour

AI has long been used in trading and market surveillance, but wider adoption changes the risk profile. If many firms rely on similar data sources and similar models, their responses to market signals can become correlated. That raises the risk of feedback loops that amplify volatility.

From a systemic perspective, the concern is not only individual model failure, but interaction effects across the market. This is why supervisors tend to focus on resilience, governance and system-wide impacts as adoption grows.

Model risk management: the line between use and control

Across all these areas, the dividing line between responsible adoption and reckless deployment is model risk management. Regulators expect firms to know which models they are using, what they are for, how critical they are, and how they are controlled.

In practice, that means clear ownership, an inventory of models, tiering by importance, independent validation, monitoring for drift, and disciplined change management. AI adds pressure because models can be harder to interpret, more sensitive to data shifts, and more dependent on third parties. When something goes wrong, regulators will still expect a clear account of what happened and who was responsible.

Bias and explainability are operational requirements

Bias in financial AI is not just about protected characteristics. It can arise from selection effects, historical decisions, measurement error and proxy variables. A model can be accurate by conventional metrics and still produce unfair outcomes.

Explainability is therefore not a technical nicety. Customers need reasons they can understand and contest. Firms need explanations they can audit and defend. Supervisors need assurance that decisions are consistent with legal and conduct obligations. No single fairness or explainability metric settles this. Effective governance relies on multiple tests, outcome monitoring and scenario analysis.

How to read “AI-powered” claims with scepticism

The safest way to assess claims about AI in finance is to look past the intelligence and towards the controls. Ask what the system actually does in practice, whether humans can meaningfully review and override it, how it is tested and monitored, what data it uses and where that data goes, and what happens when the model changes.

Related reading

• The economics of AI: why inference costs matter more than flashy demos
• AI translation and speech tools: accuracy, bias, and when to trust them
• NVIDIA unveils Rubin platform as blueprint for next-generation DGX SuperPOD systems

“Proprietary black box” should be treated as a warning, not a reassurance. If a firm cannot explain its systems at a high level, it is unlikely to be able to govern its failures.

The bottom line

AI is already delivering real benefits in financial services, particularly in fraud detection and operational efficiency. The danger is not that models exist, but that firms rely on them without adequate oversight. In finance, accountability cannot be automated away. Performance without governance is not innovation. It is a risk that will surface eventually, and usually at the worst possible moment.