Multi-Modal Fraud Detection

In the UAE, motor insurance fraud is estimated to cost insurers around AED 500 million a year, and that cost ends up in every honest driver's premium. Catching it means an investigator comparing three sources by hand: the damage photos, the claimant's story and the police report. It's slow and it doesn't scale.

Estimate: United Insurance Brokers, UAE Motor Insurance Fraud, November 2025.

The photos go to two vision models: one finds the damage, the other works out which part of the car it's on. The story and the police report go to a language model that turns each of them into the same set of structured facts. A consistency check lines the three sources up, a scoring model turns what it finds into one risk score, and the investigator sees why.

The models only extract facts. The fraud decision comes from comparing those facts, and the final call stays with the investigator.

A Mask R-CNN (torchvision v2, ResNet-50 FPN), started from COCO weights and fine-tuned on CarDD, a research dataset of about four thousand photos of damaged cars. It knows six types of damage: dent, scratch, crack, shattered glass, broken lamp and flat tyre. For each one it returns the type, a confidence, the outline and how much of the photo it covers.

On the CarDD test set it scores 53.6 box AP, against 49.2 for the paper's own Mask R-CNN. It got there on fewer training images (2,534 against 2,816), half the epochs and a free Colab GPU. The gains are on the classes the paper names as hardest: dents, scratches and cracks.

Two caveats. It's a newer Mask R-CNN than the paper's, so the fair claim is that a modern Mask R-CNN beats the paper's baseline, not that the same model does better; the paper's best model, DCN+, still leads at 58.8. And cracks are found but outlined poorly, which the paper reports too.

A second Mask R-CNN outlines 17 parts of a car, with left and right from the car's own point of view, trained on car-seg (CC BY 4.0). Each damage is matched to the part it overlaps most, so the system can say “dent on rear bumper”, or “part unclear” when it can't tell. On whole-car test photos it reaches 92.0 mask AP50.

Real claim photos are harder. On a small hand-checked set it named the right part for 7 of 22 damages: tight side close-ups are the weak spot, and fenders, quarter panels and the roof aren't in the training data at all. The fixes in progress are a 70% confidence floor before naming a part, and comparing regions (front, rear, left, right) rather than exact parts in the fraud check, so a missing class can't raise a false alarm.

A language model (gpt-oss-20b, on Groq) reads the claimant's story and the police report separately and fills in the same form for both: date, time, location, vehicles and drivers, fault, incident type and which vehicle hit which, on which side. Anything the text doesn't state stays empty, so the model can't invent a mismatch. Every answer is checked against a strict schema and retried once if it fails.

On an unseen story and two police reports, nearly every field came out right. The misses were useful: a single impact zone couldn't hold “front left”, so the model left it empty, and a car reversing out of a parking bay fit none of the five incident types. Both are being fixed.

Then the sources are lined up. The damage zone in the photos is checked against the story and the police report, the story against the report on time, place and where the car was hit, and the size of the damage against the amount claimed. Each disagreement becomes a mismatch flag.

Instead of a hard yes or no, the impact location is a probability across front, rear, left and right, so a corner hit or a vague statement is handled fairly. When the story and the police report disagree, the claim is flagged for review.

In development. A scoring model, trained on claims where we know which were fraudulent, combines the mismatch flags with the claim record into one risk score. The investigator sees why: SHAP for the claim record, attention heatmaps over the photo and text, and an LLM that writes the reason in plain words using only the flags and scores, so it can't make reasons up.

The build order is rule checks and a simple scoring model with SHAP first. The full version, where SigLIP 2 encodes photo and text and a fusion layer joins them to the claim record, gets compared against it.

This is a team project at the University of Wollongong in Dubai, and I lead the architecture decisions. The code stays private because CarDD is licensed for research only.