How We Measure Accuracy (And Why We Don't Publish a Single Number)

What powers the analysis

CatfishTracker uses Claude, Anthropic's large multimodal AI model, to visually analyse uploaded photos. The model examines the image across eight analysis dimensions: signs of AI generation, deepfake artefacts, beauty app usage, facial reshaping, skin smoothing, background manipulation, stock or model photo indicators, and lighting inconsistencies. It returns a score from 0–100 and a verdict: authentic, suspicious, manipulated, or AI generated.

The model was not trained by us we did not build a custom classifier. We prompt a state-of-the-art vision model with dating-specific photo analysis instructions and it applies its broader knowledge of visual manipulation, generative AI artefacts, and photo editing patterns. This is both a strength and a constraint.

What it's genuinely good at

• ▸Detecting obvious AI-generated faces faces from tools like Midjourney, DALL-E, and Stable Diffusion have characteristic artefacts (skin texture, ear shape, background coherence) that the model identifies reliably.
• ▸Flagging heavy Facetune and beauty app usage excessive skin smoothing, digitally enlarged eyes, and reshaping leave visible traces the model catches well.
• ▸Identifying studio or stock photos used as profile pictures professional lighting, unnatural perfection, and watermark artefacts are strong signals.
• ▸Spotting inconsistent lighting between a face and background a common tell in composite or AI-generated images.
• ▸Conversation scam pattern detection the conversation analyser is trained on documented scammer scripts and detects love bombing, off-platform requests, money asks, and urgency tactics with high consistency.

Where it struggles

• ▸Subtle edits small amounts of skin smoothing or minor facial adjustments on otherwise genuine photos may not be flagged. The model is tuned to avoid over-flagging, which means very light edits can pass.
• ▸High-quality AI images the best modern AI generators are improving faster than detection methods. A carefully generated, high-resolution face with consistent lighting will score better than it should.
• ▸Low-resolution or heavily compressed photos JPEG artefacts and low resolution reduce the amount of visual information available for analysis, making verdicts less reliable.
• ▸Selfies vs. professional photos the model can interpret a professional photographer's clean work as suspicious because it looks "too good", while a poor-quality real photo might score lower than expected.
• ▸It cannot verify identity a real photo of a real person can still be stolen and used as a fake profile. The tool detects photo manipulation, not impersonation.

Why we don't publish a headline accuracy figure

Accuracy depends entirely on what you're measuring. Accuracy on professional AI-generated headshots? High. Accuracy on lightly filtered real photos? Lower. Accuracy on decade-old compressed screenshots from a fake Facebook profile? Different again. A single number averaged across all of these would be technically true and practically meaningless.

There's also the base rate problem. On any given dating app, the overwhelming majority of photos are real. A tool that labels everything "authentic" would have extremely high accuracy by raw numbers and be completely useless. What matters is not overall accuracy but false negative rate (missed fakes) and false positive rate (flagging real photos). We optimise to catch clear fakes while keeping false positives low enough that genuine people aren't repeatedly told their photos look suspicious.

Known edge cases to be aware of

• ▸Filters and Snapchat lenses: light filters don't reliably trigger a flag. If someone always uses heavy filters, ask for an unfiltered photo.
• ▸Older photos: a photo that is five years old might be genuine but no longer look like the person. The tool analyses the photo, not whether it's current.
• ▸Screenshots of photos: adding a screenshot border, watermark, or platform UI around a photo can affect the analysis.
• ▸Very dark or backlit photos: insufficient lighting reduces the quality of analysis quality significantly.

What we're working toward

We're building a labelled dataset from user-reported outcomes to eventually measure our own false negative and false positive rates properly. When we have enough data to publish numbers that are honest and meaningful, we will. Until then, we'd rather explain how the tool works than hide behind a percentage.