Why I built it
Phishing used to take skill. Now a generative model can spin up a convincing lure in seconds, clone a brand pixel for pixel, and ship fluent copy that sails past filters that only read text.
I wanted a detector that looks at an email the way a careful human would: read the language, glance at the logos, notice the weird sender domain, then decide. Heron is that system. Paste an email (or drop a .html / .eml file) and you get a phishing-or-legitimate call with a confidence score and the signals that drove it.
One click is all it takes. Heron catches it before you do.
The problem in one picture
Text-only filters fail when the scam looks clean on the page but wears a stolen brand mark. Image-only filters fail when there is no logo at all. Metadata heuristics alone are noisy. Real phishing mixes all three.
Architecture
So Heron runs three specialists on every email, then lets a fusion head learn which specialist to trust for that particular message.
Input
Visible text is extracted, markup stripped, hyperlink patterns and HTML stats computed. Remote image URLs are never fetched.
Three specialists
Weighs language, urgency, and vocabulary from the words alone.
Catches visual brand impersonation in embedded logos. No logo means a zero-filled tensor.
Shortened URLs, caps ratio, suspicious domains, timing patterns.
Decision
Learns to lean on text, logos, or metadata depending on the email. Softmax gives phishing vs legitimate with a confidence score.
Readable signals (urgency language, lookalike domains) ship with the score so the call is inspectable.
- Input
- Tower
- Fusion
Hover a block for what it does. Three specialists look at the same email, then fuse into one verdict.
How a request travels
The app is a FastAPI service with the model in-process. No separate serving tier. One hop: parse, preprocess, classify, return.
From .html, .eml, or pasted text.
Only embedded images. Skipping remote URLs avoids SSRF on a public endpoint.
Text to 512 tokens, images to 224×224 ImageNet-normalized, 20 metadata features scaled.
Towers run, concatenate, classifier returns label and confidence.
One FastAPI process does parse, preprocess, and classify. No separate serving tier.
Parse
The email arrives as pasted text,
.html, or.eml. Visible text comes out, markup goes away, and hyperlink patterns plus HTML statistics get computed.Extract logos
Only embedded brand logos are decoded. Remote image URLs are never fetched. That keeps a public endpoint from becoming an SSRF toy. No logo means a zero-filled tensor for the image tower.
Preprocess
Text is tokenized against the training vocabulary and padded or truncated to 512 tokens. Images resize to 224×224 and get ImageNet normalization. Twenty metadata features are extracted and scaled.
Fuse
The three towers run. Their outputs concatenate into an 832-d vector. The fusion classifier produces two class logits.
Verdict
A softmax turns those into phishing vs legitimate with a 0 to 100% confidence, plus readable signals like shortened URLs, suspicious domains, and urgency language.
- Python
- PyTorch
- FastAPI
- ML
- Security
What the numbers say
I trained and compared six models on a balanced set of 76,346 emails. The dual-tower fusion model lands at 99.45% accuracy and 0.999 AUC-ROC, past every single-modality baseline.
Baselines
Classic feature baseline.
Classic feature baseline.
Specialists
Phase 1 text specialist.
Phase 1 image specialist. Weak alone, useful in fusion.
Transfer-learning comparison baseline.
Final
23 false positives and 32 false negatives on ~10k validation emails.
Dual-tower fusion beats every single-modality baseline on the held-out split.
| Model | Accuracy | Notes |
|---|---|---|
| KNN (text features) | 81.71% | Baseline |
| Logistic Regression | 80.00% | Baseline |
| Custom Text CNN | 98.96% | Phase 1 text specialist |
| Custom Image CNN | 76.30% | Phase 1 image specialist |
| ResNet18 (transfer learning) | 97.43% | Comparison baseline |
| Dual-tower fusion | 99.45% | Final model |
On the phishing class (the expensive one to miss):
| Metric | Value |
|---|---|
| Precision | 99.5% |
| Recall | 99.4% |
| F1 | 99.4% |
| False positives / negatives | 23 / 32 on ~10k validation emails |
Why the image tower still earns its seat
76.30% alone looks weak next to the text CNN. In fusion it still helps, because brand impersonation shows up in pixels when the wording is carefully bland.
The data underneath
The email corpus mixes four public sources (CEAS_08, Enron, Nazario, Nigerian Fraud) into 76,346 deduplicated, labelled emails, roughly 48% phishing and 52% legitimate. The logo corpus is OpenLogo: 72,652 images across 352 brand classes.
Real phishing HTML with embedded logos is scarce, so the test set was also augmented with generative-AI phishing emails that keep authentic structure and logos. That is the world the model has to survive in.
Weights ship from a public Hugging Face repo at startup. The service is containerized and runs on Cloud Run.
Design choices I would defend
Learned fusion, not a fixed rule
The fusion layer weights each modality per email. It leans on text when the language screams urgency, on logos when the brand mark and the sender domain disagree, and on metadata for clean-looking edge cases.
Graded confidence
A binary flag hides uncertainty. Every verdict carries a percentage and the signals that mattered, so a reviewer can see why.
No remote image fetch
Skipping remote logos costs some recall on image-only cues. It also keeps a public scan endpoint from fetching arbitrary URLs for an attacker.
In-process model
One FastAPI process owns parse through classify. Fewer moving parts, one log stream, one deploy.
Design rationale
Why dual-tower instead of one big encoder? Phishing cues are heterogeneous. A single encoder tends to under-represent one of the modalities. Separate towers let each pathway specialize, and fusion combines the strengths.
How do you validate beyond a single accuracy number? Accuracy, AUC, precision, recall, and F1 on the phishing class, plus a raw false-positive / false-negative count on the held-out split. Thresholds get chosen for the cost of a miss, not for a pretty aggregate.
What about the precision / recall tradeoff? Missing a phishing email is worse than bothering someone with a false alarm, but flooding people with false alarms kills trust. The operating point sits where recall stays high without making the product unusable.
What I would tell you next
If you want the full math and ablations, the research paper and architecture report are linked as chips at the top. The live scan surface is at heron.v-ai.org.
Sole builder
I designed, trained, evaluated, documented, and shipped every part of this. Spec-driven work against my own written specs, with an append-only session log and a teaching journal in the repo.