Robustness of Phishing Detection
Under Adversarial Unicode Obfuscation
I evaluated how a modern ML-based phishing vs benign classifier behaves when phishing emails are adversarially obfuscated using Unicode homoglyphs and zero-width characters, and compared its robustness with brittle keyword-based methods from Project 1.
Total emails (view)
4,209
4,151 phishing BEC + 58 benign
Adversarially modified phishing
1,187
Unicode homoglyph + zero-width attacks
Modified phishing in test set
230
Used for focused robustness analysis
Phishing recall (clean vs poisoned)
100%
No degradation in this setting
Experiment Pipeline
How I constructed the phishing vs benign dataset and tested robustness to Unicode obfuscation.
- 01
Dataset construction
Combined 4,151 synthetic BEC phishing emails (clean + obfuscated) with 58 benign emails from the Hugging Face dataset `UniqueData/email-spam-classification` (using only 'not spam' samples).
- 02
Phishing vs benign setup
Defined a binary classification task: label 1 for BEC phishing, label 0 for benign emails. Built two parallel views: clean (body_clean) and poisoned (body_poisoned).
- 03
Model training on clean text
Trained a TF-IDF (unigram + bigram) + Logistic Regression classifier on the CLEAN view only, with class weighting to address phishing/benign imbalance.
- 04
Robustness evaluation
Evaluated the same model on two test views: clean text vs Unicode-obfuscated phishing text, keeping benign emails unchanged.
- 05
Modified-only analysis
Isolated the 230 phishing emails that were actually adversarially modified and compared predictions using their clean vs obfuscated versions.
Clean vs Poisoned PerformanceRobustness Check
Clean-trained model evaluated on clean vs Unicode-obfuscated phishing emails.
In this configuration, the ML classifier remains robust to Unicode obfuscation, even though keyword-based detectors from Project 1 suffered heavy degradation.
Robustness Summary
Same model, same train split, evaluated under clean vs adversarially obfuscated phishing emails to measure robustness.
| Scenario | Phishing recall | Benign recall | Accuracy | Notes |
|---|---|---|---|---|
| Baseline: Clean test set | 100% | ≈ 83% | ≈ 99.8% | Model trained and evaluated on clean text; strongly separates phishing from benign emails. |
| Robustness: Poisoned test set | 100% | ≈ 83% | ≈ 99.8% | Same model evaluated on Unicode-obfuscated phishing emails; performance remains unchanged. |
| Modified phishing only (clean vs poisoned) | 100% → 100% | N/A | 100% | On the 230 adversarially modified phishing emails, the classifier still flags all correctly. |
Key Insights
- A clean-trained word-level TF-IDF + Logistic Regression classifier achieves ≈ 99.8% accuracy with effectively perfect recall on phishing emails.
- Replacing phishing emails with their Unicode-obfuscated versions (homoglyph + zero-width) does not measurably change performance in this setup.
- On the 230 adversarially modified phishing emails in the test set, the classifier still flags 100% correctly in both clean and obfuscated form.
- This contrasts with Project 1, where keyword-based detectors suffered an 81.3% evasion rate under the same obfuscation attack.
- The results suggest that richer ML representations can be comparatively robust to Unicode obfuscation, but must still be combined with layered defenses and monitoring.
Tech Stack
Tools and concepts used to evaluate robustness of phishing detection under Unicode attacks.