A hybrid machine learning and deep learning system for phishing email detection in webmail platforms

Abstract

Phishing emails continue to represent a major cybersecurity threat, exploit ing social engineering techniques and deceptive language to compromise user credentials, financial data, and organizational systems. As phishing campaigns become increasingly sophisticated and automated, traditional rule-based detection methods struggle to identify new attack patterns. In response to this challenge, this research proposes a hybrid phishing email detection framework that combines classical machine learning and deep learning techniques for integration within webmail platforms. The objec tive of the study is not only to improve phishing detection accuracy but also to demonstrate the operational feasibility of deploying intelligent de tection systems directly in a real-world email environment.
The proposed framework is trained on a multi-source dataset consisting of more than11,000 labeled emails balanced between legitimate and phish ing messages. The dataset combines legitimate emails from the Enron Email Corpus with phishing samples obtained from public phishing repositories and AI-generated phishing emails. This diverse dataset allows the model to capture both traditional phishing patterns and emerging AI-generated attack styles. To enhance classification performance, the system integrates textual representations extracted from email subject and body with engi neered structural indicators such as suspicious domain patterns, URL entro py, keyword flags, subdomain counts, and other metadata-based features. Two modeling pipelines were implemented and evaluated. The first em ploys classical machine learning algorithms, including Logistic Regression, Random Forest, and Support Vector Machine, trained using TF-IDF textual features combined with engineered attributes. The second pipeline utilizes deep learning architectures, specifically Bidirectional Long Short-Term Memory (Bi-LSTM) networks and the DistilBERT transformer model, to capture contextual language patterns and semantic relationships present in phishing messages. DistilBERT was selected due to its balance between strong predictive capability and relatively low computational cost, ena bling near real-time email analysis.
Experimental results demonstrate strong classification performance across all evaluated models using metrics such as accuracy, precision, recall, F1 score, and ROC-AUC. The best-performing model was integrated into the Roundcube webmail platform to enable real-time phishing detection. In coming emails are automatically analyzed and suspicious messages can be redirected to a dedicated phishing folder, demonstrating the practical applicability of hybrid AI-based detection systems for strengthening op erational email security.