Explainable Deepfake Detection.
Deepfake technology uses deep learning to create hyper-realistic fake videos and images, often by altering facial features or synthesising entirely new faces. As generation methods improve, telling genuine content from fabricated content gets harder, which raises concerns about misinformation, privacy violations and threats to individuals and institutions. This project addresses the problem by developing and evaluating CNN-based detectors (ResNet50 and InceptionV3) to improve the accuracy and reliability of deepfake detection, and uses GradCAM to make each prediction explainable.
Reports
Highlights
Data Extraction.
Original and deepfaked videos (250 of each) were taken from FaceForensics++. Every frame of each video was extracted into a folder named after its video ID. Because consecutive frames are nearly identical, only 1 in every 100 frames was kept, which cut computation and reduced overfitting to individual faces. A GPU-based MTCNN (facenet-pytorch) then cropped each retained frame so that only the face was saved. The result was about 2,300 face images, split 50:50 between real and manipulated, and divided 70/20/10 into training, validation and test sets.
Model Performance and Visualisation.
Three models were trained for 20 epochs on ImageNet-normalised inputs: [1] a pretrained ResNet50, [2] a pretrained InceptionV3, and [3] a ResNet50 trained from scratch. The pretrained ResNet50 performed best, reaching 99.25% training accuracy and 96.51% test accuracy (test loss 0.0662). The pretrained InceptionV3 followed closely at 95.63% test accuracy (test loss 0.1089), while the ResNet50 trained from scratch reached only 86.03% (test loss 0.3832). With only a few thousand images, transfer learning made a large difference: the model trained from scratch learned more slowly and generalised worst. Validation curves fluctuated for both ResNet models, which suggests some overfitting, while InceptionV3 trained the most stably. GradCAM was then used to show which regions drove each prediction. With its last convolutional layer as the target, ResNet50 focused on the eyes and mouth of real faces, and on deepfakes it highlighted the manipulated regions and traced their boundaries. Even the ResNet50 trained from scratch localised the manipulations well, despite its lower accuracy. InceptionV3 (target layer Mixed_7c), by contrast, highlighted the whole centre of the face with little differentiation, and this less targeted attention may explain why it scored lower than ResNet50.
Future Improvements.
The current models predict manipulation with high accuracy, but the reasoning behind a prediction can still look abstract to a human viewer. Combining Vision Transformers (ViTs) with Momentum Contrast (MoCo) learning could produce clearer, more interpretable explanations by showing which image patches matter most for the classification. A possible implementation is to: [1] pre-train a ViT with MoCo on a large image dataset, using instance discrimination so the model learns rich feature representations by telling different images apart, [2] fine-tune the pre-trained ViT on the deepfake detection dataset, [3] extract the ViT’s attention maps to see which image patches it focuses on, and [4] apply GradCAM to the ViT’s attention outputs and overlay the resulting maps on the original images. Splitting the data by video, before extracting frames, would also stop near-identical frames from the same video appearing in both the training and test sets, a leakage that may currently be making the reported test accuracy look better than it is.
