Project overview

The project examined how deepfakes are reshaping online safety risks, in particular how this technology is being used to deceive and manipulate people from scams and fraud to harassment and impersonation. This technology is developing quickly, and many users remain poorly equipped to judge whether content is genuine. It also remains unclear how far labelling or authentication measures can improve people’s understanding and perception of harm.

The secondment involved a rapid review of academic research and grey literature, alongside real-world case studies of deepfake misuse. The evidence demonstrated that responses focused narrowly on detection or transparency are important but insufficient on their own. People are generally poor at spotting deepfakes, and often overestimate their own ability to do so. Automated detection tools help, but struggle to keep pace as deepfake technology improves. Warnings and labels can reduce belief in a deepfake, but people can still be influenced by fabricated content even after being told that it is fake. Deepfakes can also erode trust more broadly, making people doubt genuine content as well as false content. The harms involved go beyond any one type of manipulation, spanning fraud, harassment, sexual exploitation, and scams.

To date, this work has led to two further strands of work. First, the rapid review fed into a broader academic paper on how people become vulnerable to malicious online influence. The paper takes Williams, Beardmore and Joinson’s (2017) model of scam susceptibility, and updates it to account for AI-generated content, using deepfakes as a case study. Its central argument is that as deepfakes become more convincing, the usual warning signs people rely on to spot manipulation, such as urgency or unfamiliar sources, become less reliable, because the content itself can look and sound entirely genuine. The paper also examines the limits of detection tools and content authentication as safeguards.

Second, the work has informed a series of public engagement workshops with adults over the age of 55, reflecting prior research showing this group faces heightened risk from online scams. These workshops will help identify what protections and interventions matter most to this group, informing future research.

Objectives

The secondment had the following objectives:

1. Explore how deepfakes are shaping misinformation and online safety.

2. Develop a clearer understanding of current challenges, including how people interpret and engage with AI-generated content, and how labelling or authentication affects trust, comprehension, and perceived harm.

3. Identify priority questions for research and policy.

4. Produce a short briefing note/scoping report outlining key findings, gaps, and next steps, to inform the co-development of longer-term initiatives linked to online harms, detection tools, and transparency in content moderation.

Two things changed from the original objectives. First, in practice the work centred on deepfakes as a tool for deception and malicious manipulation, such as scams, fraud, harassment, and impersonation, rather than misinformation specifically. Second, rather than a standalone briefing note or scoping report, the evidence gathered during the secondment was developed into a full academic paper. This serves the same purpose, setting out key findings, gaps, and next steps, but represents a more substantial output than originally scoped.

Activities

The secondment ran over 8 days within Ofcom's Online Safety team and combined knowledge exchange and rapid evidence synthesis.

Knowledge exchange:

• Delivered an invited talk to Ofcom colleagues on the behavioural evidence underpinning AI bot influence.

• Contributed academic insight to Understanding how people and businesses can benefit from AI in telecoms markets, an Ofcom document published during the secondment period inviting stakeholder comments on AI’s impact on telecoms customers, with Ofcom’s findings expected in the second half of 2026.

• Attended meetings with colleagues across Ofcom's Online Safety, Technology Policy, and Behavioural Insights teams to understand priority challenges and identify areas of mutual interest.

Evidence synthesis:

• Conducted a rapid review of deepfakes and synthetic media, drawing on academic research, grey literature, policy documents, and case studies of malicious use, focusing on how deepfakes are used to deceive and manipulate people including scams, fraud, harassment, and impersonation.

• Mapped key research questions on authenticity judgement, the persuasiveness of deepfake content, and mitigation strategies (warnings, provenance indicators, media literacy, human–AI detection).

• This review directly informed the development of an academic paper extending the Williams, Beardmore & Joinson (2017) influence-susceptibility framework to the deepfake/AI context (see Outcomes below).

Relationship-building and future planning:

• Identified Ofcom as a partner for future grant applications, knowledge exchange, and student placements.

Outcomes and outputs

• Delivered a talk to Ofcom colleagues on a scoping review of AI-mediated influence and online persuasion, part of complementary, ongoing work examining AI-generated influence more broadly.

• Academic contribution to Ofcom's published report Understanding how people and businesses can benefit from AI in telecoms markets, an Ofcom document published during the secondment period inviting stakeholder comments on AI’s impact on telecoms customers, with Ofcom’s findings expected in the second half of 2026.  

• Academic paper (in preparation for submission in August 2026): Walker, E.J. & Hinds, J. Protecting users from malevolent online influence in an evolving AI-era: A theoretical update for an uncertain future. The paper takes Williams, Beardmore and Joinson’s (2017) model of scam susceptibility and updates it to account for AI-generated content, using deepfakes as a case study.

• A series of public engagement workshops with adulted (aged 55+), informed by the work conducted during the secondment. The insights gained will feed into a future research funding application.

• Ofcom identified as a partner for future grant applications, knowledge exchange, and student placements.

Impact

The present research found that detection technology alone cannot address the harms involved. For harms like fraud and scams, deception is central, as synthetic content becomes more convincing, the usual warning signs people rely on to spot manipulation become less reliable. But other harms, such as harassment and sexual exploitation, do not necessarily rely on deception, the harm comes from the content itself, regardless of whether anyone believes it is real. This distinction has since fed into a broader academic contribution: an updated model of how people become vulnerable to malicious online influence generally, using deepfakes as a worked example.

Key findings underpinning this impact include:

• Humans are generally poor at detecting deepfakes, and often overestimate their own ability to do so.

• utomated detection tools help, but struggle to keep pace as deepfake technology improves.

• Warnings and labels can reduce belief in a deepfake, but do not remove its influence entirely; people can still be influenced by fabricated content even after being told it's fake.

• Deepfakes can erode trust more broadly, making people doubt genuine content as well as false content.

• The harms involved go beyond any one type of manipulation, spanning fraud, harassment, sexual exploitation, and scams.

Academic impact:

The secondment's rapid review fed directly into an academic paper that updates Williams, Beardmore and Joinson's (2017) model of scam susceptibility to account for AI-generated content, using deepfakes as a case study. Its central contribution is identifying that the usual cues people rely on to judge authenticity become less reliable as AI-generated content improves, a distinction with direct relevance for how platforms and regulators design detection, labelling, and credentialing interventions.

Policy impact: 

No policy impact has resulted from the secondment to date. However, the evidence base developed during this project will be drawn upon over the next 18 months as part of a UKRI Policy Fellowship, currently being undertaken at the Government Office for Science on a foresight project, providing evidence as this work continues to develop.

Organisational impact:

Strengthened the working relationship between the University of Bath's Institute of Digital Security and Behaviour and Ofcom, creating a foundation for future joint grant applications and student/researcher placements.

Social impact:

The secondment has informed a series of public engagement workshops with adults (55+), which will be used to co-design priorities for future research into deepfake-related harms. As this work is still ongoing, its impact cannot yet be assessed, but it represented a direct extension of the secondment’s focus into public-facing research practice.

Future work

Several strands of future work are planned, some of which build directly on open questions raised in the accompanying academic paper:

• Testing the model. The academic paper’s MOI-AI framework (Malevolent Online Influence in the AI era) distinguishes between traditional influence techniques (e.g., urgency, authority) and authenticity cues (e.g., how convincing the message’s content looks or sounds), a factor the paper argues becomes more critical as AI-generated content grows more credible. The next step is testing this empirically: examining how these factors interact to affect people’s judgments and behaviour, across different individuals, contexts, and types of message.

• Developing and testing interventions. Building and evaluating interventions informed by the paper’s framework, media literacy, warnings and labelling, content provenance, and human-AI collaborative detection, to understand which approaches genuinely reduce harm without unintended consequences such as reduced trust in legitimate content.

• Public engagement workshops with adults (aged 55+). A series of workshops is planned to run in September and October. The findings from the workshops will inform a future research funding application.

• UKRI Policy Fellowship. An 18-month secondment, currently being undertaken at the Government Office for Science is focusing on a foresight project. The rapid review, and subsequent ongoing research will be drawn upon as a means to provide evidence on how AI is likely to reshape fraud and financial crime, and to inform the technology and behavioural science dimensions of the fellowship’s policy recommendations.

• Further Bath-Ofcom collaboration. Consistent with the original secondment proposal, the University of Bath and Ofcom intend to pursue further collaborative funding, alongside PhD and postdoctoral placements.