Advances in generative AI have dramatically lowered the barriers to creating and distributing sexual “deepfakes” or non-consensual intimate images (NCII) — a form of technology-facilitated gender-based violence (TFGBV) that disproportionately targets women and marginalized populations. Yet as the technology enabling NCII scales, the online reporting systems meant to address it remain opaque, inconsistent, and burdensome for survivors. Absent clear structural changes to platform design — changes we anticipate would align with the legislative intent of the United States’ TAKE IT DOWN Act and complementary regulations in the United Kingdom and other global contexts — the current approach maintains the burden of harm on survivors.
This white paper, co-authored by Grace Harlan, Alice Jo, and advisor Cailin Crockett presents a comparative analysis of NCII reporting interfaces and policies of seven major online platforms. The findings reveal that prevailing approaches by most platforms to reporting and image removal processes are structurally burdensome, lack transparency and follow-through, and thereby replicate the loss of control and agency survivors experience when their images are shared and created without their consent. They find that platform design for managing NCII — interfaces, categorization, architecture — is entirely reactive, with insufficient investments in proactive, privacy-by-design settings to reduce harm. For example, dominant platforms privilege keeping content initially flagged as NCII visible during the process to verify reports, rather than choosing presumptive removal to protect victim-survivors in the interim. Most platforms fail to provide transparency around content removal and accountability measures, if any, for perpetrators. The authors call for platforms and regulators to center survivors in image-removal and takedown request processes by standardizing reporting categories across platforms, implementing trauma-informed communication practices on report status, and increasing transparency for all users around decisions to remove or keep content. With the May 19 enforcement of TAKE IT DOWN Act requirements for certain platforms in the US, and Ofcom’s updated illegal content codes in the UK, these improvements are more possible now than ever—but hinge on effective implementation and enforcement. Overall, the authors argue that comprehensively addressing NCII requires a fundamental shift: from treating it as an unfortunate but inevitable byproduct of online life to establishing coordinated standards for its rapid detection, removal, and prevention. Moving forward, preventive measures across the technological ecosystem are necessary to limit the demand for, creation, distribution, and monetization of image-based abuse, especially in the era of generative AI.
With contributions from:
Malvika Dwiveldi, Meg Marco, Chelsea McGovern, Matt Marino, Kalie Mayberry, Sarah Moon, Shelby El Otmani, Isabella Roden, Jess Weaver