The history of audiovisual media is the history of audiovisual manipulation. The first camera came into the world in 1816, although it was not until 1888 when American businessman George Eastman started marketing a device under the name of “Kodak.” Cameras were commercialized more widely in the early twentieth century, but even before the average consumer was able to take their own photographs, the world had already seen its first trial for audiovisual manipulation back in 1869. William Mumler, a jewelry engraver from Boston, took a selfie (or, back then, a “self-portrait”) that revealed the shape of his deceased cousin on the image. As the story goes, initially Mumler shared the ghostly photograph with a friend as a joke, but seeing the amazement of his colleague, he thought he could make a lucrative business out of “spirit photography” by taking images of people and conjuring their loved ones to appear on camera. These manipulations seem to have been the result of nondivine intervention, showing us an early example of double exposure, by which a previous image made its way into another photograph that used the same glass plate for producing the negative. After a few years of cashing in on the grief that the American Civil War had brought, he was accused of fraud. During the trial, another photographer testified as a witness, having produced himself a fabricated image of a client with the “ghost” of Abraham Lincoln to demonstrate the manipulation technique. Mumler was eventually acquitted.
February 20, 2024 Feature
Deepfakes in the Dock: Preparing International Criminal Justice for Deepfakes and Generative AI
Raquel Vazquez Llorente
The Mumler trial not only represents one of the earliest recorded cases of malicious audiovisual manipulation, but it also highlights two powerful dynamics that carry throughout history. First, many of us are willing to place trust in what we see as long as it aligns with our worldview. Second, even when demonstrating the ease of forgery, it may still be difficult to prove in court that any editing took place on a given image. Advancements in audiovisual manipulation, specifically deepfake technology, combined with the commercialization of artificial intelligence (AI) tools that allow anyone with an internet connection to create realistic synthetic images or audio, are now bringing novel challenges to the courtroom. While the impact of generative AI and synthetic media in the information landscape has been the subject of heightened scrutiny since shortly after ChatGPT was made publicly available in November 2022, the effects that deepfake technology can have on international criminal justice are yet to be discussed with similar intensity. This article aims to start a deeper, more informed conversation about how to prepare international courts and tribunals, especially the International Criminal Court (ICC or the Court), for developments in the field of digital evidence and media synthesis.
Demystifying the Tech: GANs vs. Diffusion Models
Deepfakes, a portmanteau of “deep learning” and “fake,” have evolved rapidly since they first appeared in 2017 in a Reddit group that was trading content of celebrities whose faces had been swapped into videos (including pornographic movies). Deepfake technology harnesses AI to create realistic images, videos, and audio recordings that can oftentimes be indistinguishable from reality. Some of the most common techniques leverage Generative Adversarial Networks (GANs) and diffusion models. GANs involve two neural networks that are pitted against each other. One is the generator that creates the examples; the other one is the discriminator that evaluates whether they are real or fake. With each iteration, the generator refines the output. A GAN trained on images of faces can produce new synthetic faces that look realistic.
On the other hand, diffusion models are a type of deep generative model that add noise to the training data and then reconstruct the data by reversing this process. While GANs are like a legal debate where each side sharpens the other’s skills, diffusion models are more about taking a broad idea and refining it into something clear and detailed. They are remarkably adept at generating high-quality images that can sometimes beat the capabilities of GANs. Whereas GANs excel in creating faces and expressions, diffusion models can craft detailed and realistic textures and patterns, making them a powerful technique to fabricate convincing environments or contexts. Diffusion models are responsible for much of the progress over the past two years in the image domain, and they are behind tools now known to the public, such as Dall-E or Stable Diffusion.
The Impact of Generative AI on International Justice
Deepfakes, or claims of AI manipulation, have begun to surface in countries embroiled in armed conflicts, experiencing mass violence, or under the thumb of authoritarian regimes. In late 2018, a video of President Ali Bongo of Gabon, released by his office partly to counter claims of his ill health, was dismissed as a deepfake by the opposition, precipitating an attempted military coup. Experts have not been able to conclude unanimously whether the video is the result of AI manipulation (although it was probably not a deepfake). In Myanmar in 2021, Phyo Min Thein, former Chief Minister of Yangon Region, confessed on camera to having paid gold and cash to Aung San Suu Kyi as part of a corruption scheme. The video was likely to have been produced under duress rather than by AI intervention.
In Venezuela and Burkina Faso, governments or their sympathizers have misused commercial software intended to be for company training and product demos, to produce favorable propaganda. In Mexico, candidates in the 2024 electoral cycle have seen themselves or their voices deepfaked into videos or audios. More notoriously, in Ukraine, deepfaked President Zelenskyy called the troops to surrender in a video rapidly debunked by the government. In Sudan, “leaked recordings” of Omar Al Bashir, the former leader who has not been seen in public for a year, were suspected of being manipulated. The organization I work for, WITNESS, has also received tips of two audio files and a video circulated on official social media channels of the Rapid Support Forces (RSF) or of their leader, Mohamed Hamdan Dagalo, who is believed to be dead. We consulted AI media forensic experts, and they all concluded that there did not seem to be indications of AI manipulation—with the caveat that models do not perform to the same level of accuracy for non-English languages. Neither does this mean the absence of other forms of manipulation, such as more traditional editing. In the recent conflict between Palestine and Israel, synthetic media portrays Israeli refugee camps, crowds marching in support of Israel, military attacks taken from video games, and children in need.
These are just a few examples of how AI is creating confusion and mistrust in what we see or hear. As deepfake technology grows more accessible, the volume of synthetic media is increasing, swamping the information ecosystem and impacting most countries, including those under the jurisdiction of the ICC. Gabonese officials and opposition members were once under preliminary examination for crimes against humanity and incitement to genocide. The situation in Myanmar/Bangladesh is under investigation. The prosecutor of the ICC resumed in 2023 their investigation into the situation in Venezuela (for transparency, I was one of the lawyers involved in the communication submitted in 2022 by the Clooney Foundation for Justice and Foro Penal to contribute to the the ICC’s investigation). While the Court has yet to act on the allegations of crimes against humanity in Mexico, victims’ groups and civil society have been calling for an investigation into mass disappearances for nearly a decade. The ICC has issued arrest warrants for war crimes in Ukraine against Vladimir Putin, president of the Russian Federation, and Maria Lvova-Belova, commissioner for Children’s Rights in the Office of the President of the Russian Federation. Al Bashir was the first sitting head of state with a warrant of arrest by the ICC and the first person to be charged by the Court for the crime of genocide. The prosecution is also investigating alleged crimes by the RSF. According to a pre-trial Chamber decision from 2021, the ICC has jurisdiction over Rome Statute crimes in Gaza and the West Bank, including East Jerusalem, and over crimes committed by Palestinian nationals or the nationals of any state parties on Israeli territory. For the past two years, the Office of the Prosecutor has been “steadily increasing the resources and personnel for the Palestine investigation.”
Looking at the countries listed above, it is not a question of whether an institution like the ICC will face a deepfake problem, but how badly it will impact justice. U.S. Ambassador-at-Large for Global Criminal Justice Beth Van Schaak has highlighted the urgency of the situation: “Now we have the ability to use synthetic media, generative AI, [. . .] and so you can imagine creating pieces of evidence that could actually infect a legal process.” While the proliferation of deepfakes risks false information being accepted as true, the larger threat is that the possibility of AI manipulation is making it easier for the public or those in power to dismiss real content as fake. If this general mistrust creeps into the courtroom, it may poison the well of all audiovisual digital evidence.
The ascent of deepfakes raises the specter of doubt, turning audiovisual content from important probative material into potential red herrings. While deepfakes could be used to frame innocent parties or for creating “evidence” of atrocities that never occurred, looking back in history and at the jurisprudence at the ICC, synthetic media will most likely have two effects on judicial processes for core international crimes if left unaddressed. First, the burgeoning volume of deepfakes will overwhelm the capacity for analysis of institutions in charge of justice and accountability. The conflict in Syria already sounded alarm bells and catalyzed the application of object recognition and frame analysis technology to images circulated mostly online; however, traditional verification methods are no longer sufficient. Second, the leap forward in deepfake technology will lead to acquittals of those guilty of crimes by inserting doubt on authentic evidence.
Future-Proofing Triage and Analysis at the International Criminal Court
The ICC prosecutor has the obligation to investigate both incriminating and exonerating circumstances equally. Hence, sifting through the avalanche of content to identify authentic evidence is critical. In an academic article I wrote with Lindsay Freeman, we examined how the volume, velocity, and volatility of digital evidence would increasingly affect trial proceedings at the ICC, and we detailed specific recommendations to help the Court address some of their most immediate needs. With generative AI rapidly reshaping the landscape of truth in armed conflicts and situations of mass violence, differentiating genuine content from manipulated media is now assuming a new urgency. This is not just a technical challenge but a fundamental threat to the course of justice.
Institutional responses must evolve in lockstep with technological innovations. As deepfake technology becomes increasingly advanced and accessible, the ICC can counteract the threats this brings by investing in three areas: provenance infrastructure, detection technology, and analytical and forensic expertise. Given the significant resources required to develop and deploy state-of-the-art technology and the capacity gap on media forensics (not unique to the ICC), these approaches will require multistakeholder collaboration, notably with the private sector. None of these solutions will be sufficient on their own, but combined they can be what tips the balance from impunity to justice.
Provenance Infrastructure
The first line of defense against deepfakes is provenance infrastructure—a suite of technology solutions that can help trace the origin of a piece of content and any modifications since its creation. Provenance tools embed metadata into a piece of media, creating “content credentials” that can help with the verification. The human rights sector has been a pioneer in piloting apps for human rights defenders documenting abuses that automatically collect metadata, for instance, CameraV and Proofmode. In the last few years, we have seen provenance technology gaining traction in the form of standards spearheaded mainly by the Coalition for Content Provenance and Authenticity (C2PA), of which WITNESS is part.
Most of the solutions available can provide helpful markers to verify content but have crucial limitations from a litigation point of view. For international criminal justice, the most useful approach to provenance is what is known as “controlled capture,” specialized software that incorporates data points at the moment of recording an image or audio file and maintains the chain of custody throughout the life of the footage. To date, only one solution has been developed that meets these criteria off the shelf and has successfully been tested in a court of law: the eyeWitness to Atrocities app, incubated at the International Bar Association (in the spirit of full disclosure, I was involved in setting up the organization and was one of the lawyers working on the first case that used content collected with the eyeWitness app). Regardless of the criteria that provenance tools are trying to meet, there is a blatant barrier to their effectiveness. These technology solutions are ultimately only as strong as their adoption, so they require intense outreach to put the tools in the hands of those documenting international crimes. Furthermore, without robust capacity-building programs that give those behind the camera the knowledge to capture what is relevant to a case, and those receiving the material the expertise to interpret the metadata, the potential of provenance technology will be crippled.
Detection Technology
Just as deepfakes grow more sophisticated, so does the need for tools that can detect whether a piece of audiovisual content has been AI generated or manipulated. It is unlikely that the ICC will be able to develop, and keep up the maintenance of, their own state-of-the-art detection tools. For this reason, partnerships with leading companies can help fill this detection gap (the Office of the Prosecutor already has an ongoing collaboration with Microsoft). However, it is important to note three shortcomings that may likely arise.
Generally speaking, detection tools are still in their infancy and have yet to be deployed at scale in a sufficiently reliable fashion. While there has been dramatic progress in audio synthesis over the year 2023, audio biometrics, which could help identify whether someone uttered certain words, still lag behind. As Sam Gregory, executive director at WITNESS, puts it, “[i]t’s no use having a tool to spot whether content is generated by one company when the same tool would give a false negative on fake audio created by one of the many other tools on the market.” When AI detection tools are trained on data not specific to an individual, their accuracy is similar to that of humans. If trained on the biometrics of a person, such as President Zelenskyy, their accuracy in determining the authenticity of a given audio can jump to 99%. This said, current deepfake detection technology has been largely optimized for facial recognition, and it is less adept at discerning tampering in nonhuman elements of footage. This limitation is critical: the manipulation of structures or the environment in a video—like a doctored image of a bombed building or the insertion of a falsified object such as illegal ammunition in a civilian area—can significantly alter the course of an investigation.
Second, algorithms trained to spot inconsistencies in images, videos, or audios underperform in non-English languages, noisy environments, and complex scenarios like conflict settings. Grainy footage, low-resolution, or lacking a representative biometric sample will challenge computational assessments. Additionally, automated analysis is more effective when the file is as close as possible to the original version. Social media platforms, by compressing images and modifying file metadata upon uploading, can obscure the digital footprint, thereby confounding automated systems. As long as audiovisual content continues to be posted online, the digital footprint will get muddled, confounding automated systems. This is why detection algorithms must be paired with contextual analysis and corroborating material to ascertain the authenticity of content. To enhance the sensitivity of these models, there is a pressing need for datasets of manipulated videos and audio, specifically derived from social media. Collaboration with these platforms is then essential to help researchers refine detection algorithms. Similarly, the integration of biometric information is necessary. However, we should also understand that acquiring potentially sensitive data could put at risk those to whom we are trying to bring justice, should this information fall in the hands of malicious actors or oppressive governments. History teaches us that mass atrocities are often preceded by mass surveillance, enabled by the collection of data and the monitoring of targeted communities.
Last, from a legal standpoint, the deployment of automated detection technology raises complex questions. If such evidence is to be used in court, how will it be presented and understood by the parties? The fact that a model did not find manipulation does not mean a specific piece of content is not a deepfake; it just means that the system did not detect what it was looking for. For example, it could be a deepfake generated in a new way that the detection tool had not yet been trained to identify. Legal practitioners will also have to grapple with the multiple variations of a manipulation. A video could be entirely synthetic, or the deception could lurk only in a fraction of the frames or the voice-over. Current models struggle with pinpointing targeted manipulations, such as the addition of an object or the alteration of a background. Furthermore, the credibility of a witness could be called into question as a sophisticated deepfake could shake the confidence in the authenticity of legitimate evidence. If it is possible to falsify one event, what is stopping any evidence from being doubted? This broad basis for mistrust threatens the core of international justice—the establishment of an objective truth.
Analytical and Forensic Expertise
Given these shortcomings, detection tools will most likely be better employed to conduct the initial triaging of audiovisual content, instead of aiming at producing conclusive verification reports. Ascertaining authenticity will require investing in human analysis in a twofold manner, by strengthening the knowledge and capacity of staff, particularly analysts and the judiciary, but also through secondment of expertise from governments, other accountability bodies, specialized institutions like Justice Rapid Response (JRR), and the private sector. Interpreting the events under question will also necessitate updating the Court Registry’s roster to incorporate expert witnesses who are skilled in media synthesis and forensic analysis and are able to testify in court to explain provenance techniques such as digital signatures, provenance standards like C2PA, detection techniques, and other technical aspects related to the authenticity of evidence.
As deepfake technology becomes more accessible and convincing, the ICC and accountability institutions must pivot towards sophisticated approaches to provenance, detection, and expertise that can keep pace with the evolving technology. The sheer quantity of potentially AI-generated or manipulated content threatens to overwhelm investigators and analysts tasked with triaging audiovisual content, potentially leading to miscarriages of justice. We owe it to the victims and survivors of these crimes to invest now in preparing our international justice system for a future that is already here.