Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center

James Wen, Sahil Nalawade, Zhiwei Liang, Catherine Bielick, Marisa Ferrara Boston +36 more
6/26/2025
cs.CYcs.AI

No confident code implementation found for this paper.

Abstract

Background: Generative artificial intelligence (AI) deployment in academic medical settings raises copyright compliance concerns. Dana-Farber Cancer Institute implemented GPT4DFCI, an internal generative AI tool utilizing OpenAI models, that is approved for enterprise use in research and operations. Given (1) the exceptionally broad adoption of the tool in our organization, (2) our research mission, and (3) the shared responsibility model required to benefit from Customer Copyright Commitment in Azure OpenAI Service products, we deemed rigorous copyright compliance testing necessary. Case Description: We conducted a structured red teaming exercise in Nov. 2024, with 42 participants from academic, industry, and government institutions. Four teams attempted to extract copyrighted content from GPT4DFCI across four domains: literary works, news articles, scientific publications, and access-restricted clinical notes. Teams successfully extracted verbatim book dedications and near-exact passages through various strategies. News article extraction failed despite jailbreak attempts. Scientific article reproduction yielded only high-level summaries. Clinical note testing revealed appropriate privacy safeguards. Discussion: The successful extraction of literary content indicates potential copyrighted material presence in training data, necessitating inference-time filtering. Differential success rates across content types suggest varying protective mechanisms. The event led to implementation of a copyright-specific meta-prompt in GPT4DFCI; this mitigation has been in production since Jan. 2025. Conclusion: Systematic red teaming revealed specific vulnerabilities in generative AI copyright compliance, leading to concrete mitigation strategies. Academic medical institutions deploying generative AI should implement continuous testing protocols to ensure legal and ethical compliance.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

DataCook: Crafting Anti-Adversarial Examples for Healthcare Data Copyright Protection

85% match2024

Generative Artificial Intelligence for Academic Research: Evidence from Guidance Issued for Researchers by Higher Education Institutions in the United States

85% match2025

Academ-AI: documenting the undisclosed use of generative artificial intelligence in academic publishing

84% match1 repo2024

U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AI

83% match1 repo42024

Copyright Protection in Generative AI: A Technical Perspective

82% match2024

Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AI

81% match2024

Cite this paper

@article{wen2025teaming,
  title  = {Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center},
  author = {James Wen and Sahil Nalawade and Zhiwei Liang and Catherine Bielick and Marisa Ferrara Boston and Alexander Chowdhury and Adele Collin and Luigi De Angelis and Jacob Ellen and Heather Frase and Rodrigo R. Gameiro and Juan Manuel Gutierrez and Pooja Kadam and Murat Keceli and Srikanth Krishnamurthy and Anne Kwok and Yanan Lance Lu and Heather Mattie and Liam G. McCoy and Katherine Miller and Allison C. Morgan and Marlene Louisa Moerig and Trang Nguyen and Alexander Owen-Post and Alex D. Ruiz and Sreekar Reddy Puchala and Soujanya Samineni and Takeshi Tohyama and Varun Ullanat and Carmine Valenza and Camilo Velez and Pengcheng Wang and Anna Wuest and Yuxiang Zhou and Yingde Zhu and Jason M. Johnson and Naomi Lenane and Jennifer Willcox and Francis J. Vitiello and Leo Anthony G. Celi and Renato Umeton},
  year   = {2025},
  eprint = {2506.22523},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2506.22523v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2506.22523.svg)](https://codeofpaper.com/paper/2506.22523)

Discussion