By David Thomas

Jan 13 (Reuters) - Minnesota Attorney General Keith Ellison cannot rely on a misinformation expert whose court filing included made-up citations generated by artificial intelligence, a federal judge ruled in a case involving a "deepfake" parody of Vice President Kamala Harris.

The Friday decision from U.S. District Judge Laura Provinzino in Minnesota federal court stems from an expert declaration Ellison's office submitted in November. Ellison is defending a Minnesota law that bans people from using deepfakes – videos, pictures or audio clips made with AI to look real – to influence an election.

But one of Minnesota's experts in the case, Jeff Hancock, a misinformation specialist and a Stanford University communication professor, used fake article citations generated by AI to support the state's arguments, the court found.

Hancock told the judge he used ChatGPT-4o while drafting his declaration, which likely "hallucinated" two citations he made in his filing, and apologized for the oversight.

Although Provinzino said she does not believe Hancock intentionally cited fake sources generated by AI, it "shatters his credibility with this court," she wrote on Friday.

The judge noted the "irony" that Hancock, "a credentialed expert on the dangers of AI and misinformation, has fallen victim to the siren call of relying too heavily on AI — in a case that revolves around the dangers of AI, no less."

Provinzino said she would exclude Hancock's expert testimony in deciding whether to grant a preliminary injunction blocking the Minnesota deepfakes law, and prohibited Ellison from filing amended testimony from Hancock. Provinzino declined to block the law in a separate Friday order.

Hancock and Ellison's office did not immediately respond to requests for comment.

The law, which was enacted in 2023, is being challenged as unconstitutional by Minnesota Republican state lawmaker Mary Franson and Christopher Kohls, a political satirist who operates under the screenname "Mr Reagan."

Franson and Kohls' lawyers at the Upper Midwest Law Center and the Hamilton Lincoln Law Institute did not immediately respond to requests for comment.

Kohls created a parody video showing the first presidential campaign ad of Harris, a Democrat, with AI-generated narration that sounded like Harris. The video was posted on X by Elon Musk, the social media site's billionaire owner, and reposted by Franson.

Kohls is also challenging the constitutionality of two California laws regulating AI-generated deepfakes about elections and electoral candidates. Those laws are also being challenged by Musk's X Corp and the Babylon Bee, a satirical website.

The case is Christopher Kohls, et al. v. Keith Ellison, et al., U.S. District Court of Minnesota, 0:24-cv-03754

For Christopher Kohls and Mary Franson: Alexandra Howell, Douglas Seaton and James Dickey, of Upper Midwest Law Center, and M. Frank Bednarz, of Hamilton Lincoln Law Institute

For Keith Ellison: Allen Barr, Angela Behrens, Elizabeth Kramer and Peter Farrell, of the Minnesota Attorney General's Office

For Chad Larson: Kristin Nierengarten and Zachary Cronen, of Rupp, Anderson, Squires & Waldspurger

(Reporting by David Thomas)