A California attorney must pay a $10,000 fine for filing a state court appeal full of fake quotations generated by the artificial intelligence tool ChatGPT.

The fine appears to be the largest issued over AI fabrications by a California court and came with a blistering opinion stating that 21 of 23 quotes from cases cited in the attorney’s opening brief were made up. It also noted that numerous out-of-state and federal courts have confronted attorneys for citing fake legal authority.

“We therefore publish this opinion as a warning,” it continued. “Simply stated, no brief, pleading, motion, or any other paper filed in any court should contain any citations— whether provided by generative AI or any other source—that the attorney responsible for submitting the pleading has not personally read and verified.”

The opinion, issued 10 days ago in California’s 2nd District Court of Appeal, is a clear example of why the state’s legal authorities are scrambling to regulate the use of AI in the judiciary. The state’s Judicial Council two weeks ago issued guidelines requiring judges and court staff to either ban generative AI or adopt a generative AI use policy by Dec. 15. Meanwhile, the California Bar Association is considering whether to strengthen its code of conduct to account for various forms of AI following a request by the California Supreme Court last month.

The Los Angeles-area attorney fined last week, Amir Mostafavi, told the court that he did not read text generated by the AI model before submitting the appeal in July 2023, months after OpenAI marketed ChatGPT as capable of passing the bar exam. A three-judge panel fined him for filing a frivolous appeal, violating court rules, citing fake cases, and wasting the court’s time and the taxpayers money, according to the opinion.

Mostafavi told CalMatters he wrote the appeal and then used ChatGPT to try and improve it. He said that he didn’t know it would add case citations or make things up. 

He thinks it is unrealistic to expect lawyers to stop using AI. It’s become an important tool just as online databases largely replaced law libraries and, until AI systems stop hallucinating fake information, he suggests lawyers who use AI to proceed with caution.

“In the meantime we’re going to have some victims, we’re going to have some damages, we’re going to have some wreckages,” he said. “I hope this example will help others not fall into the hole. I’m paying the price.”

The fine issued to Mostafavi is the most costly penalty issued to an attorney by a California state court and one of the highest fines ever issued over attorney use of  AI, according to Damien Charlotin, who teaches a class on AI and the law at a business school in Paris. He tracks instances of attorneys citing fake cases, primarily in Australia, Canada, the United States, and the United Kingdom.

In a widely-publicized case in May, a U.S. district court judge in California ordered two law firms to pay $31,100 in fees to defense counsel and the court for costs associated with using “bogus AI-generated research.” In that ruling, the judge described feeling misled, said they almost cited fake material in a judicial order and said “Strong deterrence is needed to make sure that attorneys don’t succumb to this easy shortcut.”

Charlotin thinks courts and the public should expect to see an exponential rise in these cases in the future. When he started tracking court filings involving AI and fake cases earlier this year, he encountered a few cases a month. Now he sees a few cases a day. Large language models confidently state falsehoods as facts, particularly when there are no supporting facts.

“The harder your legal argument is to make, the more the model will tend to hallucinate, because they will try to please you,” he said. “That’s where the confirmation bias kicks in.”

A May 2024 analysis by Stanford University’s RegLab found that although three out of four lawyers plan to use generative AI in their practice, some forms of AI generate hallucinations in one out of three queries. Detecting fake material cited in legal filings could get harder as models grow in size.

Another tracker of cases where lawyers cite nonexistent legal authority due to use of AI identifies 52 such cases in California and more than 600 nationwide. That amount is expected to increase in the near future because AI innovation is outpacing the education of attorneys, said Nicholas Sanctis, a law student at Capital University Law School in Ohio.

Jenny Wondracek, who leads the tracker project, said she expects this trend to get worse because she still regularly encounters lawyers who don’t know that AI makes things up or believe that legal tech tools can eliminate all fake or false material generated by language models. 

“I think we’d see a reduction if (lawyers) just understood the basics of the technology,” she said.

Like Charlotin, she suspects there are more instances of made up cases generated by AI in state court filings than in federal courts, but a lack of standard filing methods makes it difficult to verify that. She said she encounters fake cases most often among overburdened attorneys or people who choose to represent themselves in family court.

She suspects the number of arguments filed by attorneys that use AI and cite fake cases will continue to go up, but added that not just attorneys engage in the practice. In recent weeks, she’s documented three instances of judges citing fake legal authority in their decisions.

As California considers how to treat generative AI and fake case citations, Wondracek said they can consider approaches taken by other states, such as temporary suspensions, requiring attorneys who get caught to take courses to better understand how to ethically use AI, or requiring them to teach law students how they can avoid making the same mistake.

Mark McKenna, codirector of the UCLA Institute of Technology, Law & Policy praised fines like the one against Mostafavi as punishing lawyers for “an abdication of your responsibility as a party representing someone.” He thinks the problem “will get worse before it gets better,” because there’s been a rush among law schools and private firms to adopt AI without thinking through the appropriate way to use them.

UCLA School of Law professor Andrew Selbst agrees, pointing out that clerks that work for judges are recent law school graduates, and students are getting bombarded with the message that they must use AI or get left behind. Educators and other professionals report feeling similar pressures.

“This is getting shoved down all our throats,” he said. “It’s being pushed in firms and schools and a lot of places and we have not yet grappled with the consequences of that.”

For the record: The fine issued to Mostafavi was for $10,000. Due to an editing error, an earlier version of this article had an incorrect figure.