Episode 027 Is it Safe to Go to the Rave?

A lawyer running a massive document production doesn't read every page anymore — she can't, and nobody expects her to. So when does that stop being lawyering and start being blind faith? Jim Casey, Chief Privacy Officer at Akamai Technologies, joins Ron and Heather to argue there's a real answer, one that depends entirely on whether a bad AI output can still be caught before someone acts on it. When is an AI mistake actually reversible, and when have you already handed your judgment to the machine?

In this episode:

• Jim Casey's framework for deciding when a lawyer can trust an AI-generated answer, drawn from his two published pieces On Knowledge in An Agentic World and Agentic Workflows in Compliance

• The "Wilma Simpson" hypothetical — a bankruptcy lawyer using AI to summarize a 180-page court order, and where the real risk actually sits

• Ron and Jim's sharp disagreement over whether a "reversible" AI mistake stays reversible once a client acts on the lawyer's advice built from it

• Why reviewing a massive document production (sorting, summarizing, deduplicating) shifts trust from individual verification to system-level design

• An e-commerce company's real-world case study training a model on product-safety complaint data — six months of retraining to reach 90–95% accuracy

• The malpractice waiver hypothetical: what an in-house lawyer does when outside counsel wants to disclaim liability for AI-assisted work

• Client consent and data privacy limits on using AI to process a client's own personal information

• The difference between "human in the loop," "human on the loop," and "human out of the loop" — a framework borrowed from autonomous weapons policy

• "Compliance theater" — why an audit log nobody actually reviews isn't a safeguard

• The "non-delegable duty to think" — what AI can never take off a lawyer's plate

We also discuss:

• The Flintstones/Simpsons/Jetsons framework applied to matching lawyers with the right clients

• A Reddit-sourced Practice Signals case: a kind, hardworking associate who struggles with vocabulary and comprehension — does AI help, or just hide the problem?

• Chris Ryan's AI training tool for new associates, Benchsim.ai, now SOC 2 compliant and video-capable

• The Season 3 shift from Flintstones/Simpsons/Jetsons to a new "Users and Builders" framework

• First tips for builders and users: "make failure visible" and "ask and ask again"

Key Takeaway

Jim Casey's real point isn't that lawyers can or can't trust AI — it's that trust isn't one setting you flip on. The question changes at every step in the chain: is this output feeding the lawyer's own thinking, or is it about to become the advice a client acts on? Reversibility disappears the moment a lawyer hands that answer downstream. Everything else — audit logs, compliance sign-off, vendor demos — is theater unless someone is actually positioned to catch the mistake before it moves.

This matters most for Simpsons lawyers who've adopted a tool but haven't thought through where accountability actually sits in their own workflow. Flintstones lawyers aren't there yet, and Jetsons lawyers like Jim are already building the guardrails. But the solo or small firm lawyer running document review or drafting client advice off an AI summary needs a concrete answer to Jim's closing question: what do you do with the confidently wrong answer?

Mentioned in This Episode

• Jim Casey, VP & Chief Privacy Officer, Akamai Technologies

• Jim Casey's articles: Knowledge in an Agentic World and Agentic Workflows: A New Frontier for Compliance

• ChatGPT

• Claude

• Harvey (AI legal tool)

• Legora (referenced as an enterprise legal AI tool)

• Chris Ryan and his AI training tool for new lawyers (previously featured on the show)

• Reddit (source of this episode's Practice Signals story)

Highlight Reels on YouTube and Instagram

[email protected]