Believe it or not, A.I. tools like ChatGPT can produce incredible looking legal research results in a fraction of the time that it takes a measly human lawyer to complete - just don't rely on the results for accuracy or legal advice.
Written by Joseph Ur, JSU LAW
Pictured above: an A.I.-generated depiction of ChatGPT, trying its best
Since its launch in late 2022, ChatGPT has been a hot topic (and participant!) of conversations the world over, and has renewed many debates about the integration of A.I. in day-to-day life and business.
In many sectors of the economy, including in the legal world, people have been eager to integrate the power behind ChatGPT and similar services into their work or professional practices. Articles with enticing and click-baity titles like ‘Leveraging ChatGPT to save advertising professionals' time’, or ‘Seven ways to use ChatGPT at Work to Boost Your Productivity, Make Your Job Easier, and Save a Ton of Time!” are pervasive on the internet – and why not! ChatGPT is an incredible tool, and it only takes a few minutes of “speaking” to it to realize how powerful and useful a tool it might be.
So, like with any good click-bait, I surrendered to ChatGPT’s allure. I employed ChatGPT to help with some legal research, which seemed like an obvious and effective way to save time in what can be a more tedious area of my practice. I help lots of my lawyer clients with research in many different areas of the law to help them advise their own clients, or to work up a legal argument for their court documents. Any timesaving on that front is very valuable for me and therefore to my clients too. I was sure any bot worth its circuits would be able to handle a quick research request.
However – and this is something I’ve heard anecdotally from many friends and colleagues – what I discovered is that ChatGPT’s output is very hit or miss, and can often be the substantive equivalent of a Potemkin village: it looks great from far away, but once you delve deeper or have some knowledge of the inner workings, you realize it’s just a façade.
"I apologize for my mistake earlier..." - ChatGPT
For example, I asked ChatGPT to review s.320.22 of Criminal Code for me, which deals with aggravating factors for sentencing purposes for various motor vehicle (conveyance) related offences. ChatGPT confidently, and in no time at all, listed off the factors under s.320.22 and suggested I look at three specific case decisions to help me in my research. It even gave me the case citations, a brief description of each case, and how the court applied the aggravating factors in their decisions.
Unbelievable! The output was concise and looked frighteningly similar to some emails I’ve sent to or received from colleagues in response to research requests.
Then I delved deeper.
The list of factors ChatGPT listed under s.320.22 was incomplete. In fact, some of them looked confusingly wrong. Also, the cases it suggested were all related to drug possession.
I politely asked ChatGPT to fix the confusion, and reconsider the same query but with specific references to conveyance offences this time. It told me with an eerie HAL9000-like subtext that it was correct, and rather I was the one who was confused – s.320.22 doesn’t relate conveyances. “I’m sorry Joe, I’m afraid I can’t do that for you.”
I thought this was strange, not only because I was being gaslit by a machine, but also because it doesn’t take much more than a quick Google search to look up and spit out the contents of a statute, especially one as ubiquitous as the Canadian Criminal Code.
Eventually ChatGPT admitted to its errors: “I apologize for my mistake earlier. You are correct that Section 320.22 of the Criminal Code of Canada does relate to aggravating factors for offenses relating to conveyances”. Suitably humbled, ChatGPT then rattled off the correct list of aggravating factors, and provided me with another set of cases to check out.
However, the new aggravating factors it listed were wrong again! Every time I asked for it to recheck its results, ChatGPT went on a loop of apologizing for its mistake, and providing me with a new and consistently incorrect list of factors, like a malfunctioning Westworld host.
ChatGPT's research results were made up out of whole cloth... it was empty content dressed up to look like professional advice.
This broken back and forth continued until I copied and pasted the true list of factors into my query, to which ChatGPT responded with another apology, the correct list of factors, and another set of cases. At this point, I was getting worried about how much handholding my new-hire required… I asked ChatGPT for cases where 3 particular aggravating factors were considered in the court’s decision, and ChatGPT obliged my request with another nice list of cases for my review.
Then I looked into the cases: they didn’t exist.
ChatGPT's research results were made up out of whole cloth. The case citations were either completely fabricated, or they were real but led to completely different cases than ChatGPT had described, entirely unrelated to the area of law I was researching. It was empty content dressed up to look like professional advice. Unfortunately, this was cause for dismissal. I duly fired ChatGPT without notice, did the research myself, and didn’t bill for ChatGPT’s time.
It turns out A.I. is about as good at legal research as it is at drawing a human hand. ChatGPT is great at predicting what legal research results should look like, but beneath its results is a 13-fingered monstrosity.
Don’t get me wrong, I really like the idea of using A.I. and new technological advances to save time and to make my work more efficient. However, I think bots and tools like ChatGPT are a few more beta-tests away from being useful for more nuanced requests. As far as legal work goes, I won’t fear the Skynet-style A.I. apocalypse, or a ‘great A.I. replacement’ of my job just yet, and nor should you if you work in this field.
As for my business - rest assured that if you are one of my clients looking for legal support or advice, I’ll be handling the research work myself for the time being.
Written by Joseph Ur, JSU LAW
The views and content of this blog are provided for informational purposes only, and should not be construed as legal advice.