The new artificial intelligence program ChatGPT came up short last month on the multiple-choice portion of the bar exam.

The free chatbot from OpenAI performed better than predicted, however, earning passing scores on the subjects of evidence and torts. The expectation is that, at the current rate of advancement, it may be able to pass the attorney licensing test someday.

Some law professors are alarmed by ChatGPT’s performance since its November release. The program generates sophisticated, human-like responses based on requests from users

Daniel Linna, Director of Law and Technology at Northwestern University Pritzker School of Law, said most law professors thinking about language-based AI are concerned with students passing off work generated by the chatbot as their own.

Jake Heller, Chief Executive Officer of Casetext, said law schools should encourage students to use ChatGPT and similar tools as a starting point for documents and a way to generate ideas.

“It’s no different than turning to a friend in the law library late and night and saying, ‘Hey, I’m struggling with this idea,’” Heller said. “It’s like using a calculator in math.”

Andrew Perlman, Dean of Suffolk University Law School, said he would like to see first-year legal research and writing classes cover the use of tools like ChatGPT, just as they teach students to conduct research on Westlaw and LexisNexis.

“We’re at a very interesting inflection point,” Perlman said. “It would not surprise me if professionals of the future will be expected to make queries to chatbots and other tools to at least get an initial draft of a document.”

ChatGPT is not yet sophisticated enough to earn a law student a good grade on an assignment without additional work, said Northwestern’s Linna. There are also law-focused AI tools that do a better job on specific tasks, he added.

In their paper on the AI’s performance on the bar exam, Chicago-Kent College of Law professor Daniel Martin Katz and Michigan State University College of Law adjunct Michael Bommarito found that the program got answers on the Multistate Bar Exam correct about only half the time, compared to roughly 68% for human test takers.

Those limitations are not enough to soothe many skeptics. Among them is South Texas College of Law Houston law professor Josh Blackman, who urged professors to rethink take-home exams.

“This technology should strike fear in all academics,” he wrote, noting that ChatGPT produces original text that cannot be identified by existing plagiarism detection software.

Heller predicted that law schools will soon begin to amend their codes of conduct and professors will need to clarify that simply turning in a paper produced by a chatbot is the same as plagiarism. Law professors may begin to ask students to disclose what specific technology tools they used, Perlman added.

You can read more about this story here.

There is still over a month remaining until the February bar exam. And there is time still to get help understanding concepts that you still don’t quite “get“ or how to earn extra points on your essays even when you don’t know the applicable law. We can still provide this assistance to get you on track to pass before it’s too late. You can learn more about our famed bar exam tutoring program here.