Do search algorithmic differences, particularly in Westlaw and LexisNexis, require a law firm to maintain both platforms for legal information needs?
We were recently asked about the impact of search algorithms on research results by a client who is considering aligning with one provider for most of the firm’s legal research needs. It’s a procurement strategy more library and information center directors have been considering.
For this particular client, the issue stemmed from Susan Nevelow Mart’s ABA article titled, “Results may vary in legal research databases.” The premise of this article is conveyed in that title, but after consulting with various legal researchers within LAC Group, we believe the takeaway is that lazy, incomplete searches could bury some relevant information, with the potential for some dire consequences.
What is a search algorithm?
According to Technopedia, a search algorithm is “the step-by-step procedure used to locate specific data among a collection of data. It is considered a fundamental procedure in computing. In computer science, when searching for data, the difference between a fast application and a slower one often lies in the use of the proper search algorithm.”
In advanced intelligent search technologies, a programming technique called informed search is used. Informed search strategies exploit problem-specific knowledge as much as possible. An excellent overview of AI search, including informed search, is on the wiki books website.
Natural language and algorithmic search deficiencies
Algorithms are not going to do the heavy lifting in legal research…at least not yet. In a nutshell: Attorneys and paralegals performing legal research need to be cognizant of the deficiencies in natural language and algorithmic search programs; but these deficiencies wouldn’t necessarily make a difference if the firm had access to only one legal information service.
One LAC researcher stated his concerns about the impact of algorithms on different depths of searches:
“I am troubled about this issue. From my own research experience, I find differences that are very fact and case specific. A shallow search might be similar, but the deeper, more involved research queries can result in many differences.”
The question got us thinking about how important this issue might be for other firms. We were also intrigued to find out if any clients are interested in legal research services alignment, which providers they are considering for elimination and why. We decided the search algorithmic conundrum was a perfect opportunity to reach out to them through a survey of questions.
Legal research algorithm survey
We conducted a brief survey the end of March 2018 on research platforms and search accuracy, inviting 100 clients to participate. Clients represented in the survey were law firms of various sizes.
Respondents included librarians (26%), researchers (7%), library managers (52%), and other professionals identified as directors or higher (15%).
The participation rate was above average for an external survey with 28% responding, providing feedback on the following topics:
- Legal research platforms they support and why.
- The importance of algorithmic accountability in choosing to keep a platform.
- How frequently errors or omissions are found and how they were identified.
- Whether case law search results are monitored on a regular basis and how.
- Which platform the respondent believes presents the most complete results.
- The platform they would eliminate in the future, if necessary.
Survey highlights
The majority of respondents use more than one information database, primarily Westlaw, Lexis and BLAW, with a very small minority also using FastCase. Our findings were similar to the ALM’s Survey of Law Firm Management, Library and Research Professionals (2017), which listed similar tools in use.
According to Kathy Skinner and Oz Benamram in their white paper, Are Your Vendors Captive? How to Optimize your Research Dollars, 25 to 35 percent of Am Law 200 firms rely on a sole legal information provider.
Which information provider would firms be most likely to eliminate?
Our findings reveal that a larger proportion would choose Westlaw as the potential elimination candidate, as opposed to LexisNexis. We assume this shift may have to with Thomson Reuters’ persistence in placing their clients into one of their predetermined plans—not exactly “one size fits all” but also not the flexibility most organizations prefer and desire.
As to the reasons for supporting more than one legal research platform, responses were pretty evenly split between more accurate, comprehensive results and the need to support user preferences. Yet it does add time and expense. As one of the legal researchers on LAC’s own virtual research team stated while reviewing the survey results,
“One thing about having access to two legal research services is there is often an expectation to do the research in both platforms. The amount of time needed to compare and de‐dupe results incurs additional costs in both billable time and online expenses.”
But most firms believe that the value of having more than one information resource outweighs the added cost.
Should algorithms be held accountable?
As to the importance of algorithmic accountability, the majority of respondents believe it’s very important or somewhat important. Weighing in on the different research outcomes that can be derived from different sources, one of our experienced LAC legal researchers had this to say:
“It’s not just deep versus shallow searching, but also how much the database is leaning into natural language versus traditional Boolean searching. How searches are crafted, which attorney you happen to work with and other factors will all affect the results.”
While search algorithmic accountability and budget considerations matter, neither one is so momentous as to dissuade firms into downsizing or adjusting their legal research portfolios.
As we examine the legal community’s reliance on search algorithms and machine learning, we’ll assert our expertise to both challenge and correct the technologies supporting them.
Additional information pertaining to search algorithms:
Robyn Rebollo
Latest posts by Robyn Rebollo (see all)
- New options for electronic information resources - December 10, 2019
- Mapping information resources to user needs - October 23, 2019
- Essential resources for M&A due diligence - August 5, 2019
