If your financial service company's artificial intelligence tools are not litigation-ready, discovery in a lawsuit attacking those tools can become a substantial burden. One, your company could suffer enormous distractions and decreased productivity. Two, you could be subjected to onerous court orders that interfere with your ability to conduct your core business. And three, you could even suffer adverse judgment on claims that lack merit.

Preparing for disputes over machine learning tools is a crucial consideration for any financial services company that uses these tools. This is especially true because these tools often support important decisions on:

  • Whether to extend credit to potential borrowers;
  • Whether to flag a transaction as a money-laundering or fraud risk; and
  • How to balance a portfolio.

As every decision maker knows, you can't please all the people all the time. That means litigation is a matter of when—not if. And an explainable and defensible tool is easier to defend than a black box.

Machine Learning Tools and Discovery

Once litigation starts, it may well reach discovery. At that point, a plaintiff is likely to seek production of sensitive information residing in the machine learning tool. Generally, this information falls into two categories: inputs and outputs. Each presents its own difficulties.


Plaintiffs may request training sets used to develop the tool's logic. As a practical matter, these training sets may well be too large to fit on common storage media. And general business-grade cloud storage solutions lack necessary security and access-logging features. Furthermore, many inputs into machine learning tools will contain third-party confidential information or personal data. Plaintiffs may not understand the liability risk they assume if they become responsible for protecting this data from disclosure. Tracking third-party confidential information and personal data from the beginning of the project helps mitigate these difficulties.

Plaintiffs—and, eventually, the judge and jury—may also misjudge the difficulty of using inputs from a machine learning tool. Each tool will have its own data format and the tool itself is likely to evolve rapidly. Designing for explainability and defensibility can help mitigate this risk. An explainable tool with explainable inputs isn't a black box.


Finally, many machine learning tools create huge and complex outputs. For a machine learning tool, the old rule “garbage in, garbage out” becomes “a giant data set in, a giant output out.” A complex dataset subjected to complex processing creates a complex output.

Beyond complexity, there are also practical concerns driven by scale. A large output set will often take days or weeks to search. Reporting may take even longer. Plaintiffs like to assume that if a computer is involved, you simply press the button and get an output. That's rarely the case for machine learning tools.

Finally, machine tool outputs suffer from the “panda problem.” Pandas thrive in their native habitat but do poorly in other environments. Like pandas, machine learning outputs may be usable and comprehensible when combined with the tools that created them but tend to be burdensome to remove from that environment. For example, the AI system outputs may be stored in deep storage and would thus need to be moved to fast storage before they can be searched. And AI outputs may be proprietary formats that are difficult to translate into common formats like Excel or comma-separated values. And the tools for searching the outputs may well be unique, in-house solutions developed as part of the machine learning project. Again, designing for explainability and defensibility can mitigate these risks.

What Is to Be Done: Design for Explainability and Defensibility

A key step toward reducing the litigation risk of machine learning tools is to design for explainability. In other words, the model owner needs to be able to explain how the tool works. Explainability is often described as having three core technical aspects:

  • First, the tool should be transparent. It should be easy to identify the key factors in the tool's logic.
  • Second, the tool should be interpretable. It should be easy to track the weights assigned to the key factors.
  • Third, the data inputs for the tool should have clear provenance. It should be easy to identify the source of the data and assess its reliability.

A tool that is transparent, interpretable and has clear data provenance will be easier to explain to regulators, a judge and a jury than one that lacks these factors.

But there's still more a company might do. To move beyond simple explainability to defensibility, a company might, for example, identify subject matter experts (”SMEs”) skilled at addressing non-specialist audiences. These SMEs can serve as internal and external spokespeople and advocates for the tool. And, in the event of litigation, they can put their subject matter expertise to use to defend the tool. In the end, judges and juries rely on witnesses, and a machine learning tool can't testify for itself. A good spokesperson, though, can speak for it.