Make Contract AI 100x Cheaper, Faster, and More Reliable

Transcript

Hello and welcome back. My name is Josh and I am the founder and CEO of Nomio. I am the only person in the world who has spent the last seven years of their life building and maintaining contract databases so that you don’t have to.

Today we’re going to look end-to-end at what happens when you put a contract into Claude and what some of the problems might be and then how to solve them.

So you put a contract into Claude, you ask a question, you get an amazing answer. These models are incredible. You do the same thing with five contracts and you get a similar result.

If you put in 500, things start getting hairy. And the reason is that Claude only has a finite amount of memory and it cannot fit all that many contracts in before it starts to struggle.

So, we can break it out the problem into lots of pieces and solve them independently, combine the answers. But even when we do that, the thing is very expensive, it’s slow, and it’s unreliable.

And a very big reason for this is that we are Asking a question and then having to look through our entire contractual estate, every word of every page of every document to try and figure out the answer.

When in fact, we only care about a small slice of that text for any given question. So, we’re looking at a whole load of irrelevant material every time we ask a question.

That makes it very expensive and slow. We’re paying a huge cost to Anthropic for the privilege. And the unreliability comes as an inherent, unavoidable consequence of using AI, which is that it is a probabilistic machine.

It hallucinates, which means it produces information from thin air that is not correct. And as a result, if you’re working in a field that requires you to have certainty in your answers, you have to go and audit all of these answers yourself.

But the whole reason you’re using a system like Claude is so that you can avoid having to do a whole load of that work yourself.

So it just shows that it shifts a bunch of the manual work from the actual work to validating the work.

Now, if we wanted to solve this, we would need to find a way to take that big square of all of the text of our contracts, and before we give anything to Claude, figure out which parts of those are relevant that little red dot that you see, and then we pass that to Claude, and now Claude has a much smaller, much more manageable packet of information that it can answer your question with, and that is going to be much cheaper, much faster, and much more reliable because when there is a lot less information for it to look at, it is going to make far fewer mistakes with it.

And this is where the spreadsheet becomes really valuable. Spreadsheets get a lot of hate, but actually they have a huge number of advantages.

You can fit the whole thing inside an LLM, the answers are structured already, so that you can share it with other systems really easily, you do the work once, and then you can answer questions however many times you like.

You always know that if you ask the same question, you’re going to get the same answer, and anyone can read it.

But, it’s brittle. We don’t have a link back to where the information came from in the document that contains it.

Which means that, over time, we lose trust. You need to be able to validate that any one of those cells is correct if you’re going to trust the spreadsheet over the long term.

We only have the information that we’ve thought about in advance, so what happens when we have a new question. And then we’ve got a bunch of other problems, but really it comes down to having to put in manual effort to keep the thing up to date when things change.

And, wherever we’ve got manual effort involved, with a complex process, we have key person risk, what happens when they leave the business.

So, alternatively, we can build an AI database that doesn’t involve any human intervention, and can fix a bunch of the problems that we have with a spreadsheet.

So, the way that this works is that we take a contract, and we break it up into a whole load of chunks of text, and then we classify each chunk by its theme, so that, when we have a question, we only need to look at the relevant parts.

This solves pretty much all of the problems that we had with the spreadsheet. But it introduces a whole load of new problems, and those problems revolve around the fact that we’ve delegated all of our control and visibility back to an AI system, which is the reason why it was unreliable to begin with.

What we really want to do is combine both of these solutions together, because collectively, they cover all of these bases.

However, even if we do that, we’re left with three big problems. The first one is one that we inherit from spreadsheets, which is that the moment we have any manual work involved, we take on key person risk.

But there’s a new kind of key person risk here, which is that the moment we’re combining this AI system and the spreadsheet, we’ve got a complex system on our hands that itself needs maintaining and knowledge to work on.

And then we’ve got two new problems that neither of our existing solutions addressed.

The first one is that we have been living in a make-believe world of contracts. We actually have documents, and this complicates everything.

Because a contract is not the same as a document. A contract is made up of one or more documents. But before we even do that, remember that we’re dealing with a file system.

I have never seen a clean file system. First, we have to isolate what’s relevant, and then get rid of anything that isn’t relevant.

And then when we’ve done that, we have to figure out how do all of these documents combine to make up the contracts.

And the contracts are the objects that we want to extract data out of. But that data is only going to make sense when it is considered in the context of the contract as a whole, not just the individual documents.

We also want to make sure that no one sees information that they’re not meant to see, and there is no way to do that with a probabilistic layer.

You must have a hard gate that makes sure that any information that you pass to AI is entirely filtered based on what the person asking the question is able to see.

So, we take all of these problems, and we solve them, and we end up with a contract database. We have something that has, at its core, the documents all correctly organised.

We then have something like a spreadsheet sitting on top, where all of the data is linked back to those individual documents.

And then we have this fallback layer, so that if we can’t answer a question with a spreadsheet, guess what? We can answer it with our clever AI database way of representing all the text in our contracts.

And critically, every single query that we make goes through this hard filter that makes sure that no one can see anything that they’re not meant to see.

So in an ideal world, yes, we would just be able to slap AI on top of a big pile of documents.

But in reality, we are going to have a very expensive, slow, unreliable mess. Unless we insert a layer like this in between your contracts and Claude.

Thank you very much, and I will see you in the next video.