AI Agreement Comparison Prototype

The following is a brief I wrote to stakeholders within DocuSign to prove out the effectiveness of Large Language Models (LLM) in common customer pain points. This type of rapid prototyping and experimentation plays a large part in how I approach a problem. Based off the work in this brief, this project was given further funding, and is in active development.

"DocuSign launches AI solution to effortlessly track agreements as they move across multiple hands"

-PR Headline

Recap of the Current Situation 
DocuSign customers have reported a frequent pain-point that delays time to agreement completion and degrades the overall signing experience. This pain-point is most commonly seen in B2B sales situations where the signing party requires that the agreement be signed from their own account for compliance purposes.

To give an example, imagine an enterprise organization (in this example we’ll call them Tally Inc.) hires a marketing agency (Digital Solutions) to redesign their corporate website. Tally and Digital Solutions agree on the scope of work, deadlines, and the price. However, before Tally can sign the agreement, they ask Digital Solutions to hand over a copy of the final agreement, then upload this agreement into Tally’s DocuSign account to be signed by all parties. While this ensures that Tally’s agreement processes are followed, it requires Digital Solutions to do due diligence that the agreement matches the version that they originally sent over.

In talking to several customers that experienced this problem, we learned that the current solutions to compare the Docusign document to the original are laborious and error prone. As one customer put it, they perform what they call a “stare and compare” against both documents, manually looking at each line of both documents to make sure that it matches the original version. There are also software solutions that are sometimes used, like Microsoft Word’s Compare docs feature, but these tools have several drawbacks. Word’s Comparison Feature was largely designed to do a word by word comparison, and struggles with formatted data (i.e tables) and requires several steps to configure the comparison of both documents. 

Large language models: a magic wand solution
With the recent advances in Large Language Models (LLM), I was curious if this problem could be solved with AI. It turns out that LLM are well-suited for doc comparison tasks, as they are adept at detecting changes, while also contextualizing to the user what has changed. What makes LLM incredibly different from current doc comparison tools is that LLM encodes text into “tokens”, or numerical strings that represent words, punctuation, and even special characters. These tokens have been trained through machine learning and tuned to “teach” the model to understand the relationship between tokens in a meaningful way. This results in a significant revolution from a tool that does word by word comparison, to a tool that compares the meaning of agreements.

To test how well a LLM could perform a doc comparison scenario, I created a Zapier automation tool that took the inputs of two sample Statement of Work contracts. I then prompted OpenAI’s ChatGPT to compare both documents and note where the documents differ. Both samples were identical, except for a few very minor details that I had changed, such as a different deadline, removing a few words from a clause, or adding an additional clause. 

In all of the testing, ChatGPT detected all of my changes, and correctly summarized the difference. To take it even further, I asked ChatGPT to grade the perceived severity of the change, which could give greater confidence to a User that the system is helping them catch errors.

Try it out:  https://agreementtest.zapier.app/