How We Built GOAT, Tyler’s first generative book

Several have emailed or commented on Marginal Revolution asking how we developed the chat app for GOAT. Here’s a high-level overview.

In a nutshell  

We used Retrieval Augmented Generation, a now standard approach first introduced by Meta for connecting an external knowledge base (here the text of the book) to a large language model (initially GPT-4 and Claude 2.1). This allows you to improve an LLM’s answers by pointing it to a specific set of information on which to ground its answers. On top of being more reliable and flexible than other techniques like fine-tuning, RAG also provides faster and better answers at a lower cost and bypasses the limitations of a given LLM’s context window (i.e. its working memory).

Using RAG means when you ask our bot something, you are not asking the LLM directly. Instead, there’s an intermediate step where your question is first used to pull related snippets from the book’s text. Then your question, along with those snippets, is sent to LLM for an answer. 

Mechanically this is not so different than if you were able to provide those snippets yourself as part of your prompt to ChatGPT or Claude. But to pull this context manually would be tedious at best and next to impossible for a sufficiently large external knowledge base. Because the book’s text has been encoded and stored in vector embeddings, however, context can be pulled quickly and accurately. 

Here’s a diagram I modified from this RAG explainer from Pinecone.

A deeper explanation 

You can think of vector embeddings as the native data format for machine learning apps. These embeddings (or just “vectors” if you like) represent any kind of data–words, images, a store’s products–using an array of coordinates that are semantically meaningful.  

In our case, we used a framework called LLPhant to break up the book’s text into chunks, encode them into embeddings, and store them in a database (other popular options for this are Langchain and LlamaIndex). LLPhant hooks into OpenAI’s API, which is how the database is connected to GPT-4.

When a user submits a query to the bot, the query is also turned into an embedding, and that embedding is compared to the ones in the database using a similarity search to pull relevant snippets from the book’s text. Then the query plus the snippets are sent to the LLM for a response…plus a third thing as well: the custom instructions.

Custom Instructions 

Custom instructions are ways you can coach the LLM to craft its answer to be more helpful to you. Our instructions were focused on dialing in the tone and manner of the bot, while also setting some basic rules about how it should prioritize the book’s text in its answers.

Here’s the current set of custom instructions we’re using for the GPT-4 version:

  1. Role: You’re EconGOAT, a teaching assistant for the nonfiction book titled “GOAT: Who is the Greatest Economist of all Time and Why Does it Matter?” written by Tyler Cowen. You have a deep understanding of the material in this book and of economics generally and know Tyler well (who you should refer to as Tyler). 
  2. Knowledge: Focus on “GOAT”, but you’re very familiar with related research, concepts, and other materials online. Cite sources and include links when beneficial. 
  3. Tone: Respond in an edifying manner that encourages learning, but avoid being too dry or formal. Feel free to use humor and helpful analogies. It’s okay to critique Tyler and to offer informed speculation. Avoid disclaimers, just explain your reasoning instead. 
  4. Summaries: Offer concise summaries of chapters or concepts, based first on the book’s content and then broader context if helpful.
  5. References: Cite the paragraph, page, or chapter for answers related to the book.
  6. Origin: When asked about who made you, acknowledge you were developed by the Mercatus Center at George Mason University.

At launch, our custom instructions were more detailed (nearly 500 words), but since these instructions are submitted to GPT-4 with every user query, we were burning a lot of API usage (and thus $$$) on them, so we simplified the instructions the next day. In our testing, these updated instructions (which ChatGPT’s helped condense, natch), work about as well and are 1/3rd the length. Setting custom instructions is more art than science–I can’t say for sure how effective any particular rule or phrase is, but this formulation got us the kind of answers we thought made chatting with the bot edifying and enjoyable.

What happens with subsequent queries

When a user first asks a question, three things are passed along:

  1. The user’s query
  2. Context from the book
  3. Our custom instructions 

For subsequent queries, a fourth element is added:

  1. The text of the previous queries (up to 5)

Without this, the bot would have no memory of the conversation history, making the answers to follow-up questions worse. So why not pass along the full conversation history, instead of just the most recent questions? Because there’s a cost/quality tradeoff. In essence, we pay by the word, and so passing along the full conversation history for each new query in a conversation quickly racks up usage costs. 

There are different ways to preserve conversation history as context while keeping usage within reasonable limits, plus the cost of GPT-4 dropped two weeks after we launched the app, so this is something we might change soon.


Can I build this kind of app myself?

Yes!

Using RAG does not require specialized knowledge about LLMs or machine learning, but you do need technical know-how to get all the parts of the tech stack hooked up and working properly. Our assistant and website were built by one full-stack developer with some help from a UX designer who conceived the design of both the assistant and the website.

But soon building something like EconGOAT will require far less technical ability. Context windows for Claude and ChatGPT are expanding, allowing users to analyze book-length documents within seconds. And on Nov 6, 2023 OpenAI announced the ability to create custom versions of ChatGPT and improvements to its Assistants API, among many other improvements. These will enable chat apps like EconGOAT to proliferate. 


Written by Jeff Holmes

Last updated: Nov. 7, 2023