In this blog, we will describe building an AI based app utilizing multi-agent and Human-In-the-Loop collaboration to perform a task. In this specific case, the task is to create an AI teacher agent to generate a set of questions to test students on a given lesson. However, the concept can be extended to build any app requiring multi agents and human collaboration tasks.

The app uses langgraph to orchestrate the task. langgraph is a library for building stateful, multi-agent applications with LLMs, built on top of LangChain.

Lang Graph

Below diagram shows the graph set up and the flow for this app. Data Source

The flow is started with the dummy StartAgent. Start node is connected to the input node that has the sole responsibility of reading the lesson text. The lesson is added to the state which is passed on to the next connected node. The node is create_test which invokes set_test function of TeacherAgent to generate test questions from LLM. Next is the crtique node which invokes crtique function of CritiqueAgent to generate critiques for the generated test questions. At this stage, the graph state stores LLM generated questions and crtique in addition to the input lesson.

Next in the flow is human-in-the-loop input. This is represented by the connected node human_review which is responsible for receiving user input for revising the AI generated test questions. User has an option to edit AI generated critique, enter own critique or provide no critique. HumanReviewAgent is responsible for updating the state to indicate whether user wants to provide critique or not. Conditional edges allow branching logic for the flow. If the critique was provided, the flow is redirected to create_test node to repeat the process to revise test questions. If no critique was provided and user accepts the generated questions, the flow will be directed to OutputAgent which simply prints output and the flow completes.

Programmatically, Node, Edge and State are building blocks of a Lang Graph.

Nodes

In this app, we use below Nodes: StartAgent - The node to start the flow. InputAgent (input) - The node is responsible for reading lesson text from the file. TeacherAgent (create_test) - The node is responsible for generating or revising test questions for a given lesson using LLM. CritiqueAgent (critique) - The node is responsible for generating critiques for the given test questions. OutputAgent (output) - The node to simply print generated output. HumanReviewAgent (human_review) - The node is responsible for receiving Human-in-the-Loop response from the user.

Edges

Following normal edges are defined:

start -> input

input -> create_test

create_test -> critique

critique -> human_review

In addition, there are conditional edges:

human-review -> output if no critique is present

human-review -> create_test if there is critique

State

To manage state, default TypedDict is used although it is possible to use a Pydantic BaseModel.

Checkpointer & Thread

Using LangGraph’s built-in persistence layer, implemented through a checkpointer, it is possible to save or retreive a checkpoint of the graph state. Checkpointer helps build human-in-the-loop workflow by allowing humans to view, inspect and approve/interrupt steps.

Additionally, it is possible to define a thread with the checkpointer to maintaining separate states as necessary for multi tenant workflow. Essentially, a thread is a unique ID assigned to a series of checkpoints saved by a checkpointer.

In this example, we are using a simple in-memory sqllite to persist graph and a single id for checkpointer to support single workflow at a time.

See the code available in teach_agent.py for implementation details.

Deployment

Source code is available here: https://github.com/sekharkafle/teacher-agent teach_agent.py code can be deployed to any python enabled environment. In our case, we have deployed the code to StreamLit server. Click https://teacher-agent.streamlit.app/ to access the application. You need OpenAI API key to use the application.

teach_st.py has the code needed to deploy to streamlit server.