Class 11: Testing Agent-Driven Queries
Duration: ~15 minutes | Difficulty: Advanced | Prerequisites: Class 10 completed; Java 21 and Maven.
This lesson carries on from Class 10. If you followed along, keep working on the project you already have. If you skipped it, clone the agents_class_10 branch to start from the same place:
git clone --branch agents_class_10 https://github.com/graphqlguy/graphQLBeyondMCP.git
Code blocks are titled with the file they belong to, and the note in brackets says what to do with it. (new file) is a file to create. (edited, whole file) replaces a file you already have. Anything else names the one part to add or replace, as in (edited, the call method). Where a block shows part of a new file, the complete file follows in a collapsed block underneath.
Each earlier lesson repaired a real failure, and each repair left a behaviour that the project now depends on. There are five:
- A translation rule per GraphQL type, so a non-null string becomes a required property and an enum becomes a closed list.
- The drift gate promised in Class 2: a schema change that alters the catalog has to fail a build.
- A denial path, where a refused write returns an explanation that the model can act on, so that it stops retrying.
- A budget ceiling that stops a run when it is crossed.
- The unwrap that repairs the flattened arguments small models send.
The build cannot check any of those behaviours yet, because no test describes them. A future refactor, made for a good reason, could undo any of them while the build stays green. A test records why a behaviour is there. This lesson turns the course's guarantees into eight JUnit assertions across three layers. It ends with a harder question that a deterministic suite cannot answer alone: how do we test the part where a model is involved?
One principle decides how fast these tests run and how far we can trust them: test each layer against the contract of the layer below it, and bring in a model only where you are testing the model's own judgment.
Almost everything this course built is ordinary code around a model:
- generators
- mappers
- callbacks
- budgets
- gates
That code gives the same answer every time, so its tests can too. They run in milliseconds, because each test builds everything it uses. They run without Ollama, the Movie Database service or the network.
Layer One: The Translation Rules
Class 2's GraphQL-to-JSON-Schema mapping is a set of promises to the model. Each promise is now one test, and the test's name states the promise. A failing test prints its name, so the name says what should happen:
The complete JsonSchemaMapperTest.java, collapsed because it runs to 67 lines
package com.graphqlguy.moviedb.agents.toolgen;
import graphql.schema.GraphQLArgument;
import graphql.schema.GraphQLFieldDefinition;
import graphql.schema.GraphQLObjectType;
import graphql.schema.GraphQLSchema;
import graphql.schema.idl.SchemaParser;
import graphql.schema.idl.UnExecutableSchemaGenerator;
import org.junit.jupiter.api.Test;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The translation rules of Class 2, pinned as executable facts. Each test names one
* rule, so a refactor that bends a rule fails with the promise it broke.
* <p>
* Every schema below is a small slice of the Movie Database rather than a made-up
* one, so the rule under test is visible against a field the course actually
* generates a tool from.
*/
class JsonSchemaMapperTest {
private final JsonSchemaMapper mapper = new JsonSchemaMapper();
@Test
void enums_shouldBecomeClosedValueLists() {
Map<String, Object> jsonSchema = argumentSchema("""
type Query { movies(genre: Genre!): [Movie!]! }
enum Genre { DRAMA COMEDY HORROR }
type Movie { id: ID! }
""", "movies", "genre");
assertThat(jsonSchema.get("type")).isEqualTo("string");
assertThat(stringList(jsonSchema, "enum")).containsExactly("DRAMA", "COMEDY", "HORROR");
}
@Test
void nonNull_shouldMoveToTheParentsRequiredList() {
Map<String, Object> jsonSchema = argumentSchema("""
type Query { watchList(input: AddWatchListItemInput!): String }
input AddWatchListItemInput { watchListId: ID! userNotes: String }
""", "watchList", "input");
assertThat(stringList(jsonSchema, "required")).containsExactly("watchListId");
assertThat(jsonSchema.get("additionalProperties")).isEqualTo(false);
}
@Test
void ids_shouldTravelAsStringsAndListsAsArrays() {
Map<String, Object> jsonSchema = argumentSchema(
"type Query { moviesByIds(ids: [ID!]!): String }", "moviesByIds", "ids");
assertThat(jsonSchema.get("type")).isEqualTo("array");
assertThat(itemSchema(jsonSchema).get("type")).isEqualTo("string");
}
/** Parses the SDL, finds one argument of one query field, and maps its type. */
private Map<String, Object> argumentSchema(String sdl, String fieldName, String argumentName) {
GraphQLSchema schema = UnExecutableSchemaGenerator.makeUnExecutableSchema(
new SchemaParser().parse(sdl));
GraphQLObjectType queryType = schema.getQueryType();
GraphQLFieldDefinition field = queryType.getFieldDefinition(fieldName);
GraphQLArgument argument = field.getArgument(argumentName);
return mapper.toJsonSchema(argument.getType());
}
/** JSON Schema keeps its lists untyped, so the cast lives here and nowhere else. */
@SuppressWarnings("unchecked")
private static List<String> stringList(Map<String, Object> jsonSchema, String key) {
return (List<String>) jsonSchema.get(key);
}
/** The schema an array's entries follow, under the "items" key. */
@SuppressWarnings("unchecked")
private static Map<String, Object> itemSchema(Map<String, Object> jsonSchema) {
return (Map<String, Object>) jsonSchema.get("items");
}
}
Each test parses a short piece of SDL, runs the mapper and checks the map that comes back. Copy the naming style: when a refactor breaks a rule, the failure message names that rule. These tests also describe the mapping more precisely than the text of Class 2. The build runs them, so that description stays current.
Layer Two: The Drift Gate
Class 2 promised that schema drift would fail a build before it reaches an agent. This lesson keeps that promise with a golden file.
A golden file is a committed copy of the output that the code under test should produce. The test runs that code, compares the result with the committed copy and fails on any difference. Changing the output then takes a deliberate step: you regenerate the file, and the difference appears in code review like any other change. The technique is old and widely used, under names such as snapshot testing in JavaScript and approval testing in other ecosystems.
Here the output is the generated tool catalog. The golden file holds the JSON of every tool the agent may see, and the rest of this section calls it the golden catalog.
package com.graphqlguy.moviedb.agents.toolgen;
import graphql.schema.GraphQLSchema;
import graphql.schema.idl.SchemaParser;
import graphql.schema.idl.UnExecutableSchemaGenerator;
import org.junit.jupiter.api.Test;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.fail;
/**
* The drift gate Class 2 promised, as a test. The catalog is regenerated from
* the committed schema snapshot and compared byte for byte against the golden
* copy; when the schema (or the generator) changes, this test fails. The
* next step is a deliberate regeneration, reviewed in the same pull request as the
* change that caused it. A mismatch between the schema and the catalog then fails
* the build, where a person sees it before an agent does.
*/
class ToolCatalogGoldenTest {
private static final Path SCHEMA_SNAPSHOT = Path.of("src/main/resources/schema-snapshot.graphqls");
private static final Path ALLOW_LIST = Path.of("src/main/resources/tool-allowlist.txt");
private static final Path GOLDEN_CATALOG = Path.of("src/test/resources/catalog-golden.json");
/** Tools the support role may see, from the allow-list's role suffixes. */
private static final int TOOLS_FOR_SUPPORT = 5;
/** The concierge sees those plus the watch-list read and the one permitted write. */
private static final int TOOLS_FOR_CONCIERGE = 7;
private static final int WRITES_FOR_CONCIERGE = 1;
private final ToolCatalogGenerator generator = new ToolCatalogGenerator();
@Test
void generatedCatalog_shouldMatchTheCommittedGolden() throws Exception {
List<OperationTool> catalog = generateFor(null);
String json = generator.toJson(catalog);
Path golden = GOLDEN_CATALOG;
if (!Files.exists(golden)) {
// First run: write the file and fail, so a person reads the catalog before
// it becomes the thing every later run is compared against.
Files.createDirectories(golden.getParent());
Files.writeString(golden, json);
fail("Golden file created at " + golden + ". Read it, and commit it once"
+ " the catalog it describes is the catalog you meant to ship.");
}
assertThat(json.strip()).isEqualTo(Files.readString(golden).strip());
}
@Test
void roles_shouldNarrowTheCatalog() throws Exception {
assertThat(generateFor("support")).hasSize(TOOLS_FOR_SUPPORT);
assertThat(generateFor("concierge")).hasSize(TOOLS_FOR_CONCIERGE);
assertThat(generateFor("concierge").stream().filter(OperationTool::mutation))
.hasSize(WRITES_FOR_CONCIERGE);
}
/** The catalog as the application builds it: committed snapshot, committed allow-list. */
private List<OperationTool> generateFor(String role) throws Exception {
GraphQLSchema schema = UnExecutableSchemaGenerator.makeUnExecutableSchema(
new SchemaParser().parse(Files.readString(SCHEMA_SNAPSHOT)));
return generator.generate(schema, AllowList.load(ALLOW_LIST), role);
}
}
That test can pass only once the golden file exists. The catalog command already prints exactly what the generator produces, so we generate the 167 lines of JSON with it and then review them:
mvn -q spring-boot:run -Dspring-boot.run.arguments="catalog" \
> src/test/resources/catalog-golden.json
Open the result and read it before you commit it. The reading matters, because the file protects you only if a person has agreed with its contents. The file in git is then a catalog that someone approved. When a change to the schema or the generator should alter the catalog, the same command regenerates it.
The test regenerates the catalog from the committed schema snapshot and compares it with the reviewed golden file. Now each of these changes fails the build:
- a schema edit that alters a tool
- a generator refactor that changes the output
- an allow-list line that someone forgot they added
The repair is always the same deliberate step. Regenerate the golden file, so the reviewer sees the catalog diff next to the change that caused it. That is Class 2's review practice, and now the build enforces it. A second test checks the role counts the same way: five tools for support, seven for concierge, and exactly one mutation among the concierge's tools. So the allow-list's security decisions are assertions too.
Check It Works
mvn clean test
Every test so far runs without the Movie Database service and without Ollama, so the tests can run in CI. Change one line of a description in the schema snapshot and run them again: the golden test fails with a diff that names the tool that changed.
Layer Three: The Safety Controls, Against a Stub Server
The callback's safety controls need careful tests. Each of them came from a failure in Classes 4 through 8, and when one of them breaks in production, nothing reports it. The test double is small on purpose. It is the JDK's own HttpServer, about ten lines, which records the last request body and answers with fixed JSON. That is enough to check what the callback sent over the network. In the first test below, it also shows that the stub's recorded body stayed empty:
@Test
void aDeniedWrite_shouldAnswerTheModelWithoutTouchingTheServer() {
// a gate that denies everything; then:
String result = callback.call("{\"input\":{}}");
assertThat(result).contains("denied").contains("Do not retry");
assertThat(lastBody.get()).isNull(); // the stub server received no request
}
The complete GraphQlToolCallbackTest.java, collapsed because it runs to 107 lines
package com.graphqlguy.moviedb.agents.catalog;
import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.ApprovalGate;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import java.util.concurrent.atomic.AtomicReference;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/**
* The safety behaviors of Class 4, as assertions. The endpoint is a ten-line
* embedded stub recording what actually went over the wire, because the
* behaviors under test live BEFORE and AROUND the network call: the denial
* that returns without any request, the budget that throws an exception on
* the call past its ceiling, and the tolerant unwrap whose repaired shape
* must be what the server receives.
*/
class GraphQlToolCallbackTest {
private HttpServer server;
private String endpoint;
private final AtomicReference<String> lastBody = new AtomicReference<>();
/**
* The one write in the catalog, as the generator would produce it. The last two
* components carry the behaviour under test: {@code mutation} sends it through the
* approval gate, and {@code singleObjectArgument} names the argument the tolerant
* unwrap wraps a flattened payload into.
*/
private static final OperationTool ADD_WATCH_LIST_ITEM = new OperationTool(
"addWatchListItem",
"adds an item to one of the caller's watch lists",
Map.of("type", "object"),
"mutation Tool_addWatchListItem($input: AddWatchListItemInput!)"
+ " { addWatchListItem(input: $input) { id } }",
List.of(), // roles: every role may see it in this test
0, // token count: unused here
true, // mutation, so the approval gate applies
"input"); // the lone input-object argument
@BeforeEach
void startStub() throws Exception {
server = HttpServer.create(new InetSocketAddress(0), 0);
server.createContext("/graphql", exchange -> {
lastBody.set(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
byte[] response = "{\"data\":{}}".getBytes(StandardCharsets.UTF_8);
exchange.getResponseHeaders().set("Content-Type", "application/json");
exchange.sendResponseHeaders(200, response.length);
try (OutputStream out = exchange.getResponseBody()) {
out.write(response);
}
});
server.start();
endpoint = "http://localhost:" + server.getAddress().getPort() + "/graphql";
}
@AfterEach
void stopStub() {
server.stop(0);
}
@Test
void aDeniedWrite_shouldAnswerTheModelWithoutTouchingTheServer() {
ApprovalGate denyEverything = alwaysDeny();
var callback = new GraphQlToolCallback(ADD_WATCH_LIST_ITEM, endpoint,
AuthSession.anonymous(), denyEverything, null);
String result = callback.call("{\"input\":{}}");
assertThat(result).contains("denied").contains("Do not retry");
assertThat(lastBody.get()).isNull(); // the stub server received no request
}
@Test
void budget_shouldThrowAnExceptionOnTheCallPastItsCeiling() {
RunBudget budget = new RunBudget(8, 2);
var callback = new GraphQlToolCallback(ADD_WATCH_LIST_ITEM, endpoint,
AuthSession.anonymous(), null, budget);
callback.call("{\"input\":{}}");
callback.call("{\"input\":{}}");
assertThatThrownBy(() -> callback.call("{\"input\":{}}"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("budget exceeded");
}
/**
* A gate that refuses everything, reading from an empty source: the refusal happens
* before any prompt is answered, so no test ever waits on input.
*/
private static ApprovalGate alwaysDeny() {
return new ApprovalGate(new Scanner("")) {
@Override
public boolean approve(String toolName, String operationDocument, String jsonArguments) {
return false;
}
};
}
@Test
void flattenedSingleInputArguments_shouldBeWrappedBeforeTheWire() {
var callback = new GraphQlToolCallback(ADD_WATCH_LIST_ITEM, endpoint,
AuthSession.anonymous(), null, null);
callback.call("{\"watchListId\":\"2\",\"titleId\":\"5\"}"); // the flat shape small models send
assertThat(lastBody.get())
.contains("\"input\":{") // the wrap happened
.contains("\"watchListId\":\"2\"");
}
}
The other two tests follow the same pattern. The budget test checks that the call past the ceiling throws an exception: two calls pass, and the third fails with "budget exceeded" before a request is built. The unwrap test sends the flat argument shape that small models actually produce. It then checks that the body arriving at the stub has the wrapped "input": {...} form. So it checks Class 4's repair on the request that actually leaves the process.
These tests need only an approval gate, stubbed by overriding one method, and an HTTP server in the same process. So they can run on every commit, and that makes them a real check.
Testing With a Model in the Loop
The three layers above leave the model out on purpose, and that choice has a limit. Some behaviours belong to the model itself:
- Does it read before it writes?
- Does it stop retrying after a denial?
- Does it answer from the tool results?
Classes 4 through 8 caught real failures in exactly those behaviours, by running the agent and watching what it did. Three kinds of test make that watching repeatable.
Replay tests script the model. A stub ChatModel returns replies that we wrote in advance: first one that asks for a tool, then one that answers in text. The assertions are about our own code. They check that the loop:
- executed the tool
- added the result to the conversation
- stayed inside the budget
- stopped
Replay tests run in milliseconds and belong in CI.
Replay tests have a limit that follows from how they work. The replies are scripted, so the test tells us whether our loop handles those replies correctly. It cannot tell us whether a real model still produces replies of that shape, because the test runs without one.
The scripted model is three lines, and the test drives the real AgentRunner:
ChatModel scriptedModel = replying(
asksFor("myWatchLists"),
answers("You have one watch list, To watch this weekend."));
new AgentRunner(scriptedModel, "scripted").run(
"what is on my weekend list", List.of(recordingTool()), budget);
assertThat(toolCallsMade).containsExactly("myWatchLists");
assertThat(budget.summary()).contains("2 model calls");
The assertion expects two model calls, because the first call asked for a tool and the second answered. The second test in the file scripts a model that only asks for tools. It checks that the run stops after three model calls, which is the ceiling given to the budget.
The complete AgentRunnerReplayTest.java, collapsed because it runs to 85 lines
package com.graphqlguy.moviedb.agents.agent;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.DefaultToolDefinition;
import org.springframework.ai.tool.definition.ToolDefinition;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.List;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
/**
* The loop itself, driven by a scripted model. The two replies below are the shape
* a real run produces: one asking for a tool, then one answering in text. Because
* the replies are scripted, this test says whether our loop does the right thing with
* them, and it says nothing about what any model would reply today.
*/
class AgentRunnerReplayTest {
private final List<String> toolCallsMade = new ArrayList<>();
@Test
void loop_shouldExecuteTheToolAndThenReturnTheModelsAnswer() {
ChatModel scriptedModel = replying(
asksFor("myWatchLists"),
answers("You have one watch list, To watch this weekend."));
RunBudget budget = new RunBudget(8, 6);
new AgentRunner(scriptedModel, "scripted").run(
"what is on my weekend list", List.of(recordingTool()), budget);
assertThat(toolCallsMade).containsExactly("myWatchLists");
// Two model calls: the one that asked for the tool, and the one that answered.
// The budget's tool counter stays at zero because counting lives in
// GraphQlToolCallback, and this test's tool is a plain recording double.
assertThat(budget.summary()).contains("2 model calls");
}
@Test
void budget_shouldStopALoopThatKeepsAskingForTools() {
ChatModel neverFinishes = replying(
asksFor("myWatchLists"), asksFor("myWatchLists"), asksFor("myWatchLists"),
asksFor("myWatchLists"), asksFor("myWatchLists"), asksFor("myWatchLists"));
RunBudget budget = new RunBudget(3, 6);
new AgentRunner(neverFinishes, "scripted").run(
"loop forever", List.of(recordingTool()), budget);
// The ceiling is three model calls, so the fourth never happens.
assertThat(toolCallsMade).hasSize(3);
}
/** A model that replies with the given responses in order, one per call. */
private static ChatModel replying(ChatResponse... scripted) {
Deque<ChatResponse> remaining = new ArrayDeque<>(List.of(scripted));
return (Prompt prompt) -> remaining.isEmpty()
? answers("(the script ran out)")
: remaining.removeFirst();
}
private static ChatResponse asksFor(String toolName) {
AssistantMessage.ToolCall call =
new AssistantMessage.ToolCall("call-1", "function", toolName, "{}");
return new ChatResponse(List.of(new Generation(
AssistantMessage.builder().content("").toolCalls(List.of(call)).build())));
}
private static ChatResponse answers(String text) {
return new ChatResponse(List.of(new Generation(new AssistantMessage(text))));
}
/** A tool that records that it ran and returns a fixed result. */
private ToolCallback recordingTool() {
return new ToolCallback() {
@Override
public ToolDefinition getToolDefinition() {
return DefaultToolDefinition.builder()
.name("myWatchLists")
.description("lists the caller's watch lists")
.inputSchema("{\"type\":\"object\",\"properties\":{}}")
.build();
}
@Override
public String call(String toolInput) {
toolCallsMade.add("myWatchLists");
return "{\"data\":{\"myWatchLists\":[{\"id\":\"1\"}]}}";
}
};
}
}
Scenario runs with a live local model are what this course did in every lesson: a fixed task and a real Ollama. The assertions are about outcomes, and they ignore the wording of the answer. Ours checks that the agent read the watch lists before its first write, and that The Matrix is on the list afterwards. The test removes the film from the list before the run, so a film left over from an earlier run cannot make it pass. Each run takes tens of seconds, and runs differ. So put them in a scheduled suite, and run a small subset on each commit. On a local model they cost time and CPU. Against a hosted provider they also cost money, and Class 10's receipt gives the figure per run.
Keeping them out of the ordinary build takes a JUnit tag and four lines of configuration:
@Tag("scenario")
class WatchListScenarioTest {
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<!-- Scenario tests need Ollama and the Movie Database service, and take tens
of seconds each, so the ordinary build leaves them out. The scenario
profile below turns them on: mvn clean test -Pscenario -->
<groups>${test.groups}</groups>
<excludedGroups>${test.excludedGroups}</excludedGroups>
</configuration>
</plugin>
Maven needs both properties to exist before it can resolve them. Their defaults run every test except those tagged scenario:
<!-- Which JUnit tags the build runs. The scenario profile swaps them. -->
<test.groups></test.groups>
<test.excludedGroups>scenario</test.excludedGroups>
A profile swaps the two values, so the tagged tests run only when we ask for them:
<!-- mvn clean test -Pscenario runs the scenario tests, which need Ollama on
11434 and the Movie Database service on 8081, and skips everything else. -->
<profile>
<id>scenario</id>
<properties>
<test.groups>scenario</test.groups>
<test.excludedGroups></test.excludedGroups>
</properties>
</profile>
mvn clean test -Pscenario
When we verified this section, we ran it twice against the same service. The runs took 34 and 36 seconds. Each one called search, myWatchLists and addWatchListItem in that order, and ended with the film on the list. The second run first removed the entry that the first run had added. The ordinary mvn clean test runs the other eleven tests in under two seconds.
The complete WatchListScenarioTest.java, collapsed because it runs to 149 lines
package com.graphqlguy.moviedb.agents.scenario;
import com.graphqlguy.moviedb.agents.agent.AgentRunner;
import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.catalog.GraphQlToolCallback;
import com.graphqlguy.moviedb.agents.safety.ApprovalGate;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import com.graphqlguy.moviedb.agents.toolgen.AllowList;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import com.graphqlguy.moviedb.agents.toolgen.ToolCatalogGenerator;
import graphql.schema.GraphQLSchema;
import graphql.schema.idl.SchemaParser;
import graphql.schema.idl.UnExecutableSchemaGenerator;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
import org.springframework.ai.ollama.OllamaChatModel;
import org.springframework.ai.ollama.api.OllamaApi;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.web.client.RestClient;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import static org.assertj.core.api.Assertions.assertThat;
/**
* One task, a real model, and assertions about what the agent did. The wording of
* its answer is ignored. This is the tier that catches the failures a scripted model
* cannot: the model that writes before it reads, or retries a denied write.
* <p>
* It needs the Movie Database service on 8081 and Ollama on 11434, and it takes
* tens of seconds, so it carries the "scenario" tag and the build leaves it out.
* Run it deliberately:
* <pre>mvn clean test -Pscenario</pre>
*/
@Tag("scenario")
class WatchListScenarioTest {
private static final String MODEL = "qwen3:8b";
private static final String ENDPOINT = "http://localhost:8081/graphql";
/** The seeded account the course logs in as, from application.yaml. */
private static final String USERNAME = "user";
private static final String PASSWORD = "user123";
private static final String LIST = "To watch this weekend";
private static final String FILM = "The Matrix";
@Test
void agent_shouldReadTheListBeforeWritingToIt() throws Exception {
AuthSession session = AuthSession.login(ENDPOINT, USERNAME, PASSWORD);
// An earlier run may have left the film on the list. Removing it first means
// the final assertion can only pass if this run added it.
for (String entryId : filmEntries(session)) {
graphql(session, "mutation($id: ID!) { removeWatchListItem(id: $id) { success } }",
Map.of("id", entryId));
}
assertThat(filmEntries(session)).isEmpty();
RunBudget budget = new RunBudget(8, 6);
ApprovalGate approveEverything = new ApprovalGate(new Scanner("")) {
@Override
public boolean approve(String toolName, String operationDocument, String jsonArguments) {
return true;
}
};
GraphQLSchema schema = UnExecutableSchemaGenerator.makeUnExecutableSchema(new SchemaParser()
.parse(Files.readString(Path.of("src/main/resources/schema-snapshot.graphqls"))));
List<OperationTool> tools = new ToolCatalogGenerator().generate(
schema, AllowList.load(Path.of("src/main/resources/tool-allowlist.txt")), "concierge");
List<String> toolCalls = new ArrayList<>();
List<ToolCallback> callbacks = tools.stream()
.map(tool -> recording(new GraphQlToolCallback(
tool, ENDPOINT, session, approveEverything, budget), toolCalls))
.toList();
OllamaChatModel model = OllamaChatModel.builder()
.ollamaApi(OllamaApi.builder().baseUrl("http://localhost:11434").build())
.options(OllamaChatOptions.builder().model(MODEL).build())
.build();
new AgentRunner(model, MODEL).run(
"add The Matrix to my To watch this weekend watchlist", callbacks, budget);
// The assertions check the outcome. The agent read the watch lists before its
// first write, because the list id comes from that read, and the film is on the list.
assertThat(toolCalls).contains("myWatchLists", "addWatchListItem");
assertThat(toolCalls.indexOf("myWatchLists")).isLessThan(toolCalls.indexOf("addWatchListItem"));
assertThat(filmEntries(session)).hasSize(1);
}
/** The ids of the list's entries for the film, read with the session's token. */
private static List<String> filmEntries(AuthSession session) {
Map<?, ?> response = graphql(session, "{ myWatchLists { name items { id title { title } } } }", Map.of());
List<String> ids = new ArrayList<>();
if (response.get("data") instanceof Map<?, ?> data && data.get("myWatchLists") instanceof List<?> lists) {
for (Object list : lists) {
if (list instanceof Map<?, ?> watchList && LIST.equals(watchList.get("name"))
&& watchList.get("items") instanceof List<?> items) {
for (Object entry : items) {
if (entry instanceof Map<?, ?> item && item.get("title") instanceof Map<?, ?> title
&& FILM.equals(title.get("title"))) {
ids.add(String.valueOf(item.get("id")));
}
}
}
}
}
return ids;
}
private static Map<?, ?> graphql(AuthSession session, String document, Map<String, Object> variables) {
Map<?, ?> response = RestClient.create()
.post()
.uri(ENDPOINT)
.contentType(MediaType.APPLICATION_JSON)
.header(HttpHeaders.AUTHORIZATION, session.bearerOrNull())
.body(Map.of("query", document, "variables", variables))
.retrieve()
.body(Map.class);
assertThat(response).isNotNull();
assertThat(response.get("errors")).describedAs("GraphQL errors").isNull();
return response;
}
/** Wraps a callback so the test can see which tools the model called, in order. */
private static ToolCallback recording(ToolCallback callback, List<String> toolCalls) {
return new ToolCallback() {
@Override
public ToolDefinition getToolDefinition() {
return callback.getToolDefinition();
}
@Override
public String call(String toolInput) {
toolCalls.add(callback.getToolDefinition().name());
return callback.call(toolInput);
}
};
}
}
Property-style checks use the schema's own guarantees, and they cover every tool in the catalog, including tools that nobody has run. This one requires every document in the catalog to validate against the schema it was generated from:
for (OperationTool tool : catalog) {
Document document = new Parser().parseDocument(tool.operationDocument());
List<ValidationError> errors =
new Validator().validateDocument(schema, document, Locale.ENGLISH);
assertThat(errors)
.describedAs("%s sends an invalid document: %s", tool.name(), errors)
.isEmpty();
}
One assertion covers the whole catalog, and it runs without a model. It is also the reason an agent's mistakes stay cheap here. The document was written when the tool was generated. So an agent that picks a tool and fills its typed arguments always sends a document that the server can parse. Each guarantee in the code removes one thing the model could get wrong.
The complete CatalogDocumentsTest.java, collapsed because it runs to 43 lines
package com.graphqlguy.moviedb.agents.toolgen;
import graphql.language.Document;
import graphql.parser.Parser;
import graphql.schema.GraphQLSchema;
import graphql.schema.idl.SchemaParser;
import graphql.schema.idl.UnExecutableSchemaGenerator;
import graphql.validation.ValidationError;
import graphql.validation.Validator;
import org.junit.jupiter.api.Test;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Locale;
import static org.assertj.core.api.Assertions.assertThat;
/**
* A property every tool in the catalog has to hold: the operation it would send is
* valid against the schema it was generated from. One assertion covers every tool,
* including tools nobody has run, and the test runs without a model.
* <p>
* This is the guarantee that makes an agent's mistakes cheap. An agent choosing a
* tool and filling its typed arguments cannot produce a document the server refuses
* to parse, because the document was written at generation time and checked here.
*/
class CatalogDocumentsTest {
private static final Path SCHEMA_SNAPSHOT = Path.of("src/main/resources/schema-snapshot.graphqls");
private static final Path ALLOW_LIST = Path.of("src/main/resources/tool-allowlist.txt");
@Test
void everyGeneratedDocument_shouldValidateAgainstTheSchema() throws Exception {
GraphQLSchema schema = UnExecutableSchemaGenerator.makeUnExecutableSchema(
new SchemaParser().parse(Files.readString(SCHEMA_SNAPSHOT)));
List<OperationTool> catalog = new ToolCatalogGenerator()
.generate(schema, AllowList.load(ALLOW_LIST), null);
assertThat(catalog).isNotEmpty();
for (OperationTool tool : catalog) {
Document document = new Parser().parseDocument(tool.operationDocument());
List<ValidationError> errors =
new Validator().validateDocument(schema, document, Locale.ENGLISH);
assertThat(errors)
.describedAs("%s sends an invalid document: %s", tool.name(), errors)
.isEmpty();
}
}
}
What This Suite Is Defending
This section rests on weaker evidence than the rest of the course. No public survey covers how agent deployments behave in production, so what follows is what we observed while building this course. Check it against your own systems before relying on it.
Read-only assistants over curated catalogs are the easiest kind to get right, because of how they are built. When the allow-list contains only reads, the worst a confused agent can do is waste a call. That is true whatever the model does, and it is why Class 2's default-deny file is valuable, even though it is a plain list. An agent that can write needs the controls from Classes 4 and 6 to be as safe:
- a proposal that a human sees in full before it runs
- a budget that stops the run when it is crossed
- a record of what was proposed and what was approved
Most of the course's design choices come from one practical question: before a run starts, can you say in numbers what it may cost and what it may reach? A passing suite from this lesson and the meters from Class 10 let you answer it.
Summary
Test each layer against the contract of the layer below, and involve a model only where you are testing the model's judgment. Tests that give the same answer every time now check everything this course built with ordinary code:
- The translation rules are named unit tests, which describe the mapping more precisely than the text around them.
- The catalog is pinned to a committed golden file, so drift fails a build and gets resolved in review.
- The safety controls from Classes 4 through 8 are checked against a ten-line stub server, including the case where a denied write leaves the stub empty.
Behaviour that involves the model gets three kinds of test:
- Scripted replays run in CI.
- Scenario runs with a live local model run on a schedule, and Class 10's accounting prices them.
- Property-style checks use the schema's guarantees to reduce what a model can get wrong.
The build now checks every behaviour that this course added after a failure. A change that breaks one of them fails the build before it reaches production.
Further Reading
- AssertJ documentation: the assertions the suite uses, including the collection and string assertions that catalog tests need.
- JUnit 5 User Guide: the platform underneath, including the tagging and conditional-execution features that a scheduled scenario suite uses.
- graphql-java: query validation: how graphql-java validates a document, which the property-style check relies on.
- Spring Boot: testing: slice tests and the other testing tools to use when these tests become part of a real service's suite.
What's Next
The last lesson looks at where agent development may go next. Class 12 is analysis only. It covers:
- the code-execution pattern, which questions tool calling itself
- what that pattern would change about the agent we built
- which parts of our work stay useful whichever way the industry goes