Production Operations (6 Questions)

Duration: ~15 minutes | Difficulty: Advanced | Prerequisites: Federation (6 Questions)

Production operations questions are where the interview shifts from "do you know GraphQL?" to "have you operated GraphQL in production?" The questions in this chapter test your familiarity with observability, deployment processes, and the operational thinking that distinguishes a senior engineer from someone who can build features and has not yet been on call for them.

Question 1: How do you set up tracing for a GraphQL API?

Short answer: In Spring GraphQL, enable Micrometer Observation support and add a tracing backend (such as Zipkin or OpenTelemetry). Spring GraphQL automatically instruments the GraphQL execution lifecycle, including per-resolver timing, when Micrometer Observation is active. Use GraphQlTags to propagate GraphQL-specific attributes (operation name, error count) into spans.

Why this is right / what interviewers want to hear:

Tracing in a GraphQL API has two dimensions that REST tracing does not: per-field tracing and the "partial success" case.

Per-field tracing: a single GraphQL request dispatches many resolvers. A trace that only shows the total request duration is insufficient, you cannot tell which resolver is the slow one. Spring GraphQL's integration with Micrometer creates a span for each resolver invocation, nested under the root request span. The hierarchy of spans mirrors the hierarchy of query execution. The root span represents the HTTP request, and child spans represent individual resolver calls. Their durations add up to the total request time, and they often overlap, because resolvers run in parallel.

The configuration in Spring Boot takes three dependencies:

spring-boot-starter-actuator;
micrometer-tracing-bridge-otel, for OpenTelemetry;
a reporter bridge, either zipkin-reporter-brave for Zipkin or the OTLP exporter for Jaeger and the OTEL Collector.

Set management.tracing.sampling.probability=0.1 for 10% sampling in production, because sampling everything adds too much overhead. Spring GraphQL instruments the execution automatically when Micrometer is on the classpath.

The partial success case: a GraphQL response can be a 200 HTTP status with errors in the errors array. A tracing setup that only checks HTTP status codes will not detect these partial failures. The trace configuration should capture the presence of the errors array and add an attribute to the span (e.g., graphql.error.count) so that traces with partial failures are discoverable in your tracing UI. Without this, field-level errors are invisible to your monitoring system.

Propagation: in a federated graph, the trace context must be propagated from the router to each subgraph. The router adds W3C traceparent headers (or Zipkin B3 headers, depending on configuration), and each subgraph's Spring application picks up the trace context and continues the trace. This gives you end-to-end visibility across subgraphs, showing which subgraph's resolver is the source of latency.

Question 2: What GraphQL-specific metrics should you monitor in production?

Short answer: Monitor operation latency by operation name (p50, p99), error rate by operation name, resolver latency by resolver type, and, critically, query depth and complexity distributions to detect abnormal query shapes. Also monitor DataLoader hit rate and batch size as leading indicators of N+1 regressions.

Why this is right / what interviewers want to hear:

GraphQL metrics need finer granularity than REST metrics, because one endpoint handles every operation. Aggregating them all into a single "POST /graphql" latency histogram hides which operation is slow. The signal appears only once the numbers are split per operation.

Operation name metrics: Spring for GraphQL's graphql.request observation, with an operation tag, gives you a histogram per operation name. Your Movies operation and your SearchMovies operation have different expected latencies and different error rates. Treating them as the same metric means your alerting is a union of their behaviors, often masking a performance regression in one operation behind the normal behavior of the other.

Error rate by operation: the GraphQL errors array makes error monitoring more nuanced. A 200 HTTP response with errors in the payload is still an error for your SLO. Your monitoring system must parse the response or use instrumentation that exposes graphql.errors.count as a metric. Alerting on HTTP error rate alone will miss partial failures.

Resolver latency: per-resolver timing histograms are your tool for proactive N+1 detection and for finding which fields dominate request cost. A spike in Movie.director resolver duration is a signal before it becomes a user-visible latency spike. Spring for GraphQL's Micrometer integration exposes a graphql.datafetcher observation tagged with the field coordinates, and a graphql.dataloader observation for batch loading.

Query shape metrics: depth and complexity distributions tell you whether your query defenses are calibrated correctly. If your depth limit is 15 and your p99 query depth in production is 8, you have headroom. If production queries regularly hit depth 14, your limit is barely above the normal operating range and needs to be reconsidered (either relax it or investigate why queries are so deep). Tracking these distributions is a prerequisite to setting limits that mean anything.

DataLoader metrics: track the batch size distribution for each DataLoader type. A DataLoader that regularly produces batches of size 1 is behaving like the N+1 pattern it was meant to fix, something is calling resolvers outside the expected async batching context. Tracking batch size catches this before it shows up in latency.

Question 3: How do you handle schema evolution in a CI pipeline?

Short answer: Integrate a schema compatibility check (GraphQL Inspector, Rover CLI with Apollo contracts, or a custom tool) that runs on every pull request. The check compares the proposed schema to the current production schema and fails the build on breaking changes. Combine this with operation checks that scan registered client queries against the new schema to catch field removals that no usage data would otherwise surface.

Why this is right / what interviewers want to hear:

Schema evolution in CI is about catching breaking changes before they reach production, where fixing them requires either a rollback or an emergency deprecation-and-migration cycle.

The toolchain: GraphQL Inspector (graphql-inspector diff) takes two schema files and outputs a diff annotated with which changes are breaking, dangerous (potentially breaking), or safe. Rover CLI (from Apollo) does the same within the context of a registered schema registry and adds operation checks. The choice between them depends on whether you are invested in the Apollo ecosystem.

Breaking change detection: the CI check should fail the build on breaking changes. A breaking change (removed field, changed type) that reaches production will break clients immediately, without warning. The CI gate is the last defense. The PR author is told: "your change removes Movie.releaseYear, this is breaking. Deprecate it instead."

Operation checks: breaking change detection on the schema alone catches removals and type changes. It does not catch the case where a field is being deprecated with a plan to remove it after migration, and the migration is incomplete. Operation checks solve this: the check scans a registry of client query documents (populated by Apollo GraphOS, GraphQL Hive, or a build-time extraction of query files from client repositories) and reports which registered operations use the field being removed. If any operations still use Movie.releaseYear, the removal is blocked until those operations are updated.

In a federated graph: schema evolution checks must run per subgraph and at the composition level. A subgraph change that is safe in isolation may break the composed supergraph (see the federation breaking changes question). The Rover CLI rover subgraph check command handles this by checking the subgraph change against the composed supergraph, not just the subgraph schema in isolation.

Question 4: What is the difference between a depth limit, a complexity limit, and a timeout, and do you need all three?

Short answer: A depth limit rejects queries that nest too deep. A complexity limit assigns a cost to each field and rejects queries whose total cost exceeds a budget. A timeout stops execution of queries that are taking too long regardless of their declared cost. You need all three, because each catches a different class of attack or misbehavior that the other two miss.

Why this is right / what interviewers want to hear:

This is a layered defenses question, and the answer demonstrates understanding of why the defenses are orthogonal instead of redundant.

Depth limits catch queries that exploit recursive or deeply nested relationships to create exponential resolver fan-out. They operate on the query document's AST, no resolver is called, because the check happens during the validation phase. Depth limits are fast and cheap.

Complexity limits catch queries that are shallow but wide: a query that asks for 50 top-level fields, each of which is a list with subfields, can have a very low depth but a very high execution cost. Complexity is assigned per field (either with a simple count of fields, or with a cost function that weights lists more heavily than scalars), and the total is compared to a budget. Complexity limits are configurable and require careful calibration, too low and they reject legitimate queries, too high and they allow expensive queries. graphql-java provides MaxQueryComplexityInstrumentation, which Spring GraphQL applications register as a Spring bean to enable complexity limiting.

Timeouts catch queries that pass the depth and complexity checks and still run long. The cause may be a slow external dependency, a deadlock, an unexpected data shape, or any runtime behavior static analysis of the document cannot predict. The timeout is the safety net. However cheap a query looks, once it runs past N seconds the server returns an error and releases the resources. Spring GraphQL supports query timeouts via reactive timeout operators (Project Reactor's Mono.timeout()) on the subscription response.

Why you need all three: the three controls catch different things, and each one is blind to what the others see. A malicious query might be 3 levels deep (passes depth check), 100 complexity points (passes complexity check), but triggers a resolver that calls an external API that takes 60 seconds to respond. The timeout catches it. Conversely, a query might be fast per field but have so many aliases at depth 3 that it takes 5 seconds, neither depth nor timeout catches it until the alias amplification limit (a fourth, separate defense) is in place. The defenses cover different aspects of the attack surface and belong together.

Control	Runs	Catches	Misses
Depth limit	During validation, before any resolver	Deeply nested and recursive queries	The shallow query that asks for fifty expensive list fields
Complexity limit	During validation, before any resolver	Wide and expensive query shapes, weighted per field	A cheap-looking query whose resolver calls a slow dependency
Timeout	During execution	Anything still running when the clock runs out	Nothing about cost before the work starts, so it is the last line and not the first

Question 5: How do you do a zero-downtime schema migration when you need to rename a field?

Short answer: Rename via a three-phase process. Phase 1: add the new field name alongside the old one, both returning the same data. Phase 2: deprecate the old field, notify clients, and monitor usage. Phase 3: after usage reaches zero (confirmed by analytics), remove the old field. Each phase is a separate deployment.

Why this is right / what interviewers want to hear:

Field renaming in GraphQL lacks an in-place migration path. Unlike a REST endpoint where you can redirect /movies/releaseYear to /movies/year, a GraphQL field name is part of the schema contract and no HTTP-level redirect equivalent exists. The client must update its query to use the new field name; the server cannot transparently translate between old and new.

Phase 1, add the new name: implement both Movie.releaseYear and Movie.year, both resolving to the same value. The old field continues to work for all existing clients. New clients and new code should use the new field name. Both fields coexist in the schema.

Phase 2, deprecate the old name: add @deprecated(reason: "Use year instead") to Movie.releaseYear. Deploy this change. Enable field-level usage tracking (Apollo GraphOS field usage, GraphQL Hive usage reporting, or a custom Micrometer tag on the releaseYear resolver that counts invocations). Notify all teams that use releaseYear about the deprecation and the timeline for removal. In a partner-facing API, publish a deprecation notice and honor a generous migration window (GitLab uses six minor releases; Shopify uses a calendar-based window).

Phase 3, remove the old name: remove the field once usage monitoring has shown zero requests to releaseYear for a long enough observation window. With mobile clients that update rarely, that means at least one release cycle. This is the only deployment that is a breaking change, and by this point no active client should break.

The observation window is critical and often underestimated. Web clients update frequently. Mobile clients do not. If you remove a field 30 days after deprecating it, you will break users running a mobile app version from 29 days ago. For APIs with mobile clients, the removal window should be calibrated to the mobile app update lag, often 60-90 days minimum.

Question 6: What should go in a GraphQL incident runbook?

Short answer: A GraphQL incident runbook should say how to identify the offending operation, from query logs rather than HTTP logs alone. It should say how to disable or throttle a specific operation with no code deploy, and how to temporarily lower the depth or complexity limit to shed load. It should also say whom to notify for schema-related incidents, and whom for infrastructure incidents. It should also document the blast radius of schema changes.

Why this is right / what interviewers want to hear:

A runbook is only useful if it describes actions that a responder can take under pressure, without deep context. The GraphQL-specific parts of a runbook address the failure modes that REST runbooks do not cover.

Identifying the offending operation: in a REST incident, the slow endpoint is usually obvious from the URL. In a GraphQL incident, the /graphql endpoint is all you have at the HTTP layer. The runbook must describe how to find the offending operation in the structured query log:

what field to filter on: operation name, or query hash
where the logs are: which logging backend, which index, and what time range to search
what a suspicious pattern looks like: high depth, many aliases, or a missing operation name. Unnamed operations are often exploratory or attacker-generated.

Disabling a specific operation without a deploy: an operation block should be implementable in minutes, not hours. Options include:

adding the operation name to a blocklist in the application configuration, if the config hot-reloads
adding a rule in the WAF or load balancer. Operation names are not visible at that layer in a standard POST body, but an application-layer rule can extract them.
deploying an emergency safelist update, if persisted queries are in use The runbook should have the specific command or UI path for the option available in your stack.

Lowering limits under load: if a query storm is occurring (many clients sending expensive queries simultaneously), temporarily lowering the complexity or depth limit via a configuration flag can shed the most expensive queries while allowing simpler queries to proceed. This is triage, not a fix. The runbook should include the procedure for changing the limit, the expected effect, and how to restore the normal limit after the incident.

Blast radius documentation: a runbook for a schema change incident should list the clients and teams depending on the changed part of the schema. The responder then knows who to contact and what the downstream impact is. This information should be maintained as a living document (who owns what query) instead of being reconstructed during an incident.

The broader principle is this. A GraphQL incident runbook differs from a REST one because GraphQL's single endpoint and client-driven query model put the tools for isolation and diagnosis at the application layer, instead of the infrastructure layer. The runbook therefore has to name those application-layer tools explicitly.

Question 1: How do you set up tracing for a GraphQL API?​

Question 2: What GraphQL-specific metrics should you monitor in production?​

Question 3: How do you handle schema evolution in a CI pipeline?​

Question 4: What is the difference between a depth limit, a complexity limit, and a timeout, and do you need all three?​

Question 5: How do you do a zero-downtime schema migration when you need to rename a field?​

Question 6: What should go in a GraphQL incident runbook?​

Further Reading​

Question 1: How do you set up tracing for a GraphQL API?

Question 2: What GraphQL-specific metrics should you monitor in production?

Question 3: How do you handle schema evolution in a CI pipeline?

Question 4: What is the difference between a depth limit, a complexity limit, and a timeout, and do you need all three?

Question 5: How do you do a zero-downtime schema migration when you need to rename a field?

Question 6: What should go in a GraphQL incident runbook?

Further Reading