Case study demo16/16 eval pass

Employer-facing AI developer tool case study

Build Doctor Case Study

Build Doctor helps developers turn failed Next.js and Vercel build logs into a clear debugging report. The local deterministic engine owns the diagnosis; optional DeepSeek review can improve the explanation but cannot replace the source-of-truth result.

During live testing, the DeepSeek free model returned a provider rate limit. Build Doctor handled the condition safely by preserving the deterministic diagnosis, surfacing a clear provider status, and showing a cached demo review that is explicitly labeled as non-live output.

Open Build Doctor View full suite

Deterministic Workflow

Paste logs, redact secrets, classify the failure, extract evidence, show trace, suggest patch, export report.

Safety Model

Secrets are redacted before outputs or provider calls. Paid models are blocked by default and no patch is applied automatically.

Test Coverage

Fixture logs test category correctness, fix relevance, evidence extraction, redaction safety, and provider failure handling.

Problem

Modern deployment failures often produce long logs that are hard to interpret quickly. Developers need a fast way to identify the likely root cause, preserve the evidence, and create a clear report.

Solution

Build Doctor turns pasted build logs into a structured diagnosis. It redacts secrets, classifies the failure, extracts evidence, creates a deterministic trace, suggests a safe patch draft, and exports a markdown report.

Architecture

The system is deterministic-first. The local engine owns the diagnosis. Optional DeepSeek review through OpenRouter can improve the explanation, but it cannot override the source-of-truth diagnosis.

Safety

Secrets are redacted before outputs or provider calls. Paid models are blocked by default. Model output is parsed, validated, and discarded if invalid.

Result

The project demonstrates applied AI product engineering: workflow design, deterministic guardrails, optional model enrichment, eval/test coverage, and developer-tool UX.

Failure Taxonomy

Missing environment variable

MISSING_ENV_VAR

The build or runtime expects an environment variable that is unset in Vercel.

TypeScript compile error

TYPESCRIPT_ERROR

The Next.js build failed during type checking or transpilation.

Module not found

MODULE_NOT_FOUND

The bundler cannot resolve an import or package.

Next.js build or route error

NEXT_BUILD_ERROR

Next.js detected an invalid route, server/client boundary issue, or dynamic rendering violation.

Next.js static generation failure

NEXT_STATIC_GENERATION_ERROR

Static prerendering failed because a page used dynamic data or threw during generation.

Next.js App Router route handler error

APP_ROUTER_ROUTE_HANDLER_ERROR

An App Router route handler failed because of request handling, runtime, or response-shape issues.

Package install error

PACKAGE_INSTALL_ERROR

Install failed before the app could build.

pnpm lockfile mismatch

PNPM_LOCKFILE_MISMATCH

Install failed because pnpm lockfile state does not match package.json.

Package JSON parse error

PACKAGE_JSON_PARSE

The build cannot parse package.json or another package-manager manifest.

Spawn or permission failure

SPAWN_PERMISSION

The build cannot execute a script, binary, or package-manager command because of permissions or missing executable access.

Vercel environment variable missing

VERCEL_ENV_VAR_MISSING

A variable needed by Vercel build or runtime is missing from the active deployment scope.

ESLint build failure

ESLINT_BUILD_ERROR

The production build failed because lint checks reported blocking errors.

Vite build failure

VITE_BUILD_ERROR

A Vite app failed during production build or Rollup bundling.

Prisma database configuration error

PRISMA_DATABASE_ERROR

Prisma failed because database configuration or generation is missing.

Supabase configuration error

SUPABASE_CONFIG_ERROR

Supabase client setup is missing required project URL or anon/service key values.

Stripe webhook configuration error

STRIPE_WEBHOOK_ERROR

Stripe webhook verification cannot run because the signing secret or route config is wrong.

Vercel runtime mismatch

VERCEL_RUNTIME_ERROR

A route or function uses APIs incompatible with the selected Vercel runtime.

Build timeout or memory pressure

OUT_OF_MEMORY

The build or serverless function exceeded memory or duration limits.

Serverless function size or duration limit

SERVERLESS_FUNCTION_LIMIT

A Vercel serverless function exceeded bundle, memory, or execution constraints.

Unknown failure

UNKNOWN

The log does not match a known deterministic failure pattern.

Eval Results

Score: 100% | Passed: 16 | Partial: 0 | Failed: 0

Missing NEXT_PUBLIC_SUPABASE_URL

PASS

Expected SUPABASE_CONFIG_ERROR; actual SUPABASE_CONFIG_ERROR.

TypeScript property does not exist

PASS

Expected TYPESCRIPT_ERROR; actual TYPESCRIPT_ERROR.

Unresolved module import

PASS

Expected MODULE_NOT_FOUND; actual MODULE_NOT_FOUND.

Stripe webhook secret missing

PASS

Expected STRIPE_WEBHOOK_ERROR; actual STRIPE_WEBHOOK_ERROR.

Prisma DATABASE_URL invalid

PASS

Expected PRISMA_DATABASE_ERROR; actual PRISMA_DATABASE_ERROR.

Next.js dynamic server usage error

PASS

Expected NEXT_BUILD_ERROR; actual NEXT_BUILD_ERROR.

Vercel function timeout or memory issue

PASS

Expected OUT_OF_MEMORY; actual OUT_OF_MEMORY.

npm dependency conflict

PASS

Expected PACKAGE_INSTALL_ERROR; actual PACKAGE_INSTALL_ERROR.

Invalid package.json syntax

PASS

Expected PACKAGE_JSON_PARSE; actual PACKAGE_JSON_PARSE.

Build script permission denied

PASS

Expected SPAWN_PERMISSION; actual SPAWN_PERMISSION.

pnpm frozen lockfile mismatch

PASS

Expected PNPM_LOCKFILE_MISMATCH; actual PNPM_LOCKFILE_MISMATCH.

Next.js prerender failure

PASS

Expected NEXT_STATIC_GENERATION_ERROR; actual NEXT_STATIC_GENERATION_ERROR.

App Router route handler failed

PASS

Expected APP_ROUTER_ROUTE_HANDLER_ERROR; actual APP_ROUTER_ROUTE_HANDLER_ERROR.

Serverless function limit exceeded

PASS

Expected SERVERLESS_FUNCTION_LIMIT; actual SERVERLESS_FUNCTION_LIMIT.

ESLint blocks build

PASS

Expected ESLINT_BUILD_ERROR; actual ESLINT_BUILD_ERROR.

Vite import resolution failed

PASS

Expected VITE_BUILD_ERROR; actual VITE_BUILD_ERROR.

Case Study Signal

Deterministic build diagnosis with optional DeepSeek review

Eval score 100%

Deterministic-first

Local rules own the root-cause diagnosis, evidence, trace, patch draft, and report.

Evidence trail

Parser extracts fatal lines, files, warnings, and root-cause signals.

Secret safety

Logs are redacted before diagnosis, display, reports, and optional DeepSeek review.

Developer workflow

The app turns pasted logs into a safe patch draft and markdown incident report.

Developer tooling

failed build diagnosis, evidence extraction, report export

AI product engineering

deterministic source of truth with optional model review

Reliability

provider failure handling, redaction tests, browser workflow coverage

Safety

sanitized inputs, paid-model blocking, no automatic patch application