AI Idea Validator

1. Front Matter


2. Introduction & Goals

  • Problem Summary: Creators need a way to validate their quest topics before investing time in content creation. The AI Idea Validator is a Gemini-powered system that assesses topic viability and generates structured curriculum outlines including conflict points, crisis scenarios, and recommendations — all importable into the Visual Canvas editor. A July – September 2026 overhaul (PRs #674, #679, #681, #707, #721, #724, #778, #840) hardened the end-to-end flow: generated cards render with supported node types, import can merge into existing quests, the reviewer queue distinguishes AI-linked quests from manual ones, and the whole auth chain moved to Supabase Auth.

  • Goals:

    • Provide AI-powered topic validation with viability scoring (0-100)

    • Generate structured curriculum outlines with modules, in either linear (narrative) or exploration (branching) quest mode

    • Identify conflict points and crisis scenarios for engagement

    • Enable direct import of AI output to the Visual Canvas, including Smart Merge into an existing quest

    • Cache responses to reduce redundant API calls

    • Implement content safety filtering (local deterministic + Gemini AI layers)

    • Enforce platform-wide AI token quotas (sitewide, agency, member)

  • Non-Goals:

    • Real-time content generation during quest editing

    • AI-generated media assets (images, videos)

    • Automatic quest publishing without creator review

  • Glossary:

    • Viability Score: 0-100 rating of topic's educational potential

    • Curriculum Module: A unit of learning content with duration estimate

    • Conflict Point: Branching decision point for learner engagement (exploration mode)

    • Crisis Scenario: Assessment activity placed after specific modules (exploration mode)

    • Quest Mode: linear (sequential narrative) or exploration (branching with game-over detours)

    • Smart Merge: Import strategy that updates matching nodes in an existing canvas, skips duplicates, and appends genuinely new nodes

    • Canvas Metadata: React Flow nodes/edges JSON structure


3. High-Level Architecture

  • System Diagram:

  • Data Flow: Input → auth + role gate → token quota gate → Zod validation + input-quality gate → AI content safety (local hard-block → Gemini classifier) → cache check → (miss) Gemini call → parse structured JSON → mode-aware validation → persist to idea_validations → return result.

    • Import path: ValidatorOutput → deterministic lib/canvas-import conversion → deterministic layout pass (POST /api/ai/compute-canvas-layout — a TypeScript-only engine, no AI call; the client keeps the converter's positions if it fails) → sessionStorage (ai-curriculum, ai-import-decision) → quest editor → Smart Merge / Replace All → React Flow nodes → content cards.

  • Technologies Used:

    • Google Gemini — topic validation, curriculum generation, and content safety classification (canvas layout is deterministic — no AI). The active model and API key are admin-configured in gemini_config (DB default gemini-3.1-pro-preview; gemini-3.5-flash runtime fallback when no active row exists).

    • Zod — input/output schema validation, including the input-quality gate inside superRefine

    • Supabase — Postgres persistence, caching, usage tracking, Storage, Row Level Security

    • Supabase Auth — authentication and session management (Clerk was fully decommissioned in September 2026; PRs #823, #826)

    • Next.js App Router + React Flow — canvas node conversion and rendering

    • Monaco editor (@monaco-editor/react) and TipTap — code-block editor and reflection editor in the quest player

    • DOMPurify + markdownToHTML — sanitized rendering of AI markdown output


4. Detailed Design & Implementation

  • Data Model / Schema:

idea_validations Table:

Column

Type

Description

id

UUID PK

Auto-generated

creator_id

TEXT

Supabase Auth user UUID stored as text (auth.users.id; legacy column naming from Clerk, same value the auth layer passes)

topic

TEXT

Original input topic

context

TEXT

Optional context

learning_objectives

TEXT

Optional objectives

viability_score

INTEGER

0-100 score (CHECK constraint)

viability_rationale

TEXT

AI reasoning

curriculum_outline

JSONB

Array of modules (mode-aware shape)

conflict_points

TEXT[]

Exploration mode: conflict items saved by the validator; legacy rows may contain plain strings

crisis_scenarios

TEXT[]

Exploration mode: crisis items; legacy rows may contain plain strings

recommendations

TEXT[]

Array of recommendation strings

total_estimated_duration

TEXT

Estimated total duration (mirrors the numeric output field)

model_name

TEXT

Gemini model used (default 'unknown')

quest_mode

TEXT

linear or exploration (CHECK constraint, default exploration)

narrative_progression

TEXT

Linear mode only; NULL for exploration (migration 20260311)

created_at

TIMESTAMPTZ

Auto-timestamp

updated_at

TIMESTAMPTZ

Maintained by idea_validations_update_updated_at trigger

Final effective structure (after the February – March 2026 migrations):

idea_validations (
id uuid primary key default gen_random_uuid(),
creator_id text not null, -- Supabase Auth user UUID (auth.users.id)
topic text not null,
context text,
learning_objectives text,
viability_score integer not null check (viability_score between 0 and 100),
viability_rationale text not null,
curriculum_outline jsonb not null,
conflict_points text[], -- exploration mode
crisis_scenarios text[], -- exploration mode
recommendations text[],
total_estimated_duration text,
model_name text not null default 'unknown',
quest_mode text not null default 'exploration'
check (quest_mode in ('linear', 'exploration')),
narrative_progression text, -- linear mode only
created_at timestamptz not null default now(),
updated_at timestamptz not null default now()
);
 
-- RLS (20260911_supabase_rls_auth_uid.sql sweep):
-- SELECT / INSERT / UPDATE / DELETE all use: creator_id = auth.uid()::text

Implementation Notes:

  • Indexes: idx_idea_validations_creator_created (creator_id, created_at DESC) and idx_idea_validations_creator_topic_created (creator_id, topic, created_at DESC) — the history sidebar and variant dedup read patterns.

  • RLS: enabled; SELECT/INSERT/UPDATE/DELETE policies all use creator_id = auth.uid()::text after the September 2026 RLS sweep (20260911_supabase_rls_auth_uid.sql; previously auth.jwt() ->> 'user_id'). The API routes use the service-role client, so RLS is defense-in-depth and the routes also filter by the authenticated user.

  • Creator identity: creator_id stores the Supabase Auth UUID. Older migration comments say "Clerk" — that is stale naming only; never treat this value as a Clerk ID.

  • The validator only persists fresh (non-cached) results; cache hits skip the insert to avoid duplicate history rows.

Feature migrations (apply in order where not already deployed):

Migration

Purpose

20250204_create_idea_validations_table.sql

Initial table + indexes + RLS + updated_at trigger

20250205_dedup_idea_validations.sql

Dedup (creator_id, topic, context, learning_objectives), keep latest

20260311_add_narrative_progression_to_idea_validations.sql

Linear-mode narrative field

20260316_add_quest_mode_to_idea_validations.sql

quest_mode TEXT + CHECK (linear/exploration)

20260326_add_validation_id_to_quests.sql

quests.validation_id UUID REFERENCES idea_validations(id) ON DELETE SET NULL — links a quest to its source validation (reviewer report)

20260716_change_content_card_id_to_text.sql

quest_content_cards.id and activity_submissions.content_card_id uuid → text (AI-generated scenario/card IDs are synthetic strings)

Supporting Tables:

  • gemini_cache — response cache with 30-day TTL for validations; SHA-256 cache_key derived from the normalized input (including the active model); hit_count/last_accessed_at telemetry; cleanup_expired_cache() RPC; gemini_cache_stats view.

  • ai_usage_tracking — single source of truth for every AI call (feature validator / purpose topic-validation, and content-safety checks): tokens, cost, success/error, model, agency attribution. Monthly views: ai_usage_monthly_by_feature, ai_usage_monthly_by_agency, ai_usage_realtime_stats.

  • gemini_pricing — per-model pricing configuration (5-minute in-memory cache in pricing.service.ts); one active row enforced by a partial unique index.

  • gemini_config — admin-managed active model + API key (+ optional google_cloud_project_id); api_key may be NULL, in which case GOOGLE_GEMINI_API_KEY is used as fallback.

  • gemini_responses, gemini_api_logs — legacy audit tables, retained for history but no longer written by the validator flow (superseded by ai_usage_tracking).

  • API Specification:

    • POST /api/ai/validate-idea — validate a topic and generate a curriculum

      • Auth: getAgencyAuthContext() + verifyCreatorAccess() (ADMIN / CREATOR / AGENCY; LEARNER and REVIEWER blocked); Supabase Auth session required.

      • Pre-flight: enforceTokenQuota(userId, 1500) — 3-tier token gate (sitewide → agency → member).

      • Request: { topic: string (2-200 chars, required), context: string (max 1000), learningObjectives: string (max 2000), model?: "gemini-3-flash-preview" | "gemini-3.1-pro-preview" | "gemini-2.5-pro" | "gemini-2.5-flash", bypassCache?: boolean, questMode?: "linear" | "exploration" (default "exploration") }

        • topic, context, and learningObjectives also pass the input-quality gate (gibberish, excessive special chars, repetition).

        • The model field is accepted for compatibility; the service always executes the active model from gemini_config, and the cache key includes that model.

      • Response (200): { success: true, data: ValidatorOutput, error: null, message } where ValidatorOutput contains viabilityScore, viabilityRationale, curriculumOutline, recommendations, totalEstimatedDuration, cached: boolean, timing, plus mode-specific fields: narrativeProgression (linear) or conflictPoints + crisisScenarios (exploration); discriminating field questMode.

      • Errors: see the error code table below.

    • GET /api/ai/idea-validations — creator's validation history

      • Auth: Supabase session (createServerSupabaseClient()), suspension check (assertNotSuspended).

      • Query params: limit (default 50, max 50), offset (default 0), sortBy (created_at | title), sortOrder (asc | desc, default desc). Note: sortBy=title is currently ineffective — the route maps it to topic, which the params schema rejects and catches back to created_at (see Known Limitations).

      • Response (200): { success: true, data: IdeaValidationRow[], error: null, meta: { count, limit, offset } }, filtered by creator_id.

      • Note: history is read-only via API. The earlier save component (AIIdeaValidatorForm.tsx) is no longer mounted.

    • POST /api/creator/validate-idea — content-safety pre-check (lightweight)

      • Auth: getCreatorAuthContext() (PR #724) + token quota gate.

      • Request: { topic: string (1-500), context?: string (max 1000), learningObjectives?: string (max 1000) }.

      • Runs the same AI content-safety pipeline without calling Gemini for curriculum generation.

      • Response (200): { success: true, data: ContentSafetyFullResult, message: "Content validation passed" }; blocked input → 422 CONTENT_POLICY_VIOLATION with category, reasoning, flaggedFields, evasionDetected (this route's error payload does not include isLocalFallback; the main validate-idea route does).

    • POST /api/ai/compute-canvas-layout — deterministic node layout for imports

      • Auth: Supabase session + suspension check.

      • Request: { canvas: CanvasMetadata, questMode: "linear" | "exploration" }.

      • Runs the TypeScript-only layout engine (lib/ai/canvas-layout.service.ts) — no AI call — and returns the same canvas with computed x/y coordinates.

      • Response (200): { success: true, data: { canvas: CanvasMetadata } }; if the request fails, the client keeps the converter's placeholder positions (non-fatal).

    • GET /api/ai/token-quota — current month quota status for the floating token-budget widget

      • Response: { allowed, tier: "none" | "member" | "agency" | "sitewide", reason, details: { sitewideUsage, sitewideLimit, sitewideRemaining, agencyUsage, agencyLimit, agencyRemaining, agencyName, userUsage, userLimit, userRemaining } }.

Removed endpoint: the former GET /api/ai/active-model no longer exists. The active model is resolved server-side by getActiveGeminiModel() (reading gemini_config) and never exposed as a standalone endpoint.

Error Codes:

Status

Code

When

400

VALIDATION_ERROR

Malformed JSON or Zod validation failure (details[] per field)

401

AUTHENTICATION_ERROR

No Supabase Auth session (auth chain throws before the route's inline check)

403

AUTHORIZATION_ERROR / ACCOUNT_SUSPENDED

Role gate failure (LEARNER/REVIEWER) or suspended account

422

CONTENT_POLICY_VIOLATION

Local hard-block or Gemini safety classifier rejection (includes category, flaggedFields, evasionDetected, isLocalFallback)

422

INPUT_QUALITY_ERROR

Gibberish, excessive special characters, or repetition in topic/context/objectives

429

TOKEN_LIMIT_EXCEEDED

3-tier token quota exhausted (includes tier details)

429

RATE_LIMIT_EXCEEDED

Gemini quota/rate limit surfaced from the provider

500

INTERNAL_ERROR

AI parse failure, incomplete output structure, missing API key, unexpected server error

503

MODEL_NOT_FOUND

Active Gemini model misconfigured or unavailable

504

INTERNAL_ERROR

Gemini timeout (message surfaces the timeout)

  • Logic & Workflows:

Validation Flow:

  1. Creator submits topic, context, and objectives (with quest mode) on /creator/ai-validator.

  2. Auth + role gate: Supabase session resolved, verifyCreatorAccess() enforces ADMIN / CREATOR / AGENCY (LEARNER and REVIEWER blocked).

  3. Token quota gate: enforceTokenQuota(userId, 1500) — sitewide → agency → member caps; blocked → 429 with tier details. This runs before the body is parsed.

  4. Zod validates the input schema and runs the input-quality gate (content-safety.ts heuristics) inside superRefine.

  5. AI content safety: normalized evasion scan → deterministic local hard-block (authoritative) → Gemini classifier (circuit-breaker protected).

  6. Cache check — SHA-256 key of the normalized input including the active model; 30-day TTL; mode mismatch forces regeneration; bypassCache: true forces a fresh call.

  7. On miss: Gemini call with the mode-specific prompt (temperature/structured JSON output), then parseModelAwareJSON parses the response.

  8. Mode-aware output validation (shared fields + linear/exploration fields); incomplete output → 500 without persisting.

  9. Persist fresh results to idea_validations (cached hits skip persistence); token usage recorded to ai_usage_tracking with cost attribution.

  10. Return the validated result with cached and timing metadata.

Quest Mode (linear vs exploration):

  • linear: sequential narrative. Output adds narrativeProgression (2-3 sentence arc) and per-module coreConcept, practiceActivity, optional prerequisiteModules (1 activity per module on import).

  • exploration: branching. Output adds conflictPoints (title, description, related modules, resolution) and crisisScenarios (title, description, required competencies, placement, outcomes) — 3-5 activities per module on import.

  • Cached entries without a mode discriminator are treated as exploration (matching the original prompt shape); requesting the other mode regenerates.

  • Linear route responses may carry a null/absent narrativeProgression when the AI omits it — the route still succeeds, so consumers must guard for it.

Content Safety Layers:

  • Layer 1 (always on, zero latency): lib/validation/content-safety.ts — gibberish, excessive special characters, and repetition checks exported and invoked from validatorInputSchema.superRefine and the client pre-flight (validateAllFields). Result codes: CLEAN, GIBBERISH_DETECTED, EXCESSIVE_SPECIAL_CHARS, REPETITIVE_CONTENT (PROHIBITED_CONTENT is declared in the contract but is produced by the policy layers, not by these heuristics).

  • Layer 2 (deterministic hard-block): lib/validation/local-content-safety.ts — text normalization (evasion-aware) plus deterministic classifiers (EXPLICIT_SEXUAL, ILLEGAL_ACTIVITY, VIOLENCE_GLORIFICATION, HATE_SPEECH, prompt-injection → COMPANY_POLICY_VIOLATION) with educational-context allowlists. A local block is authoritative: the model can never downgrade it.

  • Layer 3 (Gemini contextual classifier): lib/ai/content-safety.service.ts — temperature 0; creator input fenced between <<<UNTRUSTED_CREATOR_INPUT>>> / <<<END_UNTRUSTED_CREATOR_INPUT>>> delimiters with fence-breakout escaping; response Zod-validated (CLEAN, EXPLICIT_SEXUAL, ILLEGAL_ACTIVITY, HATE_SPEECH, VIOLENCE_GLORIFICATION, NON_EDUCATIONAL, COMPANY_POLICY_VIOLATION).

  • Circuit breaker: 3 failures within a 2-minute window → all safety checks use the local classifier for 5 minutes; after cooldown the next call is retried against Gemini (half-open) and a success resets the breaker.

  • If the AI safety step throws an unexpected error, the route logs it and fails open (validation proceeds) — monitor ai_usage_tracking and server logs for safety-call anomalies.

Token Quota & Usage Tracking:

  • enforceTokenQuota(userId, estimatedTokens) checks three tiers in order: sitewide monthly budget (platform_config.sitewide_monthly_token_budget, with the hardcoded SITEWIDE_MONTHLY_TOKEN_BUDGET fallback in lib/constants.ts), agency aggregate, per-member cap; it fails closed on unknown state.

  • The validator pre-flight reserves ~1,500 tokens (estimate only); actual usage is measured after the call via ai.models.countTokens (input/output roles; length/4 fallback) and recorded by trackAIUsageSafe().

  • Feature attribution: validator / topic-validation for validations; content-safety / content-safety-check for safety calls.

Curriculum-to-Canvas Conversion (current pipeline: lib/canvas-import/):

  • Entry point: convertCurriculumToCanvas(validatorOutput, { questMode }) → CanvasMetadata { nodes, edges, quest_mode, folders: [] }; convertWithStats() returns conversion counters.

  • Node mapping:

    • Title → Text node (id: "title", data.isStart: true)

    • Curriculum module → Scenario node (id: "module-{order}", data.isEnd: true for the last module)

    • Deciding question (per non-last module) → Question node with choices; each wrong choice routes to a game-over branch

    • Game-over branch → Scenario node labelled "Game Over: {title}" with a Text child labelled "What Went Wrong" (exploration places these alternating above/below the flow to avoid overlaps)

    • Module children → Text, Quiz, Reflection, Code, or Link nodes depending on suggestedActivities (keyword map) with a rotating fallback (Quiz → Reflection → Code → Link)

  • Supported child types are the canvas registry types only. Since PR #681 the pipeline emits Reflection (not Essay) and Code (not Codeblock), fixing blank cards in the visual canvas, quest player, and form editor.

  • MAX_MODULES = 10; exploration emits 3-5 activities per module, linear emits 1.

  • lib/utils/curriculum-to-canvas.ts is the superseded legacy converter — no code imports it; lib/canvas-import/ is authoritative. The stale @see reference in the schema file still points at the old path.

Import Flow & Smart Merge:

  1. Creator clicks Import Curriculum → ImportDecisionDialog (create new quest vs import into an existing quest; view preference canvas or outline; quest mode preserved).

  2. AIValidator.handleImportConfirm validates importability, converts via lib/canvas-import, calls /api/ai/compute-canvas-layout (non-fatal), and stores ai-curriculum + ai-import-decision in sessionStorage.

  3. create-new: POST /api/creator/create-quest (title retried with a "(n)" suffix up to 10 times on duplicates) → PUT /api/creator/update-quest-canvas → navigate to /quest-editor/{id}/content/{visual-canvas|form-editor}.

  4. import-existing: navigate directly to the existing quest editor. On the Visual Canvas path, useCanvasFunctions reads the sessionStorage payload and opens ImportMergeDialog; on the form-editor (Outline) path, the nodes are converted to content cards instead (no merge dialog).

  5. ImportMergeDialog (PR #721 layout fix) offers Smart Merge, Replace All, or per-node decisions (cycle update → skip → append per incoming node); the dialog keeps a fixed header/footer with an internally scrolling per-node list.

  6. Smart Merge (lib/utils/smart-merge.ts): exact normalized label match → update; fuzzy token overlap ≥ 0.5 → update; otherwise append with fresh UUIDs; edges are remapped and deduplicated. (A third "type singleton" tier exists in the code but its SINGLETON_TYPES set is currently empty — the tier is disabled and always resolves as append.)

  7. Imported top-level nodes (title Text, module and game-over Scenario nodes) receive data.isAiGenerated: true and render the purple "AI" badge (PR #674); the form editor shows the badge on cards created from flagged canvas nodes (Visual Canvas import path). Child activity nodes (Question/Quiz/Reflection/Code/Link) do not currently render the badge.

  8. The form-editor path converts canvas nodes to content cards via POST /api/creator/convert-canvas-to-cards; a node's isAiGenerated flag is propagated to parent and child card content when set (Visual Canvas import path).

Rendering & Formatting Fixes (PR #681):

  • AI-generated markdown (## headings, **bold**, lists) is converted by lib/utils/markdown-to-html.ts (markdownToHTML) in the learner-facing renderers (QuestContentRenderer, ReflectionRenderer), sanitized with DOMPurify at render time. The canvas-import builders currently store raw markdown strings for canvas text nodes (no conversion inside the builders), so markdown formatting is guaranteed in the quest player/learner renderer, not necessarily on the canvas itself.

  • The quest player's code block was replaced with a Monaco-based editor (CodeblockRenderer) with language detection and a sandboxed preview; reflections use TipTap with word-count limits.

Reliability & Platform Notes (July – September 2026):

  • PR #674: AI-generated content cards carry isAiGenerated through canvas → schema → cards → UI badges; scenario edges connect each choice handle (up to 3) to the next scenario; quest_mode is propagated during AI batch import; verifyCreatorAccess performs a user_roles lookup for standalone creators; quest-generator.service logs detailed status/code/stack.

  • PR #679: quest_content_cards.id (and activity_submissions.content_card_id) migrated uuid → text so synthetic AI IDs (ai-scenario-…) flow through sync without 500s; fixed delete/reorder errors and mixed AI/manual card corruption; enhanced quest-creation error handling.

  • PR #707: the reviewer queue resolves a quest's AI report in two steps — direct quests.validation_id link first, then an exact title↔topic fallback for legacy quests; when neither matches, no report is shown ("Manual"). The PR removed the previous unsafe fallback that attached a creator's most recent validation to any quest (which made manual quests display unrelated AI reports). Reviewers additionally got fullscreen read-only enforcement, comment/analytics access, and node centering.

  • PR #724: every /api/creator/* route — including validate-idea and validate-quest-content, but excluding the intentionally public share endpoints (share/scorm, share/fetch, share/analytics) — enforces the CREATOR role via getCreatorAuthContext(); ApiResponseHelper.handleError() maps AuthorizationError → 403, AuthenticationError → 401, ServiceUnavailableError → 503 instead of 500.

  • PR #778: creator dashboard "Generate Ideas" card now links to /creator/ai-validator; dead Settings button removed; Radix modal scroll-lock fix for the sticky creator header/sidebar.

  • PR #840: middleware verifies session JWTs locally; suspension enforcement moved into the route auth chain (assertNotSuspended) — GET /api/ai/idea-validations and the other AI routes reject suspended accounts with 403 ACCOUNT_SUSPENDED; central error mapping via ApiResponseHelper.handleError.

  • Clerk → Supabase Auth: the platform completed the migration in September 2026 (#823 client UI, #826 decommission). All AIV routes authenticate with Supabase sessions; creator_id stores the Supabase Auth UUID.

  • Key Files:

Path

Role

app/api/ai/validate-idea/route.ts

Main validation endpoint (quota, safety, cache, Gemini, persistence)

app/api/ai/idea-validations/route.ts

Validation history (GET, pagination)

app/api/creator/validate-idea/route.ts

Content-safety-only pre-check endpoint

app/api/ai/compute-canvas-layout/route.ts

Deterministic canvas layout for imports (no AI call)

app/api/ai/token-quota/route.ts

Quota status for the token-budget widget

shared/schemas/ai-validator.schema.ts

Single source of truth for input/output Zod schemas (lib/schemas/ai-validator.schema.ts is a re-export barrel)

lib/ai/validator.service.ts

Prompt builders, Gemini call, parsing, mode-aware validation, token counting

lib/ai/cache.service.ts

SHA-256 cache keys, 30-day TTL, hit stats

lib/ai/content-safety.service.ts

Gemini content classifier + circuit breaker

lib/validation/content-safety.ts

Input-quality heuristics (gibberish/special/repetition)

lib/validation/local-content-safety.ts

Deterministic hard-block + prompt-injection detection

lib/ai/token-enforcement.service.ts

3-tier quota enforcement

lib/ai/ai-usage-tracking.service.ts

Usage/cost attribution to ai_usage_tracking

lib/ai/pricing.service.ts

Model pricing (5-min cache)

lib/canvas-import/index.ts

convertCurriculumToCanvas, convertWithStats, validateCurriculumForImport

lib/canvas-import/builders.ts

Node/edge builders (title, modules, deciding questions, game-over branches, children)

lib/canvas-import/constants.ts

Keyword map, fallback types, module/activity limits

lib/utils/smart-merge.ts

Import matching (analyzeSmartMerge, executeSmartMerge)

lib/utils/markdown-to-html.ts

Markdown → HTML for AI text

lib/utils/error-classifier.ts

Structured error categories for ErrorFeedbackCard

app/creator/ai-validator/page.tsx

Validator page (/creator/ai-validator)

components/creator/AIValidator.tsx

Canonical validator UI (form, progress, results, import)

components/creator/ImportDecisionDialog.tsx

Import destination & view decision

components/quest-editor/visual-canvas/ImportMergeDialog.tsx

Smart Merge / Replace All / per-node decisions

components/creator/ValidatorHistorySidebar.tsx

History sheet

components/creator/ValidationResultsModal.tsx

Validation detail modal (+ variant selector)

components/creator/ValidationVariantSelector.tsx

Switch between AI variants of the same inputs

components/creator/ErrorFeedbackCard.tsx

Structured error feedback UI

components/learner/renderers/CodeblockRenderer.tsx

Monaco code editor for code blocks

components/learner/renderers/ReflectionRenderer.tsx

TipTap reflection editor

hooks/visual-canvas/useCanvasFunctions.ts

Canvas import (import_to_flownode), merge handlers, layout

Legacy / no longer mounted: components/creator/AIIdeaValidatorForm.tsx, ValidatorForm.tsx, ValidatorResults.tsx, ValidationDiffCard.tsx, and lib/utils/curriculum-to-canvas.ts remain in the tree but are not used by the current page — AIValidator.tsx and lib/canvas-import/ are authoritative.


5. Infrastructure & Operations

  • Dependencies:

    • Google Gemini — AI inference (active model + key from gemini_config; GOOGLE_GEMINI_API_KEY fallback)

    • Supabase — Postgres persistence, caching, usage tracking, Storage, Auth

  • Monitoring & Alerting:

    • ai_usage_tracking records every AI call with tokens, cost, success/error, and agency attribution; monthly views (ai_usage_monthly_by_feature, ai_usage_monthly_by_agency) support quota reviews.

    • gemini_cache stats view tracks cache hit counts and token savings; cache expiry is handled by cleanup_expired_cache().

    • Content-safety circuit breaker logs trips (3 failures / 2 min → 5-min local fallback).

    • Persistent failures to insert history rows are logged non-fatally — alert on repeated Failed to save validation entries.

    • Token quota gate failures surface as 429s with tier details; monitor per-tier usage to avoid surprise lockouts.

  • Configuration:

    • gemini_config (admin): active model_name (DB column default gemini-3.1-pro-preview; runtime fallback gemini-3.5-flash when no active row exists), api_key (optional), google_cloud_project_id (optional).

    • Sitewide token budget: platform_config.sitewide_monthly_token_budget (hardcoded fallback SITEWIDE_MONTHLY_TOKEN_BUDGET = 5,000,000 in lib/constants.ts).

    • Env: NEXT_PUBLIC_SUPABASE_URL, NEXT_PUBLIC_SUPABASE_ANON_KEY, SUPABASE_SERVICE_ROLE_KEY, GOOGLE_GEMINI_API_KEY (fallback when the gemini_config row has no key).

  • Deployment Plan:

    1. Run the feature migrations (see table above), including 20260716_change_content_card_id_to_text.sql for AI-generated content card IDs, and the RLS sweep 20260911_supabase_rls_auth_uid.sql (creator_id = auth.uid()::text).

    2. Ensure an active gemini_config row exists (model + key) or set GOOGLE_GEMINI_API_KEY.

    3. Deploy API routes and UI components; verify /creator/ai-validator and /quest-editor/{id}/content/visual-canvas.

    4. Verify cache behavior (miss → hit), ai_usage_tracking rows, quota widget (/api/ai/token-quota), and the safety circuit breaker state.

    5. Smoke-test import: create-new, import-existing (Smart Merge + Replace All), and form-editor card conversion.


6. Testing & Quality Assurance

  • Test Strategy:

Suite

Coverage

tests/services/validator.service.test.ts

validateQuestTopic with mocked Gemini: output shape, curriculum modules, recommendations, cache bypass

tests/services/validator.service.integration.test.ts

Live Gemini integration (excluded from default Jest run; requires credentials)

tests/creator/utils/canvas-import.test.ts

Canvas conversion: sequence chain, game-over branches per wrong choice, node counts — currently outdated/red: it still asserts the pre-#681 Essay/Codeblock/Music node behavior (18 of 43 tests fail); update before citing it as pipeline verification

tests/creator/utils/learning-objectives.test.ts

Mandatory learning objectives across quest/adventure/validator schemas

tests/creator/api/content-safety.service.test.ts

Circuit breaker trip/cooldown/reset + safety result schema invariants

tests/creator/api/content-safety.prompt-injection.test.ts

Prompt-injection resistance, fencing, educational false-positive allowlists

tests/unit/content-safety.test.ts

Input-quality heuristics (gibberish, special chars, repetition)

tests/unit/local-content-safety.test.ts

Deterministic local classifier + breaker behavior

tests/api/ai-endpoints.test.ts

validate-idea endpoint auth/validation/response envelope smoke expectations

tests/unit/token-counter.test.ts

Token estimation and cost tiers

  • Run: npm test (Jest + coverage) or npx jest <path>. The integration suite is excluded by jest.config.js (\.integration\.test\.ts$) and must be run explicitly with live Gemini credentials.

  • Manual QA checklist: validate in both quest modes; verify verdict score + outline render; import create-new and import-existing (Smart Merge, Replace All, per-node cycle); confirm AI badges; check history sidebar and variant switching; confirm reviewer report only appears for AI-linked quests; confirm quota 429 surfaces cleanly.

  • Known Limitations:

    • Gemini response quality varies by topic complexity; linear mode may omit narrativeProgression (tolerated).

    • Cache invalidation is TTL-based only (30 days) plus the per-request bypassCache flag — there is no admin purge UI.

    • Content safety may produce false positives on edge cases; an unexpected safety-service exception fails open (logged).

    • Legacy conflict_points / crisis_scenarios rows may contain plain strings; the UI renders both string and structured shapes.

    • GET /api/ai/idea-validations?sortBy=title is a no-op: the route maps title → topic, which the params schema rejects and falls back to created_at.

    • quests.validation_id is not populated during quest creation yet, so AI-created quests surface their reviewer report through the exact title↔topic fallback rather than the FK; duplicate titles could match the wrong validation.

    • Orphaned UI/legacy files listed in the Key Files section can confuse navigation; treat AIValidator.tsx + lib/canvas-import/ as canonical.


7. Maintenance & Support

  • Troubleshooting:

    • 422 Content Policy Error → topic/context/objectives triggered a local hard-block or the Gemini classifier; rephrase input; inspect category / flaggedFields / isLocalFallback in the response.

    • 422 Input Quality Error → gibberish, excessive special characters, or repetition; ask for a clearer topic.

    • 429 Token Limit Exceeded → check GET /api/ai/token-quota and the reported tier (sitewide/agency/member); free budget or raise the cap.

    • 429 Rate Limit Exceeded → Gemini provider quota; retry later or check the Google project quota.

    • 503 Model Not Found → verify the active gemini_config row and model name.

    • 504 Timeout → Gemini slow/overloaded; retry or bypass cache.

    • Cache not updating → TTL-based (30 days) by design; use bypassCache: true. Switching the active model produces a different cache key.

    • Empty curriculum → topic too narrow; add context or learning objectives.

    • Blank/invisible cards after import → confirm node types are the supported registry types (Reflection/Code, not Essay/Codeblock) and that lib/canvas-import (not the legacy converter) is in use.

    • Raw markdown shown as literal text → markdown conversion lives in the learner renderers (markdownToHTML + DOMPurify in QuestContentRenderer/ReflectionRenderer); the canvas-import builders store raw markdown for canvas text nodes, so verify rendering in the quest player and do not expect the canvas to format it.

    • Smart Merge dialog overflow → layout was fixed in #721; verify the fixed-header / scrollable-body / fixed-footer structure is intact.

    • Wrong or missing AI report in the reviewer queue → the queue resolves the direct quests.validation_id link first, then an exact title↔topic fallback for legacy quests; "Manual" means neither matched. The FK is not populated at quest creation yet, so verify title uniqueness to avoid fallback mismatches.

    • Validation history sorting by title appears to do nothing → known no-op (sortBy=title degrades to created_at); use the default created_at sort.

    • 403 on validation → role gate (LEARNER/REVIEWER blocked) or suspended account; check user_roles, agency membership, and app_users.status.

  • Changelog:

    • 1.0 (Feb 2026): Initial implementation

    • 1.1 (Mar 2026): Added Quest Mode support (#285)

    • 1.2 (Mar 2026): Smart Merge & Error Feedback (#245)

    • 2.0 (Jul – Sep 2026): Reliability & rendering overhaul — supported node types instead of Essay/Codeblock (#681), markdown rendering + Monaco code blocks (#681), AI card flags & choice-based edges (#674), quest_content_cards.id uuid → text (#679), reviewer queue report matching (#707), ImportMergeDialog layout (#721), creator role enforcement on all creator routes + correct 401/403/503 mapping (#724), dashboard entry point (#778), suspension enforcement + auth round-trip cuts (#840); platform auth migrated Clerk → Supabase Auth (#823, #826).


Document Version

1.0 - Initial version, 2026-04-27 07:58 UTC

1.0.1 - Minor revisions, 2026-04-27 07:59 UTC

1.0.2 - Minor revisions, 2026-04-28 06:18 UTC

1.0.3 - Minor revisions, 2026-06-29 07:45 UTC

1.0.4 - Minor revisions, 2026-06-29 07:47 UTC

1.0.5 - Minor revisions, 2026-06-29 09:09 UTC

2.0 - Major revision by Christian Denzon: post-overhaul architecture (Supabase Auth, smart merge, canvas import pipeline, content safety + circuit breaker, 3-tier token quota, updated schema/API/tests/troubleshooting), 2026-09-30. PRs: #674, #679, #681, #707, #721, #724, #778, #840.


Was this article helpful?