AI Idea Validator

Feature Owner: scorevi (Sean Patrick Caintic)
Updated by: Christian Denzon (AI Idea Validator overhaul, July – September 2026)
Module: Diagnose
Priority: P0
Sprint #12: Fully Implemented (v2.0 reliability & rendering overhaul merged July – September 2026)
Date: 2026-06-29
Last Updated: September 30, 2026


EXECUTIVE SUMMARY

What is this feature? The AI Idea Validator provides two independent endpoints for evaluating creator content ideas: a lightweight content safety check and a comprehensive curriculum generation engine. It assesses viability, generates curriculum outlines, identifies conflict points, and produces structured recommendations. Creators can then import the generated curriculum into the Visual Canvas — either as a new quest or merged into an existing one.

Why does it matter? Creators need confidence that their educational ideas are viable before investing time in content creation. Bad ideas caught early save hours of wasted development. The validator ensures content quality and curriculum coherence from the ideation stage, and the July – September 2026 overhaul made the generated content render correctly end-to-end (supported card types, code blocks, game-over branches, and markdown formatting in the learner renderer) and importable into existing quests without breaking them.

What's the MVP scope?

  • Content safety: zero-latency input-quality pre-flight, deterministic local hard-block, and a Gemini classifier with prompt-injection/evasion detection and circuit-breaker protection.

  • Full curriculum validation with viability scoring (0-100), curriculum outlines, conflict detection, crisis scenarios, recommendations, and narrative progression.

  • Quest modes: Narrative (Linear) and Exploration (Branching) — the mode changes the generated structure.

  • Import to the Visual Canvas: create a new quest or merge into an existing quest (Smart Merge, Replace All, or per-node decisions); Outline View available for linear quests.

  • Validation history with variant switching and a 30-day response cache with an explicit "Regenerate Idea" bypass.

  • Platform constraints: CREATOR-role access via Supabase Auth (Clerk was decommissioned, September 2026) and the 3-tier token quota (sitewide → agency → member).


1. USER PAIN POINT & SOLUTION

Current State (Without Feature)

Creators submit ideas without any structured validation. Low-quality or incoherent educational content reaches development, wasting creator and platform resources.

Pain Point

Type

Impact

Emotional

Anxiety about whether an idea is "good enough" to build

Functional

No objective framework to evaluate educational viability

Business

High content rejection rates post-development; wasted AI generation costs

Future State

Creators submit an idea and receive a viability score (0-100), a structured curriculum outline, identified conflict points and crisis scenarios, actionable recommendations, and narrative progression guidance — all before writing a single line of content. The curriculum can then be imported straight into the Visual Canvas as a new quest or merged into an existing one, with AI-generated nodes clearly badged.

Marketing Hook

"Validate your educational vision in seconds. Know what works before you build."


2. 4D FRAMEWORK MAPPING

Phase

Mapping

Diagnose

Core feature — AI evaluates idea quality, viability, and curriculum structure

Design

Curriculum outline feeds into quest/adventure structure design

Develop

Recommendations and conflict resolutions guide content creation

Deliver

Validated content flows into the quest editor (Visual Canvas or form editor) as a new quest or a Smart Merge into an existing one


3. USER FLOWS

Entry Point

  • Creator Dashboard → AI Suggestion card → "Generate Ideas" (links to /creator/ai-validator; wired in PR #778).

  • Creator sidebar → Builder Tools → "AI Idea Validator".

  • Direct route: /creator/ai-validator.

Success Criteria

  • Viability score returned (0-100) with rationale.

  • Curriculum outline generated as structured JSON.

  • Conflict points and crisis scenarios identified (exploration), or narrative progression mapped (linear).

  • Curriculum importable to the Visual Canvas — as a new quest or merged into an existing quest — with AI badges on the top-level nodes and correct rendering (code blocks, reflections; markdown formatting in the learner renderer).

  • Validation history persists past results; cached results are flagged and can be regenerated.

Main Flow

  1. Creator signs in (Supabase Auth) and opens /creator/ai-validator.

  2. Creator enters a quest topic (required, 2-200 characters), optional context (max 1000 characters) and learning objectives (required in the UI, max 2000 characters), and picks a quest mode: Narrative (Linear) or Exploration (Branching).

  3. The client runs the zero-latency input-quality pre-flight (gibberish, repetition, excessive special characters).

  4. "Validate Topic" → POST /api/ai/validate-idea: role gate → 3-tier token quota → schema + input-quality validation → AI content safety → cache check → Gemini generation → mode-aware output validation.

  5. Progress phases display ("Connecting to Gemini" → "AI content safety analysis" → "Gemini thinking & generating" → "Structuring curriculum" → "Saving to database" → complete), then results render: Viability badge + rationale, Curriculum Outline (N modules), Narrative Progression (linear only), Conflict Points (N) and Crisis Scenarios (N) (exploration only), and Recommendations.

  6. If the result came from the 30-day cache, a "Regenerate Idea" action appears to bypass the cache and generate a fresh alternative.

  7. The history sidebar lists the creator's past validations; "Load to Form" restores an entry's inputs (the current results panel is cleared), while "View Details" opens the stored result in a modal; variants of the same topic can be switched.

  8. "Import Curriculum" opens the import decision: Create New Quest or Import to Existing Quest. Visual Canvas is always offered; Outline View is available for linear quests.

  9. Import converts the curriculum into canvas nodes (title, scenario modules, deciding questions, game-over branches, Text/Quiz/Reflection/Code/Link children) and runs the deterministic layout engine (falling back to the converter's positions on failure). On the Visual Canvas path, importing into an existing quest opens the Smart Merge dialog (Smart Merge / Replace All / per-node update-skip-append).

  10. Imported top-level nodes (title, module, and game-over scenario nodes) are marked with a purple "AI" badge; the form editor shows the badge on cards created from flagged canvas nodes. Fresh (non-cached) validations are saved to idea_validations for history.

Edge Cases

  • Toxic/harmful content: blocked by the deterministic local hard-block (authoritative — the model can never downgrade it) or the Gemini safety classifier; the response reports the category and flagged fields.

  • Prompt-injection attempts in context/learning objectives are neutralized (untrusted-input fencing) and cannot flip the verdict to allow.

  • Gibberish/repetition: rejected with a 422 input-quality error; ask the creator to rephrase.

  • Empty learning objectives: the UI requires them (validated at ≥2 characters); the API schema itself treats the field as optional.

  • Over-length input: topic > 200 / context > 1000 / objectives > 2000 are rejected with a 400 validation error (not truncated).

  • AI failure/timeout during safety: circuit breaker (3 failures within 2 minutes → 5-minute local fallback) keeps validation moving.

  • AI failure/timeout during generation: 504 timeout (retry) or 503 when the active model is misconfigured.

  • Repeated identical inputs: the cached result is returned for 30 days unless "Regenerate Idea" is used.

  • Import title collision: the system retries with a "(n)" suffix (up to 10 attempts).

  • Importing into a quest with many nodes: the merge dialog stays usable (fixed header/footer, per-node scrolling); Smart Merge updates matches and appends new nodes; Replace All overwrites. (Smart Merge / Replace All apply to the Visual Canvas import path; the Outline View path converts directly to content cards.)

  • Layout request failure during import: the converter's placeholder positions are kept (import still succeeds — the layout engine is deterministic and never calls AI).

Decision Points

  • Review recommendations (informational — there is no per-item accept/reject).

  • Choose the quest mode before validating — it changes the generated output shape.

  • Import as a new quest or merge into an existing quest; choose Smart Merge / Replace All / per-node decisions; choose Visual Canvas or Outline View (linear only).

  • Regenerate (bypass cache) with modified input.


4. INFORMATION ARCHITECTURE

Primary

  • Input data: topic, context, learning objectives, quest mode.

  • Viability score (0-100) and viability rationale.

  • Curriculum outline.

Secondary

  • Conflict points (exploration only), crisis scenarios (exploration only).

  • Narrative progression (linear only), recommendations, total estimated duration (shown in the import decision summary).

  • Model name (audit trail), cached flag, quest mode, validation timestamps.

Actions

  • Validate Topic, Regenerate Idea, Import Curriculum (Create New Quest / Import to Existing Quest → Smart Merge / Replace All / per-node decisions), View History / load variant.


5. WIREFRAMES

Excluded — existing UI (AIValidator.tsx).


6. WIREFLOWS

Excluded — existing UI implemented.


7. PROTOTYPE

Excluded — feature is fully implemented.


8. BACKEND SCHEMA

idea_validations table (verified columns)

Column

Type

Notes

id

UUID

PK

creator_id

TEXT NOT NULL

Supabase Auth user UUID stored as text (legacy "Clerk" naming — never a Clerk ID)

topic

TEXT NOT NULL

Creator input

context

TEXT

Optional

learning_objectives

TEXT

Optional in schema; required by the UI

viability_score

INTEGER NOT NULL

CHECK (0-100)

viability_rationale

TEXT NOT NULL

AI reasoning

curriculum_outline

JSONB NOT NULL

Structured curriculum data

conflict_points

TEXT[]

Exploration mode

crisis_scenarios

TEXT[]

Exploration mode

recommendations

TEXT[]

Array of recommendations

total_estimated_duration

TEXT

Estimated total duration

model_name

TEXT NOT NULL

DEFAULT 'unknown'

quest_mode

TEXT NOT NULL

DEFAULT 'exploration'; CHECK (linear / exploration)

narrative_progression

TEXT DEFAULT NULL

Linear only; TEXT, NOT JSONB

created_at

TIMESTAMPTZ

Auto-timestamp

updated_at

TIMESTAMPTZ

Maintained by trigger

Important notes:

  • NO content_warnings column exists.

  • NO model_variant column — use model_name instead.

  • narrative_progression is TEXT, not JSONB.

  • learningObjectives in the Zod schema: z.string().max(2000).trim().optional() — NOT an array (the UI enforces it as required).

  • Row Level Security: creators can only read/write their own rows (creator_id = auth.uid()::text).

  • quests.validation_id links a quest to its source validation for the reviewer AI report (column exists, but is not yet populated at quest creation — reviewers currently resolve reports via the exact title↔topic fallback).

  • Full DDL and migration list: see the Internal Technical Guide.


9. API ENDPOINTS

Method

Path

Auth

Purpose

File

POST

/api/creator/validate-idea

Supabase Auth + Creator role

Content safety ONLY (lightweight standalone check)

app/api/creator/validate-idea/route.ts

POST

/api/ai/validate-idea

Supabase Auth + Creator role

Full validation (quota, safety, cache, Gemini, persistence)

app/api/ai/validate-idea/route.ts

GET

/api/ai/idea-validations

Supabase Auth

Creator's validation history (pagination)

app/api/ai/idea-validations/route.ts

POST

/api/ai/compute-canvas-layout

Supabase Auth

Deterministic node layout for imports (no AI call)

app/api/ai/compute-canvas-layout/route.ts

GET

/api/ai/token-quota

Supabase Auth

Monthly token quota status (widget)

app/api/ai/token-quota/route.ts

Import target endpoints: POST /api/creator/create-quest, PUT /api/creator/update-quest-canvas (canvas save), and POST /api/creator/convert-canvas-to-cards (form-editor path).

Key distinction: TWO separate validation endpoints. The main UI action uses /api/ai/validate-idea, which runs the content-safety layers internally; /api/creator/validate-idea is a lightweight, API-only safety check (the creator UI does not call it — only the debug enforcement-test page probes it). The former GET /api/ai/active-model endpoint was removed — the active Gemini model is admin-configured server-side.

Content Safety (/api/creator/validate-idea)

  • Lightweight safety check; no curriculum generation.

  • Uses lib/ai/content-safety.service.ts plus the deterministic layers in lib/validation/.

  • Prompt-injection/evasion detection with normalization and untrusted-input fencing.

  • Circuit breaker: 3 failures in 2 minutes → 5-minute cooldown → local fallback.

Full Validation (/api/ai/validate-idea)

  • Comprehensive curriculum generation (viability scoring, outline, conflicts, recommendations).

  • Auth: Supabase Auth + Creator role gate; suspended accounts are rejected.

  • 3-tier token quota gate before generation; usage recorded to ai_usage_tracking.

  • Uses lib/ai/validator.service.ts; fresh results are persisted to idea_validations (cached hits are not re-saved).


10. DATA REQUIREMENTS

Frontend Needs

  • Topic, context, and learning objectives inputs (with character counters and limits).

  • Quest mode selection (Narrative / Exploration).

  • Viability score display (0-100) with rationale.

  • Curriculum outline for review.

  • Conflict points and crisis scenarios for creator awareness (exploration); narrative progression (linear).

  • Recommendations (display only).

  • Cached-result indicator and the Regenerate action.

  • History sidebar for past validations with variant switching.

  • Token quota display (floating budget widget).

API Calls

  • POST /api/ai/validate-idea (validation; main action).

  • GET /api/ai/idea-validations (history).

  • POST /api/ai/compute-canvas-layout (import layout).

  • POST /api/creator/create-quest + PUT /api/creator/update-quest-canvas (create-new import).

  • POST /api/creator/convert-canvas-to-cards (form-editor import).

  • (Available) POST /api/creator/validate-idea, GET /api/ai/token-quota.

Caching

  • Validation results are cached for 30 days, keyed by the normalized input including the active model; repeats return cached: true.

  • "Regenerate Idea" bypasses the cache and generates a fresh alternative for the same inputs.

  • Cached hits do not create duplicate history rows; switching the active model produces a different cache variant.


11. PERFORMANCE CONSIDERATIONS

DB Optimization

  • JSONB column for curriculum_outline allows flexible querying.

  • TEXT[] arrays for conflict points, crisis scenarios, recommendations.

  • Indexes: (creator_id, created_at DESC) and (creator_id, topic, created_at DESC).

  • Token quota is checked before any AI call (pre-flight ≈1,500 tokens for validation).

  • History queries are scoped by an explicit creator filter (the service-role client bypasses RLS; RLS remains table-level defense-in-depth).

Response Time

  • Input quality: near-instantaneous (local logic).

  • Content safety: local hard-block is zero-latency; the Gemini classifier adds AI latency.

  • Full validation: dependent on the active Gemini model's API latency; cached results return immediately.

  • Circuit breaker prevents cascading safety failures during AI outages.

  • Import layout is deterministic and instant; if the layout request fails, the converter's positions are kept.


12. SECURITY & AUTHORIZATION

Access Control

  • Supabase Auth session required on all endpoints (Clerk was decommissioned in September 2026).

  • Allowed: Platform Admin, Creator, the legacy platform AGENCY (owner) role, and agency members with the Creator role. Blocked: Learner, Reviewer (403).

  • Suspended accounts are rejected (403 ACCOUNT_SUSPENDED).

  • Creator can only view their own validation history (explicit creator filter; the service-role client bypasses RLS, which remains table-level defense-in-depth).

Auth Logic

  • creator_id is TEXT and stores the Supabase Auth user UUID (legacy Clerk string naming; no UUID FK constraint by design).

Validation

  • Learning objectives: Zod z.string().max(2000).trim().optional() (UI requires the field).

  • Content safety: input-quality heuristics, deterministic hard-block, Gemini classifier, prompt-injection/evasion detection before AI processing.

  • 3-tier token quota (sitewide → agency → member) protects against cost overruns; circuit breaker mitigates AI outages and abuse.


13. ERROR HANDLING

Error

Response

Content flagged as unsafe

422 CONTENT_POLICY_VIOLATION (category + flagged fields); generation never runs

Input quality (gibberish/repetition)

422 INPUT_QUALITY_ERROR; ask the creator to rephrase

Missing learning objectives

Allowed by the API schema; the UI prevents submission

AI API timeout/failure

504 timeout (retry) or 503 MODEL_NOT_FOUND when misconfigured

Provider rate limit

429 RATE_LIMIT_EXCEEDED

Token quota exhausted

429 TOKEN_LIMIT_EXCEEDED with the blocking tier

Malformed/incomplete AI output

500 INTERNAL_ERROR; nothing persisted

Safety service exception

Logged; validation proceeds (fail-open)

3 safety failures in 2 minutes

Circuit breaker triggers; 5-minute cooldown; local fallback

Unauthorized / forbidden

401 authentication error / 403 authorization or suspension error

Duplicate quest title on import

Automatically retried with a "(n)" suffix


14. TESTING CHECKLIST

Happy Path

  1. Sign in as a Creator; open /creator/ai-validator from the sidebar (Builder Tools) or the dashboard "Generate Ideas" card.

  2. Enter a topic, context, and learning objectives; choose Exploration (Branching); validate → score, outline, conflict points, crisis scenarios, and recommendations render.

  3. Validate the same inputs again → cached result is returned (and the "Regenerate Idea" action appears).

  4. Click "Regenerate Idea" → a fresh alternative is generated (bypasses cache).

  5. Switch to Narrative (Linear); validate → narrative progression and linear module details render.

  6. Open the history sidebar → load a previous validation; switch between variants of the same topic.

  7. Import Curriculum → Create New Quest → Visual Canvas: title, module, deciding-question, and game-over nodes appear; title/module/game-over nodes show the purple AI badge; code blocks open in the code editor; reflections render; verify markdown formatting in the quest player (learner renderer).

  8. On the Visual Canvas path, import into an existing quest via Import to Existing Quest → Smart Merge preview → merge and verify updated/appended nodes; repeat with Replace All and with per-node update/skip/append decisions.

  9. Import with the Outline View (linear mode) → content cards are created in the form editor (no merge dialog on this path; AI badges appear only for cards exported from flagged canvas nodes).

  10. Publish an AI-created quest → the reviewer queue shows its AI report (resolved via the exact title↔topic fallback until validation_id is populated); a manually created quest with no matching validation shows "Manual".

Edge Cases

  • Unauthorized account (Learner) or suspended account is denied.

  • Idea with no learning objectives (API) / attempt to submit without them (UI blocked).

  • Idea at max character limits (topic 200, context 1000, objectives 2000); over-limit input is rejected.

  • Content safety blocks harmful input; prompt-injection text in context/objectives does not flip the verdict.

  • Circuit breaker triggers after 3 rapid safety failures and recovers after cooldown.

  • Token quota exhaustion surfaces a clear 429 and the quota widget reflects usage.

  • Gemini model misconfiguration returns 503.

  • Duplicate quest title during create-new import auto-suffixes "(n)".

  • Smart Merge dialog with many nodes stays scrollable; skipped/updated/appended decisions are respected.

  • Cached results do not duplicate history rows; a model switch produces a new cache variant.


15. OPEN QUESTIONS

  • None — feature is fully implemented. (v2.0: still none outstanding for the overhaul. Engineering follow-ups — populating quests.validation_id at creation, history sort by title, and refreshing the outdated canvas-import test suite — are tracked in the Internal Technical Guide.)


16. OUT OF SCOPE

  • Multi-language curriculum validation. — Still out of scope.

  • Peer review integration for validation scores. — Partially addressed (v2.0): reviewers now see the AI validity report for AI-linked quests in the reviewer queue (PR #707); score-based peer review remains out of scope.


17. SUCCESS METRICS

  • AI validation adoption rate (% of creators validating before creating).

  • Average viability score distribution.

  • Import-to-canvas conversion rate (create-new and merge) with no rendering regressions after the v2.0 overhaul.

  • Circuit breaker trigger frequency (should be near zero).

  • Token quota block rate stays low and expected.


18. DEPENDENCIES

  • Google Gemini (validation, content safety) — active model from gemini_config; GOOGLE_GEMINI_API_KEY fallback. (Canvas layout is deterministic and does not use AI.)

  • Supabase Auth (Creator role) — migrated from Clerk, completed September 2026.

  • Supabase Database — idea_validations, gemini_cache, ai_usage_tracking, gemini_pricing, gemini_config.

  • lib/ai/validator.service.ts (validation engine).

  • lib/ai/content-safety.service.ts + lib/validation/ (safety layers).

  • lib/canvas-import/ + lib/utils/smart-merge.ts (import and merge workflow).

  • Quest Editor / Visual Canvas + form editor.

  • Token quota (platform_config budget) and the token budget widget.

  • Reviewer Queue (AI report resolution for AI-linked quests).


19. TIMELINE

Completed — Feature fully implemented in Sprint #12 (v1.0, June 2026). The v2.0 reliability & rendering overhaul merged July – September 2026 (PRs #674, #679, #681, #707, #721, #724, #778, #840).

  • Backend / Frontend (v1.0): Sean Patrick Caintic (scorevi).

  • v2.0 overhaul: Christian Denzon.

  • QA: Internal development testing.


Note: API Endpoints and Schema — please refer to its Internal Technical Guide (AI Idea Validator): https://kb.wyzquests.com/internal-technical-guides/1-2-ai-idea-validator

Document Version

1.0 - Initial version - 2026-06-29 08:11 UTC

1.1 - Added Document Version section and update author to have full name - 2026-06-29 08:43 UTC

2.0 - Hand-off updated for the reliability & rendering overhaul: correct node types + AI badges, markdown/code-block rendering, Smart Merge import flow, reviewer AI report matching, validation caching + regenerate, Supabase Auth + role/quota enforcement, and refreshed schema/API/error/QA sections. Updated by Christian Denzon, 2026-09-30. PRs: #674, #679, #681, #707, #721, #724, #778, #840.


Was this article helpful?