Discovery log

Working document. Step 1 analysis + decisions captured per interview phase. Final output: Technical Specification & System Architecture Document.

Step 1 — Architectural Critique & Tech Stack Proposal (2026-10-17)

1. Material Ingestion & AI Avatar Pipeline

Ingestion (everything -> structured Markdown + metadata)

• PDF/DOCX/PPTX -> Docling (open source, layout-aware, tables, reading order). PPTX: per-slide text + speaker notes + slide image.
• Video/audio -> faster-whisper / WhisperX (word timestamps) + scene-change keyframes (PySceneDetect) captioned by a vision model only when slides are on screen.
• Output unit: "ContentBlock" with source file, page/slide/timestamp, heading path. Everything downstream (RAG, quiz gen, scene gen) reads ContentBlocks, never raw files.

Chunking

• Structure-aware (heading hierarchy), 400–800 tokens, parent/child (retrieve child, feed parent).
• Contextual retrieval: prepend a 60–100 token LLM-written context line per chunk (whole doc in cached prompt; Haiku-class model; Batch API at 50% off).

Vector store: pgvector in Postgres (MVP)

• One DB, tenant isolation via Row-Level Security, HNSW index, filter by course_id.
• Hybrid search: pgvector + Postgres full-text (or ParadeDB BM25) + reciprocal rank fusion + cross-encoder rerank (bge-reranker-v2-m3 self-hosted or Cohere).
• Move to Qdrant only if >~10M vectors or p95 latency problems. Abstract behind a VectorIndex interface.

RAG vs long-context -> hybrid

• Per-course "Knowledge Digest" (LLM-generated, 5–10k tokens: syllabus map, key terms, module summaries) lives in the cached system prompt. Fixes classic RAG weakness on global questions.
• Specific questions -> hybrid retrieval top-k chunks with citations (page/timestamp -> deep-link into the material).
• Complex questions -> agentic retrieval (model calls a search_course tool up to N times).
• Guardrails: scope-limited to course, citation required, educator sees question log, can "pin" corrective answers -> semantic FAQ cache (cost + accuracy win).

Avatar UI tiers

• T1 (MVP): text chat with clickable citations, streaming.
• T2: voice in/out (STT Whisper/Deepgram; streaming TTS Cartesia/OpenAI/ElevenLabs; open-source Kokoro).
• T3: lightweight 3D avatar (VRM / Ready Player Me glTF) with viseme lip-sync driven by TTS phoneme timings, rendered client-side in Three.js. Avoid video talking-head APIs (~$0.10+/min). Same avatar can stand inside VR practical scenes as the guide — differentiator.

2. "Vibe-Coding" & 3D Engine Architecture

Framework decision: Three.js via React Three Fiber (R3F) + a custom declarative Scene DSL (JSON)

• LLM never writes Three.js code. It emits a JSON scene document validated by JSON Schema/Zod: entities with components (transform, model ref, physics body, trigger volume, interaction, state/event rules, UI label, success criteria).
• We write the runtime interpreter once (R3F + @react-three/xr + @react-three/rapier + drei).
• Benefits: ~10x fewer output tokens than code; schema validation before render (no runtime crashes); diff/patch edits; visual inspector for educator QA; versionable; no arbitrary code exec (security).
• Behavior = event -> condition -> action rules (like visual scripting) + a finite set of built-in interaction primitives (grab, place-into-socket, press, point-select, proximity, sequence). Escape hatch later: sandboxed JS (QuickJS WASM).
• Alternative considered: A-Frame (HTML tags, WebXR built-in, very LLM-friendly). Faster demo, weaker long-term base. Could compile the same DSL to A-Frame if ever needed. Babylon.js: strong WebXR but heavier, smaller LLM corpus. PlayCanvas: proprietary editor.

Cross-platform

• @react-three/xr: WebXR on Quest browser, visionOS Safari, Android Chrome AR. Same scene: desktop (first-person/orbit), mobile (touch + gyro), headset (controllers/hands).
• DSL interactions are abstract; the runtime maps them to each input mode.

Educational Asset Search Engine

• Catalog of permissively licensed sources, normalized to GLB (Draco/meshopt, KTX2 textures, LODs, bounding box, pivot), stored on our CDN with license record:
  • Models: Sketchfab API (CC filter), Poly Haven (CC0), Smithsonian 3D (CC0), NASA 3D, Kenney.nl & Quaternius (CC0 low-poly packs: ideal for HSE props), Khronos samples; Objaverse metadata for semantic index.
  • Simulations/widgets: PhET (embed, CC-BY), Open Source Physics / ComPADRE, Falstad circuits, GeoGebra, H5P content types (50+ open-source 2D interactives), p5.js/Matter.js sketches.
• Index: text + CLIP image embeddings in pgvector. Pipeline: Planner LLM produces Scene Spec -> Asset Resolver searches catalog per object -> fallback to primitive composition -> (later) text-to-3D (TRELLIS/Meshy, slow+costly) -> educator picks from top-N in QA stage.
• Curated starter library (~200 vetted props) for the launch vertical(s).

Vibe-coding pipeline (same pattern for 2D widgets, quizzes, 3D)

1. Intent -> Planner (learning objectives, objects, interactions, success criteria)
2. Asset resolution (search)
3. Compiler LLM -> DSL JSON (structured outputs, strict schema)
4. Validator (schema + sanity: floor, overlaps, reachability)
5. Preview -> educator chat edits = JSON Patch, not regeneration

• Large static system prompt (DSL docs, examples, ~15–20k tokens) under prompt caching.

3. Rough Cost Model (Anthropic list prices, Oct 2026)

Models: Opus 5.5 $4/$20 per MTok; Sonnet 5.5 $2/$10; Haiku 5.5 $0.10/$0.50; cache reads ~$0.20/MTok (Opus/Sonnet). Batch API -50%.

Levers: prompt caching on all static prompts; model routing (Haiku: query rewrite, grading, chunk context; Sonnet: tutor answers; Opus: scene/quiz compilation); Batch API for indexing; JSON Patch edits; semantic cache; per-tenant token budgets + metering table from day one (also feeds billing later).

4. Subject Suitability Matrix

Rule: 3D earns its cost only when space, motion, or physical procedure IS the objective. Default to 2D widgets everywhere else.

Decisions Log

Phase 1 — Core LMS, User Flows, B2B/B2C (decided 2026-10-17)

• Tenancy: Organization-first. Every educator belongs to an Organization; a solo teacher is an org of one. Public catalog/marketplace layer can be added later on top.
• Identity & enrollment: Global user identity + org membership. Students join via invite link, email list, CSV import; SSO/SCIM later. One account across orgs.
• Progression: Educator-configurable per course: linear gating or free navigation, pass thresholds, certificate of completion with optional expiry/recertification cycle.
• Roles: Platform Super-admin, Org Admin (seats, assignments, reports), Educator/Author, Student. (TA/co-author = Educator role scoped to a course.)
• Reporting (standard): per-student completion %, scores, time spent, certificate issued, CSV export. Audit-grade immutable logs deferred, but attempt table designed append-only from day one so it can be upgraded.

Phase 2 — Vibe-Coding Interface & Testing Engine (decided 2026-10-17)

• Authoring UX: Block editor + AI co-pilot side panel. Course = Modules > Lessons > Blocks (text, video, file, quiz, widget, 3D scene, iframe embed). AI generates/rewrites blocks in place; everything AI produces is an editable block document (JSON).
• Assessments at MVP: (1) Core auto-graded: MCQ, multi-select, true/false, ordering, matching, fill-blank; question banks, randomization, attempts, time limits. (2) LLM-graded open answers: short answer/essay vs educator-approved rubric, confidence score, educator override queue. (3) 3D practical scoring: scenes carry success criteria, scene events feed gradebook.
  • Deferred (recommended for v1.1): interactive 2D types: image hotspot, drag-and-drop labeling, branching scenario.
• Grounding: Grounded by default in uploaded materials with per-item source citation; per-request toggle to allow general knowledge (flagged as "unverified against materials").
• Sandbox scope: First-class = our JSON Widget DSL and Scene DSL. Second-class = sandboxed iframe embeds (PhET, GeoGebra, H5P, educator HTML) reporting scores via a small postMessage scoring protocol (xAPI-lite: started/progress/completed/score). No arbitrary server code. Custom client JS deferred to an "advanced" tier.

Phase 3 — 3D/VR Engine, Asset Search, Educator QA (decided 2026-10-17)

• 3D vertical: Subject-agnostic from day one. No hand-curated library; instead an automated Asset Quality Gate: license filter (CC0/CC-BY/CC-BY-SA with auto-attribution; reject NC/ND for commercial tenants), GLB normalization (scale to meters, pivot, Draco/meshopt, KTX2, LODs, poly budget), vision-model scoring of thumbnails vs query, and a persistent cache of approved assets that grows into the de-facto curated library. Educator picks from top-N during QA. Educators can upload their own GLB.
• VR priority: 2D screens first (desktop + mobile). WebXR via @react-three/xr enabled and smoke-tested on Meta Quest browser; no VR-only mechanics in MVP. DSL interactions stay input-agnostic so XR is a runtime concern.
• Interaction fidelity: Guided interaction, light physics. Primitives: teleport/walk, look, point-select, pick-and-place into snap sockets, press, proximity trigger, sequenced steps, timers. Rigid bodies (Rapier) only for dropped/thrown items. Full physics manipulation deferred.
• QA stage (required before publish): (1) Play-test in browser + inspector (move/scale/replace asset, edit rules via forms) + AI chat edits applied as JSON Patch. (2) Versioning: Draft -> Published, students pinned to a published version, one-click rollback. (3) Student-preview links + pilot cohort with visible attempts/feedback.
  • Deferred: AI auto-QA agent (headless play-through, reachability check). Cheap to add later because success criteria are declarative.

Phase 4 — RAG Pipeline & AI Avatar (decided 2026-10-17)

• Modality at MVP: Text chat with streaming answers and per-claim citations deep-linking to page/slide/timestamp. Voice (T2) and lip-synced 3D avatar (T3) are later tiers on the same tutor backend.
• Pedagogy: Per-course educator setting: Direct (answers + citations) or Socratic (hints, never gives assessment answers). Exam lockdown: during timed/graded assessments the tutor is disabled or restricted to clarifying question wording.
• Unknowns: Tutor states materials don't cover it; optionally gives a clearly labeled general-knowledge answer if the educator enabled the toggle; student can escalate to the educator inbox. Educator answers once and pins -> enters the course FAQ/semantic cache.
• Languages: Multilingual from day one. Auto-detect material + student language; tutor replies in the student's language. Multilingual embeddings (BGE-M3 class), Whisper language detection, per-language eval sets.
• Model hosting: Cloud APIs (Anthropic + speech/embedding vendors) as default. Must support self-hosted/open-weight models for some tenants -> provider abstraction layer with per-tenant routing policy; ingestion/retrieval stack built only on open components (Docling, WhisperX, BGE-M3, pgvector, bge-reranker) so the full pipeline can run inside a client VPC. Prompts kept model-agnostic with per-provider adapters; structured-output enforcement via schema validation + repair loop rather than provider-specific features where portability matters.

Phase 5 — Data Architecture, Billing Hooks, MVP Scope (decided 2026-10-17)

• Billing hooks (schema now, gateway later): B2B seat licensing per org; per-course purchase by student (Order + Entitlement); subscription plans (Plan, Subscription) for orgs and students; educator revenue share (Payout ledger, platform fee %, tax identity) for a future marketplace.
• AI quotas: Plan-based monthly token/credit allowance per org. Soft cap warns admin; hard cap pauses generation (educator-side) but never blocks students mid-lesson (tutor degrades to FAQ cache + cheaper model before refusing).
• Stack: TypeScript app (Next.js + React Three Fiber, Node/TS API + job orchestration, Postgres + pgvector) + Python ML worker service (Docling, WhisperX, BGE-M3, bge-reranker). Two runtimes.
• Team / timeline: Solo or duo with AI coding tools; pilot in 3–4 months. Roadmap: modular monolith, managed services (Neon/Supabase, Vercel, Fly/Railway), hosted model APIs first; provider abstraction designed in from day one, first self-hosted adapter as a post-pilot milestone.

Follow-up answers (2026-10-17)

• UI language: English primary, Russian secondary (content and tutor stay multilingual).
• Hosting region: Europe (EU-pinned managed services).
• Pilot client accepts cloud models.
• Owner wants to try subscription models first when billing is implemented.
• Pilot course topic: undecided; proposed default "Workplace fire safety and evacuation" (see SPEC.md §15).
• Design system: shadcn/ui (Radix + Tailwind v4) for the product and website, decided 2026-10-17.
• Deployment target for docs: Hostinger, domains/digi.aijam.pro/public_html/docs → https://digi.aijam.pro/docs/

Visual

• Architecture diagrams (phases 1–4): docs/architecture.html (standalone page; host on the project website)
• Full specification: docs/SPEC.md