Technical deep-dive

How it works

Study with YouTube is not a search engine. It's a structured knowledge graph of curated lecture courses — built offline with AI, served in milliseconds at search time.

Architecture
User query: "HOG"
       │
       ▼
┌─────────────────────────────────────┐
│  Abbreviation Expander              │
│  "HOG" → "Histogram of Oriented    │
│           Gradients"                │
└──────────────┬──────────────────────┘
               │
               ▼
┌─────────────────────────────────────┐
│  Semantic Search Engine             │
│  • Exact + partial title match      │
│  • ai_topics (transcript-verified)  │
│  • Abbreviation boost (+8 pts)      │
│  • Q&A penalty (−5 pts)             │
└──────────────┬──────────────────────┘
               │
      ┌────────┴────────┐
      │                 │
      ▼                 ▼
 Index Hit          No match →
 (instant,          YouTube Data API v3
  0 calls)          (free fallback)
      │
      ▼
┌─────────────────────────────────────┐
│  Sequence Context Builder           │
│  • Parse lecture numbers (9.3→9003) │
│  • Exclude Q&A from prev/next       │
│  • Build week cluster               │
└──────────────┬──────────────────────┘
               │
               ▼
    Result + prev/next navigation

🗂️ What the data looks like

Every video in the index is a structured JSON object. Here's a real example from the UCF Computer Vision course:

{
  "id": "xD2qV2Mz3W4",
  "title": "Lecture 7.2: Training Neural Networks [Backpropagation]",
  "url": "https://youtube.com/watch?v=xD2qV2Mz3W4&list=PLd3...",
  "thumbnail": "https://img.youtube.com/vi/xD2qV2Mz3W4/hqdefault.jpg",
  "course": "UCF Computer Vision",
  "course_short": "UCF CRCV",
  "playlist_id": "PLd3hlSJsX_Im0zAkTX3ogoiDN9Y7G6tSx",
  "subject": "Computer Vision",
  "position": 2,           ← real playlist order from YouTube API
  "topics": ["neural networks", "deep learning", "backpropagation"],
  "ai_topics": ["backpropagation", "gradient descent"],  ← transcript
  "is_qa": false,          ← transcript-verified (not a Q&A session)
  "ai_summary": "Today we cover backpropagation. We derive the...",
  "search_text": "lecture 7.2 training neural networks backprop..."
}

🗄️ Knowledge base — RAG on course playlists

For each of 1,200+ topics — from HOG to Dynamic Programming to Kirchhoff's Laws — the YouTube Data API v3 enumerates every video in the best-matched course playlist. This is offline RAG: instead of querying a vector store at runtime, we pre-index each video's position, topics, and playlist context into a structured retrieval layer. Search is then deterministic scoring — sub-10ms per query, zero LLM calls at runtime.

SerpAPI YouTube search → playlist ID → YouTube Data API v3 → all videos → index.json

📝 Transcript enrichment — AI topic tagging

Titles are often misleading. "Lecture 7: Q&A (Part 1)" could cover CNN, AlexNet, or anything discussed that session. The SerpAPI transcript API fetches the spoken content, then:

  • →Q&A detection: "any questions you have" → is_qa=true → excluded from prev/next navigation
  • →Topic extraction: "backpropagation", "gradient descent" found in speech → ai_topics[]
  • →Sequence hints: "last time we covered X, next we'll cover Y" → explicit ordering signal
  • →ai_summary: First substantive sentences of the lecture for search matching

🧠 Search scoring — abbreviation-aware relevance

Queries go through multi-layer scoring — no vector embeddings needed at this scale:

SignalPointsExample
Exact phrase in title+20"backpropagation" → Lecture 7.2 [Backpropagation]
ai_topics match (transcript)+15"HOG" matches ai_topics: ["HOG", "gradients"]
Exact abbreviation match+8"HOG" = abbreviation in 121-entry lookup
Topic tag match (partial)+4"gradient" in topics[]
Title word match+5"backprop" in title words
Q&A video penalty−5is_qa=true → ranked lower

🔗 Sequence intelligence — lecture graph

YouTube has no concept of "9.3 comes after 9.2." We build that graph. Each video title is parsed: Lecture 9.3 → sort key 9003. For channels without lecture numbers (Karpathy, StatQuest, 3Blue1Brown), the YouTube Data API returns correct playlist positions(0, 1, 2, 3...) which we store and use directly. Q&A videos are excluded from prev/next even if they're in the same week.

⚡ Infrastructure

Next.js 16

React Server + Client, static homepage

Vercel Edge

Global CDN, sub-50ms TTFB from India

index.json

7,500+ videos, shipped as static file

YouTube Data API v3

Free 10K units/day — live fallback

SerpAPI

Offline seeding only, not live search

shadcn/ui

Accessible components, dark mode

Built by Jai Goyal · Try it →