uzelhub / predictor / pipeline

From RSS to Trend Signal

Eight stages turn a daily pull of RSS feeds into a trend-scored knowledge graph. Archive-first ingestion, LLM extraction with quality gates, entity resolution, and four Cytoscape.js exports — each stage is inspectable, re-runnable, and reversible.

01 — Pipeline stages

Step 01 · Input

Ingest

Archive-first RSS polling.

10–20 curated sources per domain, polled daily.

Raw HTML + metadata archived on every fetch.

Nothing discarded — schemas can improve later.

url content-hash fetch-date publish-date

↺ Archive loop · step 01 → step 03

Raw content preserved — re-extraction runs back to Extract as schemas evolve. No re-fetching required.

Step 02 · Prep

Clean

Deduplication + priority scoring.

HTML stripped to plain text.

Near-duplicate detection filters echo articles before extraction budget is spent.

Priority score: source tier, signal type, recency, word count, metadata completeness.

dedupe priority-queue plain-text

Step 03 · AI Layer

Extract

LLM structured claim extraction.

Entities: orgs, people, tools, models, technologies.

Every relationship carries evidence — source snippet, doc ID, URL, confidence score.

Output validated against JSON Schema before proceeding.

entities relationships confidence-score json-schema

Step 04 · Quality

Validate

4-gate rejection before graph entry.

Evidence fidelity: snippet must appear verbatim in source text.

Orphan check: relation endpoints must match known entities.

Zero-value check: non-trivial doc must yield at least one entity.

High-confidence audit: strong claims must have strong evidence.

fidelity orphan-check hallucination-guard

Step 05 · Identity

Resolve

Entity deduplication cascade.

"GPT-4", "gpt4", "GPT-4-turbo" → one canonical entity.

Rule-based normalization → fuzzy matching → alias tracking.

All merges recorded and reversible — no silent overwrites.

canonical alias-map fuzzy-match reversible

Step 06 · Signal

Score

3 daily signals → composite trend.

Velocity: mention frequency accelerating or decelerating? (7-day window ratio)

Novelty: exponential decay + corpus-normalized rarity.

Activity: raw mention volume as baseline.

Weights configurable per domain via domain.yaml.

velocity novelty activity domain.yaml

Step 07 · Output

Export

4-view Cytoscape.js-ready JSON.

Mentions view: who talks about what.

Claims view: who asserts what about whom.

Dependencies view: what relies on what.

Trending view: what's accelerating — full metadata on every export.

cytoscape-json 4-views node-count edge-count

Step 08 · Interface

Visualize

What's Hot — interactive graph client.

Top trending entities with LLM-generated "why trending" narratives.

Filter by entity type, search, zoom neighborhoods, switch domains.

Stable node positions — emerging clusters visible over time.

graph-client LLM-narratives stable-layout multi-domain

↺ Calibration loop · step 08 → step 06

Calibration report flags anomalies — velocity gate saturation, novelty score compression, trending list churn. No auto-correction. Flagged for human review and written to tuning_log.