About
About the System
How the BigData Portal collects, indexes, and serves autonomous-driving datasets.
Pillars at a Glance
BigData Portal organizes around four technical pillars. Domains, libs, and services in the codebase map back to these.
Pillar A
Data Fabric
Cloud-distributed lake + warehouse with governance, lineage, privacy, and a standardized schema based on UTC time and absolute spatial coordinates.
Pillar B
Auto-Annotation
DL and rule-based engines extract semantics from LiDAR and imagery, then auto-label datasets with on-demand JSON / CSV / TXT exports.
Pillar C
Knowledge Graph & IR
Local domain KGs merged into a Global KG, paired with text, image, and multi-modal information retrieval (BERT / GPT, RPN + contrastive learning).
Pillar D
Sharing Portal & Open API
User portal for search, recommendation, visualization, license verification, and download — with a RESTful Open API for programmatic access.
System Architecture
Locates the Next.js portal, Postgres + pgvector stores, the embedding worker, AWS Bedrock (Titan, Nova Lite), and the read-only `adds` MySQL source for packaging metadata.
Components & Stores
Concrete Drizzle schema tables backing the architecture above — names and one-line purpose.
data_setsCatalog
Primary dataset registry, derived from the editorial content lib.
data_productsCatalog
Curated, query-shaped collections of dataset slices (ADR-003).
dataset_relationshipsCatalog
Edges between datasets — replaces a separate graph database (ADR-005).
embedding_jobsQueue
Postgres polling queue for Bedrock-backed embedding work — `FOR UPDATE SKIP LOCKED` (ADR-005).
knowledge_graph_nodesKnowledge Graph
pgvector-embedded nodes for the editorial domain knowledge graph (ADR-002).
image_indexVectors
pgvector vectors for dataset preview imagery (Titan multimodal).
text_indexVectors
pgvector vectors for dataset text descriptions (Titan text v2).
data_product_text_indexVectors
pgvector vectors over data-product copy (ADR-003).
scene_graphKnowledge Graph
VLM-derived scene descriptions (Nova Lite), keyed by frame (ADR-006).
search_benchmarkBenchmark
Curated query / result pairs for offline IR evaluation.
Ingestion Flow
How a dataset moves from MDX upload through schema mapping, embedding-job enqueue, Bedrock processing, vector write-back, and HNSW indexing into a searchable state.
Research Usage Flow
Researcher journey from a natural-language or image query through search, browse, detail, packaging, and re-annotation / download.
References
Source ADRs that shape this page. Diagram contract is frozen at the version shown in the footer.
Snapshot 2026-05-12