The data stack you learned two years ago is quietly becoming obsolete — not because it broke, but because the workloads it was designed for have fundamentally changed.
Large language models have introduced a new class of data problem: unstructured content that needs to be queryable, semantically. You can’t put a paragraph of documentation into a Postgres column and run WHERE meaning = 'authentication'. But you can embed it, store that vector, and retrieve it at inference time. That’s the skeleton of retrieval-augmented generation (RAG), and it’s reshaping what data engineers actually build.
The Problem RAG Actually Solves
LLMs have a fixed context window. They also hallucinate when asked about things they weren’t trained on — your internal docs, your product catalog, your proprietary research. RAG solves both of these constraints: instead of cramming everything into the prompt, you retrieve only the relevant pieces at query time and inject them as context.
In practice, this means your data engineering work now includes:
- Chunking raw documents into semantically meaningful segments
- Embedding those chunks using a model (OpenAI, Cohere, open-source alternatives)
- Storing and indexing vectors in a specialized database
- Orchestrating retrieval — matching a user query to relevant chunks at inference time
This pipeline runs before the LLM sees a single token. And like all pipelines, the quality of the output is entirely dependent on the quality of the data flowing in.
The New Data Stack
Let’s be concrete about what the modern LLM data stack looks like:
1. The Embedding Layer
The first transformation in any RAG pipeline is converting text into dense vectors. Models like text-embedding-ada-002 (OpenAI), embed-english-v3.0 (Cohere), or open-source options like nomic-embed-text turn arbitrary text into high-dimensional numeric representations that encode semantic meaning.
The decision here isn’t trivial. Different embedding models perform differently across domains, context lengths, and retrieval tasks. Most teams start with a hosted API and graduate to self-hosted models once cost and latency become real constraints.
2. Vector Databases
This is the most contested part of the stack. The field has exploded in the past two years:
- Pinecone — purpose-built, managed, easy to start with
- Weaviate — open-source, supports hybrid search (BM25 + vector)
- Qdrant — high-performance Rust-based, great for self-hosting
- Chroma — lightweight, ideal for local development and small projects
- pgvector — a Postgres extension that adds vector support to your existing database
For most teams, the choice comes down to operational complexity vs. performance. If you’re already running Postgres, pgvector is often the pragmatic choice. For dedicated retrieval workloads at scale, Qdrant or Weaviate tend to outperform.
3. Chunking Strategies
This is where most RAG pipelines silently fail. The naive approach — split every 500 characters — produces chunks that often start or end mid-sentence, destroying the semantic coherence the embedding model depends on.
Better approaches:
- Recursive character splitting with overlaps to preserve sentence boundaries
- Sentence-window chunking — chunk by sentence, but store surrounding sentences as context
- Semantic chunking — use an embedding model to detect topic shifts and split at semantic boundaries
- Document-aware chunking — respect headers, code blocks, lists (critical for technical docs)
The right strategy depends on your document type. A codebase needs different chunking than a legal document needs different chunking than a customer support knowledge base.
4. Metadata and Filtering
Pure vector similarity often isn’t enough. In production, you almost always need hybrid retrieval — combining semantic similarity with metadata filters. A user asking about billing should only get results from billing-related documents, even if a chunk about authentication scores slightly higher on pure similarity.
This means your ingestion pipeline needs to extract and store rich metadata: document source, creation date, author, topic category, access level. The vector database then becomes a filtered similarity search engine, not a pure nearest-neighbor lookup.
The Orchestration Layer
Tools like LangChain and LlamaIndex have emerged as de-facto orchestration layers for LLM applications. They abstract over different embedding providers, vector databases, and LLM APIs — letting you swap components without rewriting your pipeline logic.
The trade-off: they add abstraction layers that can obscure what’s actually happening, making debugging harder. Teams with more complex requirements increasingly write their own orchestration logic rather than fighting framework abstractions.
A simpler pattern — one growing in popularity — is to keep the data pipeline dumb and fast (extract → chunk → embed → store) and handle retrieval logic in application code with direct API calls to the vector DB.
Where the Real Engineering Challenges Are
Talk to any team that’s moved a RAG system from demo to production and you’ll hear the same stories.
Retrieval quality degrades silently. When embeddings drift from what users are actually asking, you don’t get an error — you get subtly wrong answers. Monitoring retrieval quality requires building eval pipelines that compare retrieved chunks to ground-truth relevance judgments.
Embedding models version and change. When you upgrade your embedding model (for better performance), your existing vector index becomes inconsistent. You need to re-embed your entire corpus. At scale, this is an expensive reindexing operation that requires careful planning.
Document freshness is hard. If a document changes, its old chunks need to be deleted and new chunks ingested. Most teams underestimate how complex document lifecycle management becomes at scale. You need to track document IDs, chunk ancestry, and update timestamps.
Context window ≠ free lunch. As LLM context windows grow (128k, 1M tokens), teams are tempted to just shove everything in and skip retrieval. This works for some use cases but fails at scale — latency spikes, cost scales linearly with context, and long-context models still struggle with needle-in-haystack retrieval for certain document structures.
What Data Engineers Actually Need to Learn
If you’re a data engineer looking at this landscape, the skills that matter:
- Vector database fundamentals — understand ANN (approximate nearest neighbor) indexing, HNSW graphs, flat vs. indexed search trade-offs
- Embedding model evaluation — how to run retrieval benchmarks (MTEB, BEIR) and interpret results
- Observability for AI systems — tools like LangSmith, Arize, or Helicone for tracing LLM calls and measuring retrieval quality
- Python over SQL, for now — the entire ecosystem lives in Python; even if you prefer SQL for data work, the orchestration and evaluation tooling is Python-first
The underlying engineering problems aren’t new: data quality, pipeline reliability, schema evolution, observability. What’s new is the domain — embeddings instead of rows, semantic similarity instead of joins, retrieval quality instead of query latency.
The Stack Is Still Settling
This ecosystem is moving fast. Best practices from 2024 are already being revised. New embedding models release monthly. Vector database vendors are adding capabilities (hybrid search, BM25, sparse vectors) that blur the lines between specialized and general-purpose stores.
The safest bet for a data team right now: pick the simplest combination that works for your current scale, build strong observability from day one, and design your pipeline interfaces to be swappable. The underlying architecture will evolve — your ability to iterate on it is the actual competitive advantage.