Load testing RAG (Retrieval-Augmented Generation)

Abstract

Retrieval-augmented generation (RAG) systems chain three latency-bearing stages — query embedding generation, approximate nearest-neighbor (ANN) search over a vector database, and autoregressive large language model (LLM) inference — into a single user-facing request. Each stage has its own throughput ceiling, its own latency distribution, and its own failure mode under load, yet these stages are rarely load-tested as an integrated system. This paper examines how vector database query latency degrades as corpus size scales past the point where the index fits in memory, how embedding-generation throughput is governed by batching and hardware placement decisions that trade off against query-time latency, and how these upstream latencies compound with, rather than merely add to, downstream LLM inference latency in a streamed response. Drawing on published benchmarking studies of hierarchical navigable small-world (HNSW) and inverted-file (IVF) indexes, embedding-serving infrastructure, and RAG operations practice, we propose a load-testing framework that evaluates RAG pipelines at the module, component, and end-to-end levels, and that treats retrieval latency as a first-class, independently scaled variable rather than a fixed offset added to generation time. We present a latency-budget taxonomy across pipeline stages and a scale-dependent vector-search latency/recall table drawn from a documented production benchmark, and we discuss why load tests that hold corpus size fixed at prototype scale systematically understate the latency a system will exhibit in production.