When Ray Serve is the right wrapper around an LLM serving runtime

**Ray Serve** is the model-serving framework from the Ray project. **vLLM** is a GPU-efficient runtime for large language models. They sit at different layers, not in competition. vLLM serves one model on a GPU. It handles batching, PagedAttention, and continuous batching for high throughput. If your deployment is just one model behind one HTTP endpoint, vLLM with a thin FastAPI wrapper is usually all you need. Ray Serve is for the case where the deployment is **composed of multiple things**. A real chat backend might run a small intent classifier on the input (your own custom model), route based on the classification to one of several LLM runtimes (vLLM for the chat model, a hosted reranker for retrieval, an embedding model), and then post-process the response with Python validation. Each of these is a separate deployment unit with its own scaling profile. Ray Serve **composes** these as Deployments in a graph, autoscales each unit independently based on its own request rate, and exposes one HTTP endpoint to the caller. The Ray cluster handles scheduling and inter-process communication. The break-even rule: when you have three or more deployment units with different scaling profiles, Ray Serve earns its overhead. For one model, the cluster overhead is more than the value.

**Ray Serve is the right wrapper when the deployment is multi-component and you need the composition graph plus per-component autoscaling.** It is the wrong wrapper for single-model serving where vLLM alone is lighter. **What Ray Serve actually does.** It is a serving layer built on top of the Ray distributed framework. The unit is a `Deployment` (a Python class with handlers); deployments can call each other, forming a graph. Each deployment autoscales independently based on its own request rate, replica count, and CPU/GPU profile. The whole graph is fronted by one HTTP or gRPC endpoint. **The composition example.** A chat backend has four moving parts: a tiny intent classifier (small custom model on CPU), three LLM runtimes (vLLM for the chat model on A100s, a hosted reranker call, an embedding model on a smaller GPU), and a Python validator that checks the response shape. The classifier sees 100x more QPS than the embedding step; the chat LLM sees the heaviest GPU load; the reranker is a network-bound external call. Each one needs different scaling. Ray Serve runs each as its own deployment with its own replica policy. **Why not plain Kubernetes.** You can do all of this with separate K8s pods and a service mesh, and many teams do. Ray Serve's pitch is that the Python-native composition (deployments call each other via local Python handles, not over HTTP) saves serialization overhead and makes the graph easier to reason about and deploy as a unit. **Why not plain vLLM.** vLLM serves one model. Pre-processing, routing, post-processing, multi-model composition are not its job. Wrapping vLLM in FastAPI for one endpoint is fine; building a multi-stage pipeline in FastAPI is exactly the work Ray Serve is built to remove. **Break-even.** Roughly three or more deployment units with different scaling profiles. Below that, the Ray cluster overhead (scheduler, head node, dashboard) costs more than the composition value.