Contributing to llm-d/inference-scheduler, Kubernetes LLM Inference Scheduling

Large language-model services must route requests across several model backends. The llm-d Endpoint Picker Plugin runs in Kubernetes and selects a vLLM backend from its KV cache, prefill location, and current load. Work on this scheduler improves shared open-source infrastructure for production inference.

The project joins several systems: Gateway API resources, Envoy request callbacks, split prefill and decode in vLLM, cache-aware scheduling, and Kubernetes deployment through Kustomize, Helm, and RBAC. Its Go 1.24+ codebase also uses tracing, Prometheus metrics, and modular filters and scorers. Contributions must preserve clear and extensible interfaces.

I plan to map how the scheduler intercepts requests and selects GPU backends. I will study ways to split prefill and decode across nodes, improve load balancing and KV-cache reuse through plugins, and locate avoidable request-processing latency.

Deepen expertise in Kubernetes operator patterns, Envoy proxy extensibility, distributed systems for ML inference, Go concurrency patterns, and production ML infrastructure design. The project provides exposure to real-world inference serving challenges at the intersection of cloud infrastructure and applied ML.

Setting up a local development environment matching the project's cluster configuration. Reviewing the codebase architecture and existing test patterns (unit, integration, and end-to-end). Familiarising with the disaggregation deployment modes and the local simulation environment for iterative development before contributing upstream.