Contributing to llm-d/inference-scheduler, Kubernetes LLM Inference Scheduling

Large language-model services must route requests across several model backends. The llm-d Endpoint Picker Plugin runs in Kubernetes and selects a vLLM backend. It considers current load, where prompt processing occurs, and the KV cache of earlier attention computations. Thus, scheduler work can improve shared open-source infrastructure for serving model responses.

The project connects Gateway API resources, Envoy request callbacks, and vLLM prompt processing and token generation on separate nodes. Moreover, it uses cache-aware scheduling and Kubernetes deployment through Kustomize, Helm, and RBAC access controls. Its Go 1.24+ codebase uses tracing, Prometheus metrics, and modular filters and scorers. Therefore, contributions must preserve clear interfaces that support extensions.

I plan to map how the scheduler intercepts requests and selects GPU backends. Then I will study ways to split prompt processing and token generation across nodes. Moreover, I will study plugins that improve load balancing and KV-cache reuse. I will also locate avoidable delays in request processing.

I want to learn more about Kubernetes operator patterns and ways to extend Envoy proxies. Moreover, I want to deepen my knowledge of distributed model serving and Go concurrency patterns. I will also study infrastructure design for production machine learning. Thus, the project exposes me to practical serving problems across cloud infrastructure and applied machine learning.

I am setting up a local development environment that matches the project’s cluster configuration. Meanwhile, I am reviewing the architecture and its unit, integration, and end-to-end tests. I am also studying deployment modes that separate prompt processing from token generation. After that, I will use local simulation to test changes before contributing upstream.