GPU Manager Ready — Self-Hosted LLM Deployment Orchestrator on RunPod

Objective

Built a full-stack infrastructure system for provisioning, monitoring, and cost-controlling on-demand GPU pods that run local LLMs (vLLM) on RunPod.

Stack

Delivery highlights

• Developed a two-component system (NestJS API + React/Vite admin web) that manages the full lifecycle of RunPod GPU pods running vLLM for self-hosted LLM inference. Implemented session-based authentication with role-based access control for the human admin path, and a separate machine-to-machine guard for the OpenAI-compatible inference path used by LiteLLM. Built deployment lifecycle management (create/start/stop/resume/delete), GPU catalog and pricing, model discovery via the Hugging Face Hub API, and an audit log of admin actions. Added a cron-based scheduler that auto-stops idle pods to prevent runaway GPU cost and reconciles pod state against RunPod on a fixed interval so the database never drifts from the real infrastructure.