Helios

Lightweight but powerful Python inference engine for Qwen3-4B on CUDA GPUs, with KV caching, prefix caching, paged attention, continuous batching, and disaggregated prefill/decode.

Source
PythonPyTorchCUDAInferenceFastAPI
Helios

Overview

Helios is a lightweight but powerful inference engine in Python. It runs Qwen3-4B on CUDA GPUs and implements KV caching, prefix caching, paged attention, continuous batching, and disaggregated prefill and decode. A FastAPI server exposes an OpenAI-style chat completions API around that serving path.

Highlights

  • Lightweight Python inference engine built for Qwen3-4B on CUDA GPUs
  • KV caching reuses attention state while decoding
  • Prefix caching reuses completed prompt blocks across requests
  • Paged attention for GPU KV memory
  • Continuous batching for concurrent generations
  • Disaggregated prefill and decode on the serving path

Stack

Python 3.11+
Runtime, engine, and server
PyTorch
Qwen3 decoder, caches, and generation
FastAPI
OpenAI-style chat completions HTTP API
Hugging Face
Tokenizer, safetensors, and model snapshot loading
CUDA
NVIDIA GPU execution