NInfer-4090

NInfer-4090 is a cutting-edge C++20/CUDA inference engine designed specifically to run the Qwen3.8-27B language model on a single 24 GB NVIDIA GeForce RTX 4090. Built from the ground up, this engine operates independently of frameworks like PyTorch or TensorRT, ensuring optimal performance tailored to the architecture of the RTX 4090.

Key Features:

• High Performance, Low Latency: Experience incredibly fast token generation rates with

  • A prefill speed reaching 3,600 tok/s for a prompt of 9,411 tokens
  • A sustained rate of 2,578 tok/s for longer prompts of up to 118K tokens.
  • Additionally, the model achieves 148.6 tok/s for MTP speculative decoding on a solo code stream, scaling up to 299 tok/s across three concurrent streams.

• Streamlined Compatibility: NInfer-4090 seamlessly integrates with a variety of coding agents via OpenAI and Anthropic APIs, while caching mechanisms optimize long conversation flows, dramatically enhancing efficiency.

• Adaptive LoRA Adapters: Support for per-request LoRA adapters allows users to select model variations dynamically, facilitating customized runs without the overhead of weight merging or complex reloads.

• Dual Decision APIs: The engine supports both OpenAI and TypeSafe System One decision APIs, enabling real-time probability outputs for complex reasoning with minimal latency.

Use Case Scenarios:

• Any Coding Agent Compatibility: NInfer can serve any coding agent requiring access to OpenAI or Anthropic communications, making it versatile for diverse applications in AI-driven tasks.

• Best Performance with Caudra: While it works with any agent, optimal functionality is achieved when paired with the Caudra application, which showcases the live prefill rates, enhances cached state routing, and leverages a local decision engine.

Example Usage:

# Start the NInfer server in a Docker container
docker run --rm --gpus all --publish 8080:8080 \
  --volume ninfer-continuations:/var/cache/ninfer \
  ninfer-4090:sm89

# Send a simple completion request
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'content-type: application/json' -d '{"model": "qwen3.8-27b", "enable_thinking": false, "messages": [{"role": "user", "content": "Say hello in five words."}]}'

NInfer-4090 stands out as a powerful tool for developers and researchers working with large language models, offering unprecedented performance metrics, flexibility in the deployment of AI solutions, and straightforward integration with existing agent frameworks.