Use an Off-the-Shelf AI Model
Choose your model, then pick an inference backend.
Step 1: Choose the Best Model for Your Local AI Use-Case
1. Determine the target VRAM and performance requirements
2. Shortlist:
Checkout our recommended recipes or filter candidates based on public benchmarks (ex. Artificial Analysis Index)
If you are using llama.cpp: prefer Q4_K_M quantized checkpoints for best performance while retaining quality
If you are using vLLM or PyTorch: Prefer NVFP4 quantized checkpoints for best performance while retaining quality
3. Evaluate:
Build a Custom Evaluation Dataset for Your Use-Case
Grade the model with Human Evaluation
For LLMs, grade the model using LLM-as-a-judge to scale
Step 2: Pick the Best Inference Backend for Your Use Case
Developers need to consider several factors before choosing a deployment path for their application. Each inference path offers specific model optimization tools and deployment mechanisms.
(Click diagram to enlarge)
