IBM’s New Granite 3.0 Generative AI Models Are Small, Yet Highly Accurate and Efficient

Today, IBM released the third generation of IBM Granite, a collection of open language models and complementary tools. Prior generations of Granite focused on domain-specific use cases; the latest IBM Granite models meet or exceed the performance of leading similarly sized open models across both academic and enterprise benchmarks. 

The developer-friendly Granite 3.0 generative AI models are designed for function calling, supporting tool-based use cases. They were developed as workhorse enterprise models capable of serving as the primary building block of sophisticated workflows across use cases including text generation, agentic AI, classification, tool calling, summarization, entity extraction, customer service chatbots, and more. 

Introducing IBM’s Granite Generation 3 family

IBM developed the Granite series, available as an NVIDIA NIM microservice, for enterprise use, prioritizing industry-leading trust, safety and cost efficiency without compromising performance. 

In its entirety, the Granite 3.0 release comprises of

• Dense, text-only LLMs: Granite 3.0 8B, Granite 3.0 2B
• Mixture of Experts (MoE) LLMs: Granite 3.0 3B-A800M, Granite 3.0 1B-A400M
• LLM-based input-output guardrail models: Granite Guardian 8B, Granite Guardian 2B

Core components of Granite’s architecture are: Group-query attention (GQA) and Rotary Position Encodings (RoPE) for positional information, multilayer perceptron (MLP) with SwiGLU activation, RMSNorm, and shared input/output embeddings. 

Optimized performance with speculative decoding

Trained on over 12 trillion tokens of carefully curated enterprise data, the new 8B and 2B models demonstrate significant improvements over their predecessors in both performance and speed. 

Speculative decoding is an optimization technique for accelerating model inference speed, helping LLMs generate text faster while using the same (or less) compute resources, and allowing more users to utilize a model at the same time. For example, in a recent IBM Research breakthrough, speculative decoding was used to cut the latency of Granite Code 20B in half while quadrupling its throughput.

In standard inferencing, LLMs process each previous token they’ve generated thus far, then generate one token at a time. In speculative decoding, LLMs also evaluate several prospective tokens that might come after the token they’re about to generate—if these “speculated” tokens are verified as sufficiently accurate, one pass can produce two or more tokens for the computational “price” of one.

Granite 3.0 8B Instruct kept pace with Mistral and Llama models on RAGBench, a benchmarking dataset consisting of 100,000 retrieval augmented generation (RAG) tasks drawn from industry corpora such as user manuals.

IBM Granite’s first MoE models

IBM Granite Generation 3 also includes Granite’s first mixture of experts (MoE) models, Granite-3B-A800M-Instruct and Granite-1B-A400-Instruct. Trained on over 10 trillion tokens of data, the Granite MoE models are ideal for deployment in on-device applications or situations requiring extremely low latency.

In this architecture, the MLP layers used by the Dense models are replaced with MoE layers. Core components of Granite MoE architecture are: Fine-grained experts, Dropless Token Routing that ensures not a single input token is dropped by the MoE router regardless of the load imbalance among experts, and Load Balancing Loss as a strategy to maintain balanced distribution of expert load. 

Granite Guardian: leading safety guardrails

The new Guardian 3.0 8B and Granite Guardian 3.0 2B are variants of their respective correspondingly sized base pre-trained Granite models, fine-tuned to evaluate and classify model inputs and outputs into various categories of risk and harm dimensions, including jailbreaking, bias, violence, profanity, sexual content, and unethical behavior.

The Granite Guardian 3.0 models also cover a range of RAG-specific concerns, evaluating for qualities like groundedness (measuring the degree to which an output is supported by the retrieved documents), context relevance (gauging whether the documents retrieved were germane to the input prompt) and answer relevance. 

The model family is developer-friendly, offered under the Apache 2.0 license and accompanied by new developer recipes available in IBM’s Granite Community on GitHub. 

Deploy Granite models anywhere with NVIDIA NIM

NVIDIA has partnered with IBM to offer the Granite family of models through NVIDIA NIM – a set of easy-to-use microservices designed for secure, reliable deployment of high performance AI model inferencing across clouds, data centers and workstations.

NIM uses inference optimization engines, industry-standard APIs, and prebuilt containers to provide high-throughput AI inference that scales with demand.

NVIDIA NIM delivers best-in-class throughput, enabling enterprises to generate more tokens, faster. For generative AI applications, token processing is the key performance metric, and increased token throughput directly translates to higher revenue for enterprises and better user experience.

Get started

Experience the Granite models with free NVIDIA cloud credits. You can start testing the model at scale and build a proof of concept (POC) by connecting your application to the NVIDIA-hosted API endpoint running on a fully accelerated stack. 

Visit the documentation page to download the models and deploy on any NVIDIA GPU-accelerated workstation, data center, or cloud platform.