Modern vision-language models (VLMs) can support tasks such as visual question answering, captioning, and image-text reasoning. In practice, however, the data needed to adapt these models may be distributed across institutions or organizations that cannot centralize their raw records.
Federated learning provides a way to coordinate training across these data-local sites. For VLMs, the challenge is not only orchestration. Sites may contribute different task or modality mixes, and model updates can be large enough to strain network bandwidth and server memory.
This post focuses on two design decisions for federated multimodal AI workflows: what model state should cross the network, and how should it be transferred and aggregated efficiently? It shows how NVIDIA FLARE coordinates federated training across sites and handles large model updates through externalization, tensor streaming, and disk-backed aggregation.
Those design questions also apply to unified multimodal models (UMMs), which support multiple modalities and tasks within a shared architecture. FedUMM, developed through a collaboration between William & Mary and NVIDIA, provides a concrete example by federating lightweight adapters over a frozen multimodal backbone.
FedUMM is supported by the NVIDIA Academic Grant Program, and received an Outstanding Student Paper Award at the FL@FM workshop at TheWebConf 2026.
Why are VLMs difficult to federate?
In a centralized vision-language experiment, images, captions, visual question answering examples, and generation prompts can feed one training pipeline. In a federated setting, those examples are distributed across sites with different data, task mixes, and operating constraints.
This creates two engineering problems. First, sites may train on different task or modality mixes, so the workflow must define what each client updates and how those updates are combined. Second, full-model updates can be expensive to serialize, transfer, and hold in server memory.
The first decision is therefore what to federate. Some approaches exchange distilled knowledge rather than model weights. Others freeze a pretrained backbone and aggregate only lightweight trainable components. CreamFL illustrates the first approach, while FedCLIP, FedPIA, and FedUMM illustrate the second. When larger updates are required, the system must also support streaming and memory-efficient aggregation.
NVIDIA FLARE, an open source, extensible Python SDK and framework for federated learning and collaborative computing, can support both parameter-efficient and full-model communication patterns.
Figure 1 shows the general workflow. Sites keep different mixes of images, text, and prompts local, while the server coordinates training and aggregates approved model updates. Large-object externalization, tensor streaming, and disk-backed aggregation help manage larger payloads.
Coordinating training across clients
Every NVIDIA FLARE job separates global coordination from local execution. The server schedules rounds and aggregates updates, while each client trains or evaluates against its local data. Site-specific preprocessing, prompt construction, and batching remain inside the client.
The NVIDIA FLARE Recipe API provides a concise starting point. The FedAvg recipe pairs a model with a client training script. The same recipe can be run in simulation or in a real provisioned multi-site deployment.
Before implementing the model, define the client update contract: what remains local, what may leave the site, which model components each client may update, and which metrics return to the server. When clients update different model components, the contract should also specify how those component-level updates are combined.
Moving and aggregating large model updates efficiently
A common baseline method for federated VLM training is fine-tuning and aggregating the whole model parameters. This will lead to large model updates. With many clients joining the federation, they create two distinct memory pressures: serializing and transferring one update, and holding several client updates in memory during aggregation. NVIDIA FLARE supports several features to tackle this challenge.
Externalize large objects
NVIDIA FLARE can replace large objects in a message with lightweight references and transfer the underlying data separately. This keeps the control message small and supports payloads that exceed the ordinary serialized-message limit. Built-in decomposers cover PyTorch tensors, NumPy arrays, and common FLARE structures; custom decomposers are needed only for application-specific object types.
Stream tensors
For PyTorch workflows, FLARE Tensor Downloader streams tensors incrementally using a pull-based protocol. Only the requested chunk is serialized at a time, reducing peak memory during model distribution. Chunk size can be tuned to balance request overhead against per-chunk memory. TensorFlow workflows use the traditional serialization path.
Offload aggregation to disk
Streaming reduces memory pressure during transfer, but the server may still need to hold several client updates during aggregation. This causes the server’s peak memory to grow linearly with the number of clients. In NVIDIA FLARE 2.8.0, tensor disk offload writes incoming PyTorch FedAvg updates to temporary safetensors files and loads them as needed, thus preventing the linear growth in CPU memory.
These mechanisms complement payload reduction from adapter-based training to enable full-model training, larger adapters, or federated learning with many clients.
For tested configuration examples, see the NVIDIA FLARE Recipe API, FLARE Tensor Downloader, and tensor disk offload documentation.
Federating lightweight adapters over a frozen VLM with FedUMM
FedUMM provides a concrete example of minimizing what crosses the network in NVIDIA FLARE. Each simulated client keeps a frozen BLIP backbone and trains LoRA adapters locally. NVIDIA FLARE coordinates the rounds and aggregates only the adapter updates. FedUMM is designed for generality with modality-specific encoders for vision, audio, and text, while its current experiments focus on vision-language.
The reported experiments evaluate VQA v2 and GenEval under Dirichlet-controlled heterogeneity with up to 16 clients. In an eight-client comparison, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round and improved VQA v2 by 0.7 points relative to full-model FedAvg. At eight clients, performance remained about 97% of the centralized reference on both benchmarks (Figure 2).
The evaluation uses simulated sites, synthetic partitions, and public general-domain benchmarks. It does not establish clinical performance or formal privacy guarantees; it shows that raw training data remains local within the simulated federated workflow.
FedUMM reduces the system burdens at its source by exchanging only small LoRA adapters. Not every AI workflow can do that. When clients must send larger updates, tensor streaming reduces memory pressure during transfer; when the server must aggregate updates from many clients, disk-backed aggregation reduces how much data must be held in memory.
Checklist for designing federated multimodal AI workflows
When designing a federated multimodal workflow, consider both what each client should contribute and how those updates will move through the system. The following checklist summarizes the key decisions for balancing model quality, communication cost, and system memory requirements.
- Define the update contract: Decide what stays local, what each client sends, and how the updates are combined.
- Minimize the payload: Exchange lightweight adapters when possible, and full-model updates only when the task requires them.
- Choose how updates move and aggregate: Use externalization and tensor streaming for large in-memory updates, together with disk offload on server when aggregation several updates would exceed server memory.
- Evaluate end-to-end: Measure model quality together with communication, runtime, memory use, data heterogeneity, and failures.
Get started building federated multimodal AI workflows
Start by defining the update contract for your workflow: what remains local, what model state each client may return, and which clients should contribute to each aggregation. Use the NVIDIA FLARE Recipe API to implement the workflow and validate it in simulation at the expected client count.
Next, choose the payload-handling mechanism that matches your bottleneck. Use large-object externalization and FLARE Tensor Downloader for large model updates, together with tensor disk offload when server-side aggregation memory becomes a constraint.
For a concrete adapter-based example, explore the paper, FedUMM: A General Framework for Federated Learning with Unified Multimodal Models, and its implementation in the NVIDIA FLARE repository. After establishing a working baseline, use Auto-FL to adapt and tune the federated experiment for your own datasets and tasks.
To learn more, join us for NVIDIA Flare Day 2026, a free online event that explores cutting-edge applications of federated learning across industries.
