Multitask Learning with Stochastic Interpolants

Multitask Learning with Stochastic Interpolants

Overview

This paper introduces a novel framework for training truly multi-task generative models based on a generalized formulation of stochastic interpolants. The key innovation is to replace the scalar time variable traditionally used in transport-based models with linear operators, enabling interpolation between random variables across multiple dimensional planes or setups. This dramatically expands the space of possible tasks a single model can perform, enabling applications like:

• Universal inpainting models that work with arbitrary masks
• Multichannel data denoisers with operators in the Fourier domain
• Posterior sampling with quadratic rewards
• Test-time dynamical optimization with rewards and interactive user feedback

Methodology

The core theoretical contributions are:

• Defining operator-based interpolants that generalize traditional scalar interpolation in dynamical generative models to higher-dimensional structures.
• Deriving multipurpose drifts and scores that enable this unified mathematical formulation to treat various generative tasks as different ways of traversing the same underlying space.
• Showing how this framework enables self-supervised generative models that can continuously learn over a wide purview of tasks without task-specific retraining.

Data & Experimental Setup

The authors evaluate their approach on several datasets and tasks:

• Image Inpainting and Sequential Generation: Tested on MNIST, CelebA, and Animal Faces HQ datasets. Used the Hadamard product interpolant for flexible inpainting and progressive generation.
• Posterior Sampling in the ϕ⁴ Model: Applied the framework to sampling from the posterior distribution of a statistical lattice field theory model.
• Maze Planning: Reformulated shortest path planning as a zero-shot inpainting problem using the Hadamard interpolant.

Architecture and training hyperparameters are provided in an appendix.

Results

• On the image inpainting benchmarks, the proposed method matched or outperformed state-of-the-art specialized inpainting approaches in both PSNR and SSIM metrics.
• For the ϕ⁴ model, the method was able to sample from the posterior distribution without retraining, as verified by the magnetization of the generated configurations.
• In the maze planning task, the method was able to generate entire trajectories between arbitrary points while respecting the maze constraints, avoiding the need for sequential Markov decision processes.

Limitations & Uncertainties

• The authors acknowledge that their approach increases the complexity of the initial learning problem, requiring models to learn a larger space of possible generation paths. However, they argue this upfront investment can be addressed through scale, with the flexibility gained compensating for the increased pretraining costs.
• The paper does not provide a comprehensive analysis of the computational costs or training time required for their proposed multi-task generative framework compared to task-specific models.

What Comes Next

The authors suggest that operator-based interpolants represent a meaningful step toward more versatile generative modeling, enabling amortization of learning across multiple tasks and post-training adaptation. Future work could explore:

• Practical considerations for balancing flexibility and computational efficiency in real-world implementations.
• Extending the framework to handle a broader range of generative tasks and application domains.
• Investigating the implications of this approach for reducing the environmental costs associated with training separate models for each task.