Text-to-image diffusion models can generate diverse, high-fidelity images based on user-provided text prompts. They operate by mapping a random sample from a high-dimensional space, conditioned on a user-provided text prompt, through a series of denoising steps. This results in a representation of the corresponding image, .

These models can also be used for more complex tasks such as image editing, learning to depict a personalized concept, or semantic data augmentation. In this context, image editing refers to the task of making local changes to a given image based on a text prompt, while the other parts of the image remain unchanged.

All these additional tasks involve a process called *inversion*: Given an image representation and its corresponding text prompt , you seek a noise seed that, when fed into the denoising process, yields the reconstructed image .

Regularized Newton-Raphson Inversion (RNRI), a novel inversion technique, was recently proposed. RNRI outperforms existing inversion approaches by balancing rapid convergence with superior accuracy, execution time, and memory efficiency, enabling real-time image editing for the first time.

## Inversion as solving an implicit equation

Inverting a diffusion model requires searching in the space of possible seeds for one that would reconstruct a given image. This search may be computationally demanding.

To understand how it can be achieved efficiently, consider first the forward (noising) process.

Sampling from diffusion models can be viewed as solving an ordinary differential equation. The popular DDIM deterministic scheduler presented in Denoising Diffusion Implicit Models denoises a latent noise vector in the following way:

Equation 1

In this equation (equation 1), , and .

## DDIM inversion

To derive inversion, the first equation is first rewritten as follows:

Equation 2

This gives an implicit equation in , that cannot be solved in closed form. DDIM inversion approximates it by replacing with :

Equation 3

DDIM inversion is a fast method but often an inaccurate inversion.

## Fixed-point and gradient descent inversion methods

Several papers improve the previous approximation using iterative methods to approximately solve the second equation. For example, directly solving the equation using fixed-point iterations is a method widely used in numerical analysis for solving implicit functions. For more information, see Effective Real Image Editing with Accelerated Iterative Diffusion Inversion.

In a related way, a more precise inversion equation can be solved, obtained by employing higher-order terms using gradient descent. For more information, see On Exact Inversion of DPM-Solvers.

Fixed-point iterations and gradient descent methods provide better accuracy than DDIM, but have a linear convergence rate and may take many seconds to compute.

## Regularized Newton-Raphson Inversion method

A faster and more accurate alternative is based on the well-known Newton-Raphson iterative method (NR).

NR is a method for iteratively finding the roots of a system of equations. A naive application of NR to the full latent space would require solving . This formulation is impractical because it requires inverting a high-dimensional Jacobian matrix.

Instead, define a multivariable scalar function :

Equation 4

Seek its roots . Because is a scalar function, the Jacobian matrix is a vector and can be computed quickly.

Solving equation 4 can be done quickly, but its solutions are not guaranteed to reconstruct the image well because the equation is underdetermined. Also, some roots of may be out of distribution for the diffusion model.

To address this issue, add a regularization term to the NR objective:

Equation 5

As each noising step in the diffusion process follows a Gaussian distribution, it is incorporated as a prior over the values of . The negative log-likelihood is added as a regularizing penalty term, forming the objective:

Equation 6

The Newton-Raphson iteration for this function can be computed efficiently using automatic differentiation engines, initializing the process with from the previous diffusion timestep. Regularized Newton Raphson Inversion (RNRI), converges in 1β2 iterations (~0.5 sec for latent consistency models).

Figure 2 compares the quality of reconstructed images (measured using PSNR) of the COCO validation set, against the time it takes to compute the inversion. It shows that RNRI improves in terms of PSNR or run time over recent methods. For a fair time comparison, run time is measured on a single NVIDIA A100 GPU for all methods. The dashed black line denotes the upper bound that is due to the inherent distortion caused by the Stable Diffusion VAE.

*Figure 2. Inversion results for PSNR compared to runtime*

Figure 3 provides a qualitative comparison between RNRI and previous state-of-the-art inversion approaches. It shows cases where RNRI accurately edits images that have high fidelity with the input image and also adhere well to the target prompt. Alternative approaches may struggle with editing these images and prompts. Baselines were run until they converged, whereas RNRI was run for only two iterations per diffusion step.

For example, in the first row, RNRI succeeds in converting the pizza into slices of bread. Other methods either fail to achieve this or incorrectly modify other elements. In the third row, all methods struggle to accurately substitute bananas with oranges or alter the background. In contrast, RNRI accurately edits the object while maintaining the original background.

*Figure 3. RNRI edits images more naturally while preserving the structure of the original image*

### Evaluation of RNRI results

Following the previous work, editing performance is measured using the following metrics:

- An LPIPS score quantifies the extent to which structure is preserved (lower is better).
- A CLIP-based score quantifies how well the generated images match the text prompt (higher is better).

Values are averages across 100 MS-COCO images. Figure 4 shows that editing with RNRI yields a superior CLIP and LPIPS score, achieving state-of-the-art editing of real images.

*Figure 4. RNRI achieves superior CLIP and LPIPS scores, indicating better compliance with text prompts and higher structure preservation*

Finally, Figure 5 shows additional real-time editing results.

*a) A laptop, a cup of milk, a black pen, and a smartphone*

*b) A cartoon of a wooden board with tomatoes, grapes, and orange on the green tablecloth*

*Figure 5. Examples of real-time image editing using RNRI*

## Conclusion

Image inversion in diffusion models is key for applications like image editing, semantic augmentation, and generating rare-concept images. Current methods often sacrifice inversion quality for computational efficiency, requiring significantly more resources for high-quality results.

Regularized Newton-Raphson Inversion (RNRI) balances rapid convergence with superior accuracy, execution time, and memory efficiency. The RNRI method outperforms existing approaches in both latent diffusion and latent consistency models, enabling real-time image editing.

For more information, see the full paper, Regularized Newton Raphson Inversion for Text-to-Image Diffusion Models. You can also try RNRI yourself.