Basic principle

Controlling caustic pattern is rather a challenging problem as very minor changes of the surface will significantly affect the quality of the pattern since light ray directions might be interfered by other light rays as they intersect with and refract through the material. This will lead to a scattered, discontinuous pattern. To tackle this problem, optimal-transport-based is one of the existing proposed methods to control caustic pattern by redirecting light's directions as it propagates through the surface of a certain transparent material. This is done by solving an inverse optimization problem based on optimal transport.^[13][14] Given a reference image of an object/pattern, the target is to formulate the mathematical description of the material surface through which light refracts and converges to the similar pattern of the reference image. This is done by rearranging/recomputing the initial light intensity until the minimum of the optimization problem is reached.

Design pipeline

Here considering only refractive caustic, the objective can be determined as follows (similar principle for reflective caustic with different output):

Input: image of pattern to be obtained after propagating lights through the material, given the light source position.

Output: caustic geometry on the receiver (flat solid surface, e.g.: floor, wall, etc...)

In order to achieve the target pattern, the surface where light refracts through and exits to the outer environment must be manufactured into certain shape to achieve desired pattern on the other side of the material.

As mentioned, given an input image, this process will produce the similar caustic pattern as the output. In principle, there are two core stages with each includes two sub-stages:

• 

  Optimal-transport-based caustic design
  Solving Optimal Transport Problem
  1. Compute Target Light Distribution
  2. Compute Mapping from Initial Distribution to Target Distribution
• Optimizing Target Surface
  1. Compute Normal Representation of Surface
  2. Surface Refinement

Solving optimal transport problem

As the case refraction occurs through a transparent surface, for instance the patterns appearing under clear water surface, 3 main phenomena can be observed:

• Very bright (condensed light intensity) points (so-called singularity)
• Curve-like objects that connect the points
• Regions with low light intensity

To perform computation, the following 3 quantities are being respectively introduced to describe the geometric characteristics of the pattern: point singularity ${\displaystyle \Phi _{p}}$ (measuring light intensity at certain highly concentrated light-point), curve singularity ${\displaystyle \Phi _{c}}$ (measuring light intensity at/around a light-curve), and irradiance measure ${\displaystyle \Phi _{I}}$ (measuring intensity in a certain poorly concentrated light-area). Putting them altogether, the following function defines the total radiant flux measure ${\displaystyle \Phi _{T}}$ at a certain section Ω on the target surface:

${\displaystyle \Phi _{T}(\Omega )=\textstyle \sum _{i=1}^{N_{\delta }}\Phi _{p}^{i}(\Omega )+\sum _{j=1}^{N_{\gamma }}\Phi _{c}^{j}(\Omega )+\Phi _{I}(\Omega )\displaystyle }$

After this step, there are two existing measures of the radiant flux measures of the source ${\displaystyle \Phi _{S}}$ (uniform distribution, by initialization) and the target ${\displaystyle \Phi _{T}}$ (computed in previous step). What remains to compute is the mapping from the source to target. In order to do this, there are several quantities to be defined. Firstly, two light intensities evaluated by probabilities: ${\displaystyle \mu _{S}}$ (light intensity evaluated by dividing ${\displaystyle \Phi _{S}}$ by the flux of the union region between ${\displaystyle \Phi _{S}}$ and ${\displaystyle \Phi _{T}}$), ${\displaystyle \mu _{T}}$ (light intensity evaluated by dividing ${\displaystyle \Phi _{T}}$ by the flux of the union region between ${\displaystyle \Phi _{S}}$ and ${\displaystyle \Phi _{T}}$) are defined. Secondly, the source mesh is generated as multiple sites ${\displaystyle s_{i}\in (\Phi _{S}\cup \Phi _{T})}$, which is later being deformed. Next, a power diagram ${\displaystyle P_{w}}$ (a set of ${\displaystyle C_{i}^{w}}$ power cells) is defined on this set of sites ${\displaystyle s_{i}}$ weighted by a weight vector ${\displaystyle \omega }$. Finally, the goal is to decide whether which power cells are going to be move. Considering all ${\displaystyle x}$ vertices on the surface, finding the minimizer ${\displaystyle \omega _{min}}$ of the following convex function will produce the matched power diagram for the target:

${\displaystyle f(\omega )=\sum _{s_{i}\in S}(\omega _{i}.\mu _{S}(C_{i}^{0})-\int \limits _{C_{i}^{\omega }}(\|x-s_{i}\|^{2}-\omega _{i})d\mu _{T}(x))}$

Optimizing target surface

Computation process

After solving optimal transport problem, the vertices are achieved. However, this gives no information about what the final surface should look like. To achieve the desired target surface given the incoming light ray ${\displaystyle d_{I}}$, outgoing light ray ${\displaystyle d_{O}}$ and power diagram from the step above, the surface normals representation can be computed according to Snell's law as:

${\displaystyle n=d_{I}+{\eta x \over \lVert d_{I}+\eta d_{T}\rVert }=d_{I}+{\eta x \over \lVert d_{I}+\eta {(x_{R}-x) \over ||x_{R}-x||}\rVert }}$

where,

${\displaystyle \eta }$: refraction coefficient

${\displaystyle x_{R}}$: target position obtained from solving above optimal transport problem

As the normal representation is obtained, surface refinement is then achieved by minimizing the following compound energy function:

${\displaystyle {\underset {x}{\operatorname {arg\,max} }}\,\omega \,\cdot [E_{int},\,E_{dir},\,E_{flux},\,E_{reg},\,E_{bar}]}$

where,

${\displaystyle E_{int}=\sum _{v\in M_{T}}\lVert n_{o}-n\rVert _{2}^{2}}$ is the integration energy that aligns the vertex normals ${\displaystyle n_{o}}$ obtained from the Optimal Transport with the target normals ${\displaystyle n}$ obtained from the Snell's law computation above.

${\displaystyle E_{dir}=\sum _{v\in M_{T}}\lVert x-{\textbf {proj}}_{(x_{S},\,d_{I})}(x)\rVert _{2}^{2}}$ as mesh generated in step Solving Optimal Transport cannot adapt to the sharp instances from the discontinuities, this energy is to penalize the vertices to not change significantly from the incoming light ray.

${\displaystyle E_{flux}=\sum _{t\in M_{T}}\lVert \Phi _{T}(t)-\Phi _{S}(t_{s})\rVert _{2}^{2}}$ is the energy measuring the flux over the triangle ${\displaystyle t}$ in the mesh.

${\displaystyle E_{reg}=\lVert {\textbf {L}}{\textbf {X}}\rVert _{2}^{2}}$ is the energy that regularizes the shape of the triangles to maintain its well-shapedness.

${\displaystyle E_{bar}=\sum _{v\in M_{T}}\lVert max(0,-log((1-n_{R}.(x-x_{R}))+d_{TH})\rVert ^{2}}$ is barrier energy to ensure that surface does not deform beyond a certain distance threshold ${\displaystyle d_{TH}}$.

Basic principle

Inverse graphics is a method of observing the data from an image and inferring all possible properties including 3D geometry, lighting, materials, and motion in order to generate a realistic image.^[15] In conventional computer graphics, to render an image with desired appearance and effects, it is given all the relevant properties/characteristics. This could be described as the forward method. On the contrary, in caustic design, the properties and characteristics of objects (especially the material surface) are not trivial. The given constraint is the target image to obtain. Therefore, the goal is to obtain its properties and characteristics by observing and inferring the target image. This can be considered the inverse/backward method.

The following is the basic loss function explaining how to optimize the parameters:

${\displaystyle L(c)=\lVert f(c)-I\rVert ^{2}}$

where,

L(c): loss function, mean square error of the rendered image and the target

c: contains elements which can influence the generated image

I: target image

Designed pipeline

Differentiable inverse rendering caustic design

At first, the target pattern is designed and the forward pass computed to get the synthetic pattern. It's compared to the target pattern and get the loss. The objection is to let the synthetic pattern is similar to the target pattern as much as possible. And then do the back propagation to get the optimized properties need to use in caustic manufacturing.

Elements contributing to generated image

• Appearance (${\displaystyle A}$): per-pixel surface appearance is modeled as product of mipmapped texture and per-pixel brightness.
• Geometry (${\displaystyle V}$): assume a 3D scene to be approximated by triangles, parameterized by vertices ${\displaystyle V}$.
• Camera (${\displaystyle C}$): focal length, the point of view, the center of the camera.

There could be more elements, for example albedo and refraction coefficient.

General differentiable framework

Introduce U as an intermediate variable indicating 2D projected vertex coordinate positions. The gradient of these properties can be derived by chain rule indirectly.

${\displaystyle {\frac {\partial f}{\partial V}}={\frac {\partial f}{\partial U}}\times {\frac {\partial U}{\partial V}}}$

${\displaystyle {\frac {\partial f}{\partial V}}={\frac {\partial f}{\partial A}}\times {\frac {\partial A}{\partial V}}}$

${\displaystyle {\frac {\partial f}{\partial C}}={\frac {\partial f}{\partial U}}\times {\frac {\partial U}{\partial C}}}$

After applying the stochastic gradient descent, the optimal ${\displaystyle A}$, ${\displaystyle V}$ and ${\displaystyle C}$ could be achieved. Subsequently, these quantities are used to carve or mill the material to generate the target pattern.

Implementation

One common approach is to utilize the ability to perform differential operations in various deep learning auto-differentiation frameworks/libraries such as: Tensorflow, PyTorch, Theano.

One more approach is to make use of the OpenDR^[16] framework to build a forward graphics model and to automatically obtain derivatives with respect to the model parameters for optimization. As optimization properties are obtained, the target image can be generated. OpenDR provides a local optimization method that can be incorporated into probabilistic programming frameworks. This can be used to solve the problem of caustic.