Linearly Transformed Spherical Harmonics

Linearly Transformed Spherical Harmonics

  • Post category:Graphics


EDIT 4.12.20: Readers pointed out, that one of the normal channels seems to be inverted in my demo renderings. Sorry for overlooking this. I’ll update my demo code and the images as soon as I’m back at my desktop PC (around mid December). Thanks to the people who noticed this and notified me.

Here I present my Bachelor’s Thesis and its implementation about Linearly Transformed Spherical Harmonics (LTSH). To keep things short I expect readers to be familiar with bidirectional reflectance distribution functions (BRDFs). If this is new to you, check out section 5.2 of my thesis.
Direct lighting with area lights is complicated, as it involves integration of the BRDF over spherical domains. Usually, modern BRDFs are too complex to “simply integrate” them. This calls for creative alternatives to solve the integral in real-time applications. Sampling the light source is too expensive for most use cases (although there is a shift happening through the advances in real-time ray tracing).
An existing technique uses linearly transformed cosines to approximate BRDFs and performs shading with them. They achieve good approximations, especially for highly specular BRDFs but the limited flexibility of the cosine function becomes apparent for rough BRDFs and at grazing angle. We swap out the rigid cosine function with more versatile spherical harmonic (SH) expansions to improve the quality of the BRDF approximation. Our technique produces realistic renderings and offers improvement in image quality compared to linearly transformed cosines, but it performs significantly slower.

 

Linearly Transformed Cosines

In 2016, Heitz et al. derived Linearly Transformed Cosines (LTC). There is a short intuitive introduction to the technique at the linked page, so I’ll only give it a quick rundown here.
The idea is to linearly transform a cosine function so it matches the BRDF as close as possible. Now we can approximate the integral of the BRDF with the integral of the linearly transformed cosine function. And the integral over the light polygon of the linearly transformed cosine is equivalent to the integral over the inversely transformed polygon of the untransformed cosine function. We need to account for a Jacobian term of our linear transformation but I’ll leave this out of here for brevity. Since the cosine function possesses an analytic integral, we can now solve the rendering integral.

 

 

Integral over light polygon of linearly transformed spherical function (left) is equivalent to integral over inversely transformed light polygon of original spherical function (right)

At a specific shading point we have the view direction and material properties. For the GGX BRDF this would mean we have four input values: the view direction consisting of two angles θ and ϕ, a surface roughness a  and an index of refraction. To reduce the dimensionality of the problem, we only look at isotropic BRDFs to drop the ϕ-component of our view direction. We also use a fixed value for the index of refraction. With this we precompute a 2D look-up table. Heitz et al. achieved good results with a 64×64 table of value pairs for the θ component of the view direction and a roughness value a and we will follow their example. 

 

Our approach

As the name of our technique suggests, LTSH is pretty close to LTC. Instead of using the cosine function to approximate the BRDF, we use spherical harmonic (SH) expansions. Spherical harmonics are “Fourier expansions on the sphere” figuratively spoken. By taking linear combinations of the SH basis functions, we can approximate any spherical function. Much like Fourier expansions, the higher the order of your SH expansion the closer your approximation gets as higher frequencies are added in. But directly using an SH expansion to approximate the BRDF would need a lot of SH bands, especially for highly specular BRDFS and is thus not feasible. However, by linearly transforming the BRDF, good approximations are found even for low order SH expansions.
Spherical harmonics become an eligible candidate because both Belcour et al. and Wang and Ramamoorthi simultaneously derived a closed form expression for the integration of SH expansions over spherical domains. In their paper, Belcour et al. state “One avenue of future work could combine these ideas to explore the expressive power of linearly-warped SH.” which is ultimately where the idea for my thesis came from. So we approximate the BRDF with LTSH and then calculate the integral over the inversely transformed light polygon of the original SH expansions. This adds some degrees of freedom to the BRDF approximation process as we now modify both the linear transformation and the SH expansion itself. A useful side fact: the cosine function is a subset of SH expansions with a degree of 1 or higher, so LTC is a subset of LTSH which means the quality of our technique is never worse than LTC in theory.

I won’t go into detail of the BRDF approximation or how the SH integral is actually solved, as it is mathematically involved and would be repetitive to what I wrote in my thesis. Refer to section 6 for BRDF fitting and section 7 for shading with LTSH.

 

The results

Now I showcase some rendering of the implementation. If you want to try it out yourself, there is a precompiled Windows release including the sun temple scene (sorry for the size :D). Go ahead and play around, move the light source and render using different techniques. Since the technique only accounts for specular shading, using the specular debug view helps in spotting differences between techniques. Here is an exemplary screenshot:

Rendering of the sun temple scene provided by Epic. Our technique (LTSH of order 4) produces plausible results with normal and roughness mapped surfaces and complex geometry.

Although such images look nice, they tell little about the quality of our technique. So let’s look at a comparison of the specular highlight on a simplistic test scene (use the arrows/swipe on mobile to switch between different techniques):

LTC
LTSH order 2
LTSH order 4
Ground Truth
previous arrow
next arrow
Slider

The shape of the specular highlight grows closer to ground truth from LTC to LTSH of order 4, but an artifact on the area behind the light source is over lit.

In the next scenario, we show significant improvement for rough surfaces. LTC has an unnatural highlight due to the rigidity of the cosine function. Even the LTSH of order 2 removes this artifact.

LTC
LTSH order 2
LTSH order 4
Ground Truth
previous arrow
next arrow
Slider

The problem is seen when looking at the BRDF approximations. The BRDF has important contributions over the whole hemisphere. The linear transformation cannot transform the cosine in a way that matches such a broad distribution. With our more versatile SH expansion however, we achieve very close approximations in this scenario.

 

 

“Backside” (opposite of the highlight) of LTC (left), LTSH of order 4 (middle),
and GGX (right); roughness = 0.63

However, our technique also introduces numerical instability. It becomes apparent when the ratio intensity/area of the light source grows large. We inherit this instability from the SH integration implementation of Wang and Ramamoorthi. For most of the cases this can be avoided by using point lights instead of tiny area lights, but using this technique to emulate line lights or other shapes with small surface areas is not advised.

To give an impression of the quality improvements of LTSH over LTC we look at the square root of the mean squared error over the whole hemisphere. Please note, that we used simple equidistant sampling in the angular domain in our BRDF fitting process. This makes our results prone to error especially for low roughness values. To be fair, the LTC errors presented here are likely worse than those of the original LTC implementation, as they used more sophisticated sampling for their BRDF fitting. But since our technique would benefit from this sampling in the same way as our LTC implementation would, I think it’s still fair to use those values for the sake of comparison here. The colormap is capped at 0.1.

Square root of mean squared error over hemisphere of LTC (left), LTSH order 2 (middle) LTSH order 4 (right). Visualization for the whole 64×64 look-up table. θ represents the view direction (90 is grazing angle) and a is the roughness. 

As you can see, a clear tendency of improvement is visible from LTC over LTSH order 2 to LTSH order 4. Especially the problems that LTC has with highly rough surfaces, as discussed earlier, are visible here. 

Another important factor are run times. Our technique introduces significantly higher run times than LTC. Here it is important to note, that our implementation is only a proof of concept. We did not have the time to go into a lot of optimizing. Also, the SH integration code provided Wang and Ramamoorthi is optimized for SH expansions of order 8 and does not provide optimal run times for lower order integration. Since the number of SH basis functions grows quadratically in the SH order, we expect the run time of our technique to also grow quadratically when increasing the number of SH bands used. The baseline is the application running without doing any lighting calculations. For more quality examples and a broader discussion visit the thesis section 7.

Baseline: 0.14ms, LTC: 0.44ms, LTSH 2: 0.95ms, LTSH 4: 1.78ms

Conclusion

Our technique offers quality improvement over LTC that comes at a cost. The frame cost for LTSH is significantly higher and grows with the number of SH bands used in a quadratic manner. But using LTSH expansions of different orders makes this approach adaptive to different needs in quality/run times. Unlike LTC, we do not offer textured area lights and alike LTC there is no support for shadows. Our technique needs further investigation and optimization before it is suitable for production use but it is an interesting alternative to LTCs. 

I had a lot of fun while researching this and hope my results prove helpful to someone in the future, so I’m happy to share it here. I’d like to thank my advisor Christoph Peters for the large amount of time he put into this thesis and providing me with useful tools for example to create visualizations of spherical functions or an SH evaluation tool. For other graphics related posts check out his blog. Also thanks to NVIDIA for providing the Falcor framework and the ORCA library to help aspiring researchers.

Jan Allmenröder

Computer Graphics and Rendering Enthusiast | Currently doing my Masters degree in Computer Science | Also interested in social/ecological topics