Neural (NuRec) Rendering#

Omniverse RTX can render NeRFs, 3D Gaussian Splats (3DGS), and 3D Gaussian Unscented Transforms (3DGUT) generated by tools such as the open-source 3DGRUT project.

Note

For best performance, it is highly recommended to run Kit with Fabric Scene Delegate (FSD) enabled. You can enable FSD either in the Edit > Preferences > Rendering menu, or with the --/app/useFabricSceneDelegate=true command-line setting.

Generating and Importing NuRec Primitives#

The easiest way to render these primitives is to use the included USDZ export in 3DGRUT.

For example, to train the bicycle example from the 3DGRUT repository with 3DGUT and USDZ export, run:

python train.py --config-name apps/colmap_3dgut.yaml \
   path=data/mipnerf360/bicycle out_dir=runs \
   experiment_name=bicycle_3dgut dataset.downsample_factor=2 \
   export_usdz.enabled=true

The USDZ assets generated this way can be loaded or referenced directly into your stage, the same as any other USD asset, such as by File > Import or by double-clicking or dragging-and-dropping the USDZ file into the stage from the content browser.

The imported NuRec primitive renders without any further steps.

Note

While the NuRec asset is loading, the viewport shows black for a few seconds. This is expected.

Converting a 3DGS .ply to USDZ#

If you have an existing 3DGS .ply model already trained, you can convert it to a USDZ for rendering in Omniverse using the following command from the 3DGRUT repository:

python -m threedgrut.export.scripts.ply_to_usd path/to/your/model.ply --output_file path/to/output.usdz

Using OmniNuRecVolumeAPI Properties in Kit#

The OmniNuRecVolumeAPI schema applied to the UsdVolVolume prim adds several properties to control how the NuRec asset is rendered:

Bounds and Offset#

Maximum and Minimum bounds define the extent of the bounding box in which the volume is rendered. By default, these are infinite. Setting them lower can be used to crop the volume.

Offset defines the position of the volume bounding box within the USD scene.

Proxy#

The Proxy relationship can be used to define up to four meshes that are used as geometric stand-ins for the NuRec volume for the purposes of shadow casting and catching, and physics. If Use Proxy Transform is enabled, then the NuRec volume is transformed by first proxy’s transform, otherwise it will use the transform of the volume prim.

The NuRec scene will always be rendered on top of the proxies, whatever their relative positions are. If Matte Object is enabled on the proxy prims, then the shadows received on the proxies are visible on the NuRec scene.

For example, this lab NuRec scene has a proxy mesh relationship with Matte Object enabled on the mesh. You can see shadows from the foreground robot cast on the NuRec floors and walls, and the robot has reflections of the NuRec scene in its chassis.

USD synthetic robot asset casting shadows onto NuRec environment

Note

For matte object compositing to be enabled, a dome light must also be present in the scene.

When matte object is enabled on the proxy mesh, then depth compositing is performed using the depth of the proxy mesh and not the depth of the NuRec scene.

NuRec primitives are not rendered in synthetic data passes, so the proxy meshes must provide synthetic data such as semantic IDs to the synthetic data AOVs.

Color Correction#

The color and brightness of an emissive field can be adjusted to help integrate it with other scene elements and post-processing pipelines.

The color-correction matrix (CCM) attributes can be used to specify a 3x3 CCM to adjust the colors in the NuRec volume:

float4 omni:nurec:ccmB = (0, 0, 1, 0)
float4 omni:nurec:ccmG = (0, 1, 0, 0)
float4 omni:nurec:ccmR = (1, 0, 0, 0)

The exposure attribute can be used to adjust the brightness of the NuRec volume using a logarithmic scale.

float omni:nurec:exposure = 0.0

*NuRec rendering of a robotics lab with default settings*#

*The same NuRec volume adjusted to blue with the CCM attributes*#

*The same NuRec volume exposed up 1 stop using the exposure attribute*#

Known Limitations#

Camera Projection Support#

Neural rendering supports pinhole (the default), OmniLensDistortionOpenCvPinholeAPI, OmniLensDistortionOpenCvFisheyeAPI and OmniLensDistortionFThetaAPI distortion models. Attempting to use other models returns an error.

Note that for 3DGS scenes, rendering with anything other than pinhole leads to artifacts since 3DGS does not support lens distortions.

Multi-GPU Support#

In Real-Time Mode (1.0) rendering a single viewport with MGPU enabled may lead to a black viewport. Disable MGPU in the Render Settings to show the NuRec rendering correctly.

Proxy Meshes#

Using the proxy mesh as a matte object in order to simulate shadows and reflections requires the geometry of the proxy mesh to be close to that of the NuRec scene. Having a NuRec scene too far from the proxy leads to white artifacts where the two do not match.

Reflections are currently simulated by raytracing against the proxy mesh, thus the quality of the reflections are dictated by the proxy mesh.

NuRec scenes are not part of the synthetic data rendering. As a consequence, synthetic data AOVs (e.g. semantics, instance IDs) do not include any NuRec information.

Tiled Rendering#

Currently, NuRec scenes cannot be rendered as part of a tiled rendering pass.