Camera Calibration (Omni.Replicator.Agent.Camera_calibration)#

Omni.Replicator.Agent.Camera_calibration provides camera information for cameras generated by the Omni.Replicator.Agent (ORA) extension. You can use this extension to select a top-view camera within the stage to create the stage layout. Details including camera direction, location, and FOV polygon information are also captured and saved as a JSON file. For each camera generated by ORA, the FOV polygon is visually represented in the stage layout.

Activate the Extension#

  1. On Window > Extension, search for “omni.replicator.agent.camera.calibration”.

  2. Enable the UI extension: omni.replicator.agent.camera_calibration.

Calibration Tool Components#

UI Layout#

an example of what the UI looks like for the camera calibration extension

Input Fields#

  • Place Info

    • Usage: Define place info in this field. Input is converted and recorded in the calibration.json file generated from the extension.

    • Default value: empty string

    • Input Format:

      • city=[city name]/building=[building name]/room=[room name]

      • Example:

        • Input: city=Santa Clara/building=NVIDIA Voyager/room=Visitor Lobby

        • Output: (in the calibration.json file)

        • {
    "place": [
        {
        "name": "city",
        "value": "Santa Clara"
        },
        {
            "name": "building",
            "value": "NVIDIA Voyager"
        },
        {
            "name": "room".
            "value": "Visitor Lobby"
        }
    ],
}

  • Output Folder Path

    • Usage: The address of the output information. Click on the folder icon to select the output folder address.

    • Default value: empty string

  • Raycast Density

    • Usage: The seed that affects the density of the ray cast. As you input a higher value in this field, it results in the generation providing a more detailed FOV contour. A seed value of N indicates N * N, which indicates that rays are uniformly distributed from each camera.

    • Default value: 100

  • Prim Path for Top View Camera Generation

    • Usage: Get prim path of the root node of the scene to calculate the top-down camera’s position and rotation.

    • Default value: empty string

  • Top View Camera Path

    • Usage: Record the prim path of the top-view camera. The view port of the top view camera is used to generate the top_view image.

    • Default value: empty string

    • Notice

      • The Top View Camera must be vertical to the ground and it must cover the position of all the calibration dots under the World/Calibration_Dots prim path and cameras under the World/Cameras prim path. Validate that the Top View Camera has a rotation [0,0,0] with projection set to “orthographic”.

      • The Top View Camera must be 1080P

  • Minimum FOV Polygon Edge Length (meter)

    • Usage: Specify the minimum length of edges in the polygon’s contour. Edges shorter than this length are ignored, and the vertices are connected to the next point that meets this criteria.

    • Default value: 0 (no simplification)

  • Minimum Area of FOV Polygon Hole to Ignore

    • Usage: When generating data, disregard holes in the FOV polygon that are smaller than this threshold value. Holes are the areas that are not included in the FOV polygon.

    • Default value: 0 (maintain every hole on the FOV polygons)

  • Create FOV Polygon Images

    • Usage: When generating top view image, render top view image with FOV polygon in debug data folder.

    • Default value: False (Do not generate the debug images)

Prerequisites#

Before generating the calibration.json file and generating the top view image, you must meet the following prerequisites (or an error message is thrown):

  • Top View Camera Path field is set up with a valid camera prim path, and the top view camera must be an orthographic projection camera with rotation [0,0,0].

  • The output folder path value is not empty.

  • The Place Info must be added and have the correct format.

  • To generate a calibration file for any camera, the camera must be set in the /World/Cameras prim path.

Button and Functions#

  • Create Top View Camera

    • Usage: Generate a top view camera prim for the scene. The camera prim is set in the /World/Top_Camera path.

    • Notice

      • Empty Prim Path for Top View Camera Generation value blocks this task.

  • Create Dot Prims

    • Usage: Generate calibration dot prims for each camera. Calibration dots must be randomly generated.

    • Notice

      • Empty Output Folder Path or Top View Camera Path values blocks this task.

  • Generate Calibration File

    • Usage: Create a file called “calibration.json” according to MDX format. Store the file in the specified output folder.

    • Notice

      • An empty Output Folder Path or Top View Camera Path value blocks this task.

      • You must run Create Dot Prims before generating the calibration file or an error message is thrown.

      • The top view camera must be set to 1080P.

  • Generate TopView Image

    • Usage

      • Generate the Top View image and store the image in the output folder.

      • A file called “imageMetadata.json” must be generated to record image information.

      • If the Create FOV Polygon Images is checked, the FOV polygon is visualized on the topview layout. The FOV images are generated in a debug data folder within the output folder.

    • Notice

      • An empty Output Folder Path or Top View Camera Path value blocks this task.

      • You must have the current camera viewport set to your top view camera.

      • The top view camera must be set to 1080P.

Walkthrough#

Here is a walk-through example that uses the calibration extension to generate camera calibration information. The Isaac Sim Full Warehouse: (.../Isaac/Environments/Simple_Warehouse/full_warehouse.usd) is used as the example scene. The following is an image of the environment:

Note

  • Stage unit must be in meters.

  • Make sure that there is a valid navmesh volume in the stage and that the navmesh only covers accessible areas.

Step 1#

Add two cameras to the /World/Cameras path as sample cameras. Ideally, adjust the camera’s position so that their view extends from the top to the ground.

  • You can use the people SDG panel by adjusting the Seed, setting Camera Number to 2, and clicking on Set Up Simulation to automatically generate two cameras in the Warehouse Scene.

  • Alternatively, you can create cameras and the /World/Cameras Xform manually. This would be the hierarchy of the camera prims inside the stage.

Tip

To create a camera in the stage manually, right-click the stage panel and then click Create > Camera.

../../_images/omni-replicator-character-camera_calibration-step1-create_camera.png

Step 2#

This step is about generating a top view with “orthographic projection” camera and “section tool” extension.

  1. Adjust the perspective camera’s height and view the scene from a far position.

  2. Search “Section” in the Window > Extension UI and enable the Section Tool Extension.

  3. Navigate to the Section option located under the Tools menu. Click on the button to open the Section panel.

    The “section tool” extension uses a horizontal plane to slice your scene.

  4. To create an optimal top view we would need to exclude objects like the ceiling, manipulate the transform of the plane /SectionTools/Section_Tool_Object.

Note

To manipulate the transform of the plane, adjust the xformOp:transform directly in Raw USD Properties under the Property Panel. You can safely ignore any error messages.

../../_images/omni-replicator-agent-camera-calibration-section-tool-workaround.png

  1. Uncheck Options > Display Section Manipulator to hide the plane after an ideal top view is generated.

  2. Add an orthographic projection top view camera to capture our top view image.

    ../../_images/omni-replicator-character-camera_calibration-step2-section-ui.png

    • To add a camera on the stage as the top view camera, set Prim Path for Top View Camera Generation to /Root, then click Create Top View Camera.

    • The Top View Camera Path is set automatically. You can then switch the viewport to top view camera.

    • Verify that your view is similar to this after switching the viewport to the top view camera:

Tip

To switch the viewport to the top-view camera, click the Camera Icon, then click the Calibration_Top_Camera camera.

../../_images/omni-replicator-character-camera_calibration-step2-switch-camera-to-top.png

  1. Adjust your viewport to 1080P, then change the Translate and Horizontal Aperture settings of the camera until it covers the entire scene.

Tip

To change the viewport to 1080P, click Settings > viewport > Render Resolution > HD1080P.

../../_images/omni-replicator-character-camera_calibration-step2-top-view-resolution.png

Step 3#

Input the scene information in the Calibration Tool panel:

  1. Input the place information in Place Info following the format:

    • For example, city=Santa Clara/building=Isaac Sim Warehouse/room=Warehouse

  2. Select the output folder path by clicking on the folder icon.

  3. Input the Calibration Seed, where the ideal range is 100 - 250. A higher value generates a more accurate FOV polygon.

  4. Input the top camera path into the Top View Camera Path field.

    • For example, /World/Top_Camera/Calibration_Top_Camera

Step 4#

Generate Calibration Dots for each Camera by clicking on the Create Dot Prims button.

Calibration dot prims are generated under /World/Calibration_Dots prim.

For each Camera prim under /World/Cameras, six calibration dots are generated. Those Xform prims are used to calculate the projection matrix for each camera.

All calibration dots are categorized under a parent prim that is named according to their target camera.

Note

You can switch your viewport to an arbitrary camera’s view to check whether all the calibration dots are visible.

Step 5#

  1. Generate a sample calibration file by clicking on Generate Calibration File button. This generates a calibration.json file at your output folder location.

  2. After the calibration.json file is generated. You can visualize the FOV in the stage by selecting your target camera.

Step 6#

To visualize the generated FOV polygon top view image, generate the image by clicking on Generate TopView Image button.

The Top View Camera’s view would be rendered and output in the result folder. An``imageMetadata.json`` file is also generated to record image information.

../../_images/omni-replicator-character-camera_calibration-step6-top.png

Note

If Create FOV Polygon Images is checked, for each camera, the FOV is recorded as a semi-transparent white FOV polygon on the top view image. The FOV polygon images would generated under [your output folder path]/Debug/fieldOfViewPolygon.

../../_images/omni-replicator-character-camera_calibration-step6-fov-polygon.png
On this page