OmniGraph#

OmniGraph is Omniverse’s visual programming framework. It provides a graph framework that connects functions from multiple systems inside Omniverse. It is also a compute framework that allows for highly customized nodes so that you can integrate your own functionality into Omniverse and automatically harness the efficient computation backend.

Inside Omniverse Isaac Sim, OmniGraph is the main engine for the Replicators, ROS and ROS2 bridges, sensor access, controllers, external input/output devices, UI, and much more.

This tutorial introduces you to the world of visual programming via OmniGraph. We highly recommend that you also read OmniGraph, because it is a key component in Omniverse Kit.

Learning Objectives#

This tutorial aims to

  • introduce you to the basic concepts of OmniGraph

  • walk you through building an action graph to control a robot in Isaac Sim, specifically, the Jetbot.

  • show you how to use the Omnigraph shortcuts to generate a differential controller graph for the Jetbot.

Getting Started#

Prerequisites

Graph Editors

There are two graph editors, the Action Graph and the Generic Graph. They can both be found under ‘Window > Visual Scripting’. For majority of cases in Isaac Sim, you will be using the Action Graph.

Node Search

There are ample nodes available in OmniGraph for your use. Once your graph editor is open, there is panel on the left hand side listing all the categories of the nodes. You can search for individual nodes by typing into the search bar on the top of the panel.

Fundamental Concepts#

Action Graph vs Push Graph

An Action graph, or sometimes called an “Execution Graph”, is executed whenever an execution node gets triggered. In many Isaac Sim cases, the trigger is a “tick” node that is set to be triggered every simulation frame tick, so that the graph doesn’t do anything when simulation isn’t running, and ticks at every rendering frame once simulation starts. If you remove the tick node from an action graph, the graph will not run upon pressing “play”.

Action Graphs can have other triggers, such as a keyboard/mouse input, or a stage event. You can find the relevant nodes under the “Event” nodes category.

A Push graph, or sometimes “Generic Graph”, will execute automatically on every rendering frame, without needing an “execution” node. Since the default viewport is always running and therefore rendering, any nodes you drag into a Push Graph editor will immediately start executing, even if the simulation isn’t “playing”. You can test this by dragging a “Print Text” node into a Push Graph editor, open your console tab and allow it to publish Informational level logs. Then the moment you finish typing something in the “Text” section of the print node, the text will get printed.

If there is a procedural order to your graph based on the connections, then a Push graph will compute the nodes in that procedural order. Otherwise, it’ll execute all of the nodes in no guaranteed order.

Try It Out#

Let’s build an action graph to control a robot in Isaac Sim the Jetbot.

Setting Up the Stage#

  1. On a new stage, start by right clicking and selecting create > Physics > Ground Plane.

  2. Use the content browser to navigate to /Isaac/Robots/JetBot.

  3. Click and drag jetbot.usd onto the stage.

  4. Position the JetBot just above the ground plane.

  5. When completed, verify that the JetBot is under /World/jetbot in the context tree and that the stage looks similar to:

../_images/isaac_tutorial_omnigraph_jetbot.png

Jetbot on the stage#

Note

Click play! Validate that the JetBot falls and lands on the stage. Click stop before continuing.

Depending on your default render settings, the camera of the JetBot may have a placeholder mesh (it looks like a gray television camera). To hide these meshes, click on the eyecon icon in the viewports and select Show By Type –> Cameras.

Building the Graph#

  1. Select Window -> Visual Scripting -> Action Graph from the dropdown menu at the top of the editor. The Graph Editor appears in the same pane as the Content browser.

  2. Click New Action Graph to open an empty graph.

  3. Type controller in the search bar of the graph editor.

  4. Drag an Articulation Controller and a Differential Controller onto the graph.

The Articulation Controller applies driver commands (in the form of force, position, or velocity) to the specified joints of any prim with an articulation root.

To tell the controller which robot it’s going to control:

  1. Select the Articulation Controller node in the graph and open up the property pane.

  2. You can either:

    • Click usePath and Type in the path to the robot /World/jetbot in robotPath, OR

    • Click Add Targets near the top of the pane for input:targetPrim and select JetBot in the pop up window.

The Differential Controller computes drive commands for a two wheeled robot given some target linear and angular velocity. Like the Articulation Controller, it also needs to be configured.

  1. Select the Differential Controller node in the graph.

  2. In the properties pane, set the wheelDistance to 0.1125, the wheelRadius to 0.03, and maxAngularSpeed to 0.2.

  3. Validate that the properties of the controllers look like the following:

../_images/isaac_tutorial_omnigraph_targetPrim_or_robotPath.png

Specifying either target prim or robot path. Not both!#

../_images/isaac_tutorial_omnigraph_differential_node.png

Differential Node#

The Articulation Controller also needs to know which joints to articulate. It expects this information in the form of a list of tokens or index values. Each joint in a robot has a name and the JetBot has exactly two. Verify this by examining the JetBot in the stage context tree. Within /World/jetbot/chassis are two revolute physics joints named left_wheel_joint and right_wheel_joint.

../_images/isaac_tutorial_omnigraph_tree.png

Stage Tree#

  1. Type token into the search bar of the graph editor.

  2. Add two Constant Token nodes to the graph.

  3. Select one and set it’s value to left_wheel_joint in the properties pane.

  4. Repeat this for the other constant token node, but set the value to right_wheel_joint.

  5. Type make array into the search bar of the graph editor.

  6. Add a Make Array node to the graph.

  7. Select the Make Array node and click on the + icon in the inputs section of the property pane menu to add a second input.

  8. Set the arraySize to 2 and set the input type to token[] from the dropdown menu in the same pane.

  9. Connect the constant token nodes to input0 and input1 of the Make Array node, and then the output of that node to the Joint Names input of the Articulation Controller node.

The last node is the event node.

  1. Search for playback in the search bar of the graph editor.

  2. Add an On Playback Tick node to the graph. This node emits an execution event for every frame, but only while the simulation is playing.

  3. Connect the Tick output of the On Playback Tick node to the Exec In input of both controller nodes.

  4. Connect the Velocity Command output of the differential controller to the Velocity Command input of the articulation controller.

  5. Validate that the graph looks similar to:

../_images/isaac_tutorial_omnigraph_jetbot_minimal.png

Simple differential control for the JetBot#

  1. Press the play button.

  2. Select the Differential Controller node in the graph.

  3. Click and drag on either the angular or linear velocity values in the properties pane to change it’s value (or just click and type in the desired value).

Note

Explore the available OmniGraph nodes and try to setup a graph to control the JetBot with the keyboard. The graph below is an example graph for controlling the JetBot with a keyboard.

../_images/isaac_tutorial_omnigraph_full.png

Keyboard control Action graph for the JetBot#

Omnigraph Shortcuts#

Putting the graph from scratch can be tedious, especially when you have to iterate. We made some shortcuts for frequently used graphs, so that within a couple clicks, you can generate a complex graph with multiple nodes and connections. They can be found under Isaac Utils -> Common Omnigraphs, and the instructions for them are in Commonly Used Omnigraph Shortcuts.

To use the Differential Controller graph from the menu shortcut:

  1. Delete (or Disable if that is an option) any previous Omnigraphs that controls the Jetbot.

  2. Go to the Menu bar and click on Isaac Utils -> Common Omnigraphs -> Differential Controller.

  3. You are prompted for the necessary parameters.

  4. Add “/World/jetbot” to Articulation Root, set the distance between wheels to 0.1125, and the wheel radius to 0.03.

  5. Given JetBot only has two controllable joints, you can leave the rest of the fields empty.

  6. Turn Use Keyboard Control (WASD) on.

  7. Click OK to generate the graph. You can open the generated graph under /Graph/differential_controller.

  8. Press Play to start simulation.

  9. Verify that you can move the JetBot using the WASD keys on the keyboard.

Summary#

This tutorial covered:

  • Basic concepts of OmniGraph

  • Setting up a stage with a robot

  • Using OmniGraph to construct interfaces to a robot

  • Using the Omnigraph shortcuts to generate differential controller graph

Further Learning#

On this page