OmniGraph: Input Devices#

Learning Objectives#

This tutorial walks you through using OmniGraph to connect input devices, such as a gamepad controller or a keyboard, to interact with a simulation in Isaac Sim. In this tutorial, you map inputs from a gamepad to operate a mobile robot in simulation and receive inputs from the keyboard to manipulate objects on the stage.

Getting Started#

Prerequisites

  • Review the GUI Tutorial series, especially OmniGraph, prior to beginning this tutorial.

  • The gamepad tutorial requires an XBox controller to run. Other gamepads may function; however their bindings may be different than described in this tutorial.

Gamepad Inputs#

Running the Example#

The NVIDIA Kaya robot is a three-wheeled holonomic drive robot designed to be as accessible as possible. It is made of 3D-printed parts and hobbyist components. It features a RealSense depth camera for mapping, localization, and object recognition. An on-board IMU and position feedback from the wheel motors provide accurate odometry information.

The Kaya’s sensor array and omni-directional motion can be simulated in Isaac Sim by completing the following steps:

  1. Connect the gamepad controller to the PC running Isaac Sim.

  2. Open the Kaya Gamepad. For example: Go to the top Menu Bar and click Isaac Examples > Input Devices > Kaya Gamepad.

  3. Press the LOAD button to load a Kaya robot into the scene.

  4. Press Play to start controlling the Kaya via the gamepad.

OmniGraph Layout#

To see the layout of the graph:

  1. Verify that the example is loaded and that there is a Kaya on the stage. Open the visual programming window: Window > Visual Scripting > Action Graph.

  2. Click on the Edit Graph icon inside the Action Graph tab.

  3. In the pop window, select /World/ActionGraph. There may be multiple action graphs for a given stage, they are all listed here.

  4. The graph seen in the image below opens. To examine the properties of each node, select the node on the graph. The corresponding properties display in the Property tab on the bottom right. The nodes are also listed on the stage tree. You can select the nodes there too.

../_images/isaac_tutorial_advanced_inputs_gamepad_graph.png

For each joystick axis or button on the gamepad, there is a dedicated node. All the mapping and hardware connections are done behind the scenes. The output of the node is the digital readout of the selected control. Those outputs are processed and fed into the holonomic controller. The holonomic controller calculates the rotational speed of the wheels. That speed is fed into the articulation controller and assigned to the robot. OmniGraph tutorial under the GUI Tutorials section has more details regarding the controllers.

Keyboard Inputs#

Running the Example#

The OmniGraph Keyboard example shows how to receive keyboard inputs and connect them. After they are connected, they can be used to modify specific attributes of the objects on the stage.

To run the example:

  1. Open the example by going to the top Menu Bar and click Isaac Examples > Input Devices > Omniverse Keyboard.

  2. Press Play. Now you can grow the size of the cube on stage by pressing A and shrink it by pressing D on the keyboard.

OmniGraph Layout#

To see the layout of the graph:

  1. Validate that the example is loaded and that there is a cube on stage. Open the visual programming window: Window > Visual Scripting > Action Graph.

  2. Click on the Edit Graph icon inside the Action Graph tab.

  3. In the window, select /World/ActionGraph. There may be multiple action graphs for a given stage, they are all listed here.

  4. The graph seen in the image below opens. To examine the properties of each node, select the node on the graph display. The corresponding properties display in the Property tab on the bottom right. The nodes are also listed on the stage tree. You can select the nodes there too.

Note

For each key on the keyboard, there is a dedicated node.

../_images/isaac_tutorial_advanced_inputs_keyboard_graph.png

Summary#

This tutorial covered the following topics:

  • Using OmniGraph to receive inputs from external devices such as gamepad and keyboard.

  • How to read and write an object’s USD attributes in OmniGraph.

Further Reading#

To learn more about using Python to script OmniGraphs: OmniGraph: Python Scripting.

On this page