Isaac ROS Image Segmentation

Overview

This repository provides NVIDIA GPU-accelerated packages for semantic image segmentation. Using a deep learned U-Net model, such as PeopleSemSegnet, and a monocular camera, the isaac_ros_unet package can generate an image mask segmenting out objects of interest.

Packages in this repository rely on accelerated DNN model inference using Triton or TensorRT from Isaac ROS DNN Inference and a pretrained model from NVIDIA GPU Cloud (NGC) or elsewhere.

Isaac ROS NITROS Acceleration

This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.

Performance

The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:

Note: The model performance without the encoder and decoder is reported

These data have been collected per the methodology described here.

Table of Contents

• Isaac ROS Image Segmentation
  • Overview
    • Isaac ROS NITROS Acceleration
  • Performance
  • Table of Contents
  • Latest Update
  • Supported Platforms
    • Docker
  • Quickstart
  • Next Steps
    • Try More Examples
    • Try NITROS-Accelerated Graph with Argus
    • Use Different Models
    • Customize your Dev Environment
  • Package Reference
    • UNetDecoderNode
      • Usage
      • ROS Parameters
      • ROS Topics Subscribed
      • ROS Topics Published
  • Troubleshooting
    • Isaac ROS Troubleshooting
    • Deep Learning Troubleshooting
  • Updates

Latest Update

Update 2022-10-19: Updated OSS licensing

Supported Platforms

This package is designed and tested to be compatible with ROS2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.

Note: Versions of ROS2 earlier than Humble are not supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.

Docker

To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.

Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.

Quickstart

1. Set up your development environment by following the instructions here.

2. Clone this repository and its dependencies under ~/workspaces/isaac_ros-dev/src.

   cd ~/workspaces/isaac_ros-dev/src

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation

   git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference

3. Pull down a ROS Bag of sample data:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_image_segmentation && \
     git lfs pull -X "" -I "resources/rosbags/"

4. Launch the Docker container using the run_dev.sh script:

   cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
     ./scripts/run_dev.sh

5. Download the PeopleSemSegNet ShuffleSeg ETLT file and the int8 inference mode cache file:

   mkdir -p /tmp/models/peoplesemsegnet_shuffleseg/1 && \
     cd /tmp/models/peoplesemsegnet_shuffleseg && \
     wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_etlt.etlt && \
     wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_cache.txt

6. Convert the ETLT file to a TensorRT plan file:

   /opt/nvidia/tao/tao-converter -k tlt_encode -d 3,544,960 -p input_2:0,1x3x544x960,1x3x544x960,1x3x544x960 -t int8 -c peoplesemsegnet_shuffleseg_cache.txt -e /tmp/models/peoplesemsegnet_shuffleseg/1/model.plan -o argmax_1 peoplesemsegnet_shuffleseg_etlt.etlt

7. Create a file called /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt by copying the sample Triton config file:

   cp /workspaces/isaac_ros-dev/src/isaac_ros_image_segmentation/resources/peoplesemsegnet_shuffleseg_config.pbtxt /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt

8. Inside the container, build and source the workspace:

   cd /workspaces/isaac_ros-dev && \
     colcon build --symlink-install && \
     source install/setup.bash

9. (Optional) Run tests to verify complete and correct installation:

   colcon test --executor sequential

10. Run the following launch files to spin up a demo of this package:

    ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax'

    Then open another terminal, and enter the Docker container again:

    cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
      ./scripts/run_dev.sh

    Then, play the ROS bag:

    ros2 bag play -l src/isaac_ros_image_segmentation/resources/rosbags/unet_sample_data/

11. Visualize and validate the output of the package by launching rqt_image_view in another terminal: In a third terminal, enter the Docker container again:

    cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \
      ./scripts/run_dev.sh

    Then launch rqt_image_view:

    ros2 run rqt_image_view rqt_image_view

    Then inside the rqt_image_view GUI, change the topic to /unet/colored_segmentation_mask to view a colorized segmentation mask.

    

    Note: The raw segmentation is also published to /unet/raw_segmentation_mask. However, the raw pixels correspond to the class labels and so the output is unsuitable for human visual inspection.

Next Steps

Try More Examples

To continue your exploration, check out the following suggested examples:

• Tutorial with Isaac Sim

Try NITROS-Accelerated Graph with Argus

If you have an Argus-compatible camera, you can launch the NITROS-accelerated graph by following the tutorial.

Use Different Models

Click here for more information about how to use NGC models.

Note: the parameters may need to be changed depending on the model used. Please verify the preprocessing steps and postprocessing steps of the model against the configuration of the launch file.

Customize your Dev Environment

To customize your development environment, reference this guide.

Package Reference

UNetDecoderNode

Usage

Triton:

ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py network_image_width:=<network_image_width> network_image_height:=<network_image_height> encoder_image_mean:=<encoder_image_mean> encoder_image_stddev:=<encoder_image_stddev> model_name:=<model_name> model_repository_paths:=<model_repository_paths> max_batch_size:=<max_batch_size> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> input_tensor_formats:=<input_tensor_formats> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> output_tensor_formats:=<output_tensor_formats> network_output_type:=<network_output_type> color_segmentation_mask_encoding:=<color_segmentation_mask_encoding> mask_width:=<mask_width> mask_height:=<mask_height>

TensorRT:

ros2 launch isaac_ros_unet isaac_ros_unet_tensor_rt.launch.py network_image_width:=<network_image_width> network_image_height:=<network_image_height> encoder_image_mean:=<encoder_image_mean> encoder_image_stddev:=<encoder_image_stddev> model_file_path:=<model_file_path> engine_file_path:=<engine_file_path> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> input_tensor_formats:=<input_tensor_formats> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> output_tensor_formats:=<output_tensor_formats> tensorrt_verbose:=<tensorrt_verbose> force_engine_update:=<force_engine_update> network_output_type:=<network_output_type> color_segmentation_mask_encoding:=<color_segmentation_mask_encoding> mask_width:=<mask_width> mask_height:=<mask_height>

ROS Parameters

Warning: the following parameters are no longer supported:

• queue_size
• frame_id as the frame_id of the header will be forwarded now
• tensor_output_order as the order will be inferred from the model. Note: the model output should be NCHW or NHWC. In this context, the C refers to the class.

Note: For the network_output_type parameter's softmax and sigmoidoption, we currently expect only 32 bit floating point values. For the argmax option, we currently expect only signed 32 bit integers.

Note: Models with greater than 255 classes are currently not supported. If a class label greater than 255 is detected, this mask will be downcast to 255 in the raw segmentation.

ROS Topics Subscribed

Limitation: All input images are required to have height and width that are both an even number of pixels.

ROS Topics Published

Troubleshooting

Isaac ROS Troubleshooting

For solutions to problems with Isaac ROS, please check here.

Deep Learning Troubleshooting

For solutions to problems with using DNN models, please check here.

Updates