Isaac ROS Image Segmentation
Overview
This repository provides NVIDIA GPU-accelerated packages for semantic image segmentation. Using a deep learned U-Net model, such as PeopleSemSegnet
, and a monocular camera, the isaac_ros_unet
package can generate an image mask segmenting out objects of interest.
Packages in this repository rely on accelerated DNN model inference using Triton or TensorRT from Isaac ROS DNN Inference and a pretrained model from NVIDIA GPU Cloud (NGC) or elsewhere.
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
Performance
The following are the benchmark performance results of the prepared pipelines in this package, by supported platform:
Pipeline | AGX Orin | Orin Nano | x86_64 w/ RTX 3060 Ti |
---|---|---|---|
PeopleSemSegNet (544p) | 260 fps 3.7ms |
128 fps 6.7ms |
300 fps 2ms |
Note: The model performance without the encoder and decoder is reported
These data have been collected per the methodology described here.
Table of Contents
- Isaac ROS Image Segmentation
Latest Update
Update 2022-10-19: Updated OSS licensing
Supported Platforms
This package is designed and tested to be compatible with ROS2 Humble running on Jetson or an x86_64 system with an NVIDIA GPU.
Note: Versions of ROS2 earlier than Humble are not supported. This package depends on specific ROS2 implementation features that were only introduced beginning with the Humble release.
Platform | Hardware | Software | Notes |
---|---|---|---|
Jetson | Jetson Orin Jetson Xavier |
JetPack 5.0.2 | For best performance, ensure that power settings are configured appropriately. |
x86_64 | NVIDIA GPU | Ubuntu 20.04+ CUDA 11.6.1+ |
Docker
To simplify development, we strongly recommend leveraging the Isaac ROS Dev Docker images by following these steps. This will streamline your development environment setup with the correct versions of dependencies on both Jetson and x86_64 platforms.
Note: All Isaac ROS Quickstarts, tutorials, and examples have been designed with the Isaac ROS Docker images as a prerequisite.
Quickstart
-
Set up your development environment by following the instructions here.
-
Clone this repository and its dependencies under
~/workspaces/isaac_ros-dev/src
.cd ~/workspaces/isaac_ros-dev/src
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation
git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_dnn_inference
-
Pull down a ROS Bag of sample data:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_image_segmentation && \ git lfs pull -X "" -I "resources/rosbags/"
-
Launch the Docker container using the
run_dev.sh
script:cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
-
Download the
PeopleSemSegNet ShuffleSeg
ETLT file and theint8
inference mode cache file:mkdir -p /tmp/models/peoplesemsegnet_shuffleseg/1 && \ cd /tmp/models/peoplesemsegnet_shuffleseg && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_etlt.etlt && \ wget https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplesemsegnet/versions/deployable_shuffleseg_unet_v1.0/files/peoplesemsegnet_shuffleseg_cache.txt
-
Convert the ETLT file to a TensorRT plan file:
/opt/nvidia/tao/tao-converter -k tlt_encode -d 3,544,960 -p input_2:0,1x3x544x960,1x3x544x960,1x3x544x960 -t int8 -c peoplesemsegnet_shuffleseg_cache.txt -e /tmp/models/peoplesemsegnet_shuffleseg/1/model.plan -o argmax_1 peoplesemsegnet_shuffleseg_etlt.etlt
-
Create a file called
/tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt
by copying the sample Triton config file:cp /workspaces/isaac_ros-dev/src/isaac_ros_image_segmentation/resources/peoplesemsegnet_shuffleseg_config.pbtxt /tmp/models/peoplesemsegnet_shuffleseg/config.pbtxt
-
Inside the container, build and source the workspace:
cd /workspaces/isaac_ros-dev && \ colcon build --symlink-install && \ source install/setup.bash
-
(Optional) Run tests to verify complete and correct installation:
colcon test --executor sequential
-
Run the following launch files to spin up a demo of this package:
ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax'
Then open another terminal, and enter the Docker container again:
cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then, play the ROS bag:
ros2 bag play -l src/isaac_ros_image_segmentation/resources/rosbags/unet_sample_data/
-
Visualize and validate the output of the package by launching
rqt_image_view
in another terminal: In a third terminal, enter the Docker container again:cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ ./scripts/run_dev.sh
Then launch
rqt_image_view
:ros2 run rqt_image_view rqt_image_view
Then inside the
rqt_image_view
GUI, change the topic to/unet/colored_segmentation_mask
to view a colorized segmentation mask.Note: The raw segmentation is also published to
/unet/raw_segmentation_mask
. However, the raw pixels correspond to the class labels and so the output is unsuitable for human visual inspection.
Next Steps
Try More Examples
To continue your exploration, check out the following suggested examples:
Try NITROS-Accelerated Graph with Argus
If you have an Argus-compatible camera, you can launch the NITROS-accelerated graph by following the tutorial.
Use Different Models
Click here for more information about how to use NGC models.
Model Name | Use Case |
---|---|
PeopleSemSegNet ShuffleSeg | Semantically segment people inference at a high speed |
PeopleSemSegNet | Semantically segment people |
Note: the parameters may need to be changed depending on the model used. Please verify the preprocessing steps and postprocessing steps of the model against the configuration of the launch file.
Customize your Dev Environment
To customize your development environment, reference this guide.
Package Reference
UNetDecoderNode
Usage
Triton:
ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py network_image_width:=<network_image_width> network_image_height:=<network_image_height> encoder_image_mean:=<encoder_image_mean> encoder_image_stddev:=<encoder_image_stddev> model_name:=<model_name> model_repository_paths:=<model_repository_paths> max_batch_size:=<max_batch_size> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> input_tensor_formats:=<input_tensor_formats> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> output_tensor_formats:=<output_tensor_formats> network_output_type:=<network_output_type> color_segmentation_mask_encoding:=<color_segmentation_mask_encoding> mask_width:=<mask_width> mask_height:=<mask_height>
TensorRT:
ros2 launch isaac_ros_unet isaac_ros_unet_tensor_rt.launch.py network_image_width:=<network_image_width> network_image_height:=<network_image_height> encoder_image_mean:=<encoder_image_mean> encoder_image_stddev:=<encoder_image_stddev> model_file_path:=<model_file_path> engine_file_path:=<engine_file_path> input_tensor_names:=<input_tensor_names> input_binding_names:=<input_binding_names> input_tensor_formats:=<input_tensor_formats> output_tensor_names:=<output_tensor_names> output_binding_names:=<output_binding_names> output_tensor_formats:=<output_tensor_formats> tensorrt_verbose:=<tensorrt_verbose> force_engine_update:=<force_engine_update> network_output_type:=<network_output_type> color_segmentation_mask_encoding:=<color_segmentation_mask_encoding> mask_width:=<mask_width> mask_height:=<mask_height>
ROS Parameters
ROS Parameter | Type | Default | Description |
---|---|---|---|
color_segmentation_mask_encoding |
string |
rgb8 |
The image encoding of the colored segmentation mask. Supported values: rgb8 , bgr8 |
color_palette |
int64_t list |
[] |
Vector of integers where each element represents the rgb color hex code for the corresponding class label. The number of elements should equal the number of classes. E.g. [0xFF0000, 0x76b900] |
network_output_type |
string |
softmax |
The type of output that the network provides. Supported values: softmax , argmax , sigmoid |
mask_width |
int16_t |
960 |
The width of the segmentation mask. |
mask_height |
int16_t |
544 |
The height of the segmentation mask. |
Warning: the following parameters are no longer supported:
queue_size
frame_id
as theframe_id
of the header will be forwarded nowtensor_output_order
as the order will be inferred from the model. Note: the model output should beNCHW
orNHWC
. In this context, theC
refers to the class.
Note: For the
network_output_type
parameter'ssoftmax
andsigmoid
option, we currently expect only 32 bit floating point values. For theargmax
option, we currently expect only signed 32 bit integers.
Note: Models with greater than 255 classes are currently not supported. If a class label greater than 255 is detected, this mask will be downcast to 255 in the raw segmentation.
ROS Topics Subscribed
ROS Topic | Interface | Description |
---|---|---|
tensor_sub |
isaac_ros_tensor_list_interfaces/TensorList | The tensor that contains raw probabilities for every class in each pixel. |
Limitation: All input images are required to have height and width that are both an even number of pixels.
ROS Topics Published
ROS Topic | Interface | Description |
---|---|---|
unet/raw_segmentation_mask |
sensor_msgs/Image | The raw segmentation mask, encoded in mono8. Each pixel represents a class label. |
/unet/colored_segmentation_mask |
sensor_msgs/Image | The colored segmentation mask. The color palette is user specified by the color_palette parameter. |
Troubleshooting
Isaac ROS Troubleshooting
For solutions to problems with Isaac ROS, please check here.
Deep Learning Troubleshooting
For solutions to problems with using DNN models, please check here.
Updates
Date | Changes |
---|---|
2022-10-19 | Updated OSS licensing |
2022-08-31 | Update to be compatible with JetPack 5.0.2 |
2022-06-30 | Removed frame_id, queue_size and tensor_output_order parameter. Added network_output_type parameter (support for sigmoid and argmax output layers). Switched implementation to use NITROS. Removed support for odd sized images. Switched tutorial to use PeopleSemSegNet ShuffleSeg and moved unnecessary details to other READMEs. |
2021-11-15 | Isaac Sim HIL documentation update |
2021-10-20 | Initial release |