Computer Vision

Dropbear's perception stack provides real-time understanding of the world through stereo cameras, depth sensing, and neural network-based scene understanding. The vision pipeline runs on-edge on the NVIDIA Jetson Orin compute module.

Vision Hardware

Intel RealSense D435i

Primary depth + RGB camera mounted in the head assembly. Provides stereo depth at up to 90fps with a 87° × 58° field of view. Includes an onboard IMU for visual-inertial odometry.

NVIDIA Jetson Orin

On-board compute for real-time inference. Runs object detection, segmentation, and pose estimation models at 30fps using TensorRT-optimized ONNX models.

Perception Pipeline

The vision pipeline processes camera data through multiple stages:

Stage	Model / Method	Output
Object Detection	YOLOv8 / RT-DETR	Bounding boxes + class labels
Instance Segmentation	Segment Anything (SAM)	Per-object masks
Depth Estimation	RealSense stereo + monocular refinement	Dense depth map
Pose Estimation	FoundationPose / MegaPose	6DoF object poses for grasping
SLAM	ORB-SLAM3 / Isaac Visual SLAM	Robot localization + 3D map
Scene Description	VLM (vision-language model)	Natural language scene understanding

ROS 2 Topics

Vision data is published on standard ROS 2 topics:

/camera/color/image_raw     # RGB image (sensor_msgs/Image)
/camera/depth/image_raw     # Depth image (sensor_msgs/Image)
/camera/aligned_depth       # Depth aligned to RGB frame
/camera/pointcloud          # 3D point cloud (sensor_msgs/PointCloud2)
/perception/detections      # Detected objects (vision_msgs/Detection3DArray)
/perception/segmentation    # Segmentation masks
/slam/pose                  # Robot pose estimate (geometry_msgs/PoseStamped)
/slam/map                   # Occupancy grid map

Using Custom Models

You can plug in your own vision models. Export to ONNX, place in the models directory, and register in the perception config. The pipeline handles TensorRT optimization automatically on first load.

📷 Hardware Note

The vision pipeline is designed for the Jetson Orin but can run on any CUDA-capable GPU for development. For simulation, Isaac Sim provides synthetic camera feeds that match the real sensor characteristics.