Skip to content

Raspberry Pi Camera Setup

The Raspberry Pi Camera Module 2 is a popular camera option for the Autonomous Driving Platform due to its compact size, ease of use, and compatibility with the Raspberry Pi Compute Module used in the car.

Raspberry Pi Camera

Setup

This only needs to be done once per camera installation. The setup script does it, and it takes effect after a reboot.

The Orin's camera connectors need a setting that says which camera is attached. The setup script picks Camera IMX219 Dual, which turns on both connectors (CAM0 and CAM1), so a single camera works on either one and a stereo pair works too.

Setting it up by hand

Run the following command in a terminal on the car:

sudo /opt/nvidia/jetson-io/jetson-io.py
  • Pick Configure Jetson 24pin CSI Connector
  • Pick Configure for compatible hardware
  • If you have one camera, plugged into port cam0, pick Camera IMX219-A
  • If you have two cameras, or want either port to work, pick Camera IMX219 Dual
  • Pick Save pin changes
  • Pick Save and reboot to reconfigure pins

Test the Camera

After the car reboots, ensure you can read the camera device, run the command v4l2-ctl --list-devices, the expected output should include:

NVIDIA Tegra Video Input Device (platform:tegra-camrtc-ca):
 /dev/media0

vi-output, imx219 9-0010 (platform:tegra-capture-vi:1):
 /dev/video0

Then, test the camera using the following command:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM), width=320, height=240, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw, format=BGRx' ! videoconvert ! 'video/x-raw, format=BGR' ! videoconvert ! xvimagesink

For a camera on CAM1, use sensor-id=1. On the car's own screen, this shorter pipeline also works for either port:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! nvvidconv ! nveglglessink

This command will open a window displaying the camera feed. You should see output similar to the following in the terminal:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3280 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3280 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0
   Camera mode  = 4
   Output Stream W = 1280 H = 720
   seconds to Run    = 0
   Frame Rate = 59.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Redistribute latency...
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Pipeline construction is invalid, please add queues.
Additional debug info:
../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.
WARNING: from element /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0: Pipeline construction is invalid, please add queues.
Additional debug info:
../libs/gst/base/gstbasesink.c(1249): gst_base_sink_query_latency (): /GstPipeline:pipeline0/GstXvImageSink:xvimagesink0:
Not enough buffering available for  the processing deadline of 0:00:00.015000000, add enough queues to buffer  0:00:00.015000000 additional data. Shortening processing latency to 0:00:00.000000000.
0:00:10.8 / 99:99:99.

Running the camera node

The teleop and navigation plans start the camera in their camera pane. To run it on its own, in its container:

cd ~/roboracer_ws
airfield package run orin_rp2_csi mono_processor

The node runs with display_mode:=none, because containers have no screen to open a window on. To see the images, use RViz on the car's remote desktop: the teleop plan shows the camera there. See VNC and RViz.

Without airfield 
ros2 run orin_rp2_csi mono_processor --ros-args -p width:=320 -p height:=240 -p framerate:=30 -p display_mode:="none"

Stereoscopic Camera Setup

Stereoscopic Camera Setup

We have several stereoscopic camera setups available for the Autonomous Driving Platform that use two Raspberry Pi Camera Modules. These setups provide the hardware to compute depth. While the pi camera nodes have the framework for computing stereo depth, it is not tested. Students interested in computational geometry and computer vision are welcome to implement and test this functionality. Calibrate the cameras first, as in Camera Calibration and LiDAR Projection.

Troubleshooting

Captured image is invalid or empty

If you see the following error when running the mono_processor node, first restart NVIDIA's camera service, which can get stuck after a crash:

sudo systemctl restart nvargus-daemon

scripts/up does this too. If that doesn't help, restart the car. If that fails, re-configure the camera.

$ ros2 run orin_rp2_csi mono_processor --ros-args -p width:=320 -p height:=240 -p framerate:=30 -p display_mode:="none"
[INFO] [1760906220.374738176] [mono_processor]: MonoProcessor constructor started
Using pipeline: nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM), width=320, height=240, framerate=30/1, format=NV12 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:794 No cameras available
[ WARN:0@0.290] global cap_gstreamer.cpp:1728 open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
[INFO] [1760906220.596398368] [mono_processor]: Successfully loaded calibration from: /home/orin/roboracer_ws/params/cameras/0/calibration.txt
MonoProcessor constructor completed
Display mode: none
Image format: raw
[INFO] [1760906220.596873920] [mono_processor]: About to initialize ImageTransport
[INFO] [1760906220.641440576] [mono_processor]: Publishing raw images to topic: camera_0/image_raw
[INFO] [1760906220.643998560] [mono_processor]: Publishing rectified images to topic: camera_0/image_rect
[INFO] [1760906220.644077184] [mono_processor]: Image format: raw
[INFO] [1760906220.644113664] [mono_processor]: About to call shared_from_this() for timer
[INFO] [1760906220.644148512] [mono_processor]: shared_from_this() succeeded
[ERROR] [1760906220.677549344] [mono_processor]: Captured image is invalid or empty