Skip to content

Cameras

RoSys provides instant camera access for object detection, remote operation and similar use cases.

Setup

USB camera devices are discovered through video4linux (v4l) and accessed with openCV. Therefore the program v4l2ctl and openCV (including python bindings) must be available. We recommend to use the RoSys Docker image which provides the full required software stack. Make sure the container can access the USB devices by starting it with --privileged or explicitly passing the specific --devices.

Show Captured Images

Using rosys.ui you can show the latest captured images from each camera:

#!/usr/bin/env python3
from nicegui import ui

from rosys.vision import SimulatedCamera

camera = SimulatedCamera(id='test_cam', width=800, height=600)

image = ui.interactive_image()
ui.timer(0.3, lambda: image.set_source(camera.get_latest_image_url()))

ui.run(title='RoSys')

The ui.timer regularly updates the source property of the ui.image. The cameras latest_image_uri property provides the URI to the latest captured image.

This example uses a SimulatedCamera for demonstration. You can directly replace the camera with a UsbCamera or RtspCamera if you know their ids or use their respective providers to discover them automatically.

Remote Operation

A fairly often required use case on real mobile robots is the remote operation. In a simple use case you may only need to visualize one camera and have some steering controls. Here we use the NEW_CAMERA event to display the first camera to control real Hardware:

#!/usr/bin/env python3
from nicegui import ui

import rosys

if rosys.hardware.SerialCommunication.is_possible():
    communication = rosys.hardware.SerialCommunication()
    robot_brain = rosys.hardware.RobotBrain(communication)
    can = rosys.hardware.CanHardware(robot_brain)
    wheels = rosys.hardware.WheelsHardware(robot_brain, can=can)
    robot = rosys.hardware.RobotHardware([can, wheels], robot_brain)
    camera_provider = rosys.vision.UsbCameraProvider()
else:
    wheels = rosys.hardware.WheelsSimulation()
    robot = rosys.hardware.RobotSimulation([wheels])
    rosys.vision.SimulatedCameraProvider.USE_PERSISTENCE = False
    camera_provider = rosys.vision.SimulatedCameraProvider()
    camera = rosys.vision.SimulatedCamera(id='test_cam', width=800, height=600)
    rosys.on_startup(lambda: camera_provider.add_camera(camera))
steerer = rosys.driving.Steerer(wheels)
odometer = rosys.driving.Odometer(wheels)


async def add_main_camera(camera: rosys.vision.Camera) -> None:
    camera_card.clear()  # remove "seeking camera" label
    with camera_card:
        main_cam = ui.interactive_image()
        ui.timer(0.1, lambda: main_cam.set_source(camera.get_latest_image_url()))

camera_provider.CAMERA_ADDED.register_ui(add_main_camera)

with ui.card().tight().style('width:30em') as camera_card:
    ui.label('seeking main camera').classes('m-8 text-center')

with ui.card().tight().style('width:30em'):
    with ui.row():
        with ui.card().tight():
            rosys.driving.joystick(steerer)
            rosys.driving.keyboard_control(steerer)
        ui.markdown('steer with joystick on the left or<br />SHIFT + arrow keys').classes('m-8 text-center')

ui.run(title='RoSys')

By adding a Joystick and KeyboardControl the robot is ready to go for remote operation.