Cameras

RoSys provides instant camera access for object detection, remote operation and similar use cases.

Setup

USB camera devices are discovered through video4linux (v4l) and accessed with openCV. Therefore the program v4l2ctl and openCV (including python bindings) must be available. We recommend to use the RoSys Docker image which provides the full required software stack. Make sure the container can access the USB devices by starting it with --privileged or explicitly passing the specific --devices.

Show Captured Images

Using rosys.ui you can show the latest captured images from each camera:

#!/usr/bin/env python3
from nicegui import ui

from rosys.vision import UsbCameraProviderSimulation, camera_provider

camera_provider = UsbCameraProviderSimulation()
camera_provider.add_camera(camera_provider.create_calibrated('test_cam', width=800, height=600))


def refresh() -> None:
    for uid, camera in camera_provider.cameras.items():
        if uid not in feeds:
            feeds[uid] = ui.interactive_image()
        feeds[uid].set_source(camera_provider.get_latest_image_url(camera))


feeds: dict[str, ui.interactive_image] = {}
ui.timer(0.3, refresh)

ui.run(title='RoSys')

The ui.timer regularly updates the source property of the ui.image. The cameras latest_image_uri property provides the URI to the latest captured image.

This example uses a UsbCameraProviderSimulation with a single simulated test camera. But you can replace the provider with a UsbCameraProviderHardware.

Remote Operation

A fairly often required use case on real mobile robots is the remote operation. In a simple use case you may only need to visualize one camera and have some steering controls. Here we use the NEW_CAMERA event to display the first camera to control real Hardware:

#!/usr/bin/env python3
from nicegui import ui

import rosys

if rosys.hardware.SerialCommunication.is_possible():
    communication = rosys.hardware.SerialCommunication()
    robot_brain = rosys.hardware.RobotBrain(communication)
    can = rosys.hardware.CanHardware(robot_brain)
    wheels = rosys.hardware.WheelsHardware(robot_brain, can=can)
    robot = rosys.hardware.RobotHardware([can, wheels], robot_brain)
    camera_provider = rosys.vision.UsbCameraProviderHardware()
else:
    wheels = rosys.hardware.WheelsSimulation()
    robot = rosys.hardware.RobotSimulation([wheels])
    camera_provider = rosys.vision.UsbCameraProviderSimulation()
    camera_provider.restore = lambda _: None  # NOTE: disable persistence
    test_cam = camera_provider.create_calibrated('test_cam', width=800, height=600)
    rosys.on_startup(lambda: camera_provider.add_camera(test_cam))
steerer = rosys.driving.Steerer(wheels)
odometer = rosys.driving.Odometer(wheels)


async def add_main_camera(camera: rosys.vision.Camera) -> None:
    camera_card.clear()  # remove "seeking camera" label
    with camera_card:
        main_cam = ui.interactive_image()
        ui.timer(0.1, lambda: main_cam.set_source(camera_provider.get_latest_image_url(camera)))

camera_provider.CAMERA_ADDED.register_ui(add_main_camera)

with ui.card().tight().style('width:30em') as camera_card:
    ui.label('seeking main camera').classes('m-8 text-center')

with ui.card().tight().style('width:30em'):
    with ui.row():
        with ui.card().tight():
            rosys.driving.joystick(steerer)
            rosys.driving.keyboard_control(steerer)
        ui.markdown('steer with joystick on the left or<br />SHIFT + arrow keys').classes('m-8 text-center')

ui.run(title='RoSys')

By adding a Joystick and KeyboardControl the robot is ready to go for remote operation.