Skip to content

Cameras

RoSys provides instant camera access for object detection, remote operation and similar use cases. Any plugged in camera becomes an entry in world.usb_cameras containing recorded images and configuration parameters like exposure.

Setup

Camera devices are discovered through video4linux (v4l) and accessed with openCV. Therefore the program v4l2ctl and openCV (including python bindings) must be available. We recommend to use the RoSys docker image which provides the full required software stack. Make sure the container can access the USB devices by starting it with privileged or explicitly passing the specific devices.

Show Captured Images

Using rosys.ui (see User Interface) you can show the latest captured images from each camera:

#!/usr/bin/env python3
import rosys
import rosys.ui
from nicegui import ui

# setup
runtime = rosys.Runtime()
runtime.with_usb_cameras()
rosys.ui.configure(ui, runtime)


async def refresh():
    for uid, camera in runtime.world.usb_cameras.items():
        if uid not in feeds:
            feeds[uid] = ui.interactive_image('', cross=False)
        await feeds[uid].set_source(camera.latest_image_uri)


# refresh timer
feeds = {}
ui.timer(0.3, refresh)

ui.run(title='RoSys', port=8080)

The ui.timer regularly updates the source property of the ui.image. The cameras latest_image_uri property provides the uri to the latest captured image.

Simulated Cameras

The above code works out-of-the-box if your camera can be discoverd with v4l2ctl. When you have no camera at hand or develop parts of your robot on a Mac or Windows system, you can simply add a simulated camera:

from rosys.actors import UsbCameraSimulator
...

camsim = runtime.get_actor(UsbCameraSimulator)
if camsim is not None and 'testcam' not in runtime.world.usb_cameras:
    runtime.world.usb_cameras['testcam'] = \
        camsim.create_calibrated('testcam', width=800, height=600)

Remote Operation

A fairly often required use case is the remote operation of a robot. In a simple use case you may only need to visualize one camera and have some steering controls. Here we use the event.Id.NEW_CAMERA to only display the first camera:

#!/usr/bin/env python3
import rosys
import rosys.ui
from nicegui import ui
from rosys.world import Camera

# setup
rosys.ui.configure(ui, rosys.Runtime())


async def add_main_camera(camera: Camera):
    camera_card.clear()  # remove "seeking cam" label
    with camera_card:
        maincam = ui.image()
        ui.timer(1, lambda: maincam.set_source(camera.latest_image_uri))
    rosys.event.unregister(rosys.event.Id.NEW_CAMERA, add_main_camera)  # we only show the first cam

with ui.card().tight().style('width:30em;') as camera_card:
    ui.label('seeking main camera').style('margin:1em')
rosys.event.register(rosys.event.Id.NEW_CAMERA, add_main_camera)

By adding joystick and keyboard_control the robot is ready to go for remote operation:

with ui.card().tight().style('width:30em;'):
    with ui.row():
        with ui.card().tight():
            rosys.ui.joystick()
            rosys.ui.keyboard_control()
        ui.markdown('steer with joystick on the left<br>or SHIFT + arrow keys') \
            .style('margin:2em;text-align:center')