Real-time tracking? From camera to stage.

Another fun Friday conceptual project

We wanted to know where a person is standing on a stage, in real time, and use that information to control lights automatically.

To do this, we built a piece of software using YOLOv8 running as an ONNX model, along with OpenCV.

The ONNX model is used to detect people in the video image. It tells us where a person is, but on its own it only works frame by frame. That is not enough for smooth tracking.

So we added OpenCV on top. Once a person is detected, OpenCV helps track them smoothly across frames.

This means we do not need to run the detection model constantly, which keeps everything from slowing down or causing delays.

We built a video interface that works with both machine vision cameras and DirectShow cameras. This made it super simple to expand out into new camera interfaces in the future, if we had a faster machine vision camera or input from a cctv camera using Infrared.

Next, we created a virtual stage using OBS, Open Broadcaster Software. Inside this stage, we placed a human standing in the middle. The software detects the person, draws a green box around them, and keeps that box locked to the person as they move.

We then added real world information to the system. This includes stage size, stage plot positions, camera field of view, lens and sensor settings, zoom levels, and camera angles like pan, tilt, and roll. With this data, the we can understand where the person is standing in actual space, not just in the video image.

By combining the person position and the camera angles, we can calculate the offset from a known zero point on the stage. This gives us an accurate position for the person at all times so we can push it into other systems.

That position data is then sent using UDP to a MagicQ lighting console. MagicQ receives the data as a tracker, which allows lights to follow the person automatically on stage.

At the moment, the system tracks one person at a time. The structure is already there to expand this in the future, allowing multiple people to be tracked and followed independently.