Implement Custom Processing Using libwebrtc

Use at Your own risk.

Introduction

TIPSTAR, Since June 2020, our online service for publicly-managed sports betting, We use WebRTC to send video data through the internet, allowing remote workers to preview videos from anywhere. This makes remote production possible.

libwebrtc library is available in various languages including C++, Go, Python, etc. Among those, libwebrtc(in C++) is implementation for Browsers such as Chrome and Firefox , Which provides stable connectivity with Browser.

libwebrtc is Highly Objectified, Abstracted, and Interfaced, allowing for simple access from those scripts on Browsers.

This time, we will explain the Custom Send/Receive Process in libwebrtc using bres as the subject.

Virtual MCU bres

fig1.virtual MCU/data sequence

• Receive multi-channel video(VP8),audio(opus) data from SFU.
• Chroma key compositing(edits) based on attributes
• Attributes(scale, position[x,y], chromakey-threshould)
• Sends compositted data to SFU as separated send-only channels.
• Works in macOS 10.15 later, can run by connecting to SFU.
• （TODO)ubuntu + GPU / EGL + HeadLess(with compile/switch)

Custom Receive

In the case of signaling with websockets, the Operations-Chain for each signaling and libwebrtc-objects related to receive Processing is as follows.

fig2.Receive Operations-Chain

VideoSinkInterface<VideoFrame> and AudioTrackSinkInterface instances implement receive event callback. Custom user logic can be implemented in VideoSinkInterface<VideoFrame>::OnFrame (video frame) and AudioTrackSinkInterface::OnData (audio data)

fig3.custom receiver

fig4.fifo

To improve the visibility of the Operations-Chain, each processing Context should be separated by Event-FIFO

• websockets signaling Events(OFFER, CANDIDATE, PING, etc.)
• webrtc::PeerConnection Events(OnFrame,OnData,OnSuccess, etc.)

fig5.Separated with FIFO

Custom Send

fig6.Send Operations-Chain

For Video-Frame, call AdaptedVideoTrackSource::OnFrame(…) at 30FPS, For audio-Data, call AudioTransport::RecordedDataIsAvailable(…) at 48K/16bit/2ch

fig7.custom sender

Image Compositing

fig8. image compositing

As shown in the figure above, Ingress Video-Frames are synchronized and composited and transmitted to another channel.

Conclusion

Hopefully this post will provide some guidance, custom user logic with libwebrtc.
We are planning to implement customization of SFU and MCU to make bres more customizable.