Inserting SEI into H.264 (AVC) Streams | v0.18.0.0+

OvenMediaEngine can insert SEI (Supplemental Enhancement Information) into live streams to deliver custom data with video content at frame-level precision.

Overview

• You can insert SEI dynamically through OvenMediaEngine's Send Event API or continuously through XML configuration.

• When inserting SEI, you can add custom data, and the inserted SEI automatically includes UUID and Timestamp.

  • The inserted SEI is generated in OvenMediaEngine-specific format containing UUID, Timestamp, and custom data.

• OvenPlayer can receive SEI inserted by OvenMediaEngine and provides functionality to automatically parse the SEI specification defined by OvenMediaEngine.

Inserting SEI via Send Event API

API Interface

You can use OvenMediaEngine's SendEvent REST API to dynamically trigger events that insert SEI into streams.

Request

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvent

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

{
    "eventFormat": "sei",
    "eventType": "video",
    "events": [
        {
            "seiType": "UserDataUnregistered",
            "data": "OvenMediaEngine"
        }
    ]
}

POST /v1/vhosts{vhost}/apps/{app}/streams/{stream}:sendEvents

Header

Authorization: Basic {credentials}

# Authorization
Credentials for HTTP Basic Authentication created with <AccessToken>

Body

[
  {
      "eventFormat": "sei",
      "eventType": "video",
      "events": [
          {
              "seiType": "UserDataUnregistered",
              "data": "OvenMediaEngine"
          }
      ]
  }
]

Parameter	Required	Description
eventFormat	Y	Specifies the event format (use sei format)
eventType	N	Specifies the event type (use video format)
events	Y	Contains event data values
event.seiType	N	Specifies SEI type (use UserDataUnregistered format)
event.data	Y	Enter the actual data to be transmitted

Response

200 Ok

Header

Content-Type: application/json

Body

{
    "message": "OK",
    "statusCode": 200
}

400 Bad Request

Header

Content-Type: application/json

Body

{
    "message": "eventFormat(string) and events(array) are required",
    "statusCode": 400
}

{
    "message": "eventFormat is not supported: [XXX]",
    "statusCode": 400
}

{
    "message": "Could not make events data",
    "statusCode": 400
}

{
    "message": "eventType must be string",
    "statusCode": 400
}

{
    "message": "eventType is not supported: [XXX]",
    "statusCode": 400
}

500 Internal Server Error

Header

Content-Type: application/json

Body

{
    "message": "Could not inject event: [XXX]",
    "statusCode": 500
}

Inserting SEI through XML Configuration

For scenarios requiring continuous SEI insertion, you can configure this behavior through XML configuration. Create an XML file defining SEI insertion events and enable EventGenerator in Server.xml.

Configuration Example

Server.xml: You can enable the EventGenerator functionality for SEI insertion by adding <Application><EventGenerator>.

<?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
  ...
  <VirtualHosts>
    <VirtualHost>
      <Applications>
        <Application>
          ...
          <EventGenerator>
            <Enable>true</Enable>
            <Path>events/send_event_info.xml</Path>
          </EventGenerator>
        </Application>
      </Applications>
    </VirtualHost>
  </VirtualHosts>
</Server>

Element	Required	Description
<Enable>	Y	Set activation status with true or false
<Path>	Y	Set the path to the XML file defining SEI insertion details. If a relative path is specified, the directory containing the Server.xml file is used as the base

XML Defining SEI Insertion Events: Create an XML file defining SEI insertion events at the path specified in Server.xml. In this example, it's send_event_info.xml.

<?xml version="1.0" encoding="UTF-8"?>
<EventInfo>
  <Event>
    <Enable>true</Enable>
    <SourceStreamName>stream*</SourceStreamName>
    <Interval>2000</Interval>
    <EventFormat>sei</EventFormat>
    <EventType>video</EventType>
    <Values>
      <SeiType>UserDataUnregistered</SeiType>
      <Data>OvenMediaEngine</Data>
    </Values>
  </Event>
</EventInfo>

Parameter	Required	Description
<Enable>	Y	Set activation status with true or false
<SourceStreamName>	Y	Specify target stream name. Supports wildcards (*) for pattern matching
<Interval>	Y	Set event occurrence interval in milliseconds (ms)
<EventFormat>	Y	Specify event format (use sei format)
<EventType>	N	Set to video
<Values><SeiType>	N	Specify SEI type (use UserDataUnregistered format)
<Values><Data>	Y	Specify custom data to be inserted into SEI

Changes made to the event definition XML file are immediately applied without needing to restart OvenMediaEngine.

OvenMediaEngine-Specific SEI Payload Format

The payload of SEI generated by OvenMediaEngine always includes UUID and Timestamp values in the following structure:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| uuid_iso_iec_11578(128)                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp (64)                                                |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Data (Payload Size - UUID(128) - Timestamp(64))               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ...                                                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Field	Size (bits)	Description
UUID	128	Indicates that the SEI Payload follows OvenMediaEngine's specification. Always set to 464d4c47-5241-494e-434f-4c4f554201
Timestamp	64	Epoch time in milliseconds
Data	Varies depending on custom data	custom data

Receiving SEI Data

OvenPlayer can parse SEI inserted by OvenMediaEngine and pass the included UUID, Timestamp, and custom data to the application.

Code Example

var player = OvenPlayer.create('player', {
  sources: [
    {
      type: 'webrtc', // Playing WebRTC stream
      file: 'wss://[YOUR_OvenMediaEngine]:3333/app/stream'
    }
  ],
  parseStream: {
    enabled: true // Enable H.264 NAL parsing
  }
});

function toAsciiString(byteArray) {
  return String.fromCharCode.apply(null, byteArray);
}

player.on('metaData', function (metadata) {
  console.log('MetaData:', metadata);
  /* Output:
    {
      type: 'sei',
      nalu: Unit8Array(33),
      sei: {
        type: 5,
        size: 39,
        payload: Unit8Array(39)
      },
      registered: true,
      uuid: '464d4c47-5241-494e-434f-4c4f-55524201',
      timecode: 1739851602778,
      userdata: Unit8Array(15)
    }
  */

  console.log(`Convert user data to string: ${toAsciiString(metadata.userdata)}`);
  /* Output:
    Convert user data to string: OvenMediaEngine
  */
});

Player Initialization:

• Call OvenPlayer.create() function to create a player in the specified div.

• Specify the stream type and URL in the sources array. SEI is only supported in WebRTC streams.

• Enable H.264 NAL (Network Abstraction Layer) parsing through parseStream.enabled: true setting. This is required for SEI metadata processing.

SEI Data Processing:

• Register a player.on('metaData', callback) event listener to process SEI whenever it is received.

• You can obtain the UUID, Timestamp, and custom data inserted by OvenMediaEngine from the event callback parameter.

metaData Event Callback Parameters

Field	Description
type	Always set to sei, indicating this is SEI metadata
nalu	Uint8Array containing raw NALU (Network Abstraction Layer Unit) data of the SEI
sei	SEI parsing result containing the following sub-fields: - type: SEI type - size: Payload size - payload: Raw SEI payload data (Uint8Array)
registered	Indicates whether the SEI was generated in the format defined by OvenMediaEngine. If true, the following additional fields are included
uuid	(when registered=true) Unique identifier inserted by OvenMediaEngine into the SEI
timecode	(when registered=true) Timestamp (milliseconds) when the SEI was inserted
userdata	(when registered=true) Uint8Array containing custom data. This data should be parsed according to the application's requirements

Appendix: Installing OvenPlayer

OvenPlayer version 0.10.39 and later can receive SEI data included in video during WebRTC stream playback.

1. OvenPlayer can be downloaded from the GitHub releases page.

2. After extracting the downloaded file, copy all files in the dist directory to your project's library folder:

├─.github
├─demo
├─dist
│      ovenplayer.js
│      ovenplayer.js.map
│      RTCTransformWorker.worker.worker.js
│      RTCTransformWorker.worker.worker.js.map
├─docs
├─packages
└─src

Key files to copy:

• ovenplayer.js: OvenPlayer core library

• RTCTransformWorker.worker.worker.js: Worker script for WebRTC processing

Due to Web Worker CORS policy, ovenplayer.js must be self-hosted, and the RTCTransformWorker.worker.worker.js file must exist in the same path as ovenplayer.js.