WebSocket APIs provide a consistent interface to process streaming image frames across our API stack.

Sample code snippets are provided in JavaScript and tested on Google Chrome

Client Workflow

Establishing Connection to a WebSocket API Service

WebSocket APIs need the WebSocketToken during connection set up. This token should be passed as a query parameter in the URL endpoint. More details about it are mentioned in WebSocketToken - Authentication for WebSocket APIs

To establish a new connection, for example, to the Face Mask API Service, the URL should look like

URL Endpoint

wss://api.skylarklabs.ai/ws/face-mask?authorization=<webSocketToken>

Sample Code to start a new connection

let webSocketToken = "f9d414eeea3a769955fd35288bd03b16189cc631877830b66b405652aa23c9c96cc9f9a0eeed99739d3db55ddda1cec6b73932054eb52ccbdcf6a31b9ab4b995";
let url = `wss://api.skylarklabs.ai/ws/face-mask?authorization=${webSocketToken}`;
var socket = new WebSocket(url);
CODE

Once connection is setup, the client can send and receive messages from the server.

Connection Configuration Options

After a connection is established the client can configure certain settings.

Batch Processing

Batch Processing is an API feature that enables processing multiple image frames to be processed by the server in a single run. Batching frames makes processing faster. This feature is useful for Real Time applications.

To control this feature a parameter batch_size which is persistent for the current connection and can be changed using the set_batch_size action. This action can be used at any point of time during connection use.

socket.send(JSON.stringify({
  action: 'set_batch_size',
  batch_size: 5
}));
CODE

Default value of batch_size for any connection is 1. When setting it to a value greater than 1, the server will only return the response once it receives the complete batch to be processed.

Sending Frames for Processing

Sending frames is a two step procedure.

Start Frame Action

First, a message is sent to the server as shown below

let initParams = {
    action: "start_frame",
    fileSize: 140945
};

{# send the params #}
socket.send(JSON.stringify(initParams));
JS

Here, action start_frame informs server that a new frame will be sent. fileSize is size of the image frame to be processed in bytes.

Maximum image frame size of 500 Kilobytes (51200 Bytes) is supported. It is advised that client applications apply suitable image compression before sending the frame for processing. This will also reduce the network latency.

Sending Image Frame as a byte array

This involves sending byte data to the server. WebSocket API will automatically decode valid image frames from byte streams. Failures, if any, break the connection in which case client will have to establish a new connection.

Receiving Results

Results are sent as JSON messages from the server. Example code to listen to message events in a callback is given below

socket.onmessage = function(event) {
    console.log(`Message Received: ${event.data}`);
};
CODE

Server can send of status messages, errors in connection state and processing results to the client app.

WebSocket APIs do NOT return any image URLs in output, but just the inferences in JSON format. Input images and processing results are NOT stored on the servers and cannot be queried later.

Sample HTML/JavaScript code

Images selected in the HTML Input are sent to Face Mask API over a WebSocket connection.

<html>
    <title>Face Mask Demo</title>
    <body>
        <input type="file" id="image"/>
        <script
          src="https://code.jquery.com/jquery-3.5.1.min.js"
          integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0="
          crossorigin="anonymous">
        </script>
        <script>
            # obtain token from the /web-socket-token API
            let webSocketToken = "f9d414eeea3a769955fd35288bd03b16189cc631877830b66b405652aa23c9c96cc9f9a0eeed99739d3db55ddda1cec6b73932054eb52ccbdcf6a31b9ab4b995";
            let url = `ws://localhost:9090/ws/face-mask?authorization=${webSocketToken}`;
            var socket = new WebSocket(url);

            socket.onopen = function(e) {
              console.log("WebSocket Connection established");
            };

            socket.onmessage = function(event) {
              console.log(`Message Received: ${event.data}`);
            };

            socket.onclose = function(event) {
              if (event.wasClean) {
                console.log(`Connection closed cleanly, code=${event.code} reason=${event.reason}`);
              } else {
                console.log('WebSocket Connection died');
              }
            };

            socket.onerror = function(error) {
              console.log(`Error: ${error.message}`);
            };

        </script>
        <script>

{#            listen to changes on the HTML file input#}
            document.querySelector('input').addEventListener('change', function() {

              var reader = new FileReader();

              reader.onload = function() {
                let arrayBuffer = this.result, array = new Uint8Array(arrayBuffer);

{#                send frame initialization parameters to intimidate server about the start of a new frame and the file size#}
                let initParams = {
                    fileSize: arrayBuffer.byteLength,
                    action: "start_frame"
                }

{#                send the params#}
                socket.send(JSON.stringify(initParams))

{#                send the byte stream for the image#}
                socket.send(array);
              }

              reader.readAsArrayBuffer(this.files[0]);

            }, false);

        </script>
    </body>
</html>
HTML