Offscreen interactivity mode messages
WebSockets API is a connected API between a client (generally a browser) and MyScript Cloud or MyScript Server. This page describes the main offscreen interactivity WebSockets messages
Opening the WebSocket
Open a websocket on this URL wss://cloud.myscript.com/api/v4.0/iink/offscreen with your applicationKey:
GET wss://cloud.myscript.com/api/v4.0/iink/document?applicationKey=XXXX-XXXXXX-XXXXX-XXXX
Authenticating
Authentication with HMAC HandShake
When the application hmac security option is set (default mode), an HMAC challenge must be sent to complete authentication.
Please find here a javascript example on how to compute the HMAC code.
Simple authentication (No HMAC HandShake)
Authentication sequence if the application hmac security option is not set (refer to Administration interface),
Setting up and using a new iink session
Once authenticated, you should use the following messages to set up your iink session.
Using a new iink session
To set up a new iink session, you should first send an initSession message
This message contains the scale to convert input coordinates unit into mm, such that (X coordinate unit * scaleX = mm) and (Y coordinate unit * scaleY = mm).
It also contains an optional configuration block: You can configure the recognition language.
Please refer to the configuration guide for an exhaustive list of configuration options.
{
"type":"initSession",
"scaleX":0.26458333333333334,
"scaleY":0.26458333333333334,
"configuration":
{
...
}
}
The server sends you a sessionDescription containing the iinkSessionId.
{
"type":"sessionDescription",
"iinkSessionId":"fb9d7464-225a-452c-9109-052023c610b2",
"contentPartCount":0}
Creating a new part
The client must declare a new "Raw Content" part with a JIIX mimeTypes which will be used for the exported message:
The server will send an export message containing the recognition result after each contentChanged message.
{
"type":"newContentPart",
"contentType":"Raw Content",
"mimeTypes":["application/vnd.myscript.jiix"]
}
{
"type":"partChanged",
"partIdx":0,
"partId":"rfvgkqum",
"partCount":1
}
{
"type":"newPart",
"idx":0,
"id":"rfvgkqum"
}
You are now ready to send strokes.
Restoring an iink session
The server saves the document periodically. If your session is interrupted, you can recover it within a few minutes.
You must send a restoreSession message containing the iinkSessionId, scale values and configuration of your disconnected session.
{
"type":"restoreSession",
"iinkSessionId":"fb9d7464-225a-452c-9109-052023c610b2",
"scaleX":0.26458333333333334,
"scaleY":0.26458333333333334,
"configuration":{...}
}
The server sends back the session description
{
"type":"sessionDescription",
"iinkSessionId":"fb9d7464-225a-452c-9109-052023c610b2",
"contentPartCount":1
}
You must then ask to reopen your part, providing its id:
{
"type":"openContentPart",
"id":"rfvgkqum"
}
{
"type":"partChanged",
"partIdx":0,
"partId":"rfvgkqum",
"partCount":1
}
You are now ready to update your ink content.
Updating content
Adding strokes
{
"type":"addStrokes",
"processGestures":true,
"strokes":[
{
"id":"stroke-e9f50fb6-4cc7-41ce-a79c-67a8911fef44",
"pointerType":"mouse",
"p":[1, ... ,0.73],
"t":[1719912978137, ... ,1719912978821],
"x":[43,...,108],
"y":[194,...,295]
}
]
}
At this point, the stroke is injected in the engine and its states is 'pending'. The engine will decide if the stroke is a gesture in which case the server sends a "Gesture detected" or not in which case it will send a "Content changed".
Erasing strokes
{
"type":"eraseStrokes",
"strokeIds":
["stroke-f88235b8-b46d-4bfe-95dd-5818fae7eeb4"]
}
Replacing strokes
{
"type":"replaceStrokes",
"oldStrokeIds":
[
"stroke-d6a5f637-9be2-447a-b3f7-021cb115bed0"
],
"newStrokes":
[
{
"id":"stroke-61b12cfd-5d17-402b-ac8f-09dc142dbb89",
"pointerType":"mouse",
"p":[1,...,0.68],
"t":[1719923017231,...,1719923017683],
"x":[417,...,500],
"y":[253,...,395]
},
{
"id":"stroke-c99e4544-a4d6-4149-8e88-c3b53d643725",
"pointerType":"mouse",
"p":[0.77],
"t":[1719923019261],
"x":[983],
"y":[332]
}
]
}
Transforming strokes
Applies a transform on strokes. Possible transforms are :
- TRANSLATE with tx the X coordinate translation and ty the Y coordinate translation.
- ROTATE with a the rotation angle in radians, (x0, y0) the coordinates of the origin point.
- SCALE with scaleX and scaleY the scaling factors.
- MATRIX with xx, yx, tx, xy, yy, ty representing a 2D affine transform, defined as a 3x3 matrix with an implicit third raw of 0 0 1: xx yx tx xy yy ty 0 0 1
Example:
{
"type":"transform",
"transformationType":"TRANSLATE",
"strokeIds":["stroke-f0d251e4-3568-465b-9f70-c9875f3a9f38","stroke-f0d251e4-3568-465b-9f70-c9875f3a9f39"],
"tx":100,
"ty":0
}
Gesture detected
Server sends a gestureDetected message to the browser every time a gesture is detected over ink content when adding strokes.
{
"type":"gestureDetected",
"gestureType":"SCRATCH",
"gestureStrokeId":"stroke-66e664d1-b48e-4694-bc90-b309b9bfb874",
"strokeIds":
["stroke-d6a5f637-9be2-447a-b3f7-021cb115bed0"],
"strokeBeforeIds":[],
"strokeAfterIds":[],
"subStrokes":[
{"id":null,
"x":[503.00003,...,983.00006],
"y":[395.0,...,332.0],
"t":[1719923017691,...,1719923019261],
"p":[0.68,...,0.77],
"pointerType":null,
"pointerId":-1,
"fullStrokeId":"stroke-d6a5f637-9be2-447a-b3f7-021cb115bed0"
}
]
}
You can then decide which action you want to perform, based on strokeIds information: for instance you may decide to erase corresponding strokes by using an eraseStroke message.
Content Changed
Server sends a contentChanged message to the browser every time the content is modified:
{
"type":"contentChanged",
"partId":"socftjfl",
"canUndo":null,
"canRedo":null,
"empty":null,
"undoStackIndex":0,
"possibleUndoCount":0
}
Client can then decide which action to proceed: add strokes, get recognition result, etc.
Getting recognition result
Before retrieving the recognition result, you must make sure that the recognition engine is idle by sending a "waitForIdle" message to the server and receiving its "idle" message.
Waiting for idle
{
"type":"waitForIdle"
}
{
"type":"idle"
}
Getting recognition result
Once recognition engine is idle, you can get recognition result by sending an export message to the server.
{
"type":"export",
"partId":"rfvgkqum",
"mimeTypes":["application/vnd.myscript.jiix"]
}
{
"type":"exported",
"partId":"rfvgkqum",
"exports":
{"application/vnd.myscript.jiix":"{\n \"type\": \"Raw Content\",\n \"bounding-box\": {\n \"x\": 25.7229156,\n \"y\": 54.0333328,\n \"width\": 64.1770782,\n \"height\": 42.7458344\n },\n \"elements\": [ {\n \"id\": \"raw-content/11\",\n \"type\": \"Text\",\n \"bounding-box\": {\n \"x\": 25.7229156,\n \"y\": 54.0333328,\n \"width\": 64.1770782,\n \"height\": 42.7458344\n },\n \"label\": \"he\",\n \"words\": [ {\n \"label\": \"he\",\n \"candidates\": [ \"he\", \"hel\", \"hee\", \"hal\", \"hl\" ],\n \"first-char\": 0,\n \"last-char\": 1,\n \"bounding-box\": {\n \"x\": 25.7229156,\n \"y\": 54.0333328,\n \"width\": 64.1770782,\n \"height\": 42.7458344\n },\n \"items\": [ {\n \"type\": \"stroke\",\n \"id\": \"stroke-f88235b8-b46d-4bfe-95dd-5818fae7eeb4\",\n \"full-id\": \"stroke-f88235b8-b46d-4bfe-95dd-5818fae7eeb4\"\n } ]\n } ],\n \"chars\": [ {\n \"label\": \"h\",\n \"candidates\": [ \"h\", \"H\", \"b\", \"n\", \"k\" ],\n \"word\": 0,\n \"grid\": [ {\n \"x\": 25.1212311,\n \"y\": 52.4228401\n }, {\n \"x\": 45.3454437,\n \"y\": 52.4228401\n }, {\n \"x\": 45.3454437,\n \"y\": 116.49025\n }, {\n \"x\": 25.1212311,\n \"y\": 116.49025\n } ],\n \"bounding-box\": {\n \"x\": 25.7229156,\n \"y\": 54.0333328,\n \"width\": 21.3145828,\n \"height\": 38.5124969\n },\n \"items\": [ {\n \"type\": \"stroke\",\n \"id\": \"00000000010064000000\",\n \"full-id\": \"stroke-f88235b8-b46d-4bfe-95dd-5818fae7eeb4\"\n } ]\n }, {\n \"label\": \"e\",\n \"candidates\": [ \"l\", \"e\" ],\n \"word\": 0,\n \"grid\": [ {\n \"x\": 45.3454437,\n \"y\": 52.4228401\n }, {\n \"x\": 88.1287155,\n \"y\": 52.4228401\n }, {\n \"x\": 88.1287155,\n \"y\": 116.49025\n }, {\n \"x\": 45.3454437,\n \"y\": 116.49025\n } ],\n \"bounding-box\": {\n \"x\": 45.0374985,\n \"y\": 64.6166611,\n \"width\": 44.8624954,\n \"height\": 32.1625061\n },\n \"items\": [ {\n \"type\": \"stroke\",\n \"id\": \"000000006500eb00ff00\",\n \"full-id\": \"stroke-f88235b8-b46d-4bfe-95dd-5818fae7eeb4\"\n } ]\n } ]\n } ],\n \"id\": \"MainBlock\",\n \"version\": \"3\"\n}"
}
}
Contextless gesture
Contextless gesture message allows you to have strokes recognized without any ink context:
{
"type": "contextlessGesture",
"scaleX":0.26458333333333334,
"scaleY":0.26458333333333334,
"stroke":
{
"id":"stroke-f0d251e4-3568-465b-9f70-c9875f3a9f38",
"pointerType":"mouse",
"p": [1,0.75,0.68,0.74],
"t":[1721834093742,1721834093800,1721834093814,1721834093819],
"x":[375,370,369,366],
"y":[255,259,262,266]
}
}
Once this message has been processed by the server, it sends a contextlessGesture message with corresponding gesture type and strokeID:
{
"type":"contextlessGesture",
"strokeId":"stroke-f0d251e4-3568-465b-9f70-c9875f3a9f38",
"gestureType":"scratch"
}
Undo/redo
Undo / redo messages allow you to undo and redo previous action granting you same recognition result. The changes array contain the list of actions to undo/redo with their strokes, for instance "addStrokes", "eraseStrokes", ...
Undo
{
"type":"undo",
"changes":
[
{
"type":"addStrokes",
"processGesture":false,
"strokes":
[
{
"id":"stroke-7abceacd-2b3d-4bcf-a0f7-2d9932966357",
"x":[340,349,370,381,395,402,407,407,404,395,390,382,379,378],
"y":[347,340,323,311,293,279,259,250,240,249,259,286,302,330],
"t":[1722240815034,1722240815110,1722240815139,1722240815150,1722240815181,1722240815199,1722240815217,1722240815247,1722240815274,1722240815373,1722240815386,1722240815416,1722240815431,1722240815464],
"p":[1,0.66,0.48,0.6,0.52,0.6,0.55,0.81,0.68,0.64,0.67,0.47,0.6,0.47]
}
]
}
]
}
Redo
{
{
"type":"redo",
"changes":
[
{
"type":"replaceStrokes",
"oldStrokeIds":["stroke-c84f9dab-2ef3-414e-a36e-ea0de77260bf"],
"newStrokes":
[
{
"id":"stroke-7abceacd-2b3d-4bcf-a0f7-2d9932966357",
"x":[340,349,370,381,395,402,407,407,404,395,390,382,379,378],
"y":[347,340,323,311,293,279,259,250,240,249,259,286,302,330],
"t":[1722240815034,1722240815110,1722240815139,1722240815150,1722240815181,1722240815199,1722240815217,1722240815247,1722240815274,1722240815373,1722240815386,1722240815416,1722240815431,1722240815464],
"p":[1,0.66,0.48,0.6,0.52,0.6,0.55,0.81,0.68,0.64,0.67,0.47,0.6,0.47]
}
]
}
]
}
}
Ping-pong
The ping pong mechanism prevents the websocket from being disconnected with a timeout (client, server or network equipment in the middle).
{
"type":"ping"
}
If server is alive, it will respond as soon as the ping message arrives with a pong message
{
"type":"pong"
}
Common behavior
On error, the server will send an error message and close the WebSocket.
{
"type": "error",
"message": "details about the error"
}