OffscreenEditor

This page explains how you can use interactivity without using iink SDK rendering.

Offscreen Interactivity use case

Up to iink SDK 2.0, it was necessary to use the iink SDK rendering in order to benefit from iink SDK interactive features. In 2.1, while keeping our classical Editor and its related objects and interfaces, we introduced the Offscreen Interactivity feature based on an OffscreenEditor.

This OffscreenEditor lets you programmatically drive the content model update by keeping the incremental recognition principle and using gesture notifications. The application can rely on its own rendering to manage the captured strokes and display its model. Recognition results are available on demand with dedicated APIs.

Offscreen Interactivity main objects

The OffscreenEditor is only available for “Raw Content” parts, but the high flexibility of “Raw Content” allows many specific behaviors depending on its configuration as described here.

It relies on these new objects and interfaces:

• OffscreenEditor is a service allowing offscreen interactivity. It is the entry point to edit an associated part and receive change notifications.
• ItemIdHelper allows working with item IDs (i.e. strokes). It is associated with an OffscreenEditor.
• IOffscreenEditorListener the listener interface for receiving OffscreenEditor events.
• IOffscreenGestureHandler The listener interface for handling gesture events.

Offscreen Interactivity undo/redo

• HistoryManager gives you access to undo/redo stack manipulation, just like the legacy Editor and content of changesets. The goal of the HistoryManager is to allow you to use undo/redo to preserve text recognition, rather than redoing strokes, which can change stroke order and recognition output.
• To help you synchronize your undo stack with ours, we also give you access to changesets, so you can know what our next undo/redo will do. A changeset contains N ChangesetOperation which each corresponds to an operation (ADD, ERASE or TRANSFORM) on a list of item IDs.
• For performance reasons, a HistoryManager is associated with an OffscreenEditor on creation, only if offscreen-editor.history-manager.enable is set to true (by default this parameter is false). It can then be accessed with OffscreenEditor getHistoryManager() method.

Math usage helper

When you configure your OffscreenEditor to classify and recognize math content in your “Raw Content”, the MathSolverController object is the entry point for interacting with your math content.

The default Math solver configuration associated with OffscreenEditor ensures that strokes generated by the MathSolverController and their bounding box are provided without any additional configuration when requesting the JIIX action output of the MathSolverController. When users write the equal sign at the right end of the equation, the bounding box and the strokes of the result can be retrieved easily by parsing the JIIX returned by the getActionOutput method.

Improve user experience

Sometimes, the block extraction or ink classification does not give the expected result, leading to a degraded user experience with some ink recognized as text instead of shape, or shape as drawing, etc.

In order to cope with such a situation, you can force a set of items to be classified as a specific type and/or grouped them with the OffscreenEditor setItemsType(). The getAvailableItemsTypes() returns the list of possible types for a set of item ids.

Setting part

The setPart API has an optional boolean parameter with a default value of false. When set to true, the part and its content fields are reconfigured using values from the current configuration, which overrides any existing values in the part. This is useful when reusing a part with different configuration settings, such as a new language or updated recognition or classification parameters.

Offscreen Interactivity API and samples

The offscreen-interactivity sample is an Android sample which shows how to integrate MyScript iink SDK interactivity with your own rendering. It drives the content model by sending the captured strokes to iink SDK and keeps the incremental recognition principle and gesture notifications. This sample uses a third-party rendering library to manage the captured strokes and display its model, and get real-time recognition results and gesture notifications.