Editing

This page introduces the role of the IINKEditor object, the central point to interact with content.

Editor creation and configuration

Creation

An IINKEditor is the main entry point to act on interactive content. It is responsible for building and updating a content model from users' input events and/or edition commands. It is instantiated from an IINKEngine object:

//create the Editor
let editor = engine.createEditor(renderer: renderer, toolController: toolcontroller)

The ToolController argument is optional. If you omit it, the editor will instantiate a ToolController with the default settings.

If you use the reference implementation, the EditorViewModel will create the editor automatically for you.

Editor-level configuration

While iink SDK can be globally configured at engine level, it is possible to override this configuration at editor level. This is particularly useful in form-like use cases, where you need to manipulate fields with different configurations.

You can access the configuration of a specific editor via the configuration property and set the values of the keys that should override the global configuration, in a cascade-like manner. Values of the keys you do not explicitly set at editor-level still follow engine-level configuration.

For example:

let globalConfig = engine.configuration
let editorConfig = editor.configuration

// Global configuration values apply ...
var globalUnit =  globalConfig.string(forKey: "math.solver.angle-unit") // -> "deg"
var editorUnit =  editorConfig.string(forKey: "math.solver.angle-unit") // -> "deg"

globalConfig.set(string: "rad", forKey: "math.solver.angle-unit")
globalUnit = globalConfig.string(forKey:"math.solver.angle-unit")   // -> "rad"
editorUnit = editorConfig.string(forKey: "math.solver.angle-unit")   // -> "rad"

// ... except if overridden at editor level
editorConfig.set(number: 4, forKey: "math.solver.fractional-part-digits")
globalConfig.set(number: 2, forKey: "math.solver.fractional-part-digits")

var editorDigits =  editorConfig.number(forKey: "math.solver.fractional-part-digits", defaultValue: 3) // -> 4
var globalDigits =  globalConfig.number(forKey: "math.solver.fractional-part-digits", defaultValue: 3) //  ->  2
Language-related settings like text.configuration.bundle and text.configuration.name behave in a particular way, in that they are only considered the very first time a given part is set to an editor and cannot be changed afterwards.

Setting a part

An IINKEditor works on an IINKContentPart. To setup the link between them, call the editor's part property and pass it the part you opened or created:

//create the content Package
contentPackage = engine.createPackage(packageName)

//create the content part
contentPart = contentPackage.createPart(with: partType)

//set the part to the editor
editor.part=contentPart
You have to make sure that you previously called setViewSize: on the editor and attached to it a font metrics provider before setting the part. If you use the reference implementation, the EditorViewModel will do it automatically for you.

Guides

Text Document and Text parts have guides set by default. Guides provide useful hints for end users to know where to write and at what size. They also improve the recognition accuracy, provided that handwriting uses them as baselines.

You can enable or disable the guides of a Text part via the text.guides.enable key of the engine or editor configuration. The vertical spacing between guides can be tuned via the text styling options.

If you know that your input will not match the guides, for instance with ink coming from an unstructured context such as a sheet of paper, you must disable them to ensure a good recognition.

Input capture

You are in charge of capturing the input events. This section gives you some hints on how to transmit the captured inputs to the editor.

Incremental input

MyScript iink SDK typically processes user input in real time. You thus have to tell how pointers are interacting with the capture surface (typically a screen or a graphical tablet).

This can be done by calling the following methods of the IINKEditor object:

  • pointerDown(point:timestamp:force:type:pointerId:) - When the pointer first touches the surface.
  • pointerMove(point:timestamp:force:type:pointerId:) - When the pointer moves while staying in contact with the surface.
  • pointerUp(point:timestamp:force:type:pointerId:) - When the pointer is lifted from the surface.

Each of these methods requires you to provide:

  • x and y - The coordinates in pixels of the pointer on the surface.
  • t - The timestamp of the pointer event
  • f - The pressure information associated to the event (normalized between 0 and 1)
  • type - The type of pointer: see the IINKPointerType
  • pointerId - An identifier for this pointer.

Example:

editor.pointerDown(point: CGPoint(x: 0.0, y: 0.0), timestamp: Int64((Date().timeIntervalSince1970 * 1000.0).rounded()), force: 0.7, type: IINKPointerType.pen, pointerId: 0)

editor.pointerMove(point: CGPoint(x: 1.4, y: 2.4), timestamp: Int64((Date().timeIntervalSince1970 * 1000.0).rounded()), force: 0.6, type: IINKPointerType.pen, pointerId: 0)

editor.pointerUp(point: CGPoint(x: 2.4, y: 4.0), timestamp: Int64((Date().timeIntervalSince1970 * 1000.0).rounded()), force: 0.5, type: IINKPointerType.pen, pointerId: 0)

Remarks:

  • The pointer events are analysed and interpreted according to the tool that is currently associated to the pointer type: when using the PEN tool, iink SDK triggers the recognition, with the SELECTOR tool iink SDK determines a selection, etc.
  • The timestamp is typically the time in ms since Jan 1st, 1970. You can set it to -1 to let iink SDK generate one for you based on the current time of the system.
  • By default, iink SDK does not use the pressure information. It is stored in the model and can be retrieved at export or when implementing your own inking. If you don't have or need this information, you can set it to 1.
  • If you only have one pointer simultaneously active, you can pass a pointer id of -1.

In the most simple case, you can write something like:

editor.pointerDown(point: CGPoint(x: 0.0, y: 0.0), timestamp: -1, force: 0, type: IINKPointerType.pen, pointerId: 0)

editor.pointerMove(point: CGPoint(x: 1.4, y: 2.4), timestamp: -1, force: 0, type: IINKPointerType.pen, pointerId: 0)

editor.pointerUp(point: CGPoint(x: 2.4, y: 4.0), timestamp: -1, force: 0, type: IINKPointerType.pen, pointerId: 0)

Series of events

In some cases, you may want to send a set of strokes to the engine in a single pass, for instance if you import ink from outside of the iink model that you want to process as a single batch.

For all types of parts but "Text Document", iink SDK provides a method to input in a single pass a series of pointer events, pointerEvents:, that take in parameter an array of IINKPointerEvent objects.

Here is an example:

// Build the pointer events array
let n = 23;
var eventArr:[IINKPointerEvent]=[]

// Stroke 1
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.down,x: 184, y: 124,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent( eventType: IINKPointerEventType.move,x: 184, y: 125,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 184, y: 128,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 184, y: 133,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 184, y: 152,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 184, y: 158,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 184, y: 163,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 183, y: 167,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 183, y: 174,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move,x: 183, y: 183,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.up,x: 183, y: 184,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))

// Stroke 2
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.down, x: 150, y: 126,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 151, y: 126,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1 ))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 152, y: 126,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 158, y: 126,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 166, y: 126,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 184, y: 126,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 190, y: 128,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 196, y: 128,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 200, y: 128,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 207, y: 128,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.move, x: 208, y: 128,t: -1, f: 0,pointerType: IINKPointerType.pen , pointerId: 1))
eventArr.append(IINKPointerEvent(eventType: IINKPointerEventType.up,x: 209, y: 128,t: -1, f: 0, pointerType: IINKPointerType.pen , pointerId: 1 ))

//provide the pointer events to the editor
editor.pointerEvents(&eventArr, count: n, doProcessGestures: false)

For the particular case of "Text Document" parts, you should:

  1. Send each batch of pointer events to a dedicated "Text", "Math", "Diagram" or "Drawing" part depending on the type of the content you want to process. Remember to disable the guides on "Text" parts if you cannot ensure that they will match the baselines of the ink words.
  2. Call waitForIdle() to ensure that the recognition is complete.
  3. Paste as a block into your "Text Document" part at the appropriate location.
When pasting a "Text" part into a "Text Document", iink SDK will attempt to automatically adjust the handwritten content to the guides.

It is possible to configure the maximum number of threads to be used by the engine for text recognition, by setting max-recognition-thread-count configuration value to the number of threads to be used (default is 1). This tuning might be relevant when processing at one time a large series of events.

Edit and decoration gestures

MyScript iink SDK supports all the standard gestures defined as part of Interactive Ink.

There is nothing particular to do to benefit from gestures when using the PEN tool. The SDK will take care of detecting and applying the effect of the gestures from the provided input without any plumbing needed.

Decorations can be styled and will be taken into account when generating some of the export formats (ex: text underlining will be ignored for a simple text export, will turn bold in the case of a docx export and be semantically tagged in a jiix export).

You can get a notification when a pen or a touch gesture is detected by attaching to the IINKEditor a delegate conforming to the IINKGestureDelegate protocol.

By default, the behaviour associated with the gesture is the same as in previous iink versions. But you can choose the action associated to the gesture to decide what to do with its stroke:

  • ADD_STROKE to add the gesture stroke as a regular stroke (if relevant) without applying the gesture behavior.
  • APPLY_GESTURE to apply the gesture behavior, as configured in the editor.
  • IGNORE to discard the gesture stroke without applying the gesture behavior.
In order to avoid any action conflict, only one IINKGestureDelegate can be used per editor. ::

Other edit operations

The following operations can be directly made on the content of a part via an Editor object:

  • Undo/Redo : iink operations handled by the undo/redo stack are operations that modify the model. Such operations include adding stroke, applying gestures, converting the content.
For further details or if you need to integrate iink SDK undo/redo with your own undo/redo stack, refer to this page (advanced)
  • Clear is applied on the whole "ContentPart".

Most operations, however, are to be done on content blocks.

Recognition feedback

Display recognized text in real time

The UI Reference Implementation comes with a "smart guide" component that lets you provide real-time text recognition feedback to your end users and allows them to select alternative interpretations from the engine.

Refer to the page describing how to work with text recognition candidates for more information.

Improve user experience

Sometimes, the block extraction or segmentation does not work properly, leading to a frustrating user experience with some ink recognized as text instead of shape, or shape as drawing, etc. In order to cope with such a situation, the lasso selector can be used to select some content, and then force the selected content to be recognized as a specific type and/or grouped with the editor setSelectionType(). The list of possible types is provided by the getAvailableSelectionTypes() method.

Monitoring changes in the model

There are cases where it makes sense to be notified of what occurs in the model. For instance, you may want to update the states of the undo/redo buttons of your interface or only permit an export when there is something to export.

You can attach a delegate conforming to the IINKEditorDelegate protocol to an IINKEditor object': the protocol has several methods that let you know when any change occurs within the model.

IINKEditorDelegate also provides an onError: method that you can implement to be notified when anything goes wrong. It is strongly recommended to implement it, as it allows detecting some frequently seen issues such as recognition assets or configurations not found by the engine.

In addition, the IINKEditor class provides other useful methods/properties, such as:

  • idle - Returns true/YES if any ongoing processing by the engine is over
  • waitForIdle - Blocks your thread until the engine is idle. It allows waiting for recognition to be complete before exporting, converting or manipulating the content.

If you rely on the reference rendering implementation provided by MyScript, ink input will be transparently managed by the EditorViewController.

At this stage, you have learnt to write in an application and see actual ink being rendered!

The topic of the next step of this guide is to illustrate how the Editor handles selections and blocks.