Editing

This page focuses on the role of the Editor object, the central point to interact with content.

Setting a part

An Editor works on a ContentPart. To setup the link between them, call the editor’s setPart() method and pass it the part you opened or created:

// Create a new package
ContentPackage contentPackage = engine.createPackage(packageNameFile);

// Create a new part
ContentPart contentPart = contentPackage.createPart("Text");

// Accessing editor
Editor editor = editorView.getEditor();

// Associate editor with the new part
editor.setPart(contentPart);

Input capture

Pen, finger and tool management

Following Interactive Ink interaction patterns, iink API considers that pen events are dedicated to write or edit content (text, math or shape content, edit gestures, etc.) and touch events to manipulate content (select, drag & drop, scroll, etc.).

As you are in charge of propagating the events to the editor - in case your users are writing with a finger or a capacitive stylus or if your application is built around a modal set of tools - you can choose what iink SDK shall consider as pen or touch events.

The type of input is defined by the PointerType enum.

You may use it to select tools with specific behaviors as well (for example the eraser).

Guides

Text Document and Text parts have guides set by default. Guides provide useful hints for end users to know where to write and at what size. They also improve the recognition accuracy, provided that handwriting uses them as baselines.

You can enable or disable the guides of a Text part via the text.guides.enable key of the engine or editor configuration. The vertical spacing between guides can be tuned via the text styling options.

Incremental input

Interactive Ink SDK typically processes user input in real time. You thus have to tell how pointers (pen, finger) are interacting with the capture surface (typically a screen or a graphical tablet).

This can be done by calling the following methods of the Editor object:

• pointerDown() - When the pointer first touches the surface.
• pointerMove() - When the pointer moves while staying in contact with the surface.
• pointerUp() - When the pointer is lifted from the surface.

Each of these methods requires you to provide:

• x and y - The coordinates in pixels of the pointer on the surface.
• t - The timestamp of the pointer event

• f - The pressure information associated to the event (normalized between 0 and 1)

• pointerType - The type of pointer (pen, finger or a predefined tool like the eraser: see the PointerType enum)

• pointerId - An identifier for this pointer.

Example:

editor.pointerDown(0.0f, 0.0f, new Date().getTime(), .7f, PointerType.PEN, 1);
editor.pointerMove(1.5f, 2.0f, new Date().getTime(), .6f, PointerType.PEN, 1);
editor.pointerUp(2.0f, 4.0f, new Date().getTime(), .5f, PointerType.PEN, 1);

Remarks:

• The timestamp is typically the time in ms since Jan 1st, 1970. You can set it to -1 to let iink SDK generate one for you based on the current time of the system.
• Interactive Ink SDK does not use the pressure information. It is stored in the model and can be retrieved at export or when implementing your own inking. If you don’t have or need this information, you can set it to 0.
• If you only have one pointer simultaneously active, you can pass a pointer id of -1.

In the most simple case, you can write something like:

final long NO_TIMESTAMP = -1;
final float NO_PRESSURE = 0.0f;
final int NO_POINTER_ID = -1;

editor.pointerDown(0.0f, 0.0f, NO_TIMESTAMP, NO_PRESSURE, PointerType.PEN, NO_POINTER_ID);
editor.pointerMove(1.5f, 2.0f, NO_TIMESTAMP, NO_PRESSURE, PointerType.PEN, NO_POINTER_ID);
editor.pointerUp(2.0f, 4.0f, NO_TIMESTAMP, NO_PRESSURE, PointerType.PEN, NO_POINTER_ID);

Series of events

In some cases, you may want to send a set of strokes to the engine in a single pass, for instance if you import ink from outside of the iink model that you want to process as a single batch.

For all types of parts but “Text Document”, iink SDK provides a method to input in a single pass a series of pointer events, pointerEvents(), that take in parameter an array of PointerEvent objects.

Here is an example:

ArrayList<PointerEvent> events = new ArrayList<PointerEvent>();

// Stroke 1
events.add(new PointerEvent().down(184.f, 124.f));
events.add(new PointerEvent().move(184.f, 125.f));
events.add(new PointerEvent().move(184.f, 128.f));
events.add(new PointerEvent().move(184.f, 133.f));
events.add(new PointerEvent().move(184.f, 152.f));
events.add(new PointerEvent().move(184.f, 158.f));
events.add(new PointerEvent().move(184.f, 163.f));
events.add(new PointerEvent().move(183.f, 167.f));
events.add(new PointerEvent().move(183.f, 174.f));
events.add(new PointerEvent().move(183.f, 183.f));
events.add(new PointerEvent().up(183.f, 184.f));

// Stroke 2
events.add(new PointerEvent().down(150.f, 126.f));
events.add(new PointerEvent().move(151.f, 126.f));
events.add(new PointerEvent().move(152.f, 126.f));
events.add(new PointerEvent().move(158.f, 126.f));
events.add(new PointerEvent().move(166.f, 126.f));
events.add(new PointerEvent().move(184.f, 126.f));
events.add(new PointerEvent().move(190.f, 128.f));
events.add(new PointerEvent().move(196.f, 128.f));
events.add(new PointerEvent().move(200.f, 128.f));
events.add(new PointerEvent().move(207.f, 128.f));
events.add(new PointerEvent().move(208.f, 128.f));
events.add(new PointerEvent().up(209.f, 128.f));

// Feed the editor
editor.pointerEvents(events.toArray(new PointerEvent[0]), false);

For the particular case of “Text Document” parts, you should:

1. Send each batch of pointer events to a dedicated “Text”, “Math”, “Diagram” or “Drawing” part depending on the type of the content you want to process. Remember to disable the guides on “Text” parts if you cannot ensure that they will match the baselines of the ink words.
2. Call waitForIdle() to ensure that the recognition is complete.
3. Paste as a block into your “Text Document” part at the appropriate location.

Edit and decoration gestures

Interactive Ink SDK supports all the standard gestures defined as part of Interactive Ink.

There is nothing particular to do to benefit from gestures. The SDK will take care of detecting and applying the effect of the gestures from the provided input without any plumbing needed.

Decorations can be styled and will be taken into account when generating some of the export formats (ex: text underlining will be ignored for a simple text export, will turn bold in the case of a docx export and be semantically tagged in a jiix export).

Other edit operations

The following operations can be directly made on the content of a part via an Editor object:

• Undo/Redo : iink operations handled by the undo/redo stack are operations that modify the model. Such operations include adding stroke, applying gestures, converting the content.

• Clear is applied on the whole “ContentPart”.

Most operations, however, are to be done on content blocks.

Recognition feedback

The UI Reference Implementation comes with a “smart guide” component that lets you provide real-time text recognition feedback to your end users and allows them to select alternative interpretations from the engine.

Refer to the page describing how to work with text recognition candidates for more information.

Monitoring changes in the model

There are cases where it makes sense to be notified of what occurs in the model. For instance, you may want to update the states of the undo/redo buttons of your interface or only permit an export when there is something to export.

You can call addListener() and removeListener() to respectively attach and remove an object of your choice that implements the IEditorListener interface: the interface has several methods that let you know when any change occurs within the model.

IEditorListener also provides an onError() method that you can implement to be notified when anything goes wrong. It is strongly recommended to implement it, as it allows detecting some frequently seen issues such as recognition assets or configurations not found by the engine.

In addition, the Editor class provides other useful methods/properties, such as:

• isIdle() - Returns true if any ongoing processing by the engine is over
• waitForIdle() - Blocks your thread until the engine is idle. It allows waiting for recognition to be complete before exporting, converting or manipulating the content.

Block management

Content blocks

A content block is a semantic subdivision of the content, and may contain data and/or other blocks. It has a unique id, a defined type (“Text”, “Math”, “Diagram”, “Drawing”, “Raw Content”, “Container”…) and a bounding box.

For example:

• A “Math” part will only contain a single block, hosting the math content itself.
• A “Text Document” part will be more complex, as it can host text paragraphs, math equations, diagrams and drawings, arranged in a complex layout, sometimes one after the other, sometimes alongside one another. This is where “Container” blocks can be used to semantically group sub-blocks together.

The following illustration shows how these different blocks relate together inside their parent parts:

When diagram.enable-sub-blocks is set to true in the configuration, “Diagram” blocks contain sub blocks of type “Text”, “Node”, “Edge” or “Polyedge” describing the content of the diagram.

Navigating the block hierarchy

The different blocks form a hierarchy, which root can be obtained by calling the getRootBlock() method on the parent part. Each block, in turn, has a getChildren() method that will return its own children, if any, and a getParent() one that will return its parent.

It is important to note that a block hierarchy is only valid at a given point in time. For instance, in the case of a Text Document, inserting new blocks, removing a text paragraph using a gesture, etc. are some examples of events that may invalidate the block hierarchy you previously retrieved.

You can check if a block is still valid by calling its isValid() method. Alternatively, watching contentChanged() events using an IEditorListener on your Editor object will provide hints that your blocks may have become invalid (the list of ids of impacted blocks is provided in parameter).

Block addition

Any part you create will contain a root block.

You may however use the addBlock() method of the editor to add a new block at a given location in compatible parts, as a way to import content (only “Text Document” parts support this feature as of now).

A dedicated addImage() method allows you to insert an image inside a Text Document part.

Operations with blocks

Some operations are possible with blocks, at a finer granularity than at the sole part level:

• hitBlock() lets you know the top-most block at a given location, if there is any. It makes it possible for example to know which block is tapped or pressed by a user.
• removeBlock() lets you remove a non-root block.
• convert() lets you convert the ink inside a given block.
• export_() lets you export the content of a specific block, including its children.
• copy() lets you copy a block into the internal clipboard. You can then paste it at a given location using paste(), much like you would add a new block. Text copy and paste works from various text block sources (“Text”, “Diagram” or “Raw Content”) to either a “Text Document” or a “Text” part. In the latter case, the part must be empty before performing the paste.
• You can monitor the events affecting a given block, as information about the impacted blocks is provided to you by the contentChanged() method of the IEditorListener interface.

Editor-level configuration

While iink SDK can be globally configured at engine level, it is possible to override this configuration at editor level. This is particularly useful in form-like use cases, where you need to manipulate fields with different configurations.

You can access the configuration of a specific editor by calling getConfiguration() and set the values of the keys that should override the global configuration, in a cascade-like manner. Values of the keys you do not explicitly set at editor-level still follow engine-level configuration.

For example:

Configuration globalConfig = engine.getConfiguration();
Configuration editorConfig = editor.getConfiguration();

// Global configuration values apply ...
String globalUnit = globalConfig.getString("math.solver.angle-unit"); // -> "deg"
String editorUnit = editorConfig.getString("math.solver.angle-unit"); // -> "deg"
globalConfig.setString("math.solver.angle-unit", "rad");
globalUnit = globalConfig.getString("math.solver.angle-unit");        // -> "rad"
editorUnit = editorConfig.getString("math.solver.angle-unit");        // -> "rad"

// ... except if overridden at editor level
editorConfig.setNumber("math.solver.fractional-part-digits", 4);
globalConfig.setNumber("math.solver.fractional-part-digits", 2);
Number editorDigits = editorConfig.getNumber("math.solver.fractional-part-digits"); // -> 4
Number globalDigits = globalConfig.getNumber("math.solver.fractional-part-digits"); // -> 2

Back to the example

You previously created an Editor object. As you rely on the reference rendering implementation provided by MyScript, ink input will be transparently managed by the EditorView. All you have to do is to load the part into the editor at the end of the openContent() method you previously wrote:

public void openContent()
{
  // Load the part in the editor
  editor.setPart(contentPart);
}

If you launch the app, you can now write in it and see actual ink being rendered!

Next, you will plug the undo, redo and clear buttons.

This can be conveniently done by calling the appropriate Editor methods in reaction to user actions:

public void undo()
{
  editor.undo();
}

public void redo()
{
  editor.redo();
}

public void clear()
{
  editor.clear();
}

As a “Math” part consists of a single root block, you will be able to convert it very easily. This will be the topic of the next step of this guide!