Ink Recognition and Ink Analysis

Being able to take handwritten notes or annotations is nice, but the real power of Tablet PCs comes from the ability to analyze and recognize digital Ink. Recognition of handwriting is important as it allows for the conversion of digital Ink into standard text strings. Ink analysis takes the concept a step further and adds spatial interpretation to the mix to apply further semantics. Gesture recognition enables the user to trigger real-time actions.

An application can have several ways to interpret and recognize digital Ink. Simple handwriting recognition returns the most likely result as a string, but you can dig a lot deeper and retrieve alternate results, confidence levels, and much more. It is often helpful to divide Ink into individual segments, such as words or paragraphs.

But you can retrieve more information from Ink beyond its simple text representation. Users can perform gestures that your application can interpret separately from the writing itself. For instance, your application could recognize a scratch out gesture to erase a section of previously written Ink.

Your application may need to analyze Ink based on its location. One example is that your application might recognize Ink written at the side of a document with a line attaching it to an area of the document as an annotation. This sort of functionality also requires recognition of primitives, such as lines, circles, triangles, or rectangles, and recognition of the spatial relationship.

Recognizing Ink as Text

Simple recognition is so straightforward that I would almost call it primitive. This, of course, is only true from the developer’s point of view after applying the functionality of the SDK. There is nothing simple or primitive about handwriting recognition at all, but unless you work for Microsoft and/or write custom recognizers, you will not have to worry about the complexity of it.

As a first Ink recognition example, I will walk you through a simple form that provides a panel area that is Ink-enabled through a simple InkCollector (or InkOverlay) object (Figure 1). The details of the implementation of the form are not particularly important (see Listing 1 if you want to follow along with the examples). What matters is that you ultimately arrive at a situation where you have an Ink object stored inside the InkCollector or InkOverlay object. In this first simple example, you’ll learn to convert that digital Ink to a text string. You’ll use a piece of code like this.

This takes all the strokes inside the Ink object and returns the most likely recognition result. A lot of work needs to happen under the hood. The system first has to determine which recognizers are installed. Almost all Tablet PC operating systems ship with multiple recognizers including (in the USA) an English handwriting recognizer and a gesture recognizer. To retrieve the most likely result, the default recognizer is used (on a US Windows Tablet PC operating system, that is likely be the English handwriting recognizer), and the collection of strokes is passed to the recognizer engine. This returns a result set that includes a whole lot of information (I will explore the details below), but most of it is ignored and only the single most likely result is returned.

You can write code to implement this approach yourself.

You can use a Recognizers object to determine the default recognizer installed (which replicates the behavior of the ToString() method. In real-life scenarios, it is more prudent to explicitly pick a specific language recognizer, such as the US-English recognizer, instead of relying on the default being set properly). Once you have that object, you have to create a private context.

By: Markus Egger

Markus is an international speaker, having presented sessions at numerous conferences in North & South America and Europe. Markus has written many articles for publications including CoDe Magazine, Visual Studio Magazine, MSDN Brazil, ASP.net Pro, FoxPro Advisor, Fuchs, FoxTalk and Microsoft Office & Database Journal. Markus is the publisher of CoDe Magazine.

Markus is also the President and Chief Software Architect of EPS Software Corp., a custom software development and consulting firm located Houston, Texas. He specializes in consulting for object-oriented development, Internet development, B2B, and Web Services. EPS does most of its development using Microsoft Visual Studio (.NET). EPS has worked on software projects for Fortune 500 companies including Philip Morris, Qualcomm, Shell, and Microsoft. Markus has also worked as a contractor on the Microsoft Visual Studio team, where he was mostly responsible for object modeling and other object- and component-related technologies.

Markus received the Microsoft MVP Award (1996-present) for his contributions to the developer community. Visual LandPro, one of the applications Markus was responsible for, was nominated three times in the Microsoft Excellence Awards.

megger@eps-software.com