CODE Magazine - Article: Windows Automation API 3.0 Overview

Reader rating:
Click here to read 3 comments about this article.
Article source: CoDe (2008 - Vol. 5 - Issue 4 - Windows Accessibility Focus)

Article Pages: < Previous - 1 2 3 - Next >

Windows Automation API 3.0 Overview (Cont.)

UI Automation Specification

The UI Automation specification provides more flexible programmatic access to UI elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. The specification can be supported across platforms other than Microsoft Windows.

The implementation of that specification in Windows is also called UI Automation. UI Automation is broader in scope than just an interface definition. UI Automation provides:
An object model and functions that make it easy for client applications to receive events, retrieve property values, and manipulate UI elements.
A core infrastructure for finding and fetching across process boundaries.
A set of interfaces for providers to express tree structure and some general properties.
A set of interfaces for providers to express other properties and functionality specific to the control type.
To improve on Microsoft Active Accessibility, UI Automation aims to address the following goals:
Enable efficient out-of-process clients, while continuing to allow in-process access.
Expose more information about the UI in a way that allows clients to be out-of-process.
Coexist with and leverage Microsoft Active Accessibility without inheriting its limitations.
Provide an alternative to IAccessible that is simple to implement.
The implementation of the UI Automation specification in Windows features COM-based interfaces and managed interfaces.

UI Automation Elements

UI Automation exposes every piece of the UI to client applications as an Automation Element. Providers supply property values for each element; invoking a method on an element invokes the corresponding method on the provider. Elements are exposed as a tree structure, with the desktop as the root element.

UI Automation offers much better performance for out-of-process client practices (400% or faster for some scenarios than Microsoft Active Accessibility running out-of-process), while adding richness and flexibility to support the latest user interface designs.

Automation elements expose common properties of the UI elements they represent. One of these properties is the control type, which describes its basic appearance and functionality (for example, a “button” or a “check box”).

UI Automation Tree

The Automation Tree represents the entire UI: the root element is the current desktop and child elements are application windows. Each of these child elements can contain elements representing menus, buttons, toolbars, and so on. These elements in turn can contain elements like list items, as shown in Figure 5.

Click for a larger version of this image.

Figure 5: This UI Automation tree represents all elements on the Run dialog.

UI Automation providers for a particular control support navigation among the child elements of that control. However, providers are not concerned with navigation between these control subtrees. This is managed by the UI Automation core, using information from the default window providers.

To help clients process UI information more effectively, the framework supports alternative views of the automation tree: (1) Raw View, (2) Control View, and (3) Content View. As Table 1 shows, the type of filtering determines the views, and the client defines the scope of a view.

UI Automation Properties

The UI Automation specification defines two kinds of properties: (1) automation element properties and (2) control pattern properties. Automation element properties apply to most controls, providing fundamental information about the element such as its name. Examples are listed in Table 2. Control pattern properties apply to control patterns, described next.

Unlike with Microsoft Active Accessibility, every UI Automation property is identified by a GUID and a programmatic name, which makes new properties easier to introduce.

UI Automation Control Patterns

A control pattern describes the attributes and functionality of an automation element. For example, a simple “clickable” control like a button or hyperlink should support the Invoke control pattern to represent the “click” action.

Each control pattern is a canonical representation of possible UI features and functions, as shown in Table 3. There are 22 control patterns defined to date, some of which are new in Windows 7, and the Windows Automation API can support custom control patterns. Unlike with Microsoft Active Accessibility role or state properties, one automation element can support multiple UI Automation control patterns.

UI Automation Control Types

A control type is another automation element property that specifies a well-known control that the element represents. Currently, UI Automation defines 38 control types, including; Button, Check Box, Combo Box, Data Grid, Document, Hyperlink, Image, ToolTip, Tree, and Window.

Before you can assign a control type to an element, the element needs to meet certain conditions, including a particular automation tree structure, property values, control patterns, and events. However, you are not limited to these. You can extend a control with custom patterns and properties as well as with the pre-defined ones.

The total number of pre-defined control types is significantly lower than Microsoft Active Accessibility accRole definitions, because you can combine UI Automation control patterns to express a larger set of features while Microsoft Active Accessibility roles cannot. You can also customize the description of control type by LocalizedControlType property while keeping the baseline type as defined.

UI Automation Events

UI Automation events notify applications of changes to and actions taken with automation elements. The four different types of UI Automation events, as listed in Table 4, do not necessarily mean that the visual state of the UI has changed. The UI Automation event model is independent of the WinEvent framework in Windows, although the Windows Automation API can make UI Automation events interoperable with the Microsoft Active Accessibility framework.

IAccessibleEx Interface

The IAccessibleEx interface helps existing applications or UI libraries extend their Microsoft Active Accessibility object model to support UI Automation without rewriting everything from scratch. With IAccessibleEx, you can implement only the differences between Microsoft Active Accessibility and UI Automation object models.

Because the MSAA-to-UIA Proxy translates the object models of IAccessibleEx-enabled Microsoft Active Accessibility servers as UI Automation object models, UI Automation clients don’t have to do any extra work. The IAccessibleEx interface can enable classic in-process Microsoft Active Accessibility clients to interact directly with UI Automation providers, too.

UI Automation Community Promise Specifications

The entire accessibility framework specification is available under Microsoft Open Specification Promise (http://www.microsoft.com/interop/osp/default.mspx). Windows Automation API 3.0 is a Windows implementation of the specification.

Find more information at the MSDN Accessibility Developer Center http://msdn.microsoft.com/accessibility/

Automation Tree	Description
Raw View	The full tree of automation element objects for which the desktop is the root.
Control View	A subset of the raw view that closely maps to the UI structure as the user perceives it.
Content View	A subset of the control view that contains content most relevant to the user, like the values in a drop-down combo box.

Table 2: Examples of UI Automation Element properties.

Property	Description
AutomationId	A string containing the UI Automation identifier (ID) for the automation element. The AutomationId property of an element is expected to be the same in any instance of the application, regardless of the local language.
BoundingRectangle	The coordinates of the rectangle that completely encloses the element. The returned rectangle is expressed in physical screen coordinates.
Name	A string for the text representation of the element. This string should always be consistent with the label text on screen. For example, the Name property must be “Browse…” for the button labeled “Browse…”.
ControlType	A ControlType of the automation element, which defines characteristics of the UI element by well known UI control primitives such as button or check box.
FrameworkId	A string for the name of the underlying UI framework. FrameworkId enables client applications to apply special cases to a particular UI framework. Examples of property values are "Win32", "WinForm", and "DirectUI".

Table 3: Examples of UI Automation Control Patterns.

Control Pattern	Description
Dock	Used for controls that can be docked in a container, like toolbars or tool palettes.
ExpandCollapse	Used for controls that can be expanded or collapsed, like the File menu.
Grid	Used for controls that support grid functionality such as sizing and moving to a specified cell, without header information. The “large icon view” in Windows® Explorer is an example of a control that follows the Grid control pattern.
GridItem	Used for controls within grids. For example, each cell in Explorer’s “details view” could follow the GridItem pattern.
Invoke	Used for controls that can be invoked, such as a button.
ItemContainer	Used for controls that hosts a number of children that may be virtualized.
MultipleView	Used for controls that can switch between multiple representations of the same set of information, data, or children. For example, a list view control where data is available in thumbnail, tile, icon, list, or detail views.
RangeValue	Used for controls that have a range of values. For example, a spinner control containing years might have a range of 1900 to 2010, while another spinner control presenting months would have a range of 1 to 12.
Scroll	Used for controls that can scroll.
Selection	Used for selection container controls. For example, list boxes and combo boxes.
Table	Used for controls those have a grid as well as header information. For example, Microsoft Excel worksheets.
Text	Used for edit controls and documents that expose textual information.
Toggle	Used for controls where the state can be toggled. For example, check boxes and checkable menu items.
Transform	Used for controls that can be resized, moved, and rotated. Typical uses for the Transform control pattern are in designers, forms, graphical editors, and drawing applications.
VirtualizedItem	Used for a virtualized object in a container that supports the ItemContainer pattern.
Window	Used for controls that provide fundamental window-based functionality within a traditional graphical user interface.

Table 4: Types of UI Automation Events.

Event	Description
Property change	Raised when a property on an UI Automation element or control pattern changes. For example, if a client needs to monitor an application's check box control, it can register to listen for a property change event on the ToggleState property. When the check box control is checked or unchecked, the provider raises the event and the client can act as necessary.
Element action	Raised when a change in the UI results from end user or programmatic activity. For example, clients can listen for the Invoked event when a user clicks a button when the InvokePattern is invoked programmatically.
Structure change	Raised when the structure of the UI Automation tree changes. The structure changes when new UI items become visible, hidden, or removed on the desktop.
General event	Raised when actions of global interest to the client occur, such as when the focus shifts from one element to another or when a window closes.

Article Pages: < Previous - 1 2 3 - Next Page: 'Which to Support: Microsoft Active Accessibility, UI Automation, or IAccessibleEx?' >>

Page 1:	Windows Automation API 3.0 Overview
Page 2:	UI Automation Specification

Page 3:

Which to Support: Microsoft Active Accessibility, UI Automation, or IAccessibleEx?

How would you rate the quality of this article?

	1	2	3	4	5
Poor						Outstanding

Tell us why you rated the content this way. (optional)

Average rating:
3.7 out of 5

22 people have rated this article.