Windows Automation API 3.0 Overview (Cont.)
UI Automation Specification
The UI Automation specification provides more flexible programmatic access to UI elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. The specification can be supported across platforms other than Microsoft Windows.
- The implementation of that specification in Windows is also called UI Automation. UI Automation is broader in scope than just an interface definition. UI Automation provides:
- An object model and functions that make it easy for client applications to receive events, retrieve property values, and manipulate UI elements.
- A core infrastructure for finding and fetching across process boundaries.
- A set of interfaces for providers to express tree structure and some general properties.
- A set of interfaces for providers to express other properties and functionality specific to the control type.
- To improve on Microsoft Active Accessibility, UI Automation aims to address the following goals:
- Enable efficient out-of-process clients, while continuing to allow in-process access.
- Expose more information about the UI in a way that allows clients to be out-of-process.
- Coexist with and leverage Microsoft Active Accessibility without inheriting its limitations.
- Provide an alternative to IAccessible that is simple to implement.
- The implementation of the UI Automation specification in Windows features COM-based interfaces and managed interfaces.
UI Automation Elements
UI Automation exposes every piece of the UI to client applications as an Automation Element. Providers supply property values for each element; invoking a method on an element invokes the corresponding method on the provider. Elements are exposed as a tree structure, with the desktop as the root element.
UI Automation offers much better performance for out-of-process client practices (400% or faster for some scenarios than Microsoft Active Accessibility running out-of-process), while adding richness and flexibility to support the latest user interface designs.
Automation elements expose common properties of the UI elements they represent. One of these properties is the control type, which describes its basic appearance and functionality (for example, a “button” or a “check box”).
UI Automation Tree
The Automation Tree represents the entire UI: the root element is the current desktop and child elements are application windows. Each of these child elements can contain elements representing menus, buttons, toolbars, and so on. These elements in turn can contain elements like list items, as shown in Figure 5.
Figure 5: This UI Automation tree represents all elements on the Run dialog.
UI Automation providers for a particular control support navigation among the child elements of that control. However, providers are not concerned with navigation between these control subtrees. This is managed by the UI Automation core, using information from the default window providers.
To help clients process UI information more effectively, the framework supports alternative views of the automation tree: (1) Raw View, (2) Control View, and (3) Content View. As Table 1 shows, the type of filtering determines the views, and the client defines the scope of a view.
UI Automation Properties
The UI Automation specification defines two kinds of properties: (1) automation element properties and (2) control pattern properties. Automation element properties apply to most controls, providing fundamental information about the element such as its name. Examples are listed in Table 2. Control pattern properties apply to control patterns, described next.
Unlike with Microsoft Active Accessibility, every UI Automation property is identified by a GUID and a programmatic name, which makes new properties easier to introduce.
UI Automation Control Patterns
A control pattern describes the attributes and functionality of an automation element. For example, a simple “clickable” control like a button or hyperlink should support the Invoke control pattern to represent the “click” action.
Each control pattern is a canonical representation of possible UI features and functions, as shown in Table 3. There are 22 control patterns defined to date, some of which are new in Windows 7, and the Windows Automation API can support custom control patterns. Unlike with Microsoft Active Accessibility role or state properties, one automation element can support multiple UI Automation control patterns.
UI Automation Control Types
A control type is another automation element property that specifies a well-known control that the element represents. Currently, UI Automation defines 38 control types, including; Button, Check Box, Combo Box, Data Grid, Document, Hyperlink, Image, ToolTip, Tree, and Window.
Before you can assign a control type to an element, the element needs to meet certain conditions, including a particular automation tree structure, property values, control patterns, and events. However, you are not limited to these. You can extend a control with custom patterns and properties as well as with the pre-defined ones.
The total number of pre-defined control types is significantly lower than Microsoft Active Accessibility accRole definitions, because you can combine UI Automation control patterns to express a larger set of features while Microsoft Active Accessibility roles cannot. You can also customize the description of control type by LocalizedControlType property while keeping the baseline type as defined.
UI Automation Events
UI Automation events notify applications of changes to and actions taken with automation elements. The four different types of UI Automation events, as listed in Table 4, do not necessarily mean that the visual state of the UI has changed. The UI Automation event model is independent of the WinEvent framework in Windows, although the Windows Automation API can make UI Automation events interoperable with the Microsoft Active Accessibility framework.
The IAccessibleEx interface helps existing applications or UI libraries extend their Microsoft Active Accessibility object model to support UI Automation without rewriting everything from scratch. With IAccessibleEx, you can implement only the differences between Microsoft Active Accessibility and UI Automation object models.
Because the MSAA-to-UIA Proxy translates the object models of IAccessibleEx-enabled Microsoft Active Accessibility servers as UI Automation object models, UI Automation clients don’t have to do any extra work. The IAccessibleEx interface can enable classic in-process Microsoft Active Accessibility clients to interact directly with UI Automation providers, too.