UI Automation activities

Semantic selectors let you indicate UI elements based on their meaning, not only on traditional selector attributes. You describe the element you want to interact with, and UiPath uses AI to identify the best matching target.

Semantic selectors support two find methods:

In DOM mode, UiPath searches the DOM to find the target element. This mode uses UiPath's DOM Extractor to collect structured information about the page, then sends that information to a large language model (LLM) to match UI elements based on their semantic description.

DOM mode is recommended when the target can be identified through information available in the DOM, such as:

DOM mode can reach any element available in the DOM, including elements that are off-screen. This makes it useful for web applications where the target element exists in the page structure even when it is not visible in the viewport.

DOM mode is less precise when the most important identifying characteristics are visual, such as a specific icon shape, color, style, or visual position. DOM mode is also not available for non-web applications.

In Image mode, Semantic selectors search the visible screen image instead of the DOM. Image mode can detect and reason over objects that are visible on the screen, including icons, colors, styles, layout, and visual relationships.

This mode uses the image grounding capabilities of the latest multimodal models. These models can understand not only that an element exists, but also where it appears on the screen and how it looks in relation to the surrounding UI.

Image mode is the better choice when the target is described more reliably by how it looks than by its structure, for example:

For example, Image mode can target "the green Sales Opportunities tile," "the blue icon next to the search field," or "the warning icon in the top-right corner."

Because Image mode works from the screen image, it is limited to what is visible. If an element is off-screen, hidden, or not visually rendered at targeting time, Image mode cannot identify it from the current screenshot.

For off-screen targets, a Mouse Scroll activity with the To element property, placed before the Semantic selector action, brings the element into view before the action executes.

Image mode is the natural evolution of UiPath's Computer Vision technology.

Computer Vision uses AI-based object detection to identify UI elements on the screen. It remains available in Unified Target and continues to be an important targeting method. Because Computer Vision is based on a dedicated object detector, it is more stable and deterministic in scenarios that require predictable visual element detection.

Image mode builds on the same direction — using visual understanding to interact with applications when traditional selectors are not enough — but introduces a more flexible approach. It uses modern multimodal models that reason over the full screen image and match elements based on natural language descriptions. The following table summarizes when to use each method:

Computer Vision remains part of Unified Target and is not replaced by the Image find method of Semantic selectors.

Semantic selectors include a Model drop-down in Project Settings > UI Automation > Semantic Selector that lets you choose the underlying model used for semantic targeting for each available find method.

For DOM mode, two models are available:

For transparency, PathMini is based on a fine-tuned Qwen model.

For Image mode, Semantic selectors can use only PathMini, because it supports image grounding and can reason over the visible screen image.

DOM mode is the default option for web applications when the target can be identified through text, labels, roles, attributes, or structure available in the DOM. DOM mode is also useful when the target exists in the DOM but is not visible on screen.

Image mode is the better option when the target is described by how it looks, when color, icon shape, style, visual position, or surrounding layout are more relevant than DOM attributes. Image mode is also useful for non-web applications, or when the DOM does not provide enough reliable information.

When using Image mode, the target must be visible on the screen at targeting time. For off-screen targets, a Mouse Scroll activity with the To element property, placed before the Semantic selector action, brings the element into view before the action executes.

For more information about Semantic selectors, refer to About Semantic selectors.