AppAgent: Autonomous Exploration and Persistent UI Knowledge

The primary technical innovation of AppAgent is its two-phase lifecycle: Exploration and Deployment.

Qualitative task evaluation of AppAgent across Google Maps, Gmail, and Lightroom.

During the exploration phase, the agent is tasked with building a "Knowledge Base" (a structured text document) for a specific application. It achieves this through two distinct strategies:

1. Autonomous Exploration: The agent interacts with UI elements via trial-and-error. For every interaction, it captures the state tuple $(I, X)$ - where $I$ is the raw screenshot and $X$ is the XML metadata - and records the outcome. If an action results in a state change (e.g., a new page loading), the agent "reflects" on the semantic meaning of that element and updates its persistent document.
2. Human Demonstration: The agent observes a human performing tasks and extracts the $(S, A, S')$ transitions.

This knowledge base acts as a long-term memory buffer. When the agent enters the Deployment Phase to execute a user request, it doesn't reason from scratch. Instead, it uses Retrieval-Augmented Generation (RAG) to query its persistent document for the relevant UI element's function. This decoupled architecture allows the agent to achieve high success rates (73.3% autonomous) without the need for real-time unconstrained exploration during task execution.

A critical implementation nuance in AppAgent is how it handles the XML-to-Visual mapping. Mobile UI elements are often densely packed; direct coordinate prediction by LMMs is prone to "spatial drift." AppAgent solves this by parsing the Android accessibility tree (the XML) to identify all interactive nodes. It then implements a Numeric Overlay system:

• Element Discovery: The system filters the XML tree for elements where clickable="true" or long-clickable="true".
• Coordinate Computation: For each valid element, the system calculates its center-point coordinate $(x_c, y_c)$ relative to the screen resolution.
• Visual Tagging: Semi-transparent numerical tags are rendered on the screenshot at these center-points.

The LLM is then restricted to an action space defined by these tags: click(5), long_press(12), swipe(3, direction). This transforms a continuous spatial reasoning task into a discrete symbolic selection task. By removing the model's responsibility for pixel precision, the framework eliminates the most common failure mode in mobile agency.

The "Knowledge Base" created by AppAgent is not a flat file but a structured document that follows a strict schema for every UI element:

• element_id: The numeric tag assigned during exploration.
• element_type: The class name from XML (e.g., android.widget.Button).
• visual_description: A semantic summary generated by the LMM during exploration.
• action_effect: A description of what happens when the element is triggered (e.g., "Opens the settings menu").

When multiple exploration steps target the same element, AppAgent performs Information Consolidation. It asks the LMM to synthesize the past $N$ observations into a single, comprehensive description. This prevents the knowledge base from becoming a noisy log and instead turns it into a high-fidelity "manual" for the application.