When AI Learns to "Book Hotels": Multi-Condition Filtering and Data Export Automation on Phones

·2026-09-04 22:13:21

1. Introduction: From "Manually Flipping Five Pages" to "One Sentence Does It All"

Imagine this scenario: the Mid-Autumn Festival is approaching, and you want to book a hotel in Beijing on Ctrip — 4-star, 500–600 yuan, suitable for 2 adults. You need to open the app, enter the city, select dates, filter by star rating, set the price range, choose the number of guests, then flip through search results page by page. By page 5, your eyes are strained, and you still need to manually copy the hotel info you like into an Excel sheet for comparison……

Now, all you need is one sentence to your computer:

"Open the Ctrip app, search for hotels in Beijing, suitable for 2 adults, check-in on Mid-Autumn Festival, 4-star, 500–600 yuan, and save the first 5 pages of results to an Excel document."

Then, your phone on the desk lights up automatically — Ctrip opens by itself, the search box auto-types "Beijing", dates auto-select Mid-Autumn Festival, filters auto-check "4-star", "500–600 yuan", "2 adults", pages flip automatically one by one, and after 5 pages, hotel names, prices, ratings, addresses, and other info are automatically extracted — a well-structured Excel file appears on your desktop.

This isn't science fiction — it's a capability already achievable with Mobile AI Use and similar GUI Agent technologies.

So what's the technical system behind this? This article will break down the complete technical pipeline from "phone multi-condition filtering → cross-page data extraction → Excel export" from architecture to implementation.

2. Overall Architecture: A Four-Layer Closed Loop from Natural Language to Excel

To achieve "one sentence filters hotels and exports Excel", a complete "perceive–understand–extract–generate" closed loop is required. The core architecture can be divided into four layers:

Layer Core Function Key Technologies
Device Connection Layer Bridge communication between browser and phone WebUSB + WebADB Protocol
Screen Perception Layer Capture and understand phone screen content Vision-Language Model (VLM) + UI Tree
Task Planning & Execution Layer Decompose complex instructions into operation sequences Multi-Agent Collaboration + Hierarchical Reflection Architecture
Data Extraction & Delivery Layer Identify list data and generate Excel Structured Extraction + Excel Generation Library

Using "filter hotels and export Excel" as an example, the entire flow can be visualized as:

graph TD
    A[User instruction: Filter Beijing hotels 4-star 500-600 Mid-Autumn check-in export Excel] --> B[AI Agent receives instruction];
    B --> C[Planner decomposes task];
    C --> D[Sub-goal 1: Open Ctrip];
    C --> E[Sub-goal 2: Search Beijing hotels];
    C --> F[Sub-goal 3: Set filter criteria];
    C --> G[Sub-goal 4: Flip pages and extract data];
    C --> H[Sub-goal 5: Generate Excel];
    
    D --> I[Execution loop];
    E --> I;
    F --> I;
    G --> I;
    I --> J[Screen perception: screenshot + UI tree];
    J --> K[Decision: next action];
    K --> L[Execute: ADB commands];
    L --> M[Reflect: verify action result];
    M --> J;
    
    G --> N[Data extraction];
    N --> O[Structured data];
    O --> P[Excel generation];
    P --> Q[Deliver to computer];

Next, let's dive into the implementation of each layer.

3. Key Technology Breakdown

1. Device Connection Layer: WebUSB + WebADB — Browser Directly Connects to Phone

To control a phone, you first need to establish a communication bridge between the computer and the phone. Traditional methods require installing drivers and ADB toolsets, while Mobile AI Use-type tools leverage the WebUSB API to accomplish this directly in the browser.

WebUSB is a W3C standard that allows web applications to communicate directly with USB devices. Users simply authorize in a browser popup, and the web page gains control of the phone.

ADB (Android Debug Bridge) is Google's official debugging bridge tool. Built on top of WebUSB, these tools implement an "ADB client in the browser" through WebADB — reconstructing ADB Client and partial Server functionality within the browser. The browser sends a CNXN handshake packet for authentication, establishing a connection with the adbd daemon on the phone.

Projects like WebDroid Agent have demonstrated the mature application of this technology: connecting Android devices via WebUSB/WebADB in the browser, capturing phone screens and sending them to vision models, then executing the model's returned actions via ADB.

This "browser-as-client" model delivers a "zero-installation, plug-and-play" experience — open a web page, connect your phone, and you can perform nearly all ADB operations.

2. Screen Perception Layer: How Vision-Language Models "Read" Ctrip Interfaces

For AI to operate the Ctrip app and extract hotel information like a human would, it must be able to "see" the screen. This relies on Vision-Language Models (VLM).

2.1 Perception Methods

The system continuously captures the phone screen via ADB (adb exec-out screencap -p) and sends screenshots to the VLM. There are two main perception approaches:

  • Pixel-level understanding: The model directly "looks at" the screenshot, understanding the meaning of icons, text, and layout. Models like ColorGUI-32B, specifically designed for mobile GUI automation, can read screen images, understand user instructions, reason about the next GUI operation, and output executable actions.
  • UI tree assistance: Obtaining the current page's UI hierarchy (View Hierarchy) through Android's AccessibilityService. This is like giving the AI a "map" of the current page, marking the coordinates, text, and properties of each UI element.

In practice, the Agent uses both screenshots and UI tree information simultaneously, sending them to the vision model along with context such as the current app, device state, full list of installed apps, and historical steps.

2.2 Coordinate System

Actions returned by the model need to precisely locate elements on screen. Current mainstream approaches support two coordinate formats:

  • Pixel coordinates: Direct screenshot pixel values (e.g., {"action": "tap", "x": 540, "y": 960})
  • Normalized coordinates: 0–1000 normalized coordinates (Vision-Pointer style), easier for model generalization

The system automatically parses, normalizes, and validates the actions returned by the model.

3. Task Planning and Execution Layer: From "Open Ctrip" to "Flip 5 Pages"

This is the core of the entire solution. When the user issues the "filter hotels and export Excel" command, the Agent needs to decompose this complex task into a series of executable sub-tasks.

3.1 Multi-Agent Collaboration Architecture

Single-agent architectures tend to fail when handling long-running, multi-step tasks — suffering from context pollution, silent failures, repetitive action loops, and other issues. Therefore, modern phone Agents commonly adopt multi-agent collaboration architectures.

Taking Minitap's mobile-use solution as an example, its six-agent decomposition architecture includes:

  • Planner: Decomposes user instructions into high-level sub-goals
  • Orchestrator: Coordinates the workflow among agents
  • Contextor: Maintains task context
  • Cortex: Handles core reasoning
  • Executor: Executes specific actions
  • Screen Analyzer: Analyzes screen content

This architecture achieves 100% success rate on the AndroidWorld benchmark through cognitive separation (each agent maintains independent context), deterministic post-verification (device state validation for operations like text input), and meta-cognitive reasoning (detecting loops and triggering strategy changes).

3.2 Hierarchical Reflection-Driven

The MobileUse project proposes a Hierarchical Reflection Architecture, enabling the Agent to self-monitor, detect, and recover from errors across multiple time scales — from individual actions to overall task completion — while maintaining efficiency through an "on-demand reflection" strategy.

Specifically for the "filter hotels" task, the Agent's Planner decomposes the instruction into the following sub-goals:

  1. Launch Ctrip app: Execute adb shell am start via ADB to launch the application
  2. Search "Beijing": Locate search box → input text → tap search
  3. Set date to Mid-Autumn Festival: Locate date picker → select Mid-Autumn Festival date
  4. Filter 4-star: Locate star rating filter → check "4-star"
  5. Filter 500–600 yuan: Locate price range → input 500 and 600
  6. Filter 2 adults: Locate guest count → select "2 adults"
  7. Flip pages and extract data: Loop "screenshot → extract → swipe to next page" until 5 pages are done
  8. Generate Excel: Assemble data → export file

Each step follows the closed loop of "observe → think → act → verify".

4. Data Extraction and Delivery Layer: From Hotel List to Excel File

Data extraction is the technical challenge of the entire task — identifying structured information from the hotel list on the phone screen and converting it to Excel format.

4.1 List Data Extraction

The Agent identifies the hotel list on screen through VLM, extracting key fields for each hotel entry:

  • Hotel name
  • Price (per night / starting from)
  • Rating
  • Address / location
  • Room type information

There are two extraction paths:

Path Implementation Applicable Scenarios
UI tree extraction Directly parse the UI hierarchy from AccessibilityService Clean list structure, data exists as text
VLM visual extraction Send screenshots to a multimodal model for semantic understanding Complex UI with images/icons and other non-text elements

In practice, both approaches are often combined.

For the "flip 5 pages" requirement, the Agent loops "extract current page data → swipe to next page" until 5 pages are done. MobileUse's Proactive Exploration Module helps the Agent better understand and navigate in unfamiliar environments.

4.2 Page Flipping and Data Stitching

The Agent needs precise control over the page-flipping operation:

  1. Extract the hotel list from the current page
  2. Execute swipe operation (adb shell input swipe)
  3. Wait for the page to finish loading
  4. Extract data from the new page
  5. Append to the existing dataset
  6. Repeat until 5 pages are done

4.3 Excel Generation and Delivery

The extracted structured data is generated into standard .xlsx files through Excel processing libraries (such as Python's openpyxl, JavaScript's SpreadJS, etc.).

The generated Excel typically includes:

  • Headers (hotel name, price, rating, address, etc.)
  • Data rows (hotel info extracted from each page)
  • Basic formatting (column widths, fonts, number formats, etc.)

Files can be delivered to the computer via:

  • Cloud sync: Operations in a cloud-based virtual phone, files directly accessible
  • Local transfer: Transfer from phone to computer via adb pull command
  • Web download: Direct download in the browser interface

4. Complete Execution Flow: "Ctrip Hotel Filtering" Example

Let's combine the above technologies to see how a complete hotel filtering and data export task is executed:

  1. Command input: User enters "Open the Ctrip app, search for hotels in Beijing, suitable for 2 adults, check-in on Mid-Autumn Festival, 4-star, 500–600 yuan, save the first 5 pages of results to an Excel document" in the web interface.

  2. Device connection: The web page establishes a connection with the USB-debugging-enabled phone via WebUSB.

  3. Task planning: The Agent's Planner decomposes the instruction into 8 sub-goals.

  4. Execution loop begins:

    • Perception: The system captures the phone screen, sending screenshots and UI tree information to the Vision-Language Model.
    • Decision: The Executor decides the next action based on the current screen state.
    • Execution: Performs specific operations via ADB (tap, input, swipe, etc.).
    • Reflection: The Cortex verifies whether the action was successful; if not, triggers retry or strategy adjustment.

  5. Specific operation sequence:

    • Launch Ctrip app (adb shell am start)
    • Tap search box, type "Beijing"
    • Tap date picker, select Mid-Autumn Festival date
    • Tap star rating filter, check "4-star"
    • Tap price range, input 500 and 600
    • Tap guest count, select "2 adults"
    • Tap search button

  6. Data extraction loop (repeated 5 times):

    • Capture current screen
    • VLM extracts hotel list data
    • Append to dataset
    • Execute swipe down
    • Wait for page to load

  7. Excel generation: Assemble 5 pages of data, generate .xlsx file via Excel library.

  8. File delivery: The file appears on the computer via cloud sync or web download.

Throughout the entire process, the user only needs to say one sentence — the Agent handles everything else automatically.

5. Technical Challenges and Outlook

Despite the broad technical prospects, phone Agents at the current stage still face some challenges:

  • UI diversity: Different apps (Ctrip, Meituan, Taobao, etc.) have vastly different UI styles; VLMs need strong generalization capabilities.
  • Long-running task reliability: Tasks involving multiple steps and pages are prone to cumulative errors, requiring hierarchical reflection and error recovery mechanisms.
  • Data extraction accuracy: Data extracted from screen needs validation, especially for price formats, date formats, etc.
  • Performance and cost: Each operation requires calling a cloud-based VLM, demanding significant network and computing resources.
  • Security: When high-risk operations like payments or orders are involved, human confirmation mechanisms are needed.

However, with the continuous enhancement of multimodal model capabilities and the increasing maturity of Agent frameworks, these challenges are being gradually overcome. MobileUse achieved a 62.9% success rate on the AndroidWorld benchmark, while Minitap's mobile-use achieved a perfect 100%. Productized intelligent agents like AutoGLM can already complete complex tasks like hotel booking on Ctrip.

From "people finding information" to "information finding people", phone GUI Agents are redefining how we interact with mobile devices. In the future, you'll only need to state your needs, and all your devices will work together, automatically completing information collection, organization, and analysis.

目录