When AI Learns to "Copy Tables": One-Click Excel Generation from Phone Lists

·2026-09-04 22:07:00

1. Introduction: From "Manual Data Entry" to "One Sentence Does It All"

Imagine this scenario: you're browsing a product list, customer roster, or data report on your phone app and need to organize that information into an Excel spreadsheet on your computer for further analysis. In the past, you'd need to repeatedly take screenshots, manually type in data, cross-check numbers…… half an hour gone before you know it.

Now, all you need is one sentence to your computer:

"Extract this product list from my phone and generate an Excel file on my computer."

Then, your phone on the desk lights up automatically — the app scrolls, takes screenshots, recognizes data on its own, and moments later, a well-structured Excel file appears on your desktop.

This isn't science fiction — it's a capability already achievable with Mobile AI Use and similar AI Agent technologies.

So what's the technical system behind this? This article will break down the complete technical pipeline from "phone screen list → desktop Excel" from a frontend engineering perspective.

2. Overall Architecture: A Four-Layer Closed Loop from Natural Language to Excel

To achieve "one sentence turns a phone list into Excel", a complete "perceive–understand–extract–generate" closed loop is required. The core architecture can be divided into four layers:

Layer Core Function Key Technologies
Device Connection Layer Bridge communication between browser and phone WebUSB + ADB Protocol
Screen Perception Layer Capture and understand phone screen content Vision-Language Model (VLM) + UI Tree
Data Extraction Layer Identify and structure data from the interface Agent Planning + Multimodal Understanding
Data Delivery Layer Generate Excel and transfer to computer Excel Generation Library + Cross-Device Sync

Using "extract product list and generate Excel" as an example, the entire flow can be visualized as:

graph TD
    A[User: "Extract list and generate Excel"] --> B[AI Agent receives instruction];
    B --> C[Capture current phone screen];
    C --> D[VLM analyzes screen content];
    D --> E{Identify list structure};
    E --> F[Extract header fields];
    E --> G[Extract each row of data];
    F --> H[Assemble structured data];
    G --> H;
    H --> I[Generate Excel file];
    I --> J[Deliver to computer];

Next, let's dive into the implementation of each layer.

3. Key Technology Breakdown

1. Device Connection Layer: WebUSB + ADB — Browser Directly Connects to Phone

To control a phone, you first need to establish a communication bridge between the computer and the phone. Traditional methods require installing drivers and ADB toolsets, while Mobile AI Use-type tools leverage the WebUSB API to accomplish this directly in the browser.

WebUSB is a W3C standard that allows web applications to communicate directly with USB devices. Users simply authorize in a browser popup, and the web page gains control of the phone.

ADB (Android Debug Bridge) is Google's official debugging bridge tool. Built on top of WebUSB, these tools implement an "ADB client in the browser" through WebADB — reconstructing ADB Client and partial Server functionality within the browser. This means all ADB commands like adb shell input tap and adb exec-out screencap -p can be executed directly in the browser.

WebADB declares the device's ADB interface via WebUSB (class 0xFF, subclass 0x42, protocol 0x01) and completes communication by reading and writing bulk data packets. The browser sends a CNXN handshake packet for authentication, establishing a connection with the adbd daemon on the phone.

This "browser-as-client" model delivers a "zero-installation, plug-and-play" experience.

WebADB's core advantage: No drivers or software installation needed — open a web page and you can perform nearly all ADB operations. For team collaboration, just share a link: "Open this page in Chrome, connect your phone, tap here."

2. Screen Perception Layer: How Vision-Language Models "Read" Phone Interfaces

For AI to extract data from phone interfaces like a human would, it must be able to "see" the screen. This relies on Vision-Language Models (VLM).

2.1 Perception Methods

The system continuously captures the phone screen via ADB (adb exec-out screencap -p) and sends screenshots to the VLM. There are two main perception approaches:

  • Pixel-level understanding: The model directly "looks at" the screenshot, understanding the meaning of icons, text, and layout.
  • UI tree assistance: Obtaining the current page's UI hierarchy (View Hierarchy) through Android's AccessibilityService. This is like giving the AI a "map" of the current page, marking the coordinates, text, and properties of each UI element.

In practice, the Agent uses both screenshots and UI tree information simultaneously, sending them to the vision model along with context such as the current app, device state, full list of installed apps, and historical steps.

2.2 Coordinate System

Actions returned by the model need to precisely locate elements on screen. Current mainstream approaches support two coordinate formats:

  • Pixel coordinates: Direct screenshot pixel values (e.g., {"action": "tap", "x": 540, "y": 960})
  • Normalized coordinates: 0–1000 normalized coordinates (Vision-Pointer style), easier for model generalization

The system automatically parses, normalizes, and validates the actions returned by the model.

3. Data Extraction Layer: How the Agent "Reads" Data from Lists

This is the core of the entire solution. When the user issues the "extract list and generate Excel" command, the Agent needs to transform from "seeing the list" to "understanding the data structure".

3.1 Agent Framework

A mature phone data extraction Agent typically includes the following roles:

  • Planner: Breaks down complex user instructions into sub-goals. For example: "extract list" → "scroll page to get complete list" → "identify headers" → "extract data row by row" → "structured output".
  • Executor: Based on the current screen state, decides the next atomic action — swipe, screenshot, tap, etc.
  • Reflector: Evaluates whether the action was successful after execution, e.g., "Did the page load more data?" If not, triggers a retry.

This multi-role collaborative framework enables AI to steadily complete data extraction tasks through the cycle of "observe → think → act → verify".

3.2 Two Paths for Data Extraction

Path Implementation Applicable Scenarios
UI tree extraction Directly parse the UI hierarchy from AccessibilityService, extract text nodes Clean list structure, data exists as text
VLM visual extraction Send screenshots to a multimodal model for understanding and structured data extraction Complex UI with images/icons and other non-text elements

In practice, both approaches are often combined — the UI tree provides precise text and coordinates, while the VLM provides understanding of complex layouts.

For list data, the Agent identifies repeating UI patterns (such as product cards or list items), extracts fields for each entry (e.g., name, price, rating), and assembles them into structured data.

Worth noting: The mobile-use project achieved 100% task completion on the AndroidWorld benchmark, ranking first and demonstrating the viability of this technical approach in real-world scenarios.

4. Data Delivery Layer: From Structured Data to Excel Files

Once data extraction is complete, the final step is generating the Excel and delivering it to the computer.

4.1 Excel Generation

Structured data is generated into standard .xlsx files through Excel processing libraries (such as Python's openpyxl, JavaScript's SpreadJS, etc.). The generated Excel can include:

  • Headers (field names identified from the UI)
  • Data rows (content extracted item by item)
  • Basic formatting (column widths, fonts, number formats, etc.)

4.2 Cross-Device Delivery

Generated files can be delivered to the computer in multiple ways:

  • Cloud sync: Operations completed in a cloud-based virtual phone, files directly accessible on the computer
  • Local transfer: Transfer files from phone to computer via ADB's adb pull command
  • Web download: Directly provide file download in the browser interface

4. Complete Execution Flow: "Extract Product List" Example

Let's combine the above technologies to see how a complete data extraction task is executed:

  1. Command input: User enters "Extract this product list from my phone and generate an Excel file on my computer" in the web interface.
  2. Device connection: The web page establishes a connection with the USB-debugging-enabled phone via WebUSB.
  3. Loop begins (Agent Loop):
    • Perception: The system captures the phone's current screen, sending screenshots and UI tree information to the Vision-Language Model.
    • Planning: The Agent's "Planner" analyzes the current screen, identifies it as a list page, and plans sub-goals: "scroll to load all data" → "identify list structure" → "extract data".
    • Execution: The system sends adb shell input swipe commands via ADB to simulate swiping and load more data.
    • Reflection: Takes another screenshot to confirm whether the list bottom has been reached.
  4. Data extraction: The Agent identifies repeating patterns in the list, extracts fields for each entry (name, price, rating, etc.), and assembles them into structured data.
  5. Excel generation: Generates .xlsx files through libraries like openpyxl.
  6. File delivery: The file appears on the computer via cloud sync or web download.

Throughout the entire process, the user only needs to say one sentence — the Agent handles everything else automatically.

5. Technical Challenges and Outlook

Despite the broad technical prospects, phone data extraction Agents at the current stage still face some challenges:

  • UI diversity: Different apps have vastly different UI styles; VLMs need strong generalization capabilities.
  • Data accuracy: Automatically extracted data needs validation, especially for number formats, date formats, etc. Some solutions add validation steps to ensure data quality.
  • Performance and cost: Each operation requires calling a cloud-based VLM, demanding significant network and computing resources.
  • Security: When data extraction is involved, user data privacy and security must be ensured.

However, with the continuous enhancement of multimodal model capabilities and the increasing maturity of Agent frameworks, these challenges are being gradually overcome. Tools like Mobile AI Use are pioneers in this transformation, showing us the prototype of "phone data autopilot" — from "people finding data" to "data finding people", the efficiency gains will be exponential.

In the future, perhaps we can truly be like in the movie Her — just state your needs, and all your devices will work together, automatically completing data collection, organization, and analysis.

Directory