Scroll Content Auto-Extraction User Guide

·2026-08-08 00:30:09

Scroll Content Auto-Extraction User Guide

Automatically scrolls the phone screen with continuous screenshots, removes overlapping areas between adjacent screenshots, stitches them into a long image, then sends it to an AI vision model for content analysis and extraction, and finally exports to Excel (.xlsx) / Word (.docx) documents.

Table of Contents

Feature Overview

Pain Points Addressed

Pain Point Solution
Per-page LLM extraction is slow and token-heavy Stitch a long image locally first, reducing LLM calls from N pages to a few segments
Items truncated across screen boundaries Overlapping stitch preserves complete items — no more discarding edge items per page
Status bar / TabBar repeating in the long image Per-row grayscale signature auto-detects and removes fixed UI regions
Duplicate data across frames Fingerprint Set global deduplication (overlap boundary duplicates caught by fingerprint fallback)
Accurate bottom-reached detection aHash perceptual hash comparison (16×16 grayscale downsampling, similarity ≥ 97% = bottom reached)
Content loading delay causing incomplete screenshots Configurable settle time after each scroll (default 800ms)

Core Capabilities

  • Fast Capture: Scrolling phase makes zero LLM calls — only screenshots + aHash bottom detection, significantly faster than per-page extraction
  • Local Long-Image Stitching: Pure Canvas computation, auto-removes fixed status bars / bottom bars, precisely aligns overlapping regions between adjacent screenshots
  • Auto-Segmentation for Oversized Images: Segments at 4096px with 300px overlap between segments to prevent items from being cut at boundaries
  • Dual Extraction Modes: Schema structured table mode (Excel / general) and richMode image-text mode (Word-specific)
  • Auto Task Completion: After successful extraction, the AI Agent task ends immediately — no further LLM decision loop

Three-Phase Workflow

flowchart TD
    START([Start]) --> P1

    subgraph P1[Phase 1: Auto-Scroll & Capture]
        CAP0[First screen screenshot] --> SWIPE[Swipe up gesture]
        SWIPE --> SETTLE[Wait settleMs]
        SETTLE --> CAPN[Post-scroll screenshot]
        CAPN --> HASH{aHash similarity >= 0.97?}
        HASH -->|No, not at bottom| SWIPE
        HASH -->|Yes, bottom reached| P1END[Capture complete]
        CAPN -->|maxPages reached| P1END
    end

    P1END --> P2

    subgraph P2[Phase 2: Overlap Removal & Stitching]
        SIG[Per-row grayscale signature] --> BARS[Detect fixed top/bottom bars]
        BARS --> OVERLAP[Match overlapping regions between adjacent screenshots]
        OVERLAP --> STITCH[Canvas stitch into virtual long image]
        STITCH --> CHUNK[Segment at 4096px
300px overlap between segments]
    end

    CHUNK --> P3

    subgraph P3[Phase 3: AI Analysis of Image Segments]
        MODE{richMode?}
        MODE -->|No, Schema mode| SCHEMA[Infer field Schema from first segment]
        SCHEMA --> EXTRACT[Per-segment structured JSON extraction]
        MODE -->|Yes, Image-text mode| RICH[Per-segment image region + text recognition]
        RICH --> CROP[Canvas crop image regions]
        EXTRACT --> DEDUP[Fingerprint deduplication]
        CROP --> DEDUP
    end

    DEDUP --> EXPORT[Export Excel / Word]
    EXPORT --> FINISH[End entire Agent task]

Comparison with Legacy Per-Page Extraction

Dimension Legacy (Per-Page) New (Long-Image Stitching)
LLM Calls 1 Schema + N pages × 1 extraction 1 Schema + segments × 1 extraction (30 screens usually → only 3–5 segments)
Scroll Speed Wait for LLM response per page Zero LLM calls during scrolling — limited only by settleMs
Truncated Items Incomplete items at top/bottom edges discarded per page Items complete after stitching — no information loss
Fixed UI Interference Every page contains status bar / TabBar Auto-detected and removed, kept only once
screenshots Return Raw per-page screenshots Stitched long-image segments (i.e. the actual images sent to AI)

Core Algorithms

1. Bottom Detection (aHash Perceptual Hash)

Input: Pre-scroll screenshot A, Post-scroll screenshot B
Processing:
  1. Downsample A and B to 16×16 pixels each
  2. Convert to grayscale (weighted: R*0.299 + G*0.587 + B*0.114)
  3. Compute mean avg of 256 grayscale values
  4. Each pixel >= avg → '1', otherwise → '0', producing a 256-bit hash
  5. similarity = 1 - Hamming_distance / 256
Decision: similarity >= 0.97 → bottom reached, stop scrolling

2. Per-Row Grayscale Signature (Basis for Stitching)

Each screenshot width downsampled to 48 columns → per-row grayscale mean → Float32Array(height)

A single getImageData call obtains row-level features for the entire image. Subsequent fixed-bar detection and overlap matching operate on this signature array, avoiding repeated pixel reads.

3. Fixed Bar Detection (detectFixedBars)

Take row signatures from pre- and post-scroll frames:
  Compare row-by-row from top — consecutive rows with |diff| < 2 → fixed top bar (status bar / nav bar)
  Compare row-by-row from bottom — same logic → fixed bottom bar (TabBar / input field)
Single-side cap at 20% of screen height to prevent false positives on solid-color backgrounds

During stitching, intermediate frames have fixed bars removed; the first frame keeps the top (with status bar), the last frame keeps the bottom — ensuring the long image has complete start and end.

4. Overlap Region Matching (findOverlapOffset)

Sliding window search within the previous frame's content area [topBar, height - bottomBar):
  Find the best match position matchY against the top 120 rows of the current frame's content area
  Match error = average grayscale difference within the window; > 8 = match failure
Match success: current frame starts new content from row topBar + (prevEnd - matchY)
Match failure: keep the current frame's full content area (minor duplicates handled by fingerprint dedup)

With early-abandon optimization (cumulative error worse than current best → exit immediately), 30-screen stitching completes in milliseconds.

5. Long-Image Segmentation (Chunk)

Virtual long image total height = sum of all stitched segment heights
Render to Canvas at CHUNK_MAX_HEIGHT = 4096px → JPEG (quality 0.85)
CHUNK_OVERLAP = 300px overlap between segments → items won't be cut at segment boundaries

Reason for segmentation: Vision models' detail recognition degrades on extremely tall images, and Canvas has size limits.

6. Cross-Segment Deduplication (Fingerprint)

Mode Fingerprint Rule
Schema mode First 2 non-empty field values trimmed and joined with |
richMode First 60 characters of trimmed item text

Tracked using a global Set<string> — items redundantly identified within the 300px overlap between segments are automatically removed.

Two Extraction Modes

Schema Mode (Default, Excel / both Format)

  1. Use the first segment long-image to let the LLM infer the list field schema (e.g. {"name":"string","price":"number"})
  2. Extract structured JSON arrays per segment according to the Schema
  3. Output result.items (table data), export to Excel / Word table

richMode Image-Text Mode (Auto-Enabled for Word Format)

  1. Skip Schema inference
  2. Have the LLM return image_region (pixel coordinates) + text (full text) for each item per segment
  3. Use Canvas to crop image regions from long-image segments → JPEG data URL
  4. Output result.richItems, export as image-text interleaved Word document

The call entry point (react.ts / phone-tools.ts) automatically passes richMode: true when outputFormat === 'word'.

Agent Tool Definition

ExtractScrollingContent is registered as a Phone Tool, callable by ReAct / Plan-and-Resolve Agents autonomously.

Tool Parameters

Parameter Type Required Description
description string Yes What data to extract, e.g. "product list with name, price, rating"
maxPages number No Maximum number of scroll pages
settleMs number No Wait time after each scroll (ms)
schemaHint string No Comma-separated field name hints
outputFormat excel | word | both No Export format, default both; word auto-enables richMode image-text mode

Tool Return Example

Extraction complete. Extracted 47 unique items across 12 page(s). Removed 5 duplicates. End reason: bottom_reached. Files downloaded: extract-2026-06-16-1789551234567.xlsx + .docx

Task Auto-Completion Behavior

After successful extraction (return starts with Extraction complete), the entire AI Agent task ends immediately:

  • ReAct mode (react.ts): No further screenshot + LLM decision loop; current step marked as success, task.conclusion takes the extraction report directly, task.status = 'finished'
  • Plan-and-Resolve mode (agent.ts): All remaining pending steps marked as skipped, skipping the Report phase LLM summary

When extraction yields no data (No data extracted...), the task continues normal flow — the LLM has the opportunity to adjust strategy (e.g. scroll to the correct page first, then retry).

Usage Examples

Example 1: Natural Language in Chat (Most Common)

Type directly in the AI chat box:

Extract the product list on the current page to Excel

Extract Meituan food delivery merchant list, including store name, rating, monthly sales, minimum order price

Save this Xiaohongshu note list as a Word document (with cover images)

The Agent will automatically recognize the intent and call the ExtractScrollingContent tool. The task ends automatically after extraction completes.

Example 2: Combined Commands (Navigate First, Then Extract)

Open Xiaohongshu, search for deepseek, extract the search results list to Excel, including title, author, likes

The Agent first performs app opening and searching, then calls the extraction tool and ends the task.

Example 3: Via phone-agent MCP

In an MCP-capable client (e.g. Qoder):

Use run_agent_task: Extract the news list from the current phone page to Excel, fields include title, source, time

After task completion, MCP returns a Markdown execution report (with per-step summary and tool calls).

Progress Message Reference

The onProgress callback will receive messages similar to the following in sequence:

Capturing initial screenshot...
Page 1: scrolling down (1 screenshot(s) captured)...
Page 2: scrolling down (2 screenshot(s) captured)...
Bottom detected (similarity=98.4%). Captured 9 screenshot(s).
Stitching 9 screenshot(s) into a long image...
Stitched into 3 image segment(s). Analyzing content with AI...
Inferring data schema from stitched image...
Segment 1/3: extracting data...
Segment 1/3: +15 new items (total 15)
Segment 2/3: extracting data...
Segment 2/3: +14 new items (total 29)
Segment 3/3: extracting data...
Segment 3/3: +11 new items (total 40)

Export Format Details

Excel (.xlsx) — Schema Mode

Feature Description
Sheet Names Data (data) + Screenshots (screenshots, generated when screenshots are provided)
First Row Column headers (field names)
Column Width Auto-fit (based on maximum content length)
Data Types Numbers stay as numbers, strings stay as strings, null → empty
Screenshot Storage Base64 of each page screenshot stored across cells (column-limited) in the Screenshots sheet (can be decoded and reassembled into images), plus downloaded as individual image files (.png/.jpg, same prefix as Excel, numbered by page)

Note: The community xlsx library (SheetJS) does not support embedding images in cells, so screenshots are saved in two ways: "Base64 split across cells + individual image file downloads" to ensure no image data is lost.

Word Table (.docx) — Schema Mode (both Format)

Feature Description
Title + Stats Row H1 heading, "Extracted N items total, exported at [time]"
Table Style Dark header, alternating row colors, auto column width
Screenshot Appendix When screenshots is provided, stitched long-image segments are embedded

Word Image-Text (.docx) — richMode (word Format)

Feature Description
Each Item Cropped image (300px wide proportional scale, if available) + multi-line text paragraphs + separator line
Text Style First line bold and slightly larger (title-like), remaining lines in normal style
Page A4, 1.25-inch left/right margins

FAQ

Q: Extraction result is empty?

Troubleshooting Steps:

  1. Confirm the phone screen is currently displaying list/card content
  2. Check whether description accurately describes the data to extract
  3. Check the console for [ScrollExtract] logs to confirm stitching segment count and Schema inference
  4. Try adding schemaHint to explicitly specify field names

Q: The stitched long image has misalignment or duplicate strips?

Possible Causes:

  • The page has parallax animation / sticky elements during scrolling, causing row signature match failure (the frame retains its full content area, resulting in duplicates but no data loss)
  • List items have large solid-color backgrounds between them, causing overlap matching to find the wrong position

Note: Duplicate items from match failures are removed by fingerprint dedup, generally not affecting final data. If misalignment is severe, increase settleMs to allow the page to fully settle before capturing.

Q: Bottom detected too early?

Possible Causes:

  • The page has a large fixed area (e.g. top banner), and the changing region after scrolling is small
  • endThreshold is set too high

Solution: Lower endThreshold (e.g. 0.95 or 0.93).

Q: Want faster extraction?

  • Reduce settleMs (e.g. 500ms) — suitable for static content pages
  • Reduce maxPages to limit capture range
  • Use a faster vision model; LLM time depends only on the number of stitched segments, not the number of scrolled pages

Q: Will segment boundaries cut items?

A 300px overlap is maintained between segments. Items cut at the boundary will appear completely in an adjacent segment. Duplicate identification is handled by fingerprint dedup.

Q: Why do Word and Excel formats have different content structures?

outputFormat: 'word' auto-enables richMode image-text mode, producing "image + free text" (richItems); excel / both use Schema mode, producing structured tables (items). The LLM prompts and outputs for the two modes are entirely different.

Q: What types of lists are supported?

List Type Support
Native Android RecyclerView Supported
React Native FlatList Supported
Flutter ListView Supported
H5 list inside WebView Supported
Paginated loading (non-infinite scroll) Supported (requires配合 page-turn clicks)
Horizontal scrolling list Not yet supported (currently vertical stitching only)

Core advantage: Based on screenshots + visual AI, independent of UI dump or XPath — universal across any rendering framework.

目录