Use OpenClaw to build an AI automatic collection system in 30 minutes: grab structured data from web pages

Tutorial Objectives

Do you often need to collect information from a web page—competitor pricing, industry news, job postings, product data—and then manually copy and paste it into a table? Not only is this a waste of time, it's also error-prone.

In this tutorial, I will teach you step by step how to use OpenClaw's browser tool and DeepSeek V4's AI capabilities to build a fully automated web page data collection system. All operations do not require writing a single line of code and can be completed in 30 minutes.

Pain point issues:

• Manually copying web page data is time-consuming and labor-intensive, and at least 1-2 hours are wasted every day.
• Manual entry is prone to errors and the data format is not uniform
• Unable to automatically update regularly, information lags behind

What will you build?

After completing this tutorial, you will have:

• ✅ A browser tool that can automatically open the target web page and capture the specified content
• ✅ An AI processing chain that can automatically organize original web text into structured tables
• ✅ A scheduled task that runs automatically every morning and outputs the results to a local file

Prerequisites:

• Node.js (v18+) and npm installed
• There is an OpenClaw runtime environment
• Understand basic command line operations

Preparation list

Account preparation:

• OpenClaw is installed and available locally
• There is an AI API key (such as DeepSeek, OpenRouter, etc.)

Knowledge Preparation:

• Basic understanding of terminal/command line operations
• Familiar with Markdown format

Budget Estimate:

• OpenClaw: free and open source, no payment required
• AI API: DeepSeek V4 costs about 0.01 yuan per acquisition, less than 1 yuan per month

Overall architecture

Data flow:

## 目标网页] → [OpenClaw浏览器工具打开页面] → [AI提取结构化数据] → [保存到本地文件

详细实施步骤

Step 1: 安装OpenClaw并验证环境

目标：在本地安装OpenClaw，确认浏览器工具和AI调用模块都能正常使用。

操作步骤：

# 全局安装OpenClaw
npm install -g openclaw

# 验证安装成功
openclaw --version

# 检查可用工具列表
openclaw tools

预期输出应该显示 openclaw 版本号和可用工具列表。如果看到 browser 工具在列表中，说明环境就绪。

注意事项：

⚠️ 关键点：如果 npm install -g 遇到权限问题，可以尝试 sudo npm install -g openclaw，或者用 npx openclaw 直接运行（无需全局安装）。

Step 2: 配置浏览器工具采集目标网页

目标：用 OpenClaw 的浏览器工具打开一个目标网页，获取页面内容。

操作步骤：

# 用浏览器工具打开目标网页
openclaw browser navigate "https://example.com/products"

# 获取页面内容快照
openclaw browser snapshot --full

浏览器工具会自动打开一个无头浏览器（Headless Browser），加载目标网页并返回页面内容。你可以指定 --full 参数获取完整的页面文本，而不是仅交互元素。

注意事项：

⚠️ 关键点：某些网站有反爬机制，如果遇到验证码或封禁，可以尝试降低访问频率或在两次请求之间加入延迟。

Step 3: 用AI将原始数据转为结构化格式

目标：将浏览器采集到的原始HTML/文本内容，通过AI自动整理为清晰的结构化数据。

操作步骤：

# 将浏览器采集的原始内容传给AI处理
openclaw ai process --input raw_content.txt --format json --schema "名称、价格、评分、链接"

配置说明：

{
  "provider": "deepseek",
  "model": "deepseek-v4-flash",
  "system_prompt": "你是一个数据提取助手。从给定的网页文本中提取指定字段，返回JSON数组。",
  "temperature": 0.1,
  "max_tokens": 2000
}

关键：AI的处理质量取决于你给它的 schema 定义是否清晰。字段名越具体，提取结果越准确。DeepSeek V4 在处理中文网页内容时表现出色，而且成本极低——100万token输入仅需12元。

常见错误：

❌ 错误：没有指定输出格式，AI返回了自然语言描述而不是结构化数据 ✅ 解决：在system prompt中明确要求 returns JSON format

Step 4: 设置自动化定时执行

目标：用 OpenClaw 的 cron 定时任务功能，让整套采集流程每天自动运行。

操作步骤：

# 创建一个定时任务，每天早上8点执行采集
openclaw cron create \
  --name "daily-data-collection" \
  --schedule "0 8 * * *" \
  --task "采集目标网页数据并保存"

验证定时任务：

# 查看所有定时任务
openclaw cron list

# 手动触发测试
openclaw cron run "daily-data-collection"

进阶配置：

// 如果想进一步自动化，可以集成 n8n 来处理采集后的数据流转
// 将OpenClaw的输出通过webhook发送到n8n，实现数据入库、通知等

常见错误：

❌ 错误：定时任务没有执行，cron表达式写错 ✅ 解决：验证cron表达式，确保时区设置为 Asia/Shanghai

❌ 错误：采集频率过高导致IP被封 ✅ 解决：将执行频率从每小时1次改为每天1次，或使用代理轮换

效果展示

完成本教程后，你将拥有：

• 每天自动采集1个目标网站的全部产品数据
• 数据输出为结构化的JSON文件，可直接导入数据库或表格
• 整个过程无需人工干预，每天节省1-2小时的手动操作时间

实际效果对比：

常见坑位排查

Q1: 浏览器工具打开页面显示空白

A: 目标页面可能依赖JavaScript渲染，但默认浏览器模式已经支持JS渲染。检查是否被反爬机制拦截，可以尝试添加 --user-agent 参数伪装成正常浏览器。

Q2: AI提取的数据格式不对

A: 检查你的system prompt是否明确指定了输出格式。建议在prompt末尾加上 "请严格按JSON格式返回，不要包含任何额外说明文字"。

Q3: 定时任务没有按预期运行

A: 确认cron表达式使用北京时间：0 8 * * * means execution at 8:00 every morning. Also check if the OpenClaw background process is running.

Advanced skills

Optimization 1 — Multi-page collection:

If you need to collect multiple pages, you can write a simple loop script to pass different URLs into the browser tool and process them one by one. Cooperating with the concurrency function of OpenClaw can greatly improve efficiency.

Optimization 2 — Data deduplication:

The results of each collection may contain duplicate data. You can add deduplication logic before output, or use n8n to connect to the database to achieve incremental updates.

Automation Enhancements:

If you want further automation, you can integrate n8n or LangGraph to orchestrate a more complex workflow - such as a fully automatic link of collection → cleaning → warehousing → notification.

Tool entry

This tutorial uses the following tools, click to view detailed introduction:

• OpenClaw: the core automation platform on which the scheduling and orchestration of all steps are completed
• DeepSeek V4: AI processing engine used to convert raw web page text into structured data
• n8n: Optional workflow orchestration tool for more complex data post-processing

Related resources

Extended reading:

• OpenClaw定时任务设置指南
• DeepSeek V4 API使用文档
• AI自动化采集完整工作流

Tool Link:

• OpenClaw: https://github.com/nousresearch/openclaw
• DeepSeek: https://platform.deepseek.com/
• n8n: https://n8n.io/