Web data extraction using 55+ verified Apify Actors for AI-driven scraping. Supports Instagram, Facebook, TikTok, YouTube, Google (SERP, Maps, Reviews) and more. Automatically selects the best Actor for each task and returns structured JSON/CSV output. Features include pagination and incremental crawling, rate limiting and retry strategies, proxy and authentication support (requires APIFY_TOKEN), scheduling and page-change monitoring, and guidance for ethical scraping and ban mitigation.
| 配置项 | 说明 |
|--------|------|
| APIFY_TOKEN | Apify API Token，在 https://console.apify.com/account/integrations 获取 |
将 Token 添加到 .env 文件：
APIFY_TOKEN=apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
| 依赖 | 用途 | 安装方式 |
|------|------|---------|
| httpx | HTTP API 调用 | pip install httpx |
| 依赖 | 用途 | 安装方式 |
|------|------|---------|
| apify-client | Apify Python SDK | pip install apify-client |
| pandas | 数据处理与导出 | pip install pandas |
curl -s "https://api.apify.com/v2/user/me?token=$APIFY_TOKEN" | python -m json.tool
Apify 平台上有数千个 Actor（即预构建的爬虫/自动化程序）。本技能聚焦 55+ 个经过验证的、面向 AI 数据提取优化的 Actor。
| 平台 | Actor | Actor ID | 主要功能 |
|------|-------|----------|---------|
| Instagram | Profile Scraper | apify/instagram-profile-scraper | 用户资料、帖子、粉丝数 |
| Instagram | Hashtag Scraper | apify/instagram-hashtag-scraper | 标签下的帖子 |
| Instagram | Comment Scraper | apify/instagram-comment-scraper | 帖子评论 |
| TikTok | Scraper | clockworks/free-tiktok-scraper | 视频、用户、标签 |
| YouTube | Scraper | bernardo/youtube-scraper | 视频信息、评论 |
| YouTube | Channel Scraper | streamers/youtube-channel-scraper | 频道数据 |
| Facebook | Posts Scraper | apify/facebook-posts-scraper | 页面帖子 |
| Facebook | Comments Scraper | apify/facebook-comments-scraper | 帖子评论 |
| Twitter/X | Scraper | apidojo/tweet-scraper | 推文搜索 |
| LinkedIn | Profile Scraper | anchor/linkedin-profile-scraper | 用户资料 |
| 平台 | Actor | Actor ID | 主要功能 |
|------|-------|----------|---------|
| Google | Search Results | apify/google-search-scraper | SERP 结果 |
| Google | Maps | compass/crawler-google-places | 商家信息、评价 |
| Google | Trends | emastra/google-trends-scraper | 搜索趋势 |
| Google | News | lhotanova/google-news-scraper | 新闻搜索 |
| Google | Shopping | epctex/google-shopping-scraper | 商品价格 |
| Bing | Search | nicefellow/bing-search-scraper | Bing 搜索结果 |
| 平台 | Actor | Actor ID | 主要功能 |
|------|-------|----------|---------|
| Amazon | Product Scraper | junglee/amazon-scraper | 商品详情、评价 |
| Amazon | Review Scraper | junglee/amazon-reviews-scraper | 商品评论 |
| eBay | Scraper | drobnikj/ebay-scraper | 商品搜索 |
| AliExpress | Scraper | epctex/aliexpress-scraper | 商品数据 |
| 功能 | Actor | Actor ID | 主要功能 |
|------|-------|----------|---------|
| 网页抓取 | Web Scraper | apify/web-scraper | 通用网页数据提取 |
| 网页截图 | Screenshot | apify/screenshot-url | 网页截图 |
| 链接提取 | Link Extractor | apify/link-extractor | 页面链接收集 |
| RSS 解析 | RSS Feed | drobnikj/rss-feed-reader | RSS 源数据 |
| AI 提取 | GPT Scraper | drobnikj/gpt-scraper | AI 驱动智能提取 |
Agent 根据用户需求自动选择最合适的 Actor：
步骤 1 — 确认需求
| 参数 | 说明 | 示例 | |------|------|------| | 平台 | 目标社交平台 | Instagram | | 数据类型 | 帖子/评论/用户/标签 | 帖子 | | 范围 | URL/关键词/用户名 | @openai | | 数量限制 | 最大抓取条数 | 100 | | 时间范围 | 时间过滤 | 最近 30 天 |
步骤 2 — 选择并配置 Actor
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run_input = {
"usernames": ["openai"],
"resultsLimit": 100,
"resultsType": "posts",
}
run = client.actor("apify/instagram-profile-scraper").call(run_input=run_input)
步骤 3 — 获取并处理结果
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
步骤 4 — 格式化输出
将数据转换为用户需要的格式（JSON/CSV/表格摘要）。
步骤 1 — 确认搜索参数
| 参数 | 说明 | 默认值 | |------|------|--------| | 关键词 | 搜索查询 | — | | 搜索引擎 | Google/Bing | Google | | 国家/语言 | 地域设置 | CN/zh | | 结果数量 | 抓取条数 | 50 | | 类型 | 网页/新闻/图片/视频 | 网页 |
步骤 2 — 调用 Actor
run_input = {
"queries": "AI agent 框架 2025",
"maxPagesPerQuery": 3,
"languageCode": "zh",
"countryCode": "cn",
"resultsPerPage": 10,
}
run = client.actor("apify/google-search-scraper").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
步骤 3 — 提取关键字段
| 字段 | 说明 |
|------|------|
| title | 结果标题 |
| url | 链接地址 |
| description | 摘要描述 |
| position | 排名位置 |
当没有专用 Actor 时，使用 AI 驱动的通用提取：
方法 A — Web Scraper（基于选择器）
run_input = {
"startUrls": [{"url": "https://example.com/products"}],
"pageFunction": """
async function pageFunction(context) {
const $ = context.jQuery;
const results = [];
$('div.product-card').each((i, el) => {
results.push({
name: $(el).find('.title').text().trim(),
price: $(el).find('.price').text().trim(),
url: $(el).find('a').attr('href'),
});
});
return results;
}
""",
"maxRequestsPerCrawl": 100,
}
run = client.actor("apify/web-scraper").call(run_input=run_input)
方法 B — GPT Scraper（AI 智能提取）
run_input = {
"startUrls": [{"url": "https://example.com/products"}],
"instructions": "Extract all product names, prices, and descriptions from this page",
"openaiApiKey": os.environ.get('OPENAI_API_KEY'),
"maxRequestsPerCrawl": 10,
}
run = client.actor("drobnikj/gpt-scraper").call(run_input=run_input)
同时从多个来源抓取数据：
步骤 1 — 列出所有抓取任务 步骤 2 — 并行启动多个 Actor 步骤 3 — 等待所有任务完成 步骤 4 — 合并结果并去重
import asyncio
from apify_client import ApifyClientAsync
async def batch_scrape(tasks):
client = ApifyClientAsync(os.environ['APIFY_TOKEN'])
results = {}
async def run_actor(name, actor_id, input_data):
run = await client.actor(actor_id).call(run_input=input_data)
items = []
async for item in client.dataset(run["defaultDatasetId"]).iterate_items():
items.append(item)
results[name] = items
await asyncio.gather(*[
run_actor(t['name'], t['actor_id'], t['input'])
for t in tasks
])
return results
{
"metadata": {
"actor": "apify/instagram-profile-scraper",
"total_items": 42,
"scraped_at": "2025-03-01T14:30:00Z",
"run_id": "abc123",
"cost_usd": 0.05
},
"data": [
{
"id": "post_12345",
"text": "帖子内容...",
"likes": 1234,
"comments": 56,
"timestamp": "2025-02-28T10:00:00Z",
"url": "https://instagram.com/p/xxx"
}
]
}
import pandas as pd
df = pd.DataFrame(items)
df.to_csv('output.csv', index=False, encoding='utf-8-sig')
当数据量较大时，先输出摘要统计：
📊 抓取完成
- Actor: Instagram Profile Scraper
- 总条数: 142 条帖子
- 时间范围: 2025-01-01 ~ 2025-03-01
- 平均点赞: 2,345
- 最高互动帖子: [URL]
- 费用: $0.12
症状：所有请求返回 401
解决：确认 .env 中的 APIFY_TOKEN 已正确设置
症状：任务长时间未完成 解决：
maxRequestsPerCrawl 或 resultsLimitmemoryMbytes 增加内存分配症状：返回 403 或空结果 解决：
不同 Actor 返回的数据结构不同。在处理数据前先检查字段：
if items:
print("Available fields:", list(items[0].keys()))
Apify 按计算单元（CU）收费。大规模抓取前：
resultsLimit: 10）确认结果质量Instagram、TikTok 等平台会动态调整反爬策略：
schedule_input = {
"actorId": "apify/google-search-scraper",
"cronExpression": "0 9 * * 1", # 每周一早 9 点
"input": {
"queries": "竞品动态",
"maxPagesPerQuery": 1,
}
}
run = client.actor("apify/web-scraper").call(
run_input=run_input,
webhooks=[{
"eventTypes": ["ACTOR.RUN.SUCCEEDED"],
"requestUrl": "https://your-server.com/webhook",
}]
)
用户可在技能同目录下创建 EXTEND.md 添加：
npx skills add openakita/openakita/skills@apify-scraper下载完整 Skill 目录，包含 SKILL.md 及所有相关文件
Search for places (restaurants, cafes, etc.) via Google Places API proxy on localhost.
Interact with GitHub using the `gh` CLI. Use `gh issue`, `gh pr`, `gh run`, and `gh api` for issues, PRs, CI runs, and advanced queries.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Start voice calls via the OpenClaw voice-call plugin.
Notion API for creating and managing pages, databases, and blocks.
Gemini CLI for one-shot Q&A, summaries, and generation.
Category:developer