1 line
118 KiB
JSON
1 line
118 KiB
JSON
{"repo": "0xmariowu/autosearch", "files": {"docs/mcp-channels-playbook.md": {"00ad54c022a3effb9e848bb70b158b79584f6014": {"t": 1785747435.862041, "n": 24403, "h": "283aa6865eb903f8", "c": "# AutoSearch Channel Research Final Playbook\n\n> Date: 2026-04-19\n> Coverage: mcpmarket.com research + hands-on testing + TikHub paid API integration\n> Positioning: This is the **final master guide**. The previously scattered docs (`mcp-channel-research.md` / `mcp-test-plan.md` / `mcp-test-results.md` / `mcp-no-cookie-inventory.md` / `mcp-final-summary.md` / `tikhub-smoke-test.md`) are consolidated here. All future channel decisions should start from this document.\n\n---\n\n## TL;DR\n\n**Three-tier conclusion**:\n\n1. **Free and usable without cookies**: Sogou WeChat, open-websearch (4 engines), Paper Search MCP (21 academic sources), PullPush (Reddit history), direct RSS reads, Jina Reader (some sites), and 13 official APIs.\n2. **5 of 8 hard targets were unblocked through paid TikHub**: Xiaohongshu, Weibo, Zhihu, Twitter/X, and Douyin. The remaining 3 (flaky Weibo upstream, Instagram/LinkedIn parameters not fully tuned) were not fully solved in this round.\n3. **Final architecture**: AutoSearch uses BYOK (users provide their own TikHub key) and declares dependencies with `requires: [env:TIKHUB_API_KEY]`. The base URL is read from an env var so environments can switch cleanly.\n\n**Cost anchor**: TikHub measured average cost is **$0.0036/request**. 1000 research-grade calls cost about $3.6/month.\n\n---\n\n## Part 1: No-Cookie Usable Inventory (41)\n\n### A. Hands-On PASS in This Round (9)\n\n| Option | Platform/Capability | Auth | Notes |\n|---|---|---|---|\n| **Sogou WeChat SERP** | WeChat Official Accounts | 🟢 None | `weixin.sogou.com/weixin?type=2&query=<q>` |\n| **open-websearch fetch-web + readability** | Any page -> clean markdown | 🟢 None | Cleaned with the Mozilla Readability algorithm |\n| **open-websearch csdn** | CSDN technical blogs | 🟢 None | Native Chinese content |\n| **open-websearch duckduckgo** | DuckDuckGo | 🟢 None | Backup general search |\n| **open-websearch startpage** | Startpage (Google proxy) | 🟢 None | Backup general search |\n| **Paper Search MCP** | arXiv + PubMed + Semantic Scholar + OpenAlex + 17 other sources | 🟢 None | One-command access to 21 academic sources, MIT |\n| **PullPush** | Reddit history from 2005 to present | 🟢 None | Full-text `q=` search is broken; subreddit/time filtering works |\n| **Substack RSS** | Substack / Medium / independent blogs | 🟢 None | All sites expose `/feed` |\n| **Jina Reader** | Bilibili / weak anti-scraping sites | 🟢 None | Zhihu, Xiaohongshu, and Weibo are blocked |\n\n### B. Official APIs Known to Work (13, Integrated in AutoSearch or One-Line Integration)\n\n| Platform | Access Method | Limit |\n|---|---|---|\n| HackerNews Algolia | Official API | Unlimited |\n| Reddit public JSON | `/r/<sub>.json` | No login |\n| Stack Exchange API | Official | 300/day without key |\n| GitHub Public Search | Official API | 10/min without token |\n| Dev.to | Public API | No key |\n| npm / PyPI registry | Official | No key |\n| OpenReview API | Official | No key |\n| arXiv API | Official | No key |\n| Xueqiu / 36kr / InfoQ Chinese | Their web/API surfaces | No key |\n| Xiaoyuzhou | RSS / public pages | No key |\n| Bilibili public search API | Official | No login |\n| YouTube transcript | yt-dlp + whisper (local) | No key |\n| Crunchbase public pages | Public search | Limited |\n\n### C. New Categories Found on mcpmarket (AutoSearch Gaps) (11)\n\n| Option | Capability | Priority |\n|---|---|---|\n| [Wikipedia MCP](https://mcpmarket.com/server/wikipedia-1) | Full encyclopedia + summaries | 🔴 Highest |\n| [Wikidata](https://mcpmarket.com/server/wikidata) | SPARQL + entities | 🔴 Highest |\n| [Google Trends Explorer](https://mcpmarket.com/server/google-trends-explorer) | Trend signals | 🔴 Highest |\n| [Google News Trends](https://mcpmarket.com/es/server/google-news-trends) | News + trending terms | 🔴 High |\n| [USPTO](https://mcpmarket.com/server/uspto) | U.S. patents + trademarks | 🟡 As needed |\n| [Google Patents](https://mcpmarket.com/server/google-patents) | Global patents | 🟡 As needed |\n| [EPSS](https://mcpmarket.com/server/epss) | CVE + vulnerability scoring | 🟡 As needed |\n| [Cybersecurity CVE](https://mcpmarket.com/es/server/cybersecurity-2) | NVD database | 🟡 As needed |\n| [OpenStreetMap](https://mcpmarket.com/server/openstreetmap) | Maps + POI | 🟢 As needed |\n| [CoinGecko](https://mcpmarket.com/server/coingecko) | Cryptocurrency | 🟢 As needed |\n| [Open Meteo](https://mcpmarket.com/tools/skills/open-meteo-weather) | Global weather | 🟢 As needed |\n\n### D. Existing Channel Reinforcements (8)\n\n| Option | Reinforces | Notes |\n|---|---|---|\n| [Package Version](https://mcpmarket.com/ja/server/package-version) | npm/PyPI/Maven/Go/Swift/Docker Hub and 9 registries total | One MCP covers 9 registries |\n| [GitLab MCP](https://mcpmarket.com/server/gitlab) | Complements GitHub | Public repos need no token |\n| [Newsfeed](https://mcpmarket.com/server/newsfeed) | `search-rss` preset categories | Free |\n| [Trend Radar](https://mcpmarket.com/tools/skills/trend-radar) | HN + GitHub signal aggregation | Free |\n| [Steam Context](https://mcpmarket.com/server/steam-context) | Games category | Steam API key is free |\n| [Zillow](https://mcpmarket.com/server/zillow) | Real-estate category | Mostly no key |\n| [AI Job Hunting Agent](https://mcpmarket.com/es/server/ai-job-hunting-agent) | Indeed + Remotive jobs | Free |\n| Sogou WeChat MCP wrappers ([ptbsare](https://github.com/ptbsare/sogou-weixin-mcp-server) / [fancyboi999](https://github.com/fancyboi999/weixin_search_mcp)) | Official Accounts | Same underlying Sogou SERP |\n\n---\n\n## Part 2: Hard-Target Breakthrough Matrix\n\n### 🔴 Integrated (Paid TikHub, 5/8)\n\n| Platform | TikHub endpoint | Measured Data |\n|---|---|---|\n| Xiaohongshu | `GET /api/v1/xiaohongshu/web/search_notes?keyword=<q>` | Claude 4.7 found 20 notes, including title/author/likes |\n| Zhihu | `GET /api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>` | LLM agent found 20 high-quality Q&A results |\n| Douyin | `GET /api/v1/douyin/web/fetch_video_search_result_v2?keyword=<q>` | DeepSeek found 12 results, highest with 540K likes |\n| Twitter/X | `GET /api/v1/twitter/web/fetch_search_timeline?keyword=<q>&search_type=Top` | 19 results, full fields (favorites/views/retweets) |\n| Bilibili | `GET /api/v1/bilibili/web/fetch_general_search?keyword=<q>&order=totalrank&page=1&page_size=10` | 8 videos + play counts |\n\n### ⚠️ Not Fully Solved in This Round (3/8)\n\n| Platform | Status | Debug Direction |\n|---|---|---|\n| **Weibo** | TikHub upstream is flaky: `web/fetch_search` returns ok=1 but `cards` is intermittently empty; `web_v2/fetch_realtime_search` 400; `app/fetch_search_all` 422 | Ask support in TikHub Discord; try `web_v2/fetch_ai_smart_search` |\n| **Instagram** | `v2/general_search` returns 200 OK but all zeroes | Try `v2/search_hashtags` or targeted keywords (#hashtag / @username) |\n| **LinkedIn** | `web/search_jobs` returns 400, even with geocode | Check TikHub docs for demo parameters; may require a special geocode format |\n\n### 🟢 Truly Impossible Without Cookies\n\n**Facebook / Instagram public personal pages / Xiaohongshu private-account content / Twitter private accounts**: even TikHub requires account support. These are not core to the research scenario, so do not invest for now.\n\n---\n\n## Part 3: TikHub Field Notes\n\n### Basic Information\n\n- **Homepage**: https://tikhub.io · **API base**: `https://api.tikhub.io/api/v1/`\n- **Auth**: `Authorization: Bearer $TIKHUB_API_KEY`\n- **OpenAPI spec**: `GET https://api.tikhub.io/openapi.json` (1058 endpoints)\n- **Billing**: pay-per-request, average **$0.0036/request** (official marketing says starts at $0.001; measured cost is 3-4x higher)\n- **Free quota**: Daily check-in gives a small amount of free credit\n\n### Platform Coverage (Confirmed Working)\n\n16 platforms, 1000+ tools:\n```\nTikTok (204) · Douyin (247) · Instagram (82) · Xiaohongshu (71)\nWeibo (64) · Bilibili (41) · YouTube (37) · Kuaishou (33)\nZhihu (32) · LinkedIn (25) · Reddit (24) · WeChat · Twitter\nThreads · TikHub Utilities · Other\n```\n\n### Key Pitfalls (In the Order Encountered)\n\n1. **Registration requires email verification**. Without verification, every endpoint returns 403 \"email not verified\".\n2. **The free tier covers only some endpoints**. Xiaohongshu/Weibo/Twitter/Douyin all return 402 (Payment Required) until paid balance is > 0. Zhihu and Bilibili are covered by the free tier.\n3. **Do not use `web_v3` for Xiaohongshu**. `/xiaohongshu/web_v3/fetch_search_notes` returns 400. Use the older `/xiaohongshu/web/search_notes`.\n4. **Do not use `app/v3` for Douyin**. `/douyin/app/v3/fetch_video_search_result` returns 400. Use `/douyin/web/fetch_video_search_result_v2`; data is under `data.business_data[i].data` (two nested layers).\n5. **Weibo endpoints are unstable across versions**. `web_v2/fetch_realtime_search` returns 400; `app/fetch_search_all` returns 422; `web/fetch_search` returns 200 but `cards` fluctuates.\n6. **400 = upstream scraping failure, not necessarily bad parameters**. TikHub's 400 body says \"please check docs and parameters\", but in practice TikHub itself failed to scrape the target site. This kind of 400 **is not billed** (Only pay for successful requests).\n7. **Twitter structure is already flattened**. `data.timeline` is a list, not the original GraphQL-style nested `instructions` structure. Iterate it directly. Fields: `screen_name / text / favorites / views / retweets / replies / created_at / tweet_id / lang`.\n\n### 🚨 Security Note: The Key Is Echoed in Error Responses\n\n**TikHub 400 / 403 / 422 response bodies echo the full request headers, including auth information**. Integration must:\n\n```python\n# BAD — printing the response body directly leaks the key\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=e.response.text) # ❌\n\n# GOOD — sanitize before logging\ndef sanitize(data: dict) -> dict:\n detail = data.get(\"detail\", {})\n if isinstance(detail, dict):\n detail.pop(\"headers\", None)\n for k in list(detail.keys()):\n if \"auth\" in k.lower() or \"token\" in k.lower():\n detail[k] = \"<redacted>\"\n return data\n\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=sanitize(e.response.json())) # ✅\n```\n\nTests must include: **assert exception string does not contain \"Bearer\"**.\n\n### MCP vs API Choice\n\nTikHub officially provides 4 transports: Stdio / SSE / Streamable HTTP / Curl(API).\n\n**For AutoSearch**: **choose direct Curl/API calls**, not MCP.\n\nReasons:\n- AutoSearch is a Python plugin installed with `pip install`. Using MCP would require users to additionally install Node.js + `mcp-remote` + the TikHub MCP server (3 dependencies)\n- AutoSearch channels are already \"tools\"; calling MCP from inside a tool is tool-calling-a-tool, and JSON-RPC over stdio adds a useless serialization layer\n- Enabling the full TikHub MCP exposes 1000+ tool schemas and injects roughly 400K tokens into Claude context, which is untenable\n- Even selecting only the 6 hard-target platforms still exposes ~470 tools (~190K tokens), which is still not ideal\n\n**Good fit for Stdio MCP**: ad hoc calls where a user in a Claude Code session wants to directly search Xiaohongshu, and this is for the user to install TikHub MCP themselves; it is unrelated to AutoSearch.\n\n---\n\n## Part 4: Final AutoSearch Integration Architecture\n\n### BYOK (Bring Your Own Key) Is the Only Reasonable Plan\n\n**Do not**:\n- ❌ Hardcode a TikHub key into the release build (a pip package exposes plaintext)\n- ❌ Share one key across all users (violates ToS and drains the balance)\n\n**Do**:\n- ✅ Users register and top up at `tikhub.io` themselves\n- ✅ `export TIKHUB_API_KEY=<your-key>`, and AutoSearch reads the env var\n- ✅ SKILL.md declares `requires: [env:TIKHUB_API_KEY]`; without a key, the channel is automatically marked unavailable while other channels continue running\n\n### Channel Structure (Validated by Pilot)\n\n```\nautosearch/lib/\n└── tikhub_client.py # Generic client + error redaction + 5xx retries\n\nautosearch/skills/channels/zhihu/\n├── SKILL.md # frontmatter declares via_tikhub method\n└── methods/\n └── via_tikhub.py # maps /zhihu/web/fetch_article_search_v3 to Evidence\n\ntests/lib/test_tikhub_client.py # client unit tests, including redaction assertion\ntests/channels/zhihu/test_via_tikhub.py # method unit tests\n```\n\n**fallback_chain order**: `[via_tikhub, api_search, api_answer_detail]` — prefer TikHub when a key is present; fall back to the original path when no key exists.\n\n### Leave Room for a Future Proxy\n\n`tikhub_client.py` reads the base URL from an env var:\n\n```python\nbase_url = os.getenv(\"TIKHUB_BASE_URL\", \"https://api.tikhub.io\")\n```\n\nIf the upstream needs to be changed later, such as to a self-hosted proxy or mirror, users only need to change two env vars with zero code changes:\n\n```bash\nexport TIKHUB_BASE_URL=https://your-proxy.example.com/tikhub\nexport TIKHUB_API_KEY=<your-proxy-token>\n```\n\n### 5 Channels Pending Integration (Priority Order)\n\n| Priority | Channel | Endpoint | Status |\n|---|---|---|---|\n| 1 | **zhihu** | `zhihu/web/fetch_article_search_v3` | ✅ pilot complete (branch `feat/tikhub-channels`) |\n| 2 | **xiaohongshu** | `xiaohongshu/web/search_notes` | Pending expansion |\n| 3 | **twitter** | `twitter/web/fetch_search_timeline` | Pending expansion |\n| 4 | **douyin** | `douyin/web/fetch_video_search_result_v2` | Pending expansion |\n| 5 | **bilibili** | `bilibili/web/fetch_general_search` | Pending expansion |\n\nAfter the pilot is validated, have Codex write the remaining 4 adapters in parallel from the same template. Each should be < 50 lines.\n\n---\n\n## Appendix\n\n### A. Test Command Quick Reference (Copy-Paste Ready)\n\n```bash\n# Sogou WeChat + readability cleanup = full-text Official Account search\ncd /tmp/mcp-smoke/ows\nnode build/index.js fetch-web \\\n \"https://weixin.sogou.com/weixin?type=2&query=<KEYWORD>\" \\\n --readability --spawn --json\n\n# open-websearch multi-engine\nnode build/index.js search \"<query>\" --engine duckduckgo --limit 5 --spawn --json\nnode build/index.js search \"<query>\" --engine csdn --limit 5 --spawn --json\n\n# Paper Search MCP academic search\ncd /tmp/mcp-smoke/paper-search-mcp\npython3 -c \"\nimport sys; sys.path.insert(0,'.')\nfrom paper_search_mcp.academic_platforms.arxiv import ArxivSearcher\ns = ArxivSearcher()\nfor p in s.search('<query>', max_results=5):\n print(p.title, p.url)\n\"\n\n# PullPush Reddit history\ncurl -s \"https://api.pullpush.io/reddit/search/submission/?subreddit=<NAME>&after=<TS>&before=<TS>&size=10\"\n\n# Jina Reader for any URL\ncurl -sL \"https://r.jina.ai/<URL>\"\n\n# Any TikHub endpoint, with redacted result inspection\ncurl -sS -H \"Authorization: Bearer $TIKHUB_API_KEY\" \\\n \"https://api.tikhub.io/api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>\" \\\n -o /tmp/t.json\npython3 -c \"\nimport json\nd = json.load(open('/tmp/t.json'))\ndetail = d.get('detail', {})\nif isinstance(detail, dict): detail.pop('headers', None)\nprint(json.dumps(d, indent=2, ensure_ascii=False)[:1000])\n\"\nrm -f /tmp/t.json # Key point: remove temp file that may contain echoed key data\n```\n\n### B. External Links\n\n- mcpmarket.com — research source for this round\n- tikhub.io — paid platform API\n- github.com/Aas-ee/open-webSearch — 8-engine free SERP\n- github.com/openags/paper-search-mcp — 21 academic sources\n- github.com/jacklenzotti/pullpush-mcp — Reddit history\n- r.jina.ai — free URL -> markdown via Jina Reader\n\n### C. Related Docs (Before This Round)\n\nConsolidated into this playbook, so **you no longer need to read them day to day**:\n- `mcp-channel-research.md` — initial research\n- `mcp-channel-test-plan.md` — test plan\n- `mcp-test-results.md` — first round of hands-on tests\n- `mcp-no-cookie-inventory.md` — no-cookie inventory\n- `mcp-final-summary.md` — phase summary\n- `tikhub-smoke-test.md` — detailed TikHub smoke test\n\nKeep them archived for traceability, but in daily use, **this playbook + `tikhub-smoke-test.md`** is enough.\n\n### D. Cost Ledger (As of 2026-04-19)\n\n- TikHub: **$0.053** (19 smoke requests)\n- Other: $0 (all free options)\n\n---\n\n**Maintenance rule**: When a new channel is integrated or a new hard target is unblocked, update the corresponding table in this playbook to keep it as the single source of truth.\n\n---\n\n## Appendix E. 2026-04-19 Supplemental Scan (New No-Key + No-Cookie Options)\n\nThis deeper mcpmarket.com pass found **12 fully no-key and no-cookie categories** missed earlier. Ranked by research value:\n\n### 🔴 High Value (Strongly Recommended)\n\n| MCP | Capability | Auth | Tested | Why It Matters |\n|---|---|---|---|---|\n| [FRED](https://mcpmarket.com/server/fred) · [Fredapi](https://mcpmarket.com/server/fredapi) · [FRED Macro Data](https://mcpmarket.com/server/fred-macroeconomic-data) | U.S. Federal Reserve economic data, 800K+ time series | 🟢 No key | Not tested | Essential macro signal for AI/startup research |\n| [World Bank Data](https://mcpmarket.com/server/world-bank-data) | Global economic indicators | 🟢 No key | Not tested | Core need for country-comparison research |\n| [Public APIs Directory](https://mcpmarket.com/es/server/public-apis) ([repo](https://github.com/zazencodes/public-apis-mcp)) | Free API directory semantic search | 🟢 No key | **✅ 04-19** | **Metatool** — future engine for AutoSearch to discover new channels |\n| [Open Library / Books](https://mcpmarket.com/server/books) · [OpenLibrary](https://mcpmarket.com/server/openlibrary) | ISBN lookup, full book catalog | 🟢 No auth | Not tested | Foundation for academic/publishing research |\n| [Word of the Day](https://mcpmarket.com/server/word-of-the-day) | Free Dictionary API with definitions, pronunciation, and examples | 🟢 No key | Not tested | Basic fact-checking utility |\n| [Dash Docset](https://mcpmarket.com/server/dash) · [Enhanced Dash](https://mcpmarket.com/server/enhanced-dash) | Local Dash docset query | 🟢 Local | Not tested | Instant technical-doc lookup |\n| [Hugging Face MCP](https://mcpmarket.com/tools/skills/hugging-face-mcp) | Hub model/dataset search | 🟢 Public search without key | **✅ 04-19** | Foundation for AI research |\n| Wikipedia API (official) | Encyclopedia search + summaries | 🟢 No key | **✅ 04-19** | Foundation for fact checking and encyclopedia enrichment |\n| Wikidata SPARQL (official) | Structured entities + graph queries | 🟢 No key | **✅ 04-19** | Entity relationship queries |\n| Google Trends via `pytrends` | Keyword trend time series | 🟢 No key | **✅ 04-19** | Topic heat and brand-comparison signals |\n\n### 🟡 Integrate As Needed\n\n| MCP | Capability | Auth | Scenario |\n|---|---|---|---|\n| [Flightradar24](https://mcpmarket.com/server/flightradar24-1) · [Flight (ADS-B)](https://mcpmarket.com/server/flight) | Real-time flight tracking | 🟢 Partly no key | Travel / supply-chain research |\n| [Etherscan MCP](https://mcpmarket.com/server/etherscan-2) · [Dune Analytics](https://mcpmarket.com/tools/skills/dune-analytics-on-chain-data) | Ethereum on-chain data | 🟢 Etherscan free key / 🟡 Dune paid | Web3 research |\n| [Ethereum JSON-RPC](https://mcpmarket.com/server/eth-1) | Native JSON-RPC chain query | 🟢 Public nodes | Deep DeFi research |\n| [TMDB](https://mcpmarket.com/server/tmdb) · [IMDb](https://mcpmarket.com/es/server/imdb-1) · [OMDB](https://mcpmarket.com/es/server/omdb) | Movie/TV data | 🟢 TMDB/OMDB free key | Content/entertainment research |\n| [Last.fm (ScrobblerContext)](https://mcpmarket.com/es/server/scrobblercontext) | Music data | 🟢 Free key | Content analysis |\n| [OpenFoodFacts](https://mcpmarket.com/server/openfoodfacts) | Global food database | 🟢 No key | Consumer goods / health research |\n\n### 🟢 Basic Utilities (Occasional Use)\n\n| MCP | Capability | Auth |\n|---|---|---|\n| [Whois Lookup](https://mcpmarket.com/server/whois) · [Domain Lookup](https://mcpmarket.com/server/domain-lookup) | WHOIS / RDAP lookup | 🟢 No key |\n| [DeepL MCP](https://mcpmarket.com/server/deepl) | Translation | 🟡 Free tier requires key |\n| [Pexels MCP](https://mcpmarket.com/es/server/pexels) | Free stock images | 🟡 Free key |\n\n### mcpmarket Has No Coverage for These Categories\n\n- **Toutiao / Baijiahao / NetEase Hao** (Chinese news aggregators)\n- **Reuters / TechCrunch / Bloomberg** (professional news, RSS only)\n- **Coursera / edX / Khan Academy / MOOC**\n- **Chinese government open data** (National Bureau of Statistics, customs)\n- **ESPN / sports data**\n- **Air quality / environmental data / real-time carbon emissions**\n\nThese can only be covered through RSS + Exa `site:`, or by waiting for future mcpmarket additions.\n\n### 2026-04-19 Hands-On Verification (5 Core MCPs)\n\nSmoke tests were run against the underlying APIs for the 5 most important MCPs for \"basic facts / trends / AI Hub / metatools.\" All passed:\n\n| MCP | Underlying API | Smoke Test | Sample |\n|---|---|---|---|\n| **Wikipedia** | `en.wikipedia.org/w/api.php?action=query&list=search&srsearch=<q>` | Search `Claude AI` -> 5 results | Claude (language model) · Anthropic · OpenAI Codex · Artificial intelligence · Project Maven |\n| **Wikidata** | `query.wikidata.org/sparql` (SPARQL) | `P31 wd:Q11660` query for AI entities -> 5 results | AlphaFold · Lee Luda · Intelligent Autonomous Systems |\n| **Google Trends** | `pytrends` Python library (unofficial, simulated requests) | 7 days hourly `Claude AI` vs `ChatGPT` -> **169 rows** | Latest hour Claude AI=3, ChatGPT=45, making the brand-awareness gap obvious |\n| **Hugging Face Hub** | `huggingface.co/api/models?search=<q>` | Search `llama` -> 5 results | meta-llama/Llama-3.1-8B-Instruct, 9.36M downloads · Llama-3.3-70B-Instruct, 490K downloads |\n| **Public APIs Directory** | Local `datastore/index.json` (1426 full APIs) | Full load + filter auth=No -> **668 no-key APIs** | 1426 APIs across dozens of categories including Animals/Crypto/Finance/Transport/Security |\n\n**Key clarification**: Public APIs Directory MCP ([zazencodes/public-apis-mcp](https://github.com/zazencodes/public-apis-mcp)) **does not depend on the now-dead `api.publicapis.org` at all**. Its data is built-in JSON (1426 entries), and the embedding index is built locally (.npz file). So even though the old API domain now fails DNS resolution, the MCP works correctly.\n\n**Pitfall warning — Google Trends**: pytrends is an unofficial simulated-request library. Google occasionally changes cookie/token mechanics and breaks it. Integration must include **retries + error tolerance** and must not treat it as a hard dependency. It is much more fragile than the other 4 official APIs.\n\n**Integration recommendation (lightest path)**: The first 4 can all be called directly through Python `httpx` against HTTP APIs; there is no need to start an MCP process. Public APIs Directory can have its `index.json` copied into AutoSearch and parsed locally. A 1426-row JSON file is small, and embeddings can be computed locally. This keeps AutoSearch free of new external process dependencies.\n\n---\n\n### Updated AutoSearch Priority After the Scan\n\nOn top of the 6 \"strongly recommended\" basics in Part 1 section C, **add 3 highest-priority items**:\n\n1. **FRED + World Bank** (macroeconomic signals, no key and no cookie in one step)\n2. **Public APIs Directory** (metatool; lets AutoSearch automatically discover new channels)\n3. **Open Library** (foundation for academic/publishing research)\n\nIn total, AutoSearch should now add **9 channels** to become the \"only full-coverage research system in the market\":\n\n```\nWikipedia + Wikidata # encyclopedia ✅ verified\nGoogle Trends # trends ✅ verified (flaky risk)\nHugging Face Hub # AI models ✅ verified\nPublic APIs Directory # metatool ✅ verified\nGoogle Patents # patents not verified\nCVE/NVD # security not verified\nFRED + World Bank # macroeconomics not verified\nOpen Library # books not verified\n```\n\nAll are 🟢 zero-key and zero-cookie. The implementation cost is small and the payoff is high.\n\n**First integration batch (4 already verified)**: Wikipedia, Wikidata, Hugging Face, Public APIs Directory — Codex can write channel adapters immediately, each < 30 lines of Python + httpx code.\n\n**Second integration batch (5 not yet smoke-tested)**: Google Trends, Google Patents, CVE/NVD, FRED, Open Library — proceed by research-scenario priority, and run a smoke test against the underlying API before each integration.\n"}, "4095d24c63933a9e65d9f3543e8aa3bab76dc592": {"t": 1785773302.9481592, "n": 24403, "h": "283aa6865eb903f8", "c": "# AutoSearch Channel Research Final Playbook\n\n> Date: 2026-04-19\n> Coverage: mcpmarket.com research + hands-on testing + TikHub paid API integration\n> Positioning: This is the **final master guide**. The previously scattered docs (`mcp-channel-research.md` / `mcp-test-plan.md` / `mcp-test-results.md` / `mcp-no-cookie-inventory.md` / `mcp-final-summary.md` / `tikhub-smoke-test.md`) are consolidated here. All future channel decisions should start from this document.\n\n---\n\n## TL;DR\n\n**Three-tier conclusion**:\n\n1. **Free and usable without cookies**: Sogou WeChat, open-websearch (4 engines), Paper Search MCP (21 academic sources), PullPush (Reddit history), direct RSS reads, Jina Reader (some sites), and 13 official APIs.\n2. **5 of 8 hard targets were unblocked through paid TikHub**: Xiaohongshu, Weibo, Zhihu, Twitter/X, and Douyin. The remaining 3 (flaky Weibo upstream, Instagram/LinkedIn parameters not fully tuned) were not fully solved in this round.\n3. **Final architecture**: AutoSearch uses BYOK (users provide their own TikHub key) and declares dependencies with `requires: [env:TIKHUB_API_KEY]`. The base URL is read from an env var so environments can switch cleanly.\n\n**Cost anchor**: TikHub measured average cost is **$0.0036/request**. 1000 research-grade calls cost about $3.6/month.\n\n---\n\n## Part 1: No-Cookie Usable Inventory (41)\n\n### A. Hands-On PASS in This Round (9)\n\n| Option | Platform/Capability | Auth | Notes |\n|---|---|---|---|\n| **Sogou WeChat SERP** | WeChat Official Accounts | 🟢 None | `weixin.sogou.com/weixin?type=2&query=<q>` |\n| **open-websearch fetch-web + readability** | Any page -> clean markdown | 🟢 None | Cleaned with the Mozilla Readability algorithm |\n| **open-websearch csdn** | CSDN technical blogs | 🟢 None | Native Chinese content |\n| **open-websearch duckduckgo** | DuckDuckGo | 🟢 None | Backup general search |\n| **open-websearch startpage** | Startpage (Google proxy) | 🟢 None | Backup general search |\n| **Paper Search MCP** | arXiv + PubMed + Semantic Scholar + OpenAlex + 17 other sources | 🟢 None | One-command access to 21 academic sources, MIT |\n| **PullPush** | Reddit history from 2005 to present | 🟢 None | Full-text `q=` search is broken; subreddit/time filtering works |\n| **Substack RSS** | Substack / Medium / independent blogs | 🟢 None | All sites expose `/feed` |\n| **Jina Reader** | Bilibili / weak anti-scraping sites | 🟢 None | Zhihu, Xiaohongshu, and Weibo are blocked |\n\n### B. Official APIs Known to Work (13, Integrated in AutoSearch or One-Line Integration)\n\n| Platform | Access Method | Limit |\n|---|---|---|\n| HackerNews Algolia | Official API | Unlimited |\n| Reddit public JSON | `/r/<sub>.json` | No login |\n| Stack Exchange API | Official | 300/day without key |\n| GitHub Public Search | Official API | 10/min without token |\n| Dev.to | Public API | No key |\n| npm / PyPI registry | Official | No key |\n| OpenReview API | Official | No key |\n| arXiv API | Official | No key |\n| Xueqiu / 36kr / InfoQ Chinese | Their web/API surfaces | No key |\n| Xiaoyuzhou | RSS / public pages | No key |\n| Bilibili public search API | Official | No login |\n| YouTube transcript | yt-dlp + whisper (local) | No key |\n| Crunchbase public pages | Public search | Limited |\n\n### C. New Categories Found on mcpmarket (AutoSearch Gaps) (11)\n\n| Option | Capability | Priority |\n|---|---|---|\n| [Wikipedia MCP](https://mcpmarket.com/server/wikipedia-1) | Full encyclopedia + summaries | 🔴 Highest |\n| [Wikidata](https://mcpmarket.com/server/wikidata) | SPARQL + entities | 🔴 Highest |\n| [Google Trends Explorer](https://mcpmarket.com/server/google-trends-explorer) | Trend signals | 🔴 Highest |\n| [Google News Trends](https://mcpmarket.com/es/server/google-news-trends) | News + trending terms | 🔴 High |\n| [USPTO](https://mcpmarket.com/server/uspto) | U.S. patents + trademarks | 🟡 As needed |\n| [Google Patents](https://mcpmarket.com/server/google-patents) | Global patents | 🟡 As needed |\n| [EPSS](https://mcpmarket.com/server/epss) | CVE + vulnerability scoring | 🟡 As needed |\n| [Cybersecurity CVE](https://mcpmarket.com/es/server/cybersecurity-2) | NVD database | 🟡 As needed |\n| [OpenStreetMap](https://mcpmarket.com/server/openstreetmap) | Maps + POI | 🟢 As needed |\n| [CoinGecko](https://mcpmarket.com/server/coingecko) | Cryptocurrency | 🟢 As needed |\n| [Open Meteo](https://mcpmarket.com/tools/skills/open-meteo-weather) | Global weather | 🟢 As needed |\n\n### D. Existing Channel Reinforcements (8)\n\n| Option | Reinforces | Notes |\n|---|---|---|\n| [Package Version](https://mcpmarket.com/ja/server/package-version) | npm/PyPI/Maven/Go/Swift/Docker Hub and 9 registries total | One MCP covers 9 registries |\n| [GitLab MCP](https://mcpmarket.com/server/gitlab) | Complements GitHub | Public repos need no token |\n| [Newsfeed](https://mcpmarket.com/server/newsfeed) | `search-rss` preset categories | Free |\n| [Trend Radar](https://mcpmarket.com/tools/skills/trend-radar) | HN + GitHub signal aggregation | Free |\n| [Steam Context](https://mcpmarket.com/server/steam-context) | Games category | Steam API key is free |\n| [Zillow](https://mcpmarket.com/server/zillow) | Real-estate category | Mostly no key |\n| [AI Job Hunting Agent](https://mcpmarket.com/es/server/ai-job-hunting-agent) | Indeed + Remotive jobs | Free |\n| Sogou WeChat MCP wrappers ([ptbsare](https://github.com/ptbsare/sogou-weixin-mcp-server) / [fancyboi999](https://github.com/fancyboi999/weixin_search_mcp)) | Official Accounts | Same underlying Sogou SERP |\n\n---\n\n## Part 2: Hard-Target Breakthrough Matrix\n\n### 🔴 Integrated (Paid TikHub, 5/8)\n\n| Platform | TikHub endpoint | Measured Data |\n|---|---|---|\n| Xiaohongshu | `GET /api/v1/xiaohongshu/web/search_notes?keyword=<q>` | Claude 4.7 found 20 notes, including title/author/likes |\n| Zhihu | `GET /api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>` | LLM agent found 20 high-quality Q&A results |\n| Douyin | `GET /api/v1/douyin/web/fetch_video_search_result_v2?keyword=<q>` | DeepSeek found 12 results, highest with 540K likes |\n| Twitter/X | `GET /api/v1/twitter/web/fetch_search_timeline?keyword=<q>&search_type=Top` | 19 results, full fields (favorites/views/retweets) |\n| Bilibili | `GET /api/v1/bilibili/web/fetch_general_search?keyword=<q>&order=totalrank&page=1&page_size=10` | 8 videos + play counts |\n\n### ⚠️ Not Fully Solved in This Round (3/8)\n\n| Platform | Status | Debug Direction |\n|---|---|---|\n| **Weibo** | TikHub upstream is flaky: `web/fetch_search` returns ok=1 but `cards` is intermittently empty; `web_v2/fetch_realtime_search` 400; `app/fetch_search_all` 422 | Ask support in TikHub Discord; try `web_v2/fetch_ai_smart_search` |\n| **Instagram** | `v2/general_search` returns 200 OK but all zeroes | Try `v2/search_hashtags` or targeted keywords (#hashtag / @username) |\n| **LinkedIn** | `web/search_jobs` returns 400, even with geocode | Check TikHub docs for demo parameters; may require a special geocode format |\n\n### 🟢 Truly Impossible Without Cookies\n\n**Facebook / Instagram public personal pages / Xiaohongshu private-account content / Twitter private accounts**: even TikHub requires account support. These are not core to the research scenario, so do not invest for now.\n\n---\n\n## Part 3: TikHub Field Notes\n\n### Basic Information\n\n- **Homepage**: https://tikhub.io · **API base**: `https://api.tikhub.io/api/v1/`\n- **Auth**: `Authorization: Bearer $TIKHUB_API_KEY`\n- **OpenAPI spec**: `GET https://api.tikhub.io/openapi.json` (1058 endpoints)\n- **Billing**: pay-per-request, average **$0.0036/request** (official marketing says starts at $0.001; measured cost is 3-4x higher)\n- **Free quota**: Daily check-in gives a small amount of free credit\n\n### Platform Coverage (Confirmed Working)\n\n16 platforms, 1000+ tools:\n```\nTikTok (204) · Douyin (247) · Instagram (82) · Xiaohongshu (71)\nWeibo (64) · Bilibili (41) · YouTube (37) · Kuaishou (33)\nZhihu (32) · LinkedIn (25) · Reddit (24) · WeChat · Twitter\nThreads · TikHub Utilities · Other\n```\n\n### Key Pitfalls (In the Order Encountered)\n\n1. **Registration requires email verification**. Without verification, every endpoint returns 403 \"email not verified\".\n2. **The free tier covers only some endpoints**. Xiaohongshu/Weibo/Twitter/Douyin all return 402 (Payment Required) until paid balance is > 0. Zhihu and Bilibili are covered by the free tier.\n3. **Do not use `web_v3` for Xiaohongshu**. `/xiaohongshu/web_v3/fetch_search_notes` returns 400. Use the older `/xiaohongshu/web/search_notes`.\n4. **Do not use `app/v3` for Douyin**. `/douyin/app/v3/fetch_video_search_result` returns 400. Use `/douyin/web/fetch_video_search_result_v2`; data is under `data.business_data[i].data` (two nested layers).\n5. **Weibo endpoints are unstable across versions**. `web_v2/fetch_realtime_search` returns 400; `app/fetch_search_all` returns 422; `web/fetch_search` returns 200 but `cards` fluctuates.\n6. **400 = upstream scraping failure, not necessarily bad parameters**. TikHub's 400 body says \"please check docs and parameters\", but in practice TikHub itself failed to scrape the target site. This kind of 400 **is not billed** (Only pay for successful requests).\n7. **Twitter structure is already flattened**. `data.timeline` is a list, not the original GraphQL-style nested `instructions` structure. Iterate it directly. Fields: `screen_name / text / favorites / views / retweets / replies / created_at / tweet_id / lang`.\n\n### 🚨 Security Note: The Key Is Echoed in Error Responses\n\n**TikHub 400 / 403 / 422 response bodies echo the full request headers, including auth information**. Integration must:\n\n```python\n# BAD — printing the response body directly leaks the key\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=e.response.text) # ❌\n\n# GOOD — sanitize before logging\ndef sanitize(data: dict) -> dict:\n detail = data.get(\"detail\", {})\n if isinstance(detail, dict):\n detail.pop(\"headers\", None)\n for k in list(detail.keys()):\n if \"auth\" in k.lower() or \"token\" in k.lower():\n detail[k] = \"<redacted>\"\n return data\n\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=sanitize(e.response.json())) # ✅\n```\n\nTests must include: **assert exception string does not contain \"Bearer\"**.\n\n### MCP vs API Choice\n\nTikHub officially provides 4 transports: Stdio / SSE / Streamable HTTP / Curl(API).\n\n**For AutoSearch**: **choose direct Curl/API calls**, not MCP.\n\nReasons:\n- AutoSearch is a Python plugin installed with `pip install`. Using MCP would require users to additionally install Node.js + `mcp-remote` + the TikHub MCP server (3 dependencies)\n- AutoSearch channels are already \"tools\"; calling MCP from inside a tool is tool-calling-a-tool, and JSON-RPC over stdio adds a useless serialization layer\n- Enabling the full TikHub MCP exposes 1000+ tool schemas and injects roughly 400K tokens into Claude context, which is untenable\n- Even selecting only the 6 hard-target platforms still exposes ~470 tools (~190K tokens), which is still not ideal\n\n**Good fit for Stdio MCP**: ad hoc calls where a user in a Claude Code session wants to directly search Xiaohongshu, and this is for the user to install TikHub MCP themselves; it is unrelated to AutoSearch.\n\n---\n\n## Part 4: Final AutoSearch Integration Architecture\n\n### BYOK (Bring Your Own Key) Is the Only Reasonable Plan\n\n**Do not**:\n- ❌ Hardcode a TikHub key into the release build (a pip package exposes plaintext)\n- ❌ Share one key across all users (violates ToS and drains the balance)\n\n**Do**:\n- ✅ Users register and top up at `tikhub.io` themselves\n- ✅ `export TIKHUB_API_KEY=<your-key>`, and AutoSearch reads the env var\n- ✅ SKILL.md declares `requires: [env:TIKHUB_API_KEY]`; without a key, the channel is automatically marked unavailable while other channels continue running\n\n### Channel Structure (Validated by Pilot)\n\n```\nautosearch/lib/\n└── tikhub_client.py # Generic client + error redaction + 5xx retries\n\nautosearch/skills/channels/zhihu/\n├── SKILL.md # frontmatter declares via_tikhub method\n└── methods/\n └── via_tikhub.py # maps /zhihu/web/fetch_article_search_v3 to Evidence\n\ntests/lib/test_tikhub_client.py # client unit tests, including redaction assertion\ntests/channels/zhihu/test_via_tikhub.py # method unit tests\n```\n\n**fallback_chain order**: `[via_tikhub, api_search, api_answer_detail]` — prefer TikHub when a key is present; fall back to the original path when no key exists.\n\n### Leave Room for a Future Proxy\n\n`tikhub_client.py` reads the base URL from an env var:\n\n```python\nbase_url = os.getenv(\"TIKHUB_BASE_URL\", \"https://api.tikhub.io\")\n```\n\nIf the upstream needs to be changed later, such as to a self-hosted proxy or mirror, users only need to change two env vars with zero code changes:\n\n```bash\nexport TIKHUB_BASE_URL=https://your-proxy.example.com/tikhub\nexport TIKHUB_API_KEY=<your-proxy-token>\n```\n\n### 5 Channels Pending Integration (Priority Order)\n\n| Priority | Channel | Endpoint | Status |\n|---|---|---|---|\n| 1 | **zhihu** | `zhihu/web/fetch_article_search_v3` | ✅ pilot complete (branch `feat/tikhub-channels`) |\n| 2 | **xiaohongshu** | `xiaohongshu/web/search_notes` | Pending expansion |\n| 3 | **twitter** | `twitter/web/fetch_search_timeline` | Pending expansion |\n| 4 | **douyin** | `douyin/web/fetch_video_search_result_v2` | Pending expansion |\n| 5 | **bilibili** | `bilibili/web/fetch_general_search` | Pending expansion |\n\nAfter the pilot is validated, have Codex write the remaining 4 adapters in parallel from the same template. Each should be < 50 lines.\n\n---\n\n## Appendix\n\n### A. Test Command Quick Reference (Copy-Paste Ready)\n\n```bash\n# Sogou WeChat + readability cleanup = full-text Official Account search\ncd /tmp/mcp-smoke/ows\nnode build/index.js fetch-web \\\n \"https://weixin.sogou.com/weixin?type=2&query=<KEYWORD>\" \\\n --readability --spawn --json\n\n# open-websearch multi-engine\nnode build/index.js search \"<query>\" --engine duckduckgo --limit 5 --spawn --json\nnode build/index.js search \"<query>\" --engine csdn --limit 5 --spawn --json\n\n# Paper Search MCP academic search\ncd /tmp/mcp-smoke/paper-search-mcp\npython3 -c \"\nimport sys; sys.path.insert(0,'.')\nfrom paper_search_mcp.academic_platforms.arxiv import ArxivSearcher\ns = ArxivSearcher()\nfor p in s.search('<query>', max_results=5):\n print(p.title, p.url)\n\"\n\n# PullPush Reddit history\ncurl -s \"https://api.pullpush.io/reddit/search/submission/?subreddit=<NAME>&after=<TS>&before=<TS>&size=10\"\n\n# Jina Reader for any URL\ncurl -sL \"https://r.jina.ai/<URL>\"\n\n# Any TikHub endpoint, with redacted result inspection\ncurl -sS -H \"Authorization: Bearer $TIKHUB_API_KEY\" \\\n \"https://api.tikhub.io/api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>\" \\\n -o /tmp/t.json\npython3 -c \"\nimport json\nd = json.load(open('/tmp/t.json'))\ndetail = d.get('detail', {})\nif isinstance(detail, dict): detail.pop('headers', None)\nprint(json.dumps(d, indent=2, ensure_ascii=False)[:1000])\n\"\nrm -f /tmp/t.json # Key point: remove temp file that may contain echoed key data\n```\n\n### B. External Links\n\n- mcpmarket.com — research source for this round\n- tikhub.io — paid platform API\n- github.com/Aas-ee/open-webSearch — 8-engine free SERP\n- github.com/openags/paper-search-mcp — 21 academic sources\n- github.com/jacklenzotti/pullpush-mcp — Reddit history\n- r.jina.ai — free URL -> markdown via Jina Reader\n\n### C. Related Docs (Before This Round)\n\nConsolidated into this playbook, so **you no longer need to read them day to day**:\n- `mcp-channel-research.md` — initial research\n- `mcp-channel-test-plan.md` — test plan\n- `mcp-test-results.md` — first round of hands-on tests\n- `mcp-no-cookie-inventory.md` — no-cookie inventory\n- `mcp-final-summary.md` — phase summary\n- `tikhub-smoke-test.md` — detailed TikHub smoke test\n\nKeep them archived for traceability, but in daily use, **this playbook + `tikhub-smoke-test.md`** is enough.\n\n### D. Cost Ledger (As of 2026-04-19)\n\n- TikHub: **$0.053** (19 smoke requests)\n- Other: $0 (all free options)\n\n---\n\n**Maintenance rule**: When a new channel is integrated or a new hard target is unblocked, update the corresponding table in this playbook to keep it as the single source of truth.\n\n---\n\n## Appendix E. 2026-04-19 Supplemental Scan (New No-Key + No-Cookie Options)\n\nThis deeper mcpmarket.com pass found **12 fully no-key and no-cookie categories** missed earlier. Ranked by research value:\n\n### 🔴 High Value (Strongly Recommended)\n\n| MCP | Capability | Auth | Tested | Why It Matters |\n|---|---|---|---|---|\n| [FRED](https://mcpmarket.com/server/fred) · [Fredapi](https://mcpmarket.com/server/fredapi) · [FRED Macro Data](https://mcpmarket.com/server/fred-macroeconomic-data) | U.S. Federal Reserve economic data, 800K+ time series | 🟢 No key | Not tested | Essential macro signal for AI/startup research |\n| [World Bank Data](https://mcpmarket.com/server/world-bank-data) | Global economic indicators | 🟢 No key | Not tested | Core need for country-comparison research |\n| [Public APIs Directory](https://mcpmarket.com/es/server/public-apis) ([repo](https://github.com/zazencodes/public-apis-mcp)) | Free API directory semantic search | 🟢 No key | **✅ 04-19** | **Metatool** — future engine for AutoSearch to discover new channels |\n| [Open Library / Books](https://mcpmarket.com/server/books) · [OpenLibrary](https://mcpmarket.com/server/openlibrary) | ISBN lookup, full book catalog | 🟢 No auth | Not tested | Foundation for academic/publishing research |\n| [Word of the Day](https://mcpmarket.com/server/word-of-the-day) | Free Dictionary API with definitions, pronunciation, and examples | 🟢 No key | Not tested | Basic fact-checking utility |\n| [Dash Docset](https://mcpmarket.com/server/dash) · [Enhanced Dash](https://mcpmarket.com/server/enhanced-dash) | Local Dash docset query | 🟢 Local | Not tested | Instant technical-doc lookup |\n| [Hugging Face MCP](https://mcpmarket.com/tools/skills/hugging-face-mcp) | Hub model/dataset search | 🟢 Public search without key | **✅ 04-19** | Foundation for AI research |\n| Wikipedia API (official) | Encyclopedia search + summaries | 🟢 No key | **✅ 04-19** | Foundation for fact checking and encyclopedia enrichment |\n| Wikidata SPARQL (official) | Structured entities + graph queries | 🟢 No key | **✅ 04-19** | Entity relationship queries |\n| Google Trends via `pytrends` | Keyword trend time series | 🟢 No key | **✅ 04-19** | Topic heat and brand-comparison signals |\n\n### 🟡 Integrate As Needed\n\n| MCP | Capability | Auth | Scenario |\n|---|---|---|---|\n| [Flightradar24](https://mcpmarket.com/server/flightradar24-1) · [Flight (ADS-B)](https://mcpmarket.com/server/flight) | Real-time flight tracking | 🟢 Partly no key | Travel / supply-chain research |\n| [Etherscan MCP](https://mcpmarket.com/server/etherscan-2) · [Dune Analytics](https://mcpmarket.com/tools/skills/dune-analytics-on-chain-data) | Ethereum on-chain data | 🟢 Etherscan free key / 🟡 Dune paid | Web3 research |\n| [Ethereum JSON-RPC](https://mcpmarket.com/server/eth-1) | Native JSON-RPC chain query | 🟢 Public nodes | Deep DeFi research |\n| [TMDB](https://mcpmarket.com/server/tmdb) · [IMDb](https://mcpmarket.com/es/server/imdb-1) · [OMDB](https://mcpmarket.com/es/server/omdb) | Movie/TV data | 🟢 TMDB/OMDB free key | Content/entertainment research |\n| [Last.fm (ScrobblerContext)](https://mcpmarket.com/es/server/scrobblercontext) | Music data | 🟢 Free key | Content analysis |\n| [OpenFoodFacts](https://mcpmarket.com/server/openfoodfacts) | Global food database | 🟢 No key | Consumer goods / health research |\n\n### 🟢 Basic Utilities (Occasional Use)\n\n| MCP | Capability | Auth |\n|---|---|---|\n| [Whois Lookup](https://mcpmarket.com/server/whois) · [Domain Lookup](https://mcpmarket.com/server/domain-lookup) | WHOIS / RDAP lookup | 🟢 No key |\n| [DeepL MCP](https://mcpmarket.com/server/deepl) | Translation | 🟡 Free tier requires key |\n| [Pexels MCP](https://mcpmarket.com/es/server/pexels) | Free stock images | 🟡 Free key |\n\n### mcpmarket Has No Coverage for These Categories\n\n- **Toutiao / Baijiahao / NetEase Hao** (Chinese news aggregators)\n- **Reuters / TechCrunch / Bloomberg** (professional news, RSS only)\n- **Coursera / edX / Khan Academy / MOOC**\n- **Chinese government open data** (National Bureau of Statistics, customs)\n- **ESPN / sports data**\n- **Air quality / environmental data / real-time carbon emissions**\n\nThese can only be covered through RSS + Exa `site:`, or by waiting for future mcpmarket additions.\n\n### 2026-04-19 Hands-On Verification (5 Core MCPs)\n\nSmoke tests were run against the underlying APIs for the 5 most important MCPs for \"basic facts / trends / AI Hub / metatools.\" All passed:\n\n| MCP | Underlying API | Smoke Test | Sample |\n|---|---|---|---|\n| **Wikipedia** | `en.wikipedia.org/w/api.php?action=query&list=search&srsearch=<q>` | Search `Claude AI` -> 5 results | Claude (language model) · Anthropic · OpenAI Codex · Artificial intelligence · Project Maven |\n| **Wikidata** | `query.wikidata.org/sparql` (SPARQL) | `P31 wd:Q11660` query for AI entities -> 5 results | AlphaFold · Lee Luda · Intelligent Autonomous Systems |\n| **Google Trends** | `pytrends` Python library (unofficial, simulated requests) | 7 days hourly `Claude AI` vs `ChatGPT` -> **169 rows** | Latest hour Claude AI=3, ChatGPT=45, making the brand-awareness gap obvious |\n| **Hugging Face Hub** | `huggingface.co/api/models?search=<q>` | Search `llama` -> 5 results | meta-llama/Llama-3.1-8B-Instruct, 9.36M downloads · Llama-3.3-70B-Instruct, 490K downloads |\n| **Public APIs Directory** | Local `datastore/index.json` (1426 full APIs) | Full load + filter auth=No -> **668 no-key APIs** | 1426 APIs across dozens of categories including Animals/Crypto/Finance/Transport/Security |\n\n**Key clarification**: Public APIs Directory MCP ([zazencodes/public-apis-mcp](https://github.com/zazencodes/public-apis-mcp)) **does not depend on the now-dead `api.publicapis.org` at all**. Its data is built-in JSON (1426 entries), and the embedding index is built locally (.npz file). So even though the old API domain now fails DNS resolution, the MCP works correctly.\n\n**Pitfall warning — Google Trends**: pytrends is an unofficial simulated-request library. Google occasionally changes cookie/token mechanics and breaks it. Integration must include **retries + error tolerance** and must not treat it as a hard dependency. It is much more fragile than the other 4 official APIs.\n\n**Integration recommendation (lightest path)**: The first 4 can all be called directly through Python `httpx` against HTTP APIs; there is no need to start an MCP process. Public APIs Directory can have its `index.json` copied into AutoSearch and parsed locally. A 1426-row JSON file is small, and embeddings can be computed locally. This keeps AutoSearch free of new external process dependencies.\n\n---\n\n### Updated AutoSearch Priority After the Scan\n\nOn top of the 6 \"strongly recommended\" basics in Part 1 section C, **add 3 highest-priority items**:\n\n1. **FRED + World Bank** (macroeconomic signals, no key and no cookie in one step)\n2. **Public APIs Directory** (metatool; lets AutoSearch automatically discover new channels)\n3. **Open Library** (foundation for academic/publishing research)\n\nIn total, AutoSearch should now add **9 channels** to become the \"only full-coverage research system in the market\":\n\n```\nWikipedia + Wikidata # encyclopedia ✅ verified\nGoogle Trends # trends ✅ verified (flaky risk)\nHugging Face Hub # AI models ✅ verified\nPublic APIs Directory # metatool ✅ verified\nGoogle Patents # patents not verified\nCVE/NVD # security not verified\nFRED + World Bank # macroeconomics not verified\nOpen Library # books not verified\n```\n\nAll are 🟢 zero-key and zero-cookie. The implementation cost is small and the payoff is high.\n\n**First integration batch (4 already verified)**: Wikipedia, Wikidata, Hugging Face, Public APIs Directory — Codex can write channel adapters immediately, each < 30 lines of Python + httpx code.\n\n**Second integration batch (5 not yet smoke-tested)**: Google Trends, Google Patents, CVE/NVD, FRED, Open Library — proceed by research-scenario priority, and run a smoke test against the underlying API before each integration.\n"}, "50b4a03f68b88050d57d5eb427830af05d531136": {"t": 1785773304.5931473, "n": 17060, "h": "3c2795c3c62e3c3b", "c": "# AutoSearch 渠道调研最终手册\n\n> 时间：2026-04-19\n> 覆盖：mcpmarket.com 调研 + 实测 + TikHub 付费 API 打通\n> 定位：这份是**最终总纲**，之前分散文档（`mcp-channel-research.md` / `mcp-test-plan.md` / `mcp-test-results.md` / `mcp-no-cookie-inventory.md` / `mcp-final-summary.md` / `tikhub-smoke-test.md`）都汇总到这里。未来所有渠道决策从这份开始。\n\n---\n\n## TL;DR\n\n**三档结论**：\n\n1. **无 cookie 免费可用**：搜狗微信、open-websearch（4 引擎）、Paper Search MCP（21 学术源）、PullPush（Reddit 历史）、RSS 直读、Jina Reader（部分站）、以及 13 个官方 API。\n2. **硬骨头 8 个中付费打通 5 个**（TikHub）：小红书、微博、知乎、Twitter/X、抖音。剩 3 个（微博上游 flaky、Instagram/LinkedIn 参数未调通）本轮未完全解决。\n3. **最终架构**：AutoSearch 走 BYOK（用户自配 TikHub key），`requires: [env:TIKHUB_API_KEY]` 声明依赖。Base URL 从 env var 读，便于在不同环境之间切换。\n\n**成本锚点**：TikHub 实测平均 **$0.0036/请求**。1000 次研究级调用 ≈ $3.6/月。\n\n---\n\n## 第一部分：无 cookie 可用清单（41 个）\n\n### A. 本次亲手实测 PASS（9 个）\n\n| 方案 | 平台/能力 | 认证 | 备注 |\n|---|---|---|---|\n| **搜狗微信 SERP** | 微信公众号 | 🟢 无 | `weixin.sogou.com/weixin?type=2&query=<q>` |\n| **open-websearch fetch-web + readability** | 任意页 → 干净 markdown | 🟢 无 | Mozilla Readability 算法清洗 |\n| **open-websearch csdn** | CSDN 技术博客 | 🟢 无 | 原生中文 |\n| **open-websearch duckduckgo** | DuckDuckGo | 🟢 无 | 备用通用搜 |\n| **open-websearch startpage** | Startpage（Google 代理） | 🟢 无 | 备用通用搜 |\n| **Paper Search MCP** | arXiv + PubMed + Semantic Scholar + OpenAlex + 其他 17 源 | 🟢 无 | 一键 21 学术源，MIT |\n| **PullPush** | Reddit 历史数据 2005-至今 | 🟢 无 | `q=` 全文搜挂，按 subreddit/时间过滤正常 |\n| **Substack RSS** | Substack / Medium / 独立博客 | 🟢 无 | 所有站暴露 `/feed` |\n| **Jina Reader** | B站 / 反爬弱的站 | 🟢 无 | 知乎小红书微博会被封 |\n\n### B. 官方 API 已知可用（13 个，AutoSearch 已集成或一行接入）\n\n| 平台 | 接入方式 | 限额 |\n|---|---|---|\n| HackerNews Algolia | 官方 API | 无限 |\n| Reddit 公开 JSON | `/r/<sub>.json` | 无登录 |\n| Stack Exchange API | 官方 | 300/day 无 key |\n| GitHub Public Search | 官方 API | 10/min 无 token |\n| Dev.to | 公开 API | 无 key |\n| npm / PyPI registry | 官方 | 无 key |\n| OpenReview API | 官方 | 无 key |\n| arXiv API | 官方 | 无 key |\n| 雪球 / 36kr / InfoQ 中文 | 各自 web/API | 无 key |\n| 小宇宙 | RSS / 公开页 | 无 key |\n| B站公开搜索 API | 官方 | 无登录 |\n| YouTube transcript | yt-dlp + whisper（本地） | 无 key |\n| Crunchbase 公开页 | 公开搜索 | 有限 |\n\n### C. mcpmarket 发现的新品类（AutoSearch 空白）（11 个）\n\n| 方案 | 能力 | 优先级 |\n|---|---|---|\n| [Wikipedia MCP](https://mcpmarket.com/server/wikipedia-1) | 百科全站 + 摘要 | 🔴 最高 |\n| [Wikidata](https://mcpmarket.com/server/wikidata) | SPARQL + 实体 | 🔴 最高 |\n| [Google Trends Explorer](https://mcpmarket.com/server/google-trends-explorer) | 趋势信号 | 🔴 最高 |\n| [Google News Trends](https://mcpmarket.com/es/server/google-news-trends) | 新闻 + 趋势词 | 🔴 高 |\n| [USPTO](https://mcpmarket.com/server/uspto) | 美国专利 + 商标 | 🟡 按需 |\n| [Google Patents](https://mcpmarket.com/server/google-patents) | 全球专利 | 🟡 按需 |\n| [EPSS](https://mcpmarket.com/server/epss) | CVE + 漏洞评分 | 🟡 按需 |\n| [Cybersecurity CVE](https://mcpmarket.com/es/server/cybersecurity-2) | NVD 数据库 | 🟡 按需 |\n| [OpenStreetMap](https://mcpmarket.com/server/openstreetmap) | 地图 + POI | 🟢 按需 |\n| [CoinGecko](https://mcpmarket.com/server/coingecko) | 加密货币 | 🟢 按需 |\n| [Open Meteo](https://mcpmarket.com/tools/skills/open-meteo-weather) | 全球天气 | 🟢 按需 |\n\n### D. 现有 channel 补强（8 个）\n\n| 方案 | 补哪个 | 备注 |\n|---|---|---|\n| [Package Version](https://mcpmarket.com/ja/server/package-version) | npm/PyPI/Maven/Go/Swift/Docker Hub 等 9 注册表 | 一个 MCP 顶 9 个 |\n| [GitLab MCP](https://mcpmarket.com/server/gitlab) | 补 GitHub 之外 | 公开 repo 无 token |\n| [Newsfeed](https://mcpmarket.com/server/newsfeed) | `search-rss` 预置分类 | 免费 |\n| [Trend Radar](https://mcpmarket.com/tools/skills/trend-radar) | HN + GitHub 信号聚合 | 免费 |\n| [Steam Context](https://mcpmarket.com/server/steam-context) | 游戏品类 | Steam API key 免费 |\n| [Zillow](https://mcpmarket.com/server/zillow) | 房产品类 | 大部分无 key |\n| [AI Job Hunting Agent](https://mcpmarket.com/es/server/ai-job-hunting-agent) | Indeed + Remotive 岗位 | 免费 |\n| 搜狗微信 MCP 包装（[ptbsare](https://github.com/ptbsare/sogou-weixin-mcp-server) / [fancyboi999](https://github.com/fancyboi999/weixin_search_mcp)） | 公众号 | 底层同搜狗 SERP |\n\n---\n\n## 第二部分：硬骨头攻克矩阵\n\n### 🔴 已打通（TikHub 付费，5/8）\n\n| 平台 | TikHub endpoint | 实测数据 |\n|---|---|---|\n| 小红书 | `GET /api/v1/xiaohongshu/web/search_notes?keyword=<q>` | Claude 4.7 搜到 20 条笔记，含 title/author/likes |\n| 知乎 | `GET /api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>` | LLM agent 搜到 20 条高质量问答 |\n| 抖音 | `GET /api/v1/douyin/web/fetch_video_search_result_v2?keyword=<q>` | DeepSeek 搜到 12 条，最高 54 万赞 |\n| Twitter/X | `GET /api/v1/twitter/web/fetch_search_timeline?keyword=<q>&search_type=Top` | 19 条，全字段（favorites/views/retweets） |\n| B站 | `GET /api/v1/bilibili/web/fetch_general_search?keyword=<q>&order=totalrank&page=1&page_size=10` | 8 条视频 + 播放量 |\n\n### ⚠️ 本轮未完全解决（3/8）\n\n| 平台 | 状态 | 排查方向 |\n|---|---|---|\n| **微博** | TikHub 上游 flaky——`web/fetch_search` 返回 ok=1 但 cards 时空时不空，重试波动；`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422 | 去 TikHub Discord 问客服；尝试 `web_v2/fetch_ai_smart_search` |\n| **Instagram** | `v2/general_search` 返回 200 OK 但全 0 | 换 `v2/search_hashtags` 或针对性关键词（#hashtag / @username） |\n| **LinkedIn** | `web/search_jobs` 400，加 geocode 也 400 | 去 TikHub docs 查 demo 参数，可能需要特殊 geocode 格式 |\n\n### 🟢 真·完全无法无 cookie 搞定\n\n**Facebook / Instagram 公开个人页 / 小红书私人号内容 / Twitter 私人账号**——即使 TikHub 也要账号支持。这些不在研究场景核心，暂不投入。\n\n---\n\n## 第三部分：TikHub 实战经验\n\n### 基础信息\n\n- **官网**：https://tikhub.io · **API base**：`https://api.tikhub.io/api/v1/`\n- **认证**：`Authorization: Bearer $TIKHUB_API_KEY`\n- **OpenAPI 规格**：`GET https://api.tikhub.io/openapi.json`（1058 个 endpoint）\n- **计费**：pay-per-request，平均 **$0.0036/次**（官方宣传 $0.001 起步，实测贵 3-4 倍）\n- **免费额度**：每日签到可领少量 free credit\n\n### 覆盖平台（确认跑通）\n\n16 平台共 1000+ tools：\n```\nTikTok (204) · Douyin (247) · Instagram (82) · Xiaohongshu (71)\nWeibo (64) · Bilibili (41) · YouTube (37) · Kuaishou (33)\nZhihu (32) · LinkedIn (25) · Reddit (24) · WeChat · Twitter\nThreads · TikHub Utilities · 其他\n```\n\n### 关键坑（按踩坑顺序）\n\n1. **注册要验证邮箱**。不验证一切 endpoint 都 403 \"邮箱未验证\"。\n2. **免费 tier 只覆盖部分 endpoint**。小红书/微博/Twitter/抖音都是 402（Payment Required）直到付费余额 > 0。知乎 / B站属于免费 tier 覆盖。\n3. **小红书别用 `web_v3`**。`/xiaohongshu/web_v3/fetch_search_notes` 返回 400。用老版 `/xiaohongshu/web/search_notes`。\n4. **抖音别用 `app/v3`**。`/douyin/app/v3/fetch_video_search_result` 返回 400。用 `/douyin/web/fetch_video_search_result_v2`，数据在 `data.business_data[i].data`（嵌套两层）。\n5. **微博 endpoint 多版本都不稳**。`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422；`web/fetch_search` 200 但 cards 波动。\n6. **400 = 上游爬取失败，不是参数错**。TikHub 的 400 body 是\"请查看文档核对参数\"，但实际上是 TikHub 自己爬目标站失败。这种 400 **不计费**（Only pay for successful requests）。\n7. **Twitter 结构已扁平化**。`data.timeline` 是 list（不是 GraphQL 原格式的 instructions 嵌套），直接遍历即可。字段：`screen_name / text / favorites / views / retweets / replies / created_at / tweet_id / lang`。\n\n### 🚨 安全注意：Key 会在错误响应里被回显\n\n**TikHub 的 400 / 403 / 422 响应体会原样回显完整请求头（含认证信息）**。接入必须：\n\n```python\n# BAD — 直接打印 response body 会泄露 key\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=e.response.text) # ❌\n\n# GOOD — 先脱敏再 log\ndef sanitize(data: dict) -> dict:\n detail = data.get(\"detail\", {})\n if isinstance(detail, dict):\n detail.pop(\"headers\", None)\n for k in list(detail.keys()):\n if \"auth\" in k.lower() or \"token\" in k.lower():\n detail[k] = \"<redacted>\"\n return data\n\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=sanitize(e.response.json())) # ✅\n```\n\n测试里必须有一条：**assert exception string 不包含 \"Bearer\"**。\n\n### MCP vs API 选择\n\nTikHub 官方提供 4 种 transport：Stdio / SSE / Streamable HTTP / Curl(API)。\n\n**对 AutoSearch 而言**：**选 Curl/API 直连**，不要走 MCP。\n\n理由：\n- AutoSearch 是 Python plugin，用户 `pip install` 安装。走 MCP 要求用户额外装 Node.js + `mcp-remote` + TikHub MCP server（3 个依赖）\n- AutoSearch 的 channel 是\"工具\"本身，再走一层 MCP 是工具里调工具，JSON-RPC over stdio 多一层序列化毫无意义\n- TikHub MCP 全勾是 1000+ tool schema，注入 Claude context ~400k token，直接炸\n- 即使只勾 6 个硬骨头平台也 ~470 tools（~190k token），依然不理想\n\n**Stdio MCP 适用场景**：用户在 Claude Code 会话里想直接 \"搜一下小红书\" 的即兴调用。这是用户自己装 TikHub MCP 的事，和 AutoSearch 无关。\n\n---\n\n## 第四部分：AutoSearch 最终接入架构\n\n### BYOK（Bring Your Own Key）是唯一合理方案\n\n**不要做的**：\n- ❌ 把 TikHub key 硬编码到发布版里（pip 包明文可提取）\n- ❌ 一个 key 共享给所有用户（违反 ToS + 余额被烧光）\n\n**要做的**：\n- ✅ 用户自己去 `tikhub.io` 注册 + 充值\n- ✅ `export TIKHUB_API_KEY=xxx`，AutoSearch 读 env var\n- ✅ SKILL.md 声明 `requires: [env:TIKHUB_API_KEY]`，没 key 时 channel 自动标记不可用，其他 channel 照跑\n\n### Channel 结构（已通过 pilot 验证）\n\n```\nautosearch/lib/\n└── tikhub_client.py # 通用客户端 + 错误脱敏 + 5xx 重试\n\nautosearch/skills/channels/zhihu/\n├── SKILL.md # frontmatter 声明 via_tikhub method\n└── methods/\n └── via_tikhub.py # 调 /zhihu/web/fetch_article_search_v3 映射到 Evidence\n\ntests/lib/test_tikhub_client.py # 客户端单测（含脱敏断言）\ntests/channels/zhihu/test_via_tikhub.py # method 单测\n```\n\n**fallback_chain 顺序**：`[via_tikhub, api_search, api_answer_detail]` — TikHub 有 key 时优先用，没 key 时 fallback 到原方案。\n\n### 为未来 proxy 留口子\n\n`tikhub_client.py` 的 base URL 从 env var 读：\n\n```python\nbase_url = os.getenv(\"TIKHUB_BASE_URL\", \"https://api.tikhub.io\")\n```\n\n这样如果以后需要换上游（比如自托管代理或镜像），用户只需改两个 env var，零代码改动：\n\n```bash\nexport TIKHUB_BASE_URL=https://your-proxy.example.com/tikhub\nexport TIKHUB_API_KEY=<your-proxy-token>\n```\n\n### 待接入的 5 个 channel（按优先级）\n\n| 优先级 | Channel | Endpoint | 状态 |\n|---|---|---|---|\n| 1 | **zhihu** | `zhihu/web/fetch_article_search_v3` | ✅ pilot 写完（branch `feat/tikhub-channels`） |\n| 2 | **xiaohongshu** | `xiaohongshu/web/search_notes` | 待扩展 |\n| 3 | **twitter** | `twitter/web/fetch_search_timeline` | 待扩展 |\n| 4 | **douyin** | `douyin/web/fetch_video_search_result_v2` | 待扩展 |\n| 5 | **bilibili** | `bilibili/web/fetch_general_search` | 待扩展 |\n\nPilot 验证通过后，让 Codex 按同一模板并行写剩下 4 个 adapter。改造量每个 < 50 行。\n\n---\n\n## 附录\n\n### A. 测试命令速查（可复制即跑）\n\n```bash\n# 搜狗微信 + readability 清洗 = 公众号全文搜索\ncd /tmp/mcp-smoke/ows\nnode build/index.js fetch-web \\\n \"https://weixin.sogou.com/weixin?type=2&query=<KEYWORD>\" \\\n --readability --spawn --json\n\n# open-websearch 多引擎\nnode build/index.js search \"<query>\" --engine duckduckgo --limit 5 --spawn --json\nnode build/index.js search \"<query>\" --engine csdn --limit 5 --spawn --json\n\n# Paper Search MCP 学术\ncd /tmp/mcp-smoke/paper-search-mcp\npython3 -c \"\nimport sys; sys.path.insert(0,'.')\nfrom paper_search_mcp.academic_platforms.arxiv import ArxivSearcher\ns = ArxivSearcher()\nfor p in s.search('<query>', max_results=5):\n print(p.title, p.url)\n\"\n\n# PullPush Reddit 历史\ncurl -s \"https://api.pullpush.io/reddit/search/submission/?subreddit=<NAME>&after=<TS>&before=<TS>&size=10\"\n\n# Jina Reader 任意 URL\ncurl -sL \"https://r.jina.ai/<URL>\"\n\n# TikHub 任意 endpoint（脱敏地看结果）\ncurl -sS -H \"Authorization: Bearer $TIKHUB_API_KEY\" \\\n \"https://api.tikhub.io/api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>\" \\\n -o /tmp/t.json\npython3 -c \"\nimport json\nd = json.load(open('/tmp/t.json'))\ndetail = d.get('detail', {})\nif isinstance(detail, dict): detail.pop('headers', None)\nprint(json.dumps(d, indent=2, ensure_ascii=False)[:1000])\n\"\nrm -f /tmp/t.json # 关键：清理可能含 key 回显的临时文件\n```\n\n### B. 外部链接\n\n- mcpmarket.com — 本次调研来源\n- tikhub.io — 付费平台 API\n- github.com/Aas-ee/open-webSearch — 8 引擎免费 SERP\n- github.com/openags/paper-search-mcp — 21 学术源\n- github.com/jacklenzotti/pullpush-mcp — Reddit 历史\n- r.jina.ai — Jina Reader 免费 URL → markdown\n\n### C. 相关文档（本次之前）\n\n已被本手册汇总、**可以不用再翻**：\n- `mcp-channel-research.md` — 初期调研\n- `mcp-channel-test-plan.md` — 测试计划\n- `mcp-test-results.md` — 第一轮实测\n- `mcp-no-cookie-inventory.md` — 无 cookie 清单\n- `mcp-final-summary.md` — 阶段总结\n- `tikhub-smoke-test.md` — TikHub 详细实测\n\n保留归档供溯源，但日常看**本手册 + `tikhub-smoke-test.md`** 即可。\n\n### D. 开销总账（截至 2026-04-19）\n\n- TikHub：**$0.053**（19 次 smoke request）\n- 其他：$0（全部免费方案）\n\n---\n\n**维护规则**：接入一个新 channel 或打通一个新硬骨头 → 更新本手册对应表格，保持单一事实来源。\n\n---\n\n## 附 E. 2026-04-19 补充扫描（免 key + 免 cookie 新增）\n\n本轮深挖 mcpmarket.com 发现了之前漏掉的 **12 个完全免 key 免 cookie 的品类**。按研究价值分档：\n\n### 🔴 高价值（强烈建议接入）\n\n| MCP | 能力 | 认证 | 实测 | 为什么值得 |\n|---|---|---|---|---|\n| [FRED](https://mcpmarket.com/server/fred) · [Fredapi](https://mcpmarket.com/server/fredapi) · [FRED Macro Data](https://mcpmarket.com/server/fred-macroeconomic-data) | 美联储经济数据 80w+ 时间序列 | 🟢 无 key | 未测 | AI/创业研究的宏观信号必备 |\n| [World Bank Data](https://mcpmarket.com/server/world-bank-data) | 全球经济指标 | 🟢 无 key | 未测 | 国别对比研究刚需 |\n| [Public APIs Directory](https://mcpmarket.com/es/server/public-apis) ([repo](https://github.com/zazencodes/public-apis-mcp)) | 免费 API 目录 semantic search | 🟢 无 key | **✅ 04-19** | **元工具** — AutoSearch 未来发现新 channel 的引擎 |\n| [Open Library / Books](https://mcpmarket.com/server/books) · [OpenLibrary](https://mcpmarket.com/server/openlibrary) | ISBN 书籍查询，全量书目 | 🟢 无 auth | 未测 | 学术/出版研究基础 |\n| [Word of the Day](https://mcpmarket.com/server/word-of-the-day) | Free Dictionary API，定义+发音+例句 | 🟢 无 key | 未测 | 事实核查基础工具 |\n| [Dash Docset](https://mcpmarket.com/server/dash) · [Enhanced Dash](https://mcpmarket.com/server/enhanced-dash) | 本地 Dash docset 查询 | 🟢 本地 | 未测 | 技术文档即时查询 |\n| [Hugging Face MCP](https://mcpmarket.com/tools/skills/hugging-face-mcp) | Hub model/dataset 搜索 | 🟢 公开搜索无 key | **✅ 04-19** | AI 研究基础 |\n| Wikipedia API（官方） | 百科搜索 + 摘要 | 🟢 无 key | **✅ 04-19** | 事实核查/百科补全基础 |\n| Wikidata SPARQL（官方） | 结构化实体 + 图谱查询 | 🟢 无 key | **✅ 04-19** | 实体关系查询 |\n| Google Trends via `pytrends` | 关键词趋势时间序列 | 🟢 无 key | **✅ 04-19** | 话题热度/品牌对比信号 |\n\n### 🟡 按需接入\n\n| MCP | 能力 | 认证 | 场景 |\n|---|---|---|---|\n| [Flightradar24](https://mcpmarket.com/server/flightradar24-1) · [Flight (ADS-B)](https://mcpmarket.com/server/flight) | 实时航班追踪 | 🟢 部分无 key | 出行 / 供应链研究 |\n| [Etherscan MCP](https://mcpmarket.com/server/etherscan-2) · [Dune Analytics](https://mcpmarket.com/tools/skills/dune-analytics-on-chain-data) | 以太坊链上数据 | 🟢 Etherscan 免费 key / 🟡 Dune 付费 | Web3 研究 |\n| [Ethereum JSON-RPC](https://mcpmarket.com/server/eth-1) | 原生 JSON-RPC 查链 | 🟢 公开节点 | DeFi 深度研究 |\n| [TMDB](https://mcpmarket.com/server/tmdb) · [IMDb](https://mcpmarket.com/es/server/imdb-1) · [OMDB](https://mcpmarket.com/es/server/omdb) | 电影/电视数据 | 🟢 TMDB/OMDB 免费 key | 内容/娱乐研究 |\n| [Last.fm (ScrobblerContext)](https://mcpmarket.com/es/server/scrobblercontext) | 音乐数据 | 🟢 免费 key | 内容分析 |\n| [OpenFoodFacts](https://mcpmarket.com/server/openfoodfacts) | 全球食品数据库 | 🟢 无 key | 消费品/健康研究 |\n\n### 🟢 基础工具（偶尔用）\n\n| MCP | 能力 | 认证 |\n|---|---|---|\n| [Whois Lookup](https://mcpmarket.com/server/whois) · [Domain Lookup](https://mcpmarket.com/server/domain-lookup) | WHOIS / RDAP 查询 | 🟢 无 key |\n| [DeepL MCP](https://mcpmarket.com/server/deepl) | 翻译 | 🟡 免费 tier 需 key |\n| [Pexels MCP](https://mcpmarket.com/es/server/pexels) | 免费 stock 图片 | 🟡 免费 key |\n\n### mcpmarket 完全空白（这些品类以后别找 mcpmarket）\n\n- **今日头条 / 百家号 / 网易号**（中文新闻聚合）\n- **Reuters / TechCrunch / Bloomberg**（专业新闻，只有 RSS）\n- **Coursera / edX / Khan Academy / MOOC**\n- **中国政府开放数据**（统计局、海关）\n- **ESPN / 体育数据**\n- **空气质量 / 环境数据 / 实时碳排放**\n\n这些只能靠 RSS + Exa `site:` 覆盖，或者等 mcpmarket 以后上新。\n\n### 2026-04-19 实测验证（5 个核心 MCP）\n\n对\"基础事实 / 趋势 / AI Hub / 元工具\"这 5 个最关键 MCP 做了底层 API smoke test。结果全 PASS：\n\n| MCP | 底层 API | Smoke test | 样本 |\n|---|---|---|---|\n| **Wikipedia** | `en.wikipedia.org/w/api.php?action=query&list=search&srsearch=<q>` | 搜 `Claude AI` → 5 条 | Claude (language model) · Anthropic · OpenAI Codex · Artificial intelligence · Project Maven |\n| **Wikidata** | `query.wikidata.org/sparql` (SPARQL) | `P31 wd:Q11660` 查 AI 实体 → 5 条 | AlphaFold · Lee Luda · Intelligent Autonomous Systems |\n| **Google Trends** | `pytrends` Python 库（非官方，模拟请求） | 7 天每小时 `Claude AI` vs `ChatGPT` → **169 行** | 最新一小时 Claude AI=3, ChatGPT=45（品牌认知度差距直观） |\n| **Hugging Face Hub** | `huggingface.co/api/models?search=<q>` | 搜 `llama` → 5 条 | meta-llama/Llama-3.1-8B-Instruct ⬇936万 · Llama-3.3-70B-Instruct ⬇49万 |\n| **Public APIs Directory** | 本地 `datastore/index.json`（1426 API 全量） | 全量加载 + 筛 auth=No → **668 个免 key** | 1426 个 API 涵盖 Animals/Crypto/Finance/Transport/Security 等几十类 |\n\n**关键澄清**：Public APIs Directory MCP（[zazencodes/public-apis-mcp](https://github.com/zazencodes/public-apis-mcp)）**完全不依赖已挂掉的 `api.publicapis.org`**。它的数据是内置 JSON（1426 条），embedding 索引本地构建（.npz 文件）。所以即使那个老 API 域名已 DNS 解析失败，MCP 功能完好。\n\n**踩坑预警 — Google Trends**：pytrends 是非官方模拟请求，Google 偶尔会改 cookie/token 机制导致失效。接入时**必加重试 + 错误容忍**，不能作为硬依赖。比其他 4 个官方 API 脆弱得多。\n\n**接入建议（最轻方案）**：前 4 个都可以**直接 Python httpx 调 HTTP API**，不必启动 MCP 进程。Public APIs Directory 可以把它的 `index.json` 拷贝到 AutoSearch 里自己解析——1426 条 JSON 没什么分量，embedding 也可以本地现算。这样 AutoSearch 对外零新依赖进程。\n\n---\n\n### 扫描后更新 AutoSearch 优先级\n\n原 playbook 第一部分 C 节的\"强烈建议接入\"6 个基础上，**追加 3 个最高优先级**：\n\n1. **FRED + World Bank**（宏观经济信号，免 key 免 cookie 一步到位）\n2. **Public APIs Directory**（元工具，加了它 AutoSearch 自动发现新 channel）\n3. **Open Library**（学术/出版研究基础）\n\n合计现在 AutoSearch 应该新增 **9 个 channel** 就能达到\"业内唯一全覆盖研究系统\"：\n\n```\nWikipedia + Wikidata # 百科 ✅ 已验证\nGoogle Trends # 趋势 ✅ 已验证（有 flaky 风险）\nHugging Face Hub # AI 模型 ✅ 已验证\nPublic APIs Directory # 元工具 ✅ 已验证\nGoogle Patents # 专利 未验证\nCVE/NVD # 安全 未验证\nFRED + World Bank # 宏观经济 未验证\nOpen Library # 书籍 未验证\n```\n\n全部 🟢 零 key 零 cookie。改造量小，收益最大。\n\n**第一批接入（已实测可用的 4 个）**：Wikipedia、Wikidata、Hugging Face、Public APIs Directory — 这 4 个可以立刻让 Codex 写 channel adapter，每个 < 30 行 Python + httpx 代码。\n\n**第二批接入（未实测 5 个）**：Google Trends、Google Patents、CVE/NVD、FRED、Open Library — 按研究场景优先级推进，每个接入前先 smoke test 底层 API。\n"}, "4385559015b76116ae030334e73aa0f7206b4870": {"t": 1785773306.2182221, "n": 17085, "h": "d5348a2026d209c5", "c": "# AutoSearch 渠道调研最终手册\n\n> 时间：2026-04-19\n> 覆盖：mcpmarket.com 调研 + 实测 + TikHub 付费 API 打通\n> 定位：这份是**最终总纲**，之前分散文档（`mcp-channel-research.md` / `mcp-test-plan.md` / `mcp-test-results.md` / `mcp-no-cookie-inventory.md` / `mcp-final-summary.md` / `tikhub-smoke-test.md`）都汇总到这里。未来所有渠道决策从这份开始。\n\n---\n\n## TL;DR\n\n**三档结论**：\n\n1. **无 cookie 免费可用**：搜狗微信、open-websearch（4 引擎）、Paper Search MCP（21 学术源）、PullPush（Reddit 历史）、RSS 直读、Jina Reader（部分站）、以及 13 个官方 API。\n2. **硬骨头 8 个中付费打通 5 个**（TikHub）：小红书、微博、知乎、Twitter/X、抖音。剩 3 个（微博上游 flaky、Instagram/LinkedIn 参数未调通）本轮未完全解决。\n3. **最终架构**：AutoSearch 走 BYOK（用户自配 TikHub key），`requires: [env:TIKHUB_API_KEY]` 声明依赖，未来可无缝切换到 hosted proxy 做商业化。\n\n**成本锚点**：TikHub 实测平均 **$0.0036/请求**。1000 次研究级调用 ≈ $3.6/月。\n\n---\n\n## 第一部分：无 cookie 可用清单（41 个）\n\n### A. 本次亲手实测 PASS（9 个）\n\n| 方案 | 平台/能力 | 认证 | 备注 |\n|---|---|---|---|\n| **搜狗微信 SERP** | 微信公众号 | 🟢 无 | `weixin.sogou.com/weixin?type=2&query=<q>` |\n| **open-websearch fetch-web + readability** | 任意页 → 干净 markdown | 🟢 无 | Mozilla Readability 算法清洗 |\n| **open-websearch csdn** | CSDN 技术博客 | 🟢 无 | 原生中文 |\n| **open-websearch duckduckgo** | DuckDuckGo | 🟢 无 | 备用通用搜 |\n| **open-websearch startpage** | Startpage（Google 代理） | 🟢 无 | 备用通用搜 |\n| **Paper Search MCP** | arXiv + PubMed + Semantic Scholar + OpenAlex + 其他 17 源 | 🟢 无 | 一键 21 学术源，MIT |\n| **PullPush** | Reddit 历史数据 2005-至今 | 🟢 无 | `q=` 全文搜挂，按 subreddit/时间过滤正常 |\n| **Substack RSS** | Substack / Medium / 独立博客 | 🟢 无 | 所有站暴露 `/feed` |\n| **Jina Reader** | B站 / 反爬弱的站 | 🟢 无 | 知乎小红书微博会被封 |\n\n### B. 官方 API 已知可用（13 个，AutoSearch 已集成或一行接入）\n\n| 平台 | 接入方式 | 限额 |\n|---|---|---|\n| HackerNews Algolia | 官方 API | 无限 |\n| Reddit 公开 JSON | `/r/<sub>.json` | 无登录 |\n| Stack Exchange API | 官方 | 300/day 无 key |\n| GitHub Public Search | 官方 API | 10/min 无 token |\n| Dev.to | 公开 API | 无 key |\n| npm / PyPI registry | 官方 | 无 key |\n| OpenReview API | 官方 | 无 key |\n| arXiv API | 官方 | 无 key |\n| 雪球 / 36kr / InfoQ 中文 | 各自 web/API | 无 key |\n| 小宇宙 | RSS / 公开页 | 无 key |\n| B站公开搜索 API | 官方 | 无登录 |\n| YouTube transcript | yt-dlp + whisper（本地） | 无 key |\n| Crunchbase 公开页 | 公开搜索 | 有限 |\n\n### C. mcpmarket 发现的新品类（AutoSearch 空白）（11 个）\n\n| 方案 | 能力 | 优先级 |\n|---|---|---|\n| [Wikipedia MCP](https://mcpmarket.com/server/wikipedia-1) | 百科全站 + 摘要 | 🔴 最高 |\n| [Wikidata](https://mcpmarket.com/server/wikidata) | SPARQL + 实体 | 🔴 最高 |\n| [Google Trends Explorer](https://mcpmarket.com/server/google-trends-explorer) | 趋势信号 | 🔴 最高 |\n| [Google News Trends](https://mcpmarket.com/es/server/google-news-trends) | 新闻 + 趋势词 | 🔴 高 |\n| [USPTO](https://mcpmarket.com/server/uspto) | 美国专利 + 商标 | 🟡 按需 |\n| [Google Patents](https://mcpmarket.com/server/google-patents) | 全球专利 | 🟡 按需 |\n| [EPSS](https://mcpmarket.com/server/epss) | CVE + 漏洞评分 | 🟡 按需 |\n| [Cybersecurity CVE](https://mcpmarket.com/es/server/cybersecurity-2) | NVD 数据库 | 🟡 按需 |\n| [OpenStreetMap](https://mcpmarket.com/server/openstreetmap) | 地图 + POI | 🟢 按需 |\n| [CoinGecko](https://mcpmarket.com/server/coingecko) | 加密货币 | 🟢 按需 |\n| [Open Meteo](https://mcpmarket.com/tools/skills/open-meteo-weather) | 全球天气 | 🟢 按需 |\n\n### D. 现有 channel 补强（8 个）\n\n| 方案 | 补哪个 | 备注 |\n|---|---|---|\n| [Package Version](https://mcpmarket.com/ja/server/package-version) | npm/PyPI/Maven/Go/Swift/Docker Hub 等 9 注册表 | 一个 MCP 顶 9 个 |\n| [GitLab MCP](https://mcpmarket.com/server/gitlab) | 补 GitHub 之外 | 公开 repo 无 token |\n| [Newsfeed](https://mcpmarket.com/server/newsfeed) | `search-rss` 预置分类 | 免费 |\n| [Trend Radar](https://mcpmarket.com/tools/skills/trend-radar) | HN + GitHub 信号聚合 | 免费 |\n| [Steam Context](https://mcpmarket.com/server/steam-context) | 游戏品类 | Steam API key 免费 |\n| [Zillow](https://mcpmarket.com/server/zillow) | 房产品类 | 大部分无 key |\n| [AI Job Hunting Agent](https://mcpmarket.com/es/server/ai-job-hunting-agent) | Indeed + Remotive 岗位 | 免费 |\n| 搜狗微信 MCP 包装（[ptbsare](https://github.com/ptbsare/sogou-weixin-mcp-server) / [fancyboi999](https://github.com/fancyboi999/weixin_search_mcp)） | 公众号 | 底层同搜狗 SERP |\n\n---\n\n## 第二部分：硬骨头攻克矩阵\n\n### 🔴 已打通（TikHub 付费，5/8）\n\n| 平台 | TikHub endpoint | 实测数据 |\n|---|---|---|\n| 小红书 | `GET /api/v1/xiaohongshu/web/search_notes?keyword=<q>` | Claude 4.7 搜到 20 条笔记，含 title/author/likes |\n| 知乎 | `GET /api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>` | LLM agent 搜到 20 条高质量问答 |\n| 抖音 | `GET /api/v1/douyin/web/fetch_video_search_result_v2?keyword=<q>` | DeepSeek 搜到 12 条，最高 54 万赞 |\n| Twitter/X | `GET /api/v1/twitter/web/fetch_search_timeline?keyword=<q>&search_type=Top` | 19 条，全字段（favorites/views/retweets） |\n| B站 | `GET /api/v1/bilibili/web/fetch_general_search?keyword=<q>&order=totalrank&page=1&page_size=10` | 8 条视频 + 播放量 |\n\n### ⚠️ 本轮未完全解决（3/8）\n\n| 平台 | 状态 | 排查方向 |\n|---|---|---|\n| **微博** | TikHub 上游 flaky——`web/fetch_search` 返回 ok=1 但 cards 时空时不空，重试波动；`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422 | 去 TikHub Discord 问客服；尝试 `web_v2/fetch_ai_smart_search` |\n| **Instagram** | `v2/general_search` 返回 200 OK 但全 0 | 换 `v2/search_hashtags` 或针对性关键词（#hashtag / @username） |\n| **LinkedIn** | `web/search_jobs` 400，加 geocode 也 400 | 去 TikHub docs 查 demo 参数，可能需要特殊 geocode 格式 |\n\n### 🟢 真·完全无法无 cookie 搞定\n\n**Facebook / Instagram 公开个人页 / 小红书私人号内容 / Twitter 私人账号**——即使 TikHub 也要账号支持。这些不在研究场景核心，暂不投入。\n\n---\n\n## 第三部分：TikHub 实战经验\n\n### 基础信息\n\n- **官网**：https://tikhub.io · **API base**：`https://api.tikhub.io/api/v1/`\n- **认证**：`Authorization: Bearer $TIKHUB_API_KEY`\n- **OpenAPI 规格**：`GET https://api.tikhub.io/openapi.json`（1058 个 endpoint）\n- **计费**：pay-per-request，平均 **$0.0036/次**（官方宣传 $0.001 起步，实测贵 3-4 倍）\n- **免费额度**：每日签到可领少量 free credit\n\n### 覆盖平台（确认跑通）\n\n16 平台共 1000+ tools：\n```\nTikTok (204) · Douyin (247) · Instagram (82) · Xiaohongshu (71)\nWeibo (64) · Bilibili (41) · YouTube (37) · Kuaishou (33)\nZhihu (32) · LinkedIn (25) · Reddit (24) · WeChat · Twitter\nThreads · TikHub Utilities · 其他\n```\n\n### 关键坑（按踩坑顺序）\n\n1. **注册要验证邮箱**。不验证一切 endpoint 都 403 \"邮箱未验证\"。\n2. **免费 tier 只覆盖部分 endpoint**。小红书/微博/Twitter/抖音都是 402（Payment Required）直到付费余额 > 0。知乎 / B站属于免费 tier 覆盖。\n3. **小红书别用 `web_v3`**。`/xiaohongshu/web_v3/fetch_search_notes` 返回 400。用老版 `/xiaohongshu/web/search_notes`。\n4. **抖音别用 `app/v3`**。`/douyin/app/v3/fetch_video_search_result` 返回 400。用 `/douyin/web/fetch_video_search_result_v2`，数据在 `data.business_data[i].data`（嵌套两层）。\n5. **微博 endpoint 多版本都不稳**。`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422；`web/fetch_search` 200 但 cards 波动。\n6. **400 = 上游爬取失败，不是参数错**。TikHub 的 400 body 是\"请查看文档核对参数\"，但实际上是 TikHub 自己爬目标站失败。这种 400 **不计费**（Only pay for successful requests）。\n7. **Twitter 结构已扁平化**。`data.timeline` 是 list（不是 GraphQL 原格式的 instructions 嵌套），直接遍历即可。字段：`screen_name / text / favorites / views / retweets / replies / created_at / tweet_id / lang`。\n\n### 🚨 安全注意：Key 会在错误响应里被回显\n\n**TikHub 的 400 / 403 / 422 响应体会原样回显完整 `Authorization` header（含 Bearer token）**。接入必须：\n\n```python\n# BAD — 直接打印 response body 会泄露 key\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=e.response.text) # ❌\n\n# GOOD — 先脱敏再 log\ndef sanitize(data: dict) -> dict:\n detail = data.get(\"detail\", {})\n if isinstance(detail, dict):\n detail.pop(\"headers\", None)\n for k in list(detail.keys()):\n if \"auth\" in k.lower() or \"token\" in k.lower():\n detail[k] = \"<redacted>\"\n return data\n\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=sanitize(e.response.json())) # ✅\n```\n\n测试里必须有一条：**assert exception string 不包含 \"Bearer\"**。\n\n### MCP vs API 选择\n\nTikHub 官方提供 4 种 transport：Stdio / SSE / Streamable HTTP / Curl(API)。\n\n**对 AutoSearch 而言**：**选 Curl/API 直连**，不要走 MCP。\n\n理由：\n- AutoSearch 是 Python plugin，用户 `pip install` 安装。走 MCP 要求用户额外装 Node.js + `mcp-remote` + TikHub MCP server（3 个依赖）\n- AutoSearch 的 channel 是\"工具\"本身，再走一层 MCP 是工具里调工具，JSON-RPC over stdio 多一层序列化毫无意义\n- TikHub MCP 全勾是 1000+ tool schema，注入 Claude context ~400k token，直接炸\n- 即使只勾 6 个硬骨头平台也 ~470 tools（~190k token），依然不理想\n\n**Stdio MCP 适用场景**：用户在 Claude Code 会话里想直接 \"搜一下小红书\" 的即兴调用。这是用户自己装 TikHub MCP 的事，和 AutoSearch 无关。\n\n---\n\n## 第四部分：AutoSearch 最终接入架构\n\n### BYOK（Bring Your Own Key）是唯一合理方案\n\n**不要做的**：\n- ❌ 把 TikHub key 硬编码到发布版里（pip 包明文可提取）\n- ❌ 一个 key 共享给所有用户（违反 ToS + 余额被烧光）\n\n**要做的**：\n- ✅ 用户自己去 `tikhub.io` 注册 + 充值\n- ✅ `export TIKHUB_API_KEY=xxx`，AutoSearch 读 env var\n- ✅ SKILL.md 声明 `requires: [env:TIKHUB_API_KEY]`，没 key 时 channel 自动标记不可用，其他 channel 照跑\n\n### Channel 结构（已通过 pilot 验证）\n\n```\nautosearch/lib/\n└── tikhub_client.py # 通用客户端 + 错误脱敏 + 5xx 重试\n\nautosearch/skills/channels/zhihu/\n├── SKILL.md # frontmatter 声明 via_tikhub method\n└── methods/\n └── via_tikhub.py # 调 /zhihu/web/fetch_article_search_v3 映射到 Evidence\n\ntests/lib/test_tikhub_client.py # 客户端单测（含脱敏断言）\ntests/channels/zhihu/test_via_tikhub.py # method 单测\n```\n\n**fallback_chain 顺序**：`[via_tikhub, api_search, api_answer_detail]` — TikHub 有 key 时优先用，没 key 时 fallback 到原方案。\n\n### 为未来 proxy 留口子\n\n`tikhub_client.py` 的 base URL 从 env var 读：\n\n```python\nbase_url = os.getenv(\"TIKHUB_BASE_URL\", \"https://api.tikhub.io\")\n```\n\n未来老板上 hosted proxy 做商业化时，用户只需改两个 env var，零代码改动：\n\n```bash\nexport TIKHUB_BASE_URL=https://api.autosearch.com/tikhub\nexport TIKHUB_API_KEY=<AutoSearch 平台发的 token>\n```\n\n### 待接入的 5 个 channel（按优先级）\n\n| 优先级 | Channel | Endpoint | 状态 |\n|---|---|---|---|\n| 1 | **zhihu** | `zhihu/web/fetch_article_search_v3` | ✅ pilot 写完（branch `feat/tikhub-channels`） |\n| 2 | **xiaohongshu** | `xiaohongshu/web/search_notes` | 待扩展 |\n| 3 | **twitter** | `twitter/web/fetch_search_timeline` | 待扩展 |\n| 4 | **douyin** | `douyin/web/fetch_video_search_result_v2` | 待扩展 |\n| 5 | **bilibili** | `bilibili/web/fetch_general_search` | 待扩展 |\n\nPilot 验证通过后，让 Codex 按同一模板并行写剩下 4 个 adapter。改造量每个 < 50 行。\n\n---\n\n## 附录\n\n### A. 测试命令速查（可复制即跑）\n\n```bash\n# 搜狗微信 + readability 清洗 = 公众号全文搜索\ncd /tmp/mcp-smoke/ows\nnode build/index.js fetch-web \\\n \"https://weixin.sogou.com/weixin?type=2&query=<KEYWORD>\" \\\n --readability --spawn --json\n\n# open-websearch 多引擎\nnode build/index.js search \"<query>\" --engine duckduckgo --limit 5 --spawn --json\nnode build/index.js search \"<query>\" --engine csdn --limit 5 --spawn --json\n\n# Paper Search MCP 学术\ncd /tmp/mcp-smoke/paper-search-mcp\npython3 -c \"\nimport sys; sys.path.insert(0,'.')\nfrom paper_search_mcp.academic_platforms.arxiv import ArxivSearcher\ns = ArxivSearcher()\nfor p in s.search('<query>', max_results=5):\n print(p.title, p.url)\n\"\n\n# PullPush Reddit 历史\ncurl -s \"https://api.pullpush.io/reddit/search/submission/?subreddit=<NAME>&after=<TS>&before=<TS>&size=10\"\n\n# Jina Reader 任意 URL\ncurl -sL \"https://r.jina.ai/<URL>\"\n\n# TikHub 任意 endpoint（脱敏地看结果）\ncurl -sS -H \"Authorization: Bearer $TIKHUB_API_KEY\" \\\n \"https://api.tikhub.io/api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>\" \\\n -o /tmp/t.json\npython3 -c \"\nimport json\nd = json.load(open('/tmp/t.json'))\ndetail = d.get('detail', {})\nif isinstance(detail, dict): detail.pop('headers', None)\nprint(json.dumps(d, indent=2, ensure_ascii=False)[:1000])\n\"\nrm -f /tmp/t.json # 关键：清理可能含 key 回显的临时文件\n```\n\n### B. 外部链接\n\n- mcpmarket.com — 本次调研来源\n- tikhub.io — 付费平台 API\n- github.com/Aas-ee/open-webSearch — 8 引擎免费 SERP\n- github.com/openags/paper-search-mcp — 21 学术源\n- github.com/jacklenzotti/pullpush-mcp — Reddit 历史\n- r.jina.ai — Jina Reader 免费 URL → markdown\n\n### C. 相关文档（本次之前）\n\n已被本手册汇总、**可以不用再翻**：\n- `mcp-channel-research.md` — 初期调研\n- `mcp-channel-test-plan.md` — 测试计划\n- `mcp-test-results.md` — 第一轮实测\n- `mcp-no-cookie-inventory.md` — 无 cookie 清单\n- `mcp-final-summary.md` — 阶段总结\n- `tikhub-smoke-test.md` — TikHub 详细实测\n\n保留归档供溯源，但日常看**本手册 + `tikhub-smoke-test.md`** 即可。\n\n### D. 开销总账（截至 2026-04-19）\n\n- TikHub：**$0.053**（19 次 smoke request）\n- 其他：$0（全部免费方案）\n\n---\n\n**维护规则**：接入一个新 channel 或打通一个新硬骨头 → 更新本手册对应表格，保持单一事实来源。\n\n---\n\n## 附 E. 2026-04-19 补充扫描（免 key + 免 cookie 新增）\n\n本轮深挖 mcpmarket.com 发现了之前漏掉的 **12 个完全免 key 免 cookie 的品类**。按研究价值分档：\n\n### 🔴 高价值（强烈建议接入）\n\n| MCP | 能力 | 认证 | 实测 | 为什么值得 |\n|---|---|---|---|---|\n| [FRED](https://mcpmarket.com/server/fred) · [Fredapi](https://mcpmarket.com/server/fredapi) · [FRED Macro Data](https://mcpmarket.com/server/fred-macroeconomic-data) | 美联储经济数据 80w+ 时间序列 | 🟢 无 key | 未测 | AI/创业研究的宏观信号必备 |\n| [World Bank Data](https://mcpmarket.com/server/world-bank-data) | 全球经济指标 | 🟢 无 key | 未测 | 国别对比研究刚需 |\n| [Public APIs Directory](https://mcpmarket.com/es/server/public-apis) ([repo](https://github.com/zazencodes/public-apis-mcp)) | 免费 API 目录 semantic search | 🟢 无 key | **✅ 04-19** | **元工具** — AutoSearch 未来发现新 channel 的引擎 |\n| [Open Library / Books](https://mcpmarket.com/server/books) · [OpenLibrary](https://mcpmarket.com/server/openlibrary) | ISBN 书籍查询，全量书目 | 🟢 无 auth | 未测 | 学术/出版研究基础 |\n| [Word of the Day](https://mcpmarket.com/server/word-of-the-day) | Free Dictionary API，定义+发音+例句 | 🟢 无 key | 未测 | 事实核查基础工具 |\n| [Dash Docset](https://mcpmarket.com/server/dash) · [Enhanced Dash](https://mcpmarket.com/server/enhanced-dash) | 本地 Dash docset 查询 | 🟢 本地 | 未测 | 技术文档即时查询 |\n| [Hugging Face MCP](https://mcpmarket.com/tools/skills/hugging-face-mcp) | Hub model/dataset 搜索 | 🟢 公开搜索无 key | **✅ 04-19** | AI 研究基础 |\n| Wikipedia API（官方） | 百科搜索 + 摘要 | 🟢 无 key | **✅ 04-19** | 事实核查/百科补全基础 |\n| Wikidata SPARQL（官方） | 结构化实体 + 图谱查询 | 🟢 无 key | **✅ 04-19** | 实体关系查询 |\n| Google Trends via `pytrends` | 关键词趋势时间序列 | 🟢 无 key | **✅ 04-19** | 话题热度/品牌对比信号 |\n\n### 🟡 按需接入\n\n| MCP | 能力 | 认证 | 场景 |\n|---|---|---|---|\n| [Flightradar24](https://mcpmarket.com/server/flightradar24-1) · [Flight (ADS-B)](https://mcpmarket.com/server/flight) | 实时航班追踪 | 🟢 部分无 key | 出行 / 供应链研究 |\n| [Etherscan MCP](https://mcpmarket.com/server/etherscan-2) · [Dune Analytics](https://mcpmarket.com/tools/skills/dune-analytics-on-chain-data) | 以太坊链上数据 | 🟢 Etherscan 免费 key / 🟡 Dune 付费 | Web3 研究 |\n| [Ethereum JSON-RPC](https://mcpmarket.com/server/eth-1) | 原生 JSON-RPC 查链 | 🟢 公开节点 | DeFi 深度研究 |\n| [TMDB](https://mcpmarket.com/server/tmdb) · [IMDb](https://mcpmarket.com/es/server/imdb-1) · [OMDB](https://mcpmarket.com/es/server/omdb) | 电影/电视数据 | 🟢 TMDB/OMDB 免费 key | 内容/娱乐研究 |\n| [Last.fm (ScrobblerContext)](https://mcpmarket.com/es/server/scrobblercontext) | 音乐数据 | 🟢 免费 key | 内容分析 |\n| [OpenFoodFacts](https://mcpmarket.com/server/openfoodfacts) | 全球食品数据库 | 🟢 无 key | 消费品/健康研究 |\n\n### 🟢 基础工具（偶尔用）\n\n| MCP | 能力 | 认证 |\n|---|---|---|\n| [Whois Lookup](https://mcpmarket.com/server/whois) · [Domain Lookup](https://mcpmarket.com/server/domain-lookup) | WHOIS / RDAP 查询 | 🟢 无 key |\n| [DeepL MCP](https://mcpmarket.com/server/deepl) | 翻译 | 🟡 免费 tier 需 key |\n| [Pexels MCP](https://mcpmarket.com/es/server/pexels) | 免费 stock 图片 | 🟡 免费 key |\n\n### mcpmarket 完全空白（这些品类以后别找 mcpmarket）\n\n- **今日头条 / 百家号 / 网易号**（中文新闻聚合）\n- **Reuters / TechCrunch / Bloomberg**（专业新闻，只有 RSS）\n- **Coursera / edX / Khan Academy / MOOC**\n- **中国政府开放数据**（统计局、海关）\n- **ESPN / 体育数据**\n- **空气质量 / 环境数据 / 实时碳排放**\n\n这些只能靠 RSS + Exa `site:` 覆盖，或者等 mcpmarket 以后上新。\n\n### 2026-04-19 实测验证（5 个核心 MCP）\n\n对\"基础事实 / 趋势 / AI Hub / 元工具\"这 5 个最关键 MCP 做了底层 API smoke test。结果全 PASS：\n\n| MCP | 底层 API | Smoke test | 样本 |\n|---|---|---|---|\n| **Wikipedia** | `en.wikipedia.org/w/api.php?action=query&list=search&srsearch=<q>` | 搜 `Claude AI` → 5 条 | Claude (language model) · Anthropic · OpenAI Codex · Artificial intelligence · Project Maven |\n| **Wikidata** | `query.wikidata.org/sparql` (SPARQL) | `P31 wd:Q11660` 查 AI 实体 → 5 条 | AlphaFold · Lee Luda · Intelligent Autonomous Systems |\n| **Google Trends** | `pytrends` Python 库（非官方，模拟请求） | 7 天每小时 `Claude AI` vs `ChatGPT` → **169 行** | 最新一小时 Claude AI=3, ChatGPT=45（品牌认知度差距直观） |\n| **Hugging Face Hub** | `huggingface.co/api/models?search=<q>` | 搜 `llama` → 5 条 | meta-llama/Llama-3.1-8B-Instruct ⬇936万 · Llama-3.3-70B-Instruct ⬇49万 |\n| **Public APIs Directory** | 本地 `datastore/index.json`（1426 API 全量） | 全量加载 + 筛 auth=No → **668 个免 key** | 1426 个 API 涵盖 Animals/Crypto/Finance/Transport/Security 等几十类 |\n\n**关键澄清**：Public APIs Directory MCP（[zazencodes/public-apis-mcp](https://github.com/zazencodes/public-apis-mcp)）**完全不依赖已挂掉的 `api.publicapis.org`**。它的数据是内置 JSON（1426 条），embedding 索引本地构建（.npz 文件）。所以即使那个老 API 域名已 DNS 解析失败，MCP 功能完好。\n\n**踩坑预警 — Google Trends**：pytrends 是非官方模拟请求，Google 偶尔会改 cookie/token 机制导致失效。接入时**必加重试 + 错误容忍**，不能作为硬依赖。比其他 4 个官方 API 脆弱得多。\n\n**接入建议（最轻方案）**：前 4 个都可以**直接 Python httpx 调 HTTP API**，不必启动 MCP 进程。Public APIs Directory 可以把它的 `index.json` 拷贝到 AutoSearch 里自己解析——1426 条 JSON 没什么分量，embedding 也可以本地现算。这样 AutoSearch 对外零新依赖进程。\n\n---\n\n### 扫描后更新 AutoSearch 优先级\n\n原 playbook 第一部分 C 节的\"强烈建议接入\"6 个基础上，**追加 3 个最高优先级**：\n\n1. **FRED + World Bank**（宏观经济信号，免 key 免 cookie 一步到位）\n2. **Public APIs Directory**（元工具，加了它 AutoSearch 自动发现新 channel）\n3. **Open Library**（学术/出版研究基础）\n\n合计现在 AutoSearch 应该新增 **9 个 channel** 就能达到\"业内唯一全覆盖研究系统\"：\n\n```\nWikipedia + Wikidata # 百科 ✅ 已验证\nGoogle Trends # 趋势 ✅ 已验证（有 flaky 风险）\nHugging Face Hub # AI 模型 ✅ 已验证\nPublic APIs Directory # 元工具 ✅ 已验证\nGoogle Patents # 专利 未验证\nCVE/NVD # 安全 未验证\nFRED + World Bank # 宏观经济 未验证\nOpen Library # 书籍 未验证\n```\n\n全部 🟢 零 key 零 cookie。改造量小，收益最大。\n\n**第一批接入（已实测可用的 4 个）**：Wikipedia、Wikidata、Hugging Face、Public APIs Directory — 这 4 个可以立刻让 Codex 写 channel adapter，每个 < 30 行 Python + httpx 代码。\n\n**第二批接入（未实测 5 个）**：Google Trends、Google Patents、CVE/NVD、FRED、Open Library — 按研究场景优先级推进，每个接入前先 smoke test 底层 API。\n"}, "aab0714fde4ff0306971a54023f85cc8d0517d13": {"t": 1785773307.8182688, "n": 17074, "h": "9bbdaff1ad1887a0", "c": "# AutoSearch 渠道调研最终手册\n\n> 时间：2026-04-19\n> 覆盖：mcpmarket.com 调研 + 实测 + TikHub 付费 API 打通\n> 定位：这份是**最终总纲**，之前分散文档（`mcp-channel-research.md` / `mcp-test-plan.md` / `mcp-test-results.md` / `mcp-no-cookie-inventory.md` / `mcp-final-summary.md` / `tikhub-smoke-test.md`）都汇总到这里。未来所有渠道决策从这份开始。\n\n---\n\n## TL;DR\n\n**三档结论**：\n\n1. **无 cookie 免费可用**：搜狗微信、open-websearch（4 引擎）、Paper Search MCP（21 学术源）、PullPush（Reddit 历史）、RSS 直读、Jina Reader（部分站）、以及 13 个官方 API。\n2. **硬骨头 8 个中付费打通 5 个**（TikHub）：小红书、微博、知乎、Twitter/X、抖音。剩 3 个（微博上游 flaky、Instagram/LinkedIn 参数未调通）本轮未完全解决。\n3. **最终架构**：AutoSearch 走 BYOK（用户自配 TikHub key），`requires: [env:TIKHUB_API_KEY]` 声明依赖，未来可无缝切换到 hosted proxy 做商业化。\n\n**成本锚点**：TikHub 实测平均 **$0.0036/请求**。1000 次研究级调用 ≈ $3.6/月。\n\n---\n\n## 第一部分：无 cookie 可用清单（41 个）\n\n### A. 本次亲手实测 PASS（9 个）\n\n| 方案 | 平台/能力 | 认证 | 备注 |\n|---|---|---|---|\n| **搜狗微信 SERP** | 微信公众号 | 🟢 无 | `weixin.sogou.com/weixin?type=2&query=<q>` |\n| **open-websearch fetch-web + readability** | 任意页 → 干净 markdown | 🟢 无 | Mozilla Readability 算法清洗 |\n| **open-websearch csdn** | CSDN 技术博客 | 🟢 无 | 原生中文 |\n| **open-websearch duckduckgo** | DuckDuckGo | 🟢 无 | 备用通用搜 |\n| **open-websearch startpage** | Startpage（Google 代理） | 🟢 无 | 备用通用搜 |\n| **Paper Search MCP** | arXiv + PubMed + Semantic Scholar + OpenAlex + 其他 17 源 | 🟢 无 | 一键 21 学术源，MIT |\n| **PullPush** | Reddit 历史数据 2005-至今 | 🟢 无 | `q=` 全文搜挂，按 subreddit/时间过滤正常 |\n| **Substack RSS** | Substack / Medium / 独立博客 | 🟢 无 | 所有站暴露 `/feed` |\n| **Jina Reader** | B站 / 反爬弱的站 | 🟢 无 | 知乎小红书微博会被封 |\n\n### B. 官方 API 已知可用（13 个，AutoSearch 已集成或一行接入）\n\n| 平台 | 接入方式 | 限额 |\n|---|---|---|\n| HackerNews Algolia | 官方 API | 无限 |\n| Reddit 公开 JSON | `/r/<sub>.json` | 无登录 |\n| Stack Exchange API | 官方 | 300/day 无 key |\n| GitHub Public Search | 官方 API | 10/min 无 token |\n| Dev.to | 公开 API | 无 key |\n| npm / PyPI registry | 官方 | 无 key |\n| OpenReview API | 官方 | 无 key |\n| arXiv API | 官方 | 无 key |\n| 雪球 / 36kr / InfoQ 中文 | 各自 web/API | 无 key |\n| 小宇宙 | RSS / 公开页 | 无 key |\n| B站公开搜索 API | 官方 | 无登录 |\n| YouTube transcript | yt-dlp + whisper（本地） | 无 key |\n| Crunchbase 公开页 | 公开搜索 | 有限 |\n\n### C. mcpmarket 发现的新品类（AutoSearch 空白）（11 个）\n\n| 方案 | 能力 | 优先级 |\n|---|---|---|\n| [Wikipedia MCP](https://mcpmarket.com/server/wikipedia-1) | 百科全站 + 摘要 | 🔴 最高 |\n| [Wikidata](https://mcpmarket.com/server/wikidata) | SPARQL + 实体 | 🔴 最高 |\n| [Google Trends Explorer](https://mcpmarket.com/server/google-trends-explorer) | 趋势信号 | 🔴 最高 |\n| [Google News Trends](https://mcpmarket.com/es/server/google-news-trends) | 新闻 + 趋势词 | 🔴 高 |\n| [USPTO](https://mcpmarket.com/server/uspto) | 美国专利 + 商标 | 🟡 按需 |\n| [Google Patents](https://mcpmarket.com/server/google-patents) | 全球专利 | 🟡 按需 |\n| [EPSS](https://mcpmarket.com/server/epss) | CVE + 漏洞评分 | 🟡 按需 |\n| [Cybersecurity CVE](https://mcpmarket.com/es/server/cybersecurity-2) | NVD 数据库 | 🟡 按需 |\n| [OpenStreetMap](https://mcpmarket.com/server/openstreetmap) | 地图 + POI | 🟢 按需 |\n| [CoinGecko](https://mcpmarket.com/server/coingecko) | 加密货币 | 🟢 按需 |\n| [Open Meteo](https://mcpmarket.com/tools/skills/open-meteo-weather) | 全球天气 | 🟢 按需 |\n\n### D. 现有 channel 补强（8 个）\n\n| 方案 | 补哪个 | 备注 |\n|---|---|---|\n| [Package Version](https://mcpmarket.com/ja/server/package-version) | npm/PyPI/Maven/Go/Swift/Docker Hub 等 9 注册表 | 一个 MCP 顶 9 个 |\n| [GitLab MCP](https://mcpmarket.com/server/gitlab) | 补 GitHub 之外 | 公开 repo 无 token |\n| [Newsfeed](https://mcpmarket.com/server/newsfeed) | `search-rss` 预置分类 | 免费 |\n| [Trend Radar](https://mcpmarket.com/tools/skills/trend-radar) | HN + GitHub 信号聚合 | 免费 |\n| [Steam Context](https://mcpmarket.com/server/steam-context) | 游戏品类 | Steam API key 免费 |\n| [Zillow](https://mcpmarket.com/server/zillow) | 房产品类 | 大部分无 key |\n| [AI Job Hunting Agent](https://mcpmarket.com/es/server/ai-job-hunting-agent) | Indeed + Remotive 岗位 | 免费 |\n| 搜狗微信 MCP 包装（[ptbsare](https://github.com/ptbsare/sogou-weixin-mcp-server) / [fancyboi999](https://github.com/fancyboi999/weixin_search_mcp)） | 公众号 | 底层同搜狗 SERP |\n\n---\n\n## 第二部分：硬骨头攻克矩阵\n\n### 🔴 已打通（TikHub 付费，5/8）\n\n| 平台 | TikHub endpoint | 实测数据 |\n|---|---|---|\n| 小红书 | `GET /api/v1/xiaohongshu/web/search_notes?keyword=<q>` | Claude 4.7 搜到 20 条笔记，含 title/author/likes |\n| 知乎 | `GET /api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>` | LLM agent 搜到 20 条高质量问答 |\n| 抖音 | `GET /api/v1/douyin/web/fetch_video_search_result_v2?keyword=<q>` | DeepSeek 搜到 12 条，最高 54 万赞 |\n| Twitter/X | `GET /api/v1/twitter/web/fetch_search_timeline?keyword=<q>&search_type=Top` | 19 条，全字段（favorites/views/retweets） |\n| B站 | `GET /api/v1/bilibili/web/fetch_general_search?keyword=<q>&order=totalrank&page=1&page_size=10` | 8 条视频 + 播放量 |\n\n### ⚠️ 本轮未完全解决（3/8）\n\n| 平台 | 状态 | 排查方向 |\n|---|---|---|\n| **微博** | TikHub 上游 flaky——`web/fetch_search` 返回 ok=1 但 cards 时空时不空，重试波动；`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422 | 去 TikHub Discord 问客服；尝试 `web_v2/fetch_ai_smart_search` |\n| **Instagram** | `v2/general_search` 返回 200 OK 但全 0 | 换 `v2/search_hashtags` 或针对性关键词（#hashtag / @username） |\n| **LinkedIn** | `web/search_jobs` 400，加 geocode 也 400 | 去 TikHub docs 查 demo 参数，可能需要特殊 geocode 格式 |\n\n### 🟢 真·完全无法无 cookie 搞定\n\n**Facebook / Instagram 公开个人页 / 小红书私人号内容 / Twitter 私人账号**——即使 TikHub 也要账号支持。这些不在研究场景核心，暂不投入。\n\n---\n\n## 第三部分：TikHub 实战经验\n\n### 基础信息\n\n- **官网**：https://tikhub.io · **API base**：`https://api.tikhub.io/api/v1/`\n- **认证**：`Authorization: Bearer $TIKHUB_API_KEY`\n- **OpenAPI 规格**：`GET https://api.tikhub.io/openapi.json`（1058 个 endpoint）\n- **计费**：pay-per-request，平均 **$0.0036/次**（官方宣传 $0.001 起步，实测贵 3-4 倍）\n- **免费额度**：每日签到可领少量 free credit\n\n### 覆盖平台（确认跑通）\n\n16 平台共 1000+ tools：\n```\nTikTok (204) · Douyin (247) · Instagram (82) · Xiaohongshu (71)\nWeibo (64) · Bilibili (41) · YouTube (37) · Kuaishou (33)\nZhihu (32) · LinkedIn (25) · Reddit (24) · WeChat · Twitter\nThreads · TikHub Utilities · 其他\n```\n\n### 关键坑（按踩坑顺序）\n\n1. **注册要验证邮箱**。不验证一切 endpoint 都 403 \"邮箱未验证\"。\n2. **免费 tier 只覆盖部分 endpoint**。小红书/微博/Twitter/抖音都是 402（Payment Required）直到付费余额 > 0。知乎 / B站属于免费 tier 覆盖。\n3. **小红书别用 `web_v3`**。`/xiaohongshu/web_v3/fetch_search_notes` 返回 400。用老版 `/xiaohongshu/web/search_notes`。\n4. **抖音别用 `app/v3`**。`/douyin/app/v3/fetch_video_search_result` 返回 400。用 `/douyin/web/fetch_video_search_result_v2`，数据在 `data.business_data[i].data`（嵌套两层）。\n5. **微博 endpoint 多版本都不稳**。`web_v2/fetch_realtime_search` 400；`app/fetch_search_all` 422；`web/fetch_search` 200 但 cards 波动。\n6. **400 = 上游爬取失败，不是参数错**。TikHub 的 400 body 是\"请查看文档核对参数\"，但实际上是 TikHub 自己爬目标站失败。这种 400 **不计费**（Only pay for successful requests）。\n7. **Twitter 结构已扁平化**。`data.timeline` 是 list（不是 GraphQL 原格式的 instructions 嵌套），直接遍历即可。字段：`screen_name / text / favorites / views / retweets / replies / created_at / tweet_id / lang`。\n\n### 🚨 安全注意：Key 会在错误响应里被回显\n\n**TikHub 的 400 / 403 / 422 响应体会原样回显完整 `Authorization` header（含 Bearer token）**。接入必须：\n\n```python\n# BAD — 直接打印 response body 会泄露 key\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=e.response.text) # ❌\n\n# GOOD — 先脱敏再 log\ndef sanitize(data: dict) -> dict:\n detail = data.get(\"detail\", {})\n if isinstance(detail, dict):\n detail.pop(\"headers\", None)\n for k in list(detail.keys()):\n if \"auth\" in k.lower() or \"token\" in k.lower():\n detail[k] = \"<redacted>\"\n return data\n\nexcept httpx.HTTPStatusError as e:\n log.error(\"TikHub failed\", body=sanitize(e.response.json())) # ✅\n```\n\n测试里必须有一条：**assert exception string 不包含 \"Bearer\"**。\n\n### MCP vs API 选择\n\nTikHub 官方提供 4 种 transport：Stdio / SSE / Streamable HTTP / Curl(API)。\n\n**对 AutoSearch 而言**：**选 Curl/API 直连**，不要走 MCP。\n\n理由：\n- AutoSearch 是 Python plugin，用户 `pip install` 安装。走 MCP 要求用户额外装 Node.js + `mcp-remote` + TikHub MCP server（3 个依赖）\n- AutoSearch 的 channel 是\"工具\"本身，再走一层 MCP 是工具里调工具，JSON-RPC over stdio 多一层序列化毫无意义\n- TikHub MCP 全勾是 1000+ tool schema，注入 Claude context ~400k token，直接炸\n- 即使只勾 6 个硬骨头平台也 ~470 tools（~190k token），依然不理想\n\n**Stdio MCP 适用场景**：用户在 Claude Code 会话里想直接 \"搜一下小红书\" 的即兴调用。这是用户自己装 TikHub MCP 的事，和 AutoSearch 无关。\n\n---\n\n## 第四部分：AutoSearch 最终接入架构\n\n### BYOK（Bring Your Own Key）是唯一合理方案\n\n**不要做的**：\n- ❌ 把 TikHub key 硬编码到发布版里（pip 包明文可提取）\n- ❌ 一个 key 共享给所有用户（违反 ToS + 余额被烧光）\n\n**要做的**：\n- ✅ 用户自己去 `tikhub.io` 注册 + 充值\n- ✅ `export TIKHUB_API_KEY=xxx`，AutoSearch 读 env var\n- ✅ SKILL.md 声明 `requires: [env:TIKHUB_API_KEY]`，没 key 时 channel 自动标记不可用，其他 channel 照跑\n\n### Channel 结构（已通过 pilot 验证）\n\n```\nautosearch/lib/\n└── tikhub_client.py # 通用客户端 + 错误脱敏 + 5xx 重试\n\nskills/channels/zhihu/\n├── SKILL.md # frontmatter 声明 via_tikhub method\n└── methods/\n └── via_tikhub.py # 调 /zhihu/web/fetch_article_search_v3 映射到 Evidence\n\ntests/lib/test_tikhub_client.py # 客户端单测（含脱敏断言）\ntests/channels/zhihu/test_via_tikhub.py # method 单测\n```\n\n**fallback_chain 顺序**：`[via_tikhub, api_search, api_answer_detail]` — TikHub 有 key 时优先用，没 key 时 fallback 到原方案。\n\n### 为未来 proxy 留口子\n\n`tikhub_client.py` 的 base URL 从 env var 读：\n\n```python\nbase_url = os.getenv(\"TIKHUB_BASE_URL\", \"https://api.tikhub.io\")\n```\n\n未来老板上 hosted proxy 做商业化时，用户只需改两个 env var，零代码改动：\n\n```bash\nexport TIKHUB_BASE_URL=https://api.autosearch.com/tikhub\nexport TIKHUB_API_KEY=<AutoSearch 平台发的 token>\n```\n\n### 待接入的 5 个 channel（按优先级）\n\n| 优先级 | Channel | Endpoint | 状态 |\n|---|---|---|---|\n| 1 | **zhihu** | `zhihu/web/fetch_article_search_v3` | ✅ pilot 写完（branch `feat/tikhub-channels`） |\n| 2 | **xiaohongshu** | `xiaohongshu/web/search_notes` | 待扩展 |\n| 3 | **twitter** | `twitter/web/fetch_search_timeline` | 待扩展 |\n| 4 | **douyin** | `douyin/web/fetch_video_search_result_v2` | 待扩展 |\n| 5 | **bilibili** | `bilibili/web/fetch_general_search` | 待扩展 |\n\nPilot 验证通过后，让 Codex 按同一模板并行写剩下 4 个 adapter。改造量每个 < 50 行。\n\n---\n\n## 附录\n\n### A. 测试命令速查（可复制即跑）\n\n```bash\n# 搜狗微信 + readability 清洗 = 公众号全文搜索\ncd /tmp/mcp-smoke/ows\nnode build/index.js fetch-web \\\n \"https://weixin.sogou.com/weixin?type=2&query=<KEYWORD>\" \\\n --readability --spawn --json\n\n# open-websearch 多引擎\nnode build/index.js search \"<query>\" --engine duckduckgo --limit 5 --spawn --json\nnode build/index.js search \"<query>\" --engine csdn --limit 5 --spawn --json\n\n# Paper Search MCP 学术\ncd /tmp/mcp-smoke/paper-search-mcp\npython3 -c \"\nimport sys; sys.path.insert(0,'.')\nfrom paper_search_mcp.academic_platforms.arxiv import ArxivSearcher\ns = ArxivSearcher()\nfor p in s.search('<query>', max_results=5):\n print(p.title, p.url)\n\"\n\n# PullPush Reddit 历史\ncurl -s \"https://api.pullpush.io/reddit/search/submission/?subreddit=<NAME>&after=<TS>&before=<TS>&size=10\"\n\n# Jina Reader 任意 URL\ncurl -sL \"https://r.jina.ai/<URL>\"\n\n# TikHub 任意 endpoint（脱敏地看结果）\ncurl -sS -H \"Authorization: Bearer $TIKHUB_API_KEY\" \\\n \"https://api.tikhub.io/api/v1/zhihu/web/fetch_article_search_v3?keyword=<q>\" \\\n -o /tmp/t.json\npython3 -c \"\nimport json\nd = json.load(open('/tmp/t.json'))\ndetail = d.get('detail', {})\nif isinstance(detail, dict): detail.pop('headers', None)\nprint(json.dumps(d, indent=2, ensure_ascii=False)[:1000])\n\"\nrm -f /tmp/t.json # 关键：清理可能含 key 回显的临时文件\n```\n\n### B. 外部链接\n\n- mcpmarket.com — 本次调研来源\n- tikhub.io — 付费平台 API\n- github.com/Aas-ee/open-webSearch — 8 引擎免费 SERP\n- github.com/openags/paper-search-mcp — 21 学术源\n- github.com/jacklenzotti/pullpush-mcp — Reddit 历史\n- r.jina.ai — Jina Reader 免费 URL → markdown\n\n### C. 相关文档（本次之前）\n\n已被本手册汇总、**可以不用再翻**：\n- `mcp-channel-research.md` — 初期调研\n- `mcp-channel-test-plan.md` — 测试计划\n- `mcp-test-results.md` — 第一轮实测\n- `mcp-no-cookie-inventory.md` — 无 cookie 清单\n- `mcp-final-summary.md` — 阶段总结\n- `tikhub-smoke-test.md` — TikHub 详细实测\n\n保留归档供溯源，但日常看**本手册 + `tikhub-smoke-test.md`** 即可。\n\n### D. 开销总账（截至 2026-04-19）\n\n- TikHub：**$0.053**（19 次 smoke request）\n- 其他：$0（全部免费方案）\n\n---\n\n**维护规则**：接入一个新 channel 或打通一个新硬骨头 → 更新本手册对应表格，保持单一事实来源。\n\n---\n\n## 附 E. 2026-04-19 补充扫描（免 key + 免 cookie 新增）\n\n本轮深挖 mcpmarket.com 发现了之前漏掉的 **12 个完全免 key 免 cookie 的品类**。按研究价值分档：\n\n### 🔴 高价值（强烈建议接入）\n\n| MCP | 能力 | 认证 | 实测 | 为什么值得 |\n|---|---|---|---|---|\n| [FRED](https://mcpmarket.com/server/fred) · [Fredapi](https://mcpmarket.com/server/fredapi) · [FRED Macro Data](https://mcpmarket.com/server/fred-macroeconomic-data) | 美联储经济数据 80w+ 时间序列 | 🟢 无 key | 未测 | AI/创业研究的宏观信号必备 |\n| [World Bank Data](https://mcpmarket.com/server/world-bank-data) | 全球经济指标 | 🟢 无 key | 未测 | 国别对比研究刚需 |\n| [Public APIs Directory](https://mcpmarket.com/es/server/public-apis) ([repo](https://github.com/zazencodes/public-apis-mcp)) | 免费 API 目录 semantic search | 🟢 无 key | **✅ 04-19** | **元工具** — AutoSearch 未来发现新 channel 的引擎 |\n| [Open Library / Books](https://mcpmarket.com/server/books) · [OpenLibrary](https://mcpmarket.com/server/openlibrary) | ISBN 书籍查询，全量书目 | 🟢 无 auth | 未测 | 学术/出版研究基础 |\n| [Word of the Day](https://mcpmarket.com/server/word-of-the-day) | Free Dictionary API，定义+发音+例句 | 🟢 无 key | 未测 | 事实核查基础工具 |\n| [Dash Docset](https://mcpmarket.com/server/dash) · [Enhanced Dash](https://mcpmarket.com/server/enhanced-dash) | 本地 Dash docset 查询 | 🟢 本地 | 未测 | 技术文档即时查询 |\n| [Hugging Face MCP](https://mcpmarket.com/tools/skills/hugging-face-mcp) | Hub model/dataset 搜索 | 🟢 公开搜索无 key | **✅ 04-19** | AI 研究基础 |\n| Wikipedia API（官方） | 百科搜索 + 摘要 | 🟢 无 key | **✅ 04-19** | 事实核查/百科补全基础 |\n| Wikidata SPARQL（官方） | 结构化实体 + 图谱查询 | 🟢 无 key | **✅ 04-19** | 实体关系查询 |\n| Google Trends via `pytrends` | 关键词趋势时间序列 | 🟢 无 key | **✅ 04-19** | 话题热度/品牌对比信号 |\n\n### 🟡 按需接入\n\n| MCP | 能力 | 认证 | 场景 |\n|---|---|---|---|\n| [Flightradar24](https://mcpmarket.com/server/flightradar24-1) · [Flight (ADS-B)](https://mcpmarket.com/server/flight) | 实时航班追踪 | 🟢 部分无 key | 出行 / 供应链研究 |\n| [Etherscan MCP](https://mcpmarket.com/server/etherscan-2) · [Dune Analytics](https://mcpmarket.com/tools/skills/dune-analytics-on-chain-data) | 以太坊链上数据 | 🟢 Etherscan 免费 key / 🟡 Dune 付费 | Web3 研究 |\n| [Ethereum JSON-RPC](https://mcpmarket.com/server/eth-1) | 原生 JSON-RPC 查链 | 🟢 公开节点 | DeFi 深度研究 |\n| [TMDB](https://mcpmarket.com/server/tmdb) · [IMDb](https://mcpmarket.com/es/server/imdb-1) · [OMDB](https://mcpmarket.com/es/server/omdb) | 电影/电视数据 | 🟢 TMDB/OMDB 免费 key | 内容/娱乐研究 |\n| [Last.fm (ScrobblerContext)](https://mcpmarket.com/es/server/scrobblercontext) | 音乐数据 | 🟢 免费 key | 内容分析 |\n| [OpenFoodFacts](https://mcpmarket.com/server/openfoodfacts) | 全球食品数据库 | 🟢 无 key | 消费品/健康研究 |\n\n### 🟢 基础工具（偶尔用）\n\n| MCP | 能力 | 认证 |\n|---|---|---|\n| [Whois Lookup](https://mcpmarket.com/server/whois) · [Domain Lookup](https://mcpmarket.com/server/domain-lookup) | WHOIS / RDAP 查询 | 🟢 无 key |\n| [DeepL MCP](https://mcpmarket.com/server/deepl) | 翻译 | 🟡 免费 tier 需 key |\n| [Pexels MCP](https://mcpmarket.com/es/server/pexels) | 免费 stock 图片 | 🟡 免费 key |\n\n### mcpmarket 完全空白（这些品类以后别找 mcpmarket）\n\n- **今日头条 / 百家号 / 网易号**（中文新闻聚合）\n- **Reuters / TechCrunch / Bloomberg**（专业新闻，只有 RSS）\n- **Coursera / edX / Khan Academy / MOOC**\n- **中国政府开放数据**（统计局、海关）\n- **ESPN / 体育数据**\n- **空气质量 / 环境数据 / 实时碳排放**\n\n这些只能靠 RSS + Exa `site:` 覆盖，或者等 mcpmarket 以后上新。\n\n### 2026-04-19 实测验证（5 个核心 MCP）\n\n对\"基础事实 / 趋势 / AI Hub / 元工具\"这 5 个最关键 MCP 做了底层 API smoke test。结果全 PASS：\n\n| MCP | 底层 API | Smoke test | 样本 |\n|---|---|---|---|\n| **Wikipedia** | `en.wikipedia.org/w/api.php?action=query&list=search&srsearch=<q>` | 搜 `Claude AI` → 5 条 | Claude (language model) · Anthropic · OpenAI Codex · Artificial intelligence · Project Maven |\n| **Wikidata** | `query.wikidata.org/sparql` (SPARQL) | `P31 wd:Q11660` 查 AI 实体 → 5 条 | AlphaFold · Lee Luda · Intelligent Autonomous Systems |\n| **Google Trends** | `pytrends` Python 库（非官方，模拟请求） | 7 天每小时 `Claude AI` vs `ChatGPT` → **169 行** | 最新一小时 Claude AI=3, ChatGPT=45（品牌认知度差距直观） |\n| **Hugging Face Hub** | `huggingface.co/api/models?search=<q>` | 搜 `llama` → 5 条 | meta-llama/Llama-3.1-8B-Instruct ⬇936万 · Llama-3.3-70B-Instruct ⬇49万 |\n| **Public APIs Directory** | 本地 `datastore/index.json`（1426 API 全量） | 全量加载 + 筛 auth=No → **668 个免 key** | 1426 个 API 涵盖 Animals/Crypto/Finance/Transport/Security 等几十类 |\n\n**关键澄清**：Public APIs Directory MCP（[zazencodes/public-apis-mcp](https://github.com/zazencodes/public-apis-mcp)）**完全不依赖已挂掉的 `api.publicapis.org`**。它的数据是内置 JSON（1426 条），embedding 索引本地构建（.npz 文件）。所以即使那个老 API 域名已 DNS 解析失败，MCP 功能完好。\n\n**踩坑预警 — Google Trends**：pytrends 是非官方模拟请求，Google 偶尔会改 cookie/token 机制导致失效。接入时**必加重试 + 错误容忍**，不能作为硬依赖。比其他 4 个官方 API 脆弱得多。\n\n**接入建议（最轻方案）**：前 4 个都可以**直接 Python httpx 调 HTTP API**，不必启动 MCP 进程。Public APIs Directory 可以把它的 `index.json` 拷贝到 AutoSearch 里自己解析——1426 条 JSON 没什么分量，embedding 也可以本地现算。这样 AutoSearch 对外零新依赖进程。\n\n---\n\n### 扫描后更新 AutoSearch 优先级\n\n原 playbook 第一部分 C 节的\"强烈建议接入\"6 个基础上，**追加 3 个最高优先级**：\n\n1. **FRED + World Bank**（宏观经济信号，免 key 免 cookie 一步到位）\n2. **Public APIs Directory**（元工具，加了它 AutoSearch 自动发现新 channel）\n3. **Open Library**（学术/出版研究基础）\n\n合计现在 AutoSearch 应该新增 **9 个 channel** 就能达到\"业内唯一全覆盖研究系统\"：\n\n```\nWikipedia + Wikidata # 百科 ✅ 已验证\nGoogle Trends # 趋势 ✅ 已验证（有 flaky 风险）\nHugging Face Hub # AI 模型 ✅ 已验证\nPublic APIs Directory # 元工具 ✅ 已验证\nGoogle Patents # 专利 未验证\nCVE/NVD # 安全 未验证\nFRED + World Bank # 宏观经济 未验证\nOpen Library # 书籍 未验证\n```\n\n全部 🟢 零 key 零 cookie。改造量小，收益最大。\n\n**第一批接入（已实测可用的 4 个）**：Wikipedia、Wikidata、Hugging Face、Public APIs Directory — 这 4 个可以立刻让 Codex 写 channel adapter，每个 < 30 行 Python + httpx 代码。\n\n**第二批接入（未实测 5 个）**：Google Trends、Google Patents、CVE/NVD、FRED、Open Library — 按研究场景优先级推进，每个接入前先 smoke test 底层 API。\n"}}}} |