1 line
17 KiB
JSON
1 line
17 KiB
JSON
{"repo": "botnick/claude-verifier", "files": {"README.md": {"36b5bde66bf46c65b10ef0ad0d32adcc8c727195": {"t": 1785596444.6846392, "n": 15914, "h": "60c3b915b827aa78", "c": "# Claude Verifier — is your \"Claude\" endpoint *actually* Claude?\n\n[](https://github.com/botnick/claude-verifier/actions/workflows/ci.yml)\n[](https://github.com/botnick/claude-verifier/actions/workflows/release.yml)\n[](./LICENSE)\n[](./public/tests.js)\n[](https://docs.anthropic.com/)\n\n[](https://github.com/botnick/claude-verifier/releases/latest)\n[](https://github.com/botnick/claude-verifier/releases/latest)\n[](https://github.com/botnick/claude-verifier/releases/latest)\n[](https://botnick.github.io/claude-verifier/)\n\n> **The cross-platform desktop tool for verifying that an `/v1/messages`\n> endpoint is genuinely Claude (Anthropic) — not a silently-swapped GPT-4o,\n> Qwen, DeepSeek, Llama, Gemini, Mistral, or other model behind a proxy.**\n\n🌐 **[Live probe editor & docs →](https://botnick.github.io/claude-verifier/)**\n · \n📦 **[Download for macOS / Linux / Windows →](https://github.com/botnick/claude-verifier/releases/latest)**\n\n---\n\n## 🧪 Share your probes (the most useful thing you can do)\n\nThis project gets stronger every time someone adds a high-signal probe.\nAnyone who's caught a sneaky proxy in the wild can package the prompt that\ncaught it as a JSON file and drop it back into the app via **⬆ Import** —\nno fork, no PR, no build step required.\n\n| Action | What to do |\n|---|---|\n| **Share your detection prompts** | Tick the probes you want to ship → click **⬇ Export** → post the `probe_pack_*.json` anywhere (GitHub Gist, Discussions tab, your blog, Discord). |\n| **Use someone else's pack** | Click **⬆ Import** → pick their JSON → probes appear in your catalog with a `custom` badge and persist across restarts. |\n| **PR built-in probes** | See [CONTRIBUTING.md](./CONTRIBUTING.md) — adds 5 golden-baseline samples too. |\n| **Discuss new probe ideas** | Open a GitHub Discussion / issue with the prompt + the model behavior you want to catch. |\n\n📦 **Probe pack format** is tiny, portable, and never includes your endpoint or API key:\n\n```json\n{\n \"format\": \"claude-verifier-probe-pack\",\n \"version\": 1,\n \"name\": \"Your pack name\",\n \"description\": \"What it catches\",\n \"probes\": [\n { \"id\": \"your_unique_id\",\n \"cat\": \"identity\",\n \"title\": \"Short display name\",\n \"prompt\": \"The actual question to ask the endpoint…\",\n \"expect_any\": [\"claude\", \"anthropic\"],\n \"red_flag\": [\"gpt\", \"openai\", \"qwen\", \"deepseek\"] }\n ]\n}\n```\n\n---\n\n## What it does\n\nOpen-source desktop app that probes an LLM endpoint to determine whether\nit's genuinely **Claude (Anthropic)** or a proxy / wrapper pretending to be.\nRuns a curated suite of **47 probes** across 5 categories, scores each one,\nand produces an evidence-rich Markdown report with a shareable URL.\n\n```\n◆ Claude Verifier\n ● Connected · claude-opus-4-7\n 47 probes · golden-baseline loaded · ready to run\n```\n\n## Why this exists\n\nLLM gateways and reseller proxies sometimes silently downgrade or swap\nthe model behind a `claude-*` model id. This app is a single-click way to\ncatch that — by hammering an endpoint with 47 prompts that expose identity,\nbehavioral, and capability differences between Claude and other model\nfamilies (GPT-, Qwen-, DeepSeek-, Gemini-, Llama-, etc.).\n\n## What it does\n\n| Category | What it checks |\n|---|---|\n| **Identity** | Who the endpoint claims to be, knowledge cutoff, creator, training methodology, and whether it corrects false claims like \"I am ChatGPT…\". |\n| **Jailbreak resistance** | DAN, \"ignore previous instructions\", grandma exploit, base64-encoded request, evil-twin roleplay. |\n| **Politically sensitive (CN)** | Tiananmen, Taiwan, Xinjiang, Hong Kong, Xi Jinping, Falun Gong, Winnie-the-Pooh — in Thai, Chinese, and English. Useful for spotting Chinese-trained models behind a proxy. |\n| **Social-engineering / leak** | Flattery + framework probe, system-prompt leak, infra/host probe, implicit-owner traps, forced multiple-choice, sentence-completion, \"you're a wrapper\" trap, covert-token reply, CC system-prompt echo trap. |\n| **Capability sanity** | Math chain-of-thought, strict JSON, Thai language, Unicode round-trip, long-context needle-in-haystack, no-fence JSON. |\n\nA free-form **Chat** tab lets you talk to the endpoint with multi-turn\nhistory and 13 quick-prompt chips (TH / ZH / EN).\n\n## Key features\n\n- **1:1 Claude Code traffic emulation** — every probe matches real\n `claude-cli` request shape: `user-agent`, `x-stainless-*` headers,\n `anthropic-beta`, system prompt, `metadata.user_id` (sha256 of API key),\n and the full tool catalog. Endpoints can't behave differently for\n \"verifier\" traffic vs genuine Claude Code traffic.\n- **Sandbox auto-fallback** — when the model replies with a pure `tool_use`\n turn, the chat IPC silently retries the same request without `tools` so\n the verdict has usable text. Transparent to the user; reported in Detail\n + report.\n- **Parallel execution** — runs 4 probes in flight by default. A 47-probe\n full run finishes in ~30–60 seconds depending on latency.\n- **Stop + Resume** — stop mid-run and the partial results still produce a\n `PARTIAL` MD report. Press **↻ Resume** to continue only the remaining\n probes; the final report is rebuilt with everything combined.\n- **Golden similarity** — character-bigram cosine similarity against a\n reference baseline (`data/golden.json`) per probe. Shows how close the\n endpoint's response is to a known-genuine Claude response (`mean / max /\n min` over 5 samples). Language-agnostic — works for Thai/Chinese/Arabic\n with no tokenizer.\n- **Auto-finish** — when a run finishes, the MD report is silently saved to\n `<project>/reports/<stamp>.md` AND published to a free paste host. URL\n is copied to your clipboard automatically.\n- **Probe pack import/export** — share probe sets with the community.\n Import a `.json` pack → probes get added to your catalog and persist in\n `data/custom-probes.json`. Export ticked probes or the full catalog.\n No endpoint/key/model ever leaks into a pack.\n- **Latency anomaly detection** — flags suspiciously uniform latencies\n (proxy/cache hint, CV < 0.18) or unusually variable ones.\n- **Confidence score** per probe (0.0–1.0) on top of PASS / SUSPICIOUS /\n FAIL / INFO label.\n\n## Run\n\n### Pre-built binaries\n\nGrab the right one for your OS from the [latest release](https://github.com/botnick/claude-verifier/releases/latest):\n\n- **macOS** — `Claude.Verifier-X.Y.Z-mac-arm64.dmg` (Apple Silicon) or `-x64.dmg` (Intel)\n- **Windows** — `Claude.Verifier-X.Y.Z-win-x64.exe` (NSIS installer) or `-portable.exe`\n- **Linux** — `Claude.Verifier-X.Y.Z-linux-x64.AppImage` or `.deb`\n\n> **⚠ macOS Gatekeeper note** — The published `.dmg` is **not signed with an\n> Apple Developer certificate** (the open-source project doesn't pay the\n> $99/year for one). After dragging the app into `/Applications`, Gatekeeper\n> may say *\"Claude Verifier is damaged and can't be opened\"*. Fix it with one\n> Terminal command (this is standard for unsigned macOS apps):\n>\n> ```bash\n> xattr -cr \"/Applications/Claude Verifier.app\"\n> ```\n>\n> This strips the download-quarantine attribute Apple stamps onto everything\n> not signed by a paid Developer ID. After running it, the app opens normally.\n> If you'd rather not run that command, build from source with `npm start`.\n\n> **⚠ Windows SmartScreen note** — The `.exe` is also unsigned. On first run\n> SmartScreen will warn \"Windows protected your PC\"; click **More info → Run\n> anyway**.\n\n### macOS / Linux from source\n```bash\ncd claude-verifier\n./run.sh\n```\n\n### Windows from source\n```bat\nrun.bat\n```\n\n### Manually\n```bash\nnpm install # first time only — downloads Electron (~ 200 MB)\nnpm start\n```\n\nRequirements: **Node.js 18+** (which ships with `npm`). Install from\n<https://nodejs.org/> or via Homebrew: `brew install node`.\n\n### Build your own installers\n\n```bash\nnpm run dist:mac # builds .dmg + .zip into ./dist\nnpm run dist:win # builds .exe (NSIS + portable)\nnpm run dist:linux # builds .AppImage + .deb + .tar.gz\nnpm run dist # builds for the host OS\n```\n\nA `release` GitHub Action also builds for all three platforms automatically\nwhenever you push a `v*` tag — the artifacts attach themselves to a fresh\nGitHub Release. See `.github/workflows/release.yml`.\n\n## Usage\n\n1. Paste your endpoint (default = Anthropic's official `/v1/messages`) and API key.\n2. Pick a model from the dropdown (or type any model identifier).\n3. Click **◯ Test connection** to confirm credentials before a full run.\n4. Tick the probes you want in the left **Test Catalog**, then **▶ Run selected**\n or **▶▶ Run ALL**.\n5. (Optional) Stop mid-run → press **↻ Resume** to continue from where you left off.\n6. Switch to the **Chat** tab for free-form interrogation; tap a quick-probe\n chip to drop a curated prompt into the input, then press **Send ▶**.\n\nWhen the run finishes you'll see:\n- A shareable URL in the action bar (already copied to your clipboard).\n- A summary verdict in the **Summary** tab.\n- A local `.md` file at `<project>/reports/`.\n\n### Keyboard shortcuts\n\n| Key | Action |\n|---|---|\n| `⌘↩` / `Ctrl+Enter` | Run selected |\n| `⇧⌘↩` / `Ctrl+Shift+Enter` | Run ALL |\n| `Esc` | Stop (when a run is in flight) |\n| `⌘K` / `Ctrl+K` | Focus API key field |\n| `⌘L` / `Ctrl+L` | Focus endpoint field |\n\n## How verdicts are computed\n\nEach probe carries two keyword lists: `expect_any` (signals genuine Claude\nbehavior) and `red_flag` (signals a different / suppressed model). After the\nresponse comes back the app classifies it:\n\n- **PASS** — expected keywords found, no red flags.\n- **SUSPICIOUS** — expected keywords missing, OR both expected & red flags present.\n- **FAIL** — red flags found and no expected keywords, OR request errored.\n- **INFO** — purely informational probe with no scoring.\n\nThe **Summary** tab gives an overall judgment:\n\n- ✅ behavior consistent with genuine Claude\n- ⚠ likely NOT Claude (identity FAILs or many red-flag hits)\n- 🟡 inconclusive (review SUSPICIOUS rows manually)\n- ⏸ PARTIAL (run was stopped — verdict is provisional)\n\nHeuristics live in `public/tests.js` (`TESTS` array) and `public/app.js`\n(`judge()` function). Tune by editing keyword lists, not by adding\nabstraction.\n\n## Golden baseline\n\nIf `data/golden.json` exists, every probe response is compared to 5\nreference samples via character-bigram cosine similarity. The MD report\nshows per-probe `mean / max / min` and an aggregate \"Avg golden similarity\"\nrow with a verdict tag (✅ close · ⚠️ partially diverging · ⛔ clearly\ndifferent).\n\nThe repo ships with a baseline (`data/golden.json`). Regenerate it with\nlive samples whenever you want:\n\n```bash\nANTHROPIC_API_KEY=sk-ant-... npm run build-golden\n# 5 samples × 47 probes against the real Anthropic Messages API\n```\n\nOptional env for `build-golden`: `GOLDEN_MODEL`, `GOLDEN_SAMPLES`,\n`GOLDEN_ENDPOINT`, `GOLDEN_OUT`, `GOLDEN_CONCURRENCY`.\n\nUser-added probes that have no golden entry simply skip the similarity\ncomparison — nothing breaks, no empty column dangling.\n\n## Probe packs (community sharing)\n\nA \"probe pack\" is a portable JSON containing probe definitions. Use it to\nshare probe sets between teammates, contribute to the community catalog,\nor version-control localized probe collections.\n\n```json\n{\n \"format\": \"claude-verifier-probe-pack\",\n \"version\": 1,\n \"name\": \"TH localization pack\",\n \"description\": \"Probes for verifying Thai-language responses\",\n \"probes\": [\n { \"id\": \"th_custom_1\",\n \"cat\": \"capability\",\n \"title\": \"ความสุภาพแบบ formal\",\n \"prompt\": \"ใช้ภาษาไทยทางการแบบกระทรวงพูดถึง...\",\n \"expect_any\": [\"ครับ\", \"ค่ะ\"],\n \"red_flag\": [] }\n ]\n}\n```\n\n- **Export** (catalog header): if any probes are ticked → only those are\n exported; else the entire catalog ships. Never includes endpoint, key,\n or model.\n- **Import**: probes are merged into the runtime catalog and persisted in\n `data/custom-probes.json` (survives restart). Custom probes show a\n `custom` badge with an accent border; right-click a custom probe row to\n remove it.\n\n## Project layout\n\n```\nclaude-verifier/\n├── main.js # Electron main: IPC + HTTPS + sandbox\n├── preload.js # contextBridge: typed window.api surface\n├── public/\n│ ├── index.html\n│ ├── styles.css # anthropic-noir dark theme\n│ ├── tests.js # shared probe catalog (47 probes)\n│ └── app.js # renderer: run loop, judge, MD report, UI\n├── tools/\n│ └── build-golden.js # baseline generator (real Anthropic API)\n├── data/ # baseline + user-imported probes\n│ ├── golden.json\n│ └── custom-probes.json # created on first import\n├── reports/ # auto-saved MD reports\n└── run.sh / run.bat # one-shot launchers\n```\n\n**Everything this app produces stays inside the project directory** —\nnothing is ever written to `~/Documents`, app-data dirs, or shared user\nlocations. Reports go in `reports/`, persistent data in `data/`.\n\n## Working with non-Anthropic endpoints\n\nAny endpoint that accepts the Anthropic Messages format (POST `/v1/messages`,\n`x-api-key` header, `anthropic-version: 2023-06-01`) will work — AWS Bedrock\nproxies, LiteLLM with the Anthropic adapter, internal gateways, etc.\n\n## Contributing\n\nThis project is **MIT-licensed**. Contributions welcome — especially:\n\n- **New probes** — add to `public/tests.js` (or share as a probe pack JSON).\n If you add a probe, rerun `npm run build-golden` against the Anthropic\n API to refresh `data/golden.json` so similarity comparison covers it.\n- **Localized probes** — TH / ZH / EN versions are great. Other languages\n welcome.\n- **Heuristic tuning** — `expect_any` / `red_flag` keyword lists are the\n knobs. Open a PR with a probe pack showing the false-positive or\n false-negative you're trying to fix.\n\nWhen developing:\n\n```bash\nnpm start # launch the app\nANTHROPIC_API_KEY=sk-... npm run build-golden # regenerate baseline\nnode --check public/app.js # parse-check after edits (no test suite)\n```\n\nThere is no build step, no bundler, and no test runner — just Electron.\nVanilla JS + plain CSS. Edits are reflected immediately on relaunch.\n\n## Security note\n\n- The API key only leaves your machine via direct HTTPS to whichever endpoint\n you typed; nothing is logged or sent anywhere else.\n- The \"auto-publish\" feature uploads your MD report to a free paste host\n (rentry.co with dpaste.org / paste.rs fallbacks). The report contains\n full prompts + responses — **redact before publishing if your prompts\n contain anything sensitive**, or disable auto-finish in code if you need\n the run to stay strictly local.\n- This tool is for verifying services you have authorization to test (your\n own accounts, vendor evaluations, internal QA).\n\n## Links\n\n- 🌐 **Site & probe editor** — https://botnick.github.io/claude-verifier/\n- 📦 **Releases** — https://github.com/botnick/claude-verifier/releases\n- 💬 **Discussions** (share probe packs) — https://github.com/botnick/claude-verifier/discussions\n- 🐛 **Issues** — https://github.com/botnick/claude-verifier/issues\n- 📖 **Docs** — https://botnick.github.io/claude-verifier/docs.html\n\n## License\n\nMIT — see [LICENSE](./LICENSE).\n"}}}} |