1 line
23 KiB
JSON
1 line
23 KiB
JSON
{"repo": "amanning3390/deepswarm", "files": {"task.yaml": {"614d635bc06f643f150d58d07c1beb823a5f3dad": {"t": 1785745198.1766267, "n": 1048, "h": "2aa79e45850d5d43", "c": "# DeepSwarm 2.0 — Sample Task Configuration\n# Copy this file and customize for your batch task.\n\n# ── What to do ──\ntask_type: generation\n\nprompt_template: |\n You are an AI assistant. Complete the following task thoroughly.\n \n Task: {{seed}}\n \n Respond with a complete, detailed answer.\n\nsystem_prompt: \"You are a helpful AI assistant. Be thorough and accurate.\"\n\n# ── Model routing (tiered delegation) ──\norchestrator_model: deepseek-v4-pro # Plans + monitors (uses DEEPSEEK_API_KEY)\nworker_model: deepseek-v4-flash # Executes batches (cheaper!)\nworker_api_base: https://api.deepseek.com/v1/chat/completions\nworker_max_tokens: 4096\n\n# ── Execution ──\nmulti_turn: false # Set true for conversation/tool-calling tasks\nmax_turns: 1 # Max turns per task (only if multi_turn: true)\n\n# ── Input (create with scripts/seed.py) ──\nseeds_file: seeds.jsonl\n\n# ── Output ──\noutput_dir: output/\noutput_format: jsonl\ncheckpoint_every: 10\n\n# ── Worker optimization (omit for auto) ──\n# workers: 8\n# stagger: 5\n# batch_size: 125\n"}, "40bd28f11701abe958c9dbfa7c4fca03321e5dc7": {"t": 1785748529.5377095, "n": 1048, "h": "2aa79e45850d5d43", "c": "# DeepSwarm 2.0 — Sample Task Configuration\n# Copy this file and customize for your batch task.\n\n# ── What to do ──\ntask_type: generation\n\nprompt_template: |\n You are an AI assistant. Complete the following task thoroughly.\n \n Task: {{seed}}\n \n Respond with a complete, detailed answer.\n\nsystem_prompt: \"You are a helpful AI assistant. Be thorough and accurate.\"\n\n# ── Model routing (tiered delegation) ──\norchestrator_model: deepseek-v4-pro # Plans + monitors (uses DEEPSEEK_API_KEY)\nworker_model: deepseek-v4-flash # Executes batches (cheaper!)\nworker_api_base: https://api.deepseek.com/v1/chat/completions\nworker_max_tokens: 4096\n\n# ── Execution ──\nmulti_turn: false # Set true for conversation/tool-calling tasks\nmax_turns: 1 # Max turns per task (only if multi_turn: true)\n\n# ── Input (create with scripts/seed.py) ──\nseeds_file: seeds.jsonl\n\n# ── Output ──\noutput_dir: output/\noutput_format: jsonl\ncheckpoint_every: 10\n\n# ── Worker optimization (omit for auto) ──\n# workers: 8\n# stagger: 5\n# batch_size: 125\n"}}, "SKILL.md": {"614d635bc06f643f150d58d07c1beb823a5f3dad": {"t": 1785746366.8842833, "n": 7748, "h": "60c9f999c67ba620", "c": "---\nname: deepswarm\ndescription: Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.\nversion: 2.0.0\nauthor: Hermes Agent\nlicense: MIT\nmetadata:\n hermes:\n tags: [orchestration, parallel, workers, api-generation, swarm, batch, multi-turn, delegation, tiered-models]\n related_skills: [tmux-agent-orchestrator, hermes-agent]\n---\n\n# DeepSwarm — Task-Agnostic Parallel Worker Orchestration\n\nSpawn N parallel API workers for **any** long-running or multi-turn batch task. Auto-calculates optimal worker count and stagger delay. Supports tiered model delegation: orchestrator plans with a frontier model (V4 Pro), workers execute with a cheaper model (V4 Flash).\n\n## Overview\n\nDeepSwarm 2.0 generalizes the proven orchestration pattern from the 19,331-trace generation project to any batch API task. You define a task — translations, reasoning traces, code reviews, summarization — and DeepSwarm parallelizes it across optimal workers with the right stagger for your API.\n\n**The core insight:** API rate limits are a function of simultaneous connections, not total volume. Auto-calculated stagger + worker count = 99.95% success.\n\n## When to Use\n\n- **Any** batch API task: generation, translation, summarization, extraction, classification\n- Long-running individual calls (30s+) that benefit from parallelization\n- Multi-turn tasks where each worker loops through conversation turns\n- Cost optimization via tiered delegation (orchestrator ≠ worker model)\n- Crash-resilient batch processing (checkpointed, idempotent)\n\n**Don't use for:**\n- Quick calls under 10s (overhead not worth it — just loop)\n- Tasks requiring inter-worker coordination (use `delegate_task`)\n- Real-time interactive sessions (use tmux-agent-orchestrator)\n\n## Quick Start\n\n```bash\n# Install\nhermes skills tap add amanning3390/deepswarm\n\n# Define your task (task.yaml)\n# Generate seeds\npython3 scripts/seed.py --task task.yaml\n\n# Launch — auto-optimizes workers, stagger, model routing\npython3 scripts/swarm.py --task task.yaml --total 1000\n\n# Filter — repair JSON, validate structure, apply length thresholds\npython3 scripts/filter.py --input-dir output/ --output clean.jsonl --errors errors.jsonl\n```\n\n## Task Definition (task.yaml)\n\n```yaml\n# What to do\ntask_type: generation # generation | translation | summarization | custom\nprompt_template: |\n You are an AI assistant. {{seed}}\n\n# Model routing (tiered delegation)\norchestrator_model: deepseek-v4-pro # Plans, monitors, handles errors\nworker_model: deepseek-v4-flash # Executes batches (cheaper!)\nworker_api_base: https://api.deepseek.com/v1/chat/completions\nworker_max_tokens: 4096\n\n# Execution control\nmulti_turn: true # Workers loop through conversation turns\nmax_turns: 20 # Max turns per worker conversation\nseeds_file: seeds.jsonl # Pre-generated task seeds\n\n# Worker optimization (auto-calculated if omitted)\nworkers: auto # auto | N\nstagger: auto # auto | seconds\nbatch_size: auto # auto | tasks per worker\n\n# Output\noutput_dir: output/\noutput_format: jsonl # jsonl | json | parquet\ncheckpoint_every: 10 # Save progress every N tasks\n\n# Optional: custom worker logic\nworker_script: custom_worker.py # Override default worker behavior\n```\n\n## Tiered Model Delegation\n\nOrchestrator (V4 Pro) and workers (V4 Flash) can use different models:\n\n```\nUser Task → V4 Pro (plans, monitors)\n ├─ V4 Flash Worker 0 → API → output/\n ├─ V4 Flash Worker 1 → API → output/\n ├─ V4 Flash Worker 2 → API → output/\n └─ ...\n```\n\n**Why tiered delegation matters:**\n- V4 Pro costs ~3× V4 Flash per token\n- Orchestrator only plans + monitors (few calls)\n- Workers make thousands of calls — use the cheapest model that works\n- Typical savings: 60-70% vs using V4 Pro for everything\n\n**When to use same model for both:**\n- Task quality requires frontier reasoning at every step\n- Worker model doesn't support the required format\n- Budget allows it and quality is paramount\n\n## Auto-Optimization\n\nWhen `workers: auto` and `stagger: auto`:\n\n1. DeepSwarm runs a single calibration call to measure call duration\n2. Calculates optimal workers: `min(8, floor(rate_limit / call_duration))`\n3. Sets stagger: `call_duration / workers × 2`\n4. Adjusts batch_size: `total / workers`\n\n**Calibration table (pre-computed):**\n\n| Call Duration | Workers | Stagger | Success | Throughput |\n|--------------|---------|---------|---------|------------|\n| <10s | 16 | 1s | 99.9% | ~5,760/hr |\n| 10-30s | 12 | 2s | 99.9% | ~1,440/hr |\n| 30-60s | 8 | 5s | 99.95% | ~440/hr |\n| 60-90s | 6 | 10s | 99.9% | ~240/hr |\n| >90s | 4 | 15s | 99.9% | ~96/hr |\n\n## Multi-Turn Task Support\n\nFor tasks requiring conversation loops (generation, debugging, interactive work):\n\n```\nWorker loop:\n for each seed:\n messages = [system_prompt, user_task]\n for turn in range(max_turns):\n response = api_call(messages, model=worker_model)\n messages.append({\"role\": \"assistant\", \"content\": response})\n if task_complete(response):\n break\n if needs_tool_call(response):\n messages.append(simulate_tool_response(response))\n```\n\nEach turn is an independent API call. Multi-turn tasks benefit most from parallelization because per-task latency is high.\n\n## Task-Agnostic Worker Design\n\nThe worker (`worker.py`) accepts a YAML task definition and executes any pipeline:\n\n```python\ndef run_task(seed, config):\n messages = build_messages(seed, config)\n for turn in range(config[\"max_turns\"]):\n response = call_api(messages, config)\n if is_complete(response, config):\n return finish(response, messages)\n if needs_continuation(response, config):\n messages = append_turn(messages, response, config)\n return messages\n```\n\nBuilt-in task types:\n- `generation` — Generate content from seed (the trace generation pattern)\n- `translation` — Translate each seed text\n- `summarization` — Summarize each seed document\n- `classification` — Classify each seed input\n- `custom` — Uses `worker_script` for completely custom logic\n\n## Common Pitfalls\n\n1. **`workers: auto` choosing too many.** If calibration call was fast but actual calls are slow, override manually.\n2. **Forgetting to stagger between workers.** Even 2 workers at same millisecond can trigger rate limits on slow APIs.\n3. **Mixing models without checking format compatibility.** Worker model must support the same prompt format as orchestrator.\n4. **Not checkpointing.** Worker dies at 120/125 = lost work. Checkpoint every 10.\n5. **Shell `&` without `wait`.** Without `wait`, shell exits early and kills child workers.\n6. **Using V4 Pro for workers when V4 Flash would work.** Check quality on 10-sample test before committing to expensive model.\n7. **Not deleting error outputs before restarting.** Error files consume indices and inflate disk.\n\n## Verification Checklist\n\n- [ ] Task YAML has valid model names and API base URL\n- [ ] API key exported for both orchestrator and worker models\n- [ ] `workers: auto` or manual count ≤ 8 per batch\n- [ ] `stagger: auto` or manual ≥ call_duration / workers × 2\n- [ ] Worker model tested on 5-sample run before full batch\n- [ ] Output directory exists and is writable\n- [ ] Seeds file exists with correct format\n- [ ] Checkpointing enabled for runs >100 tasks\n- [ ] Orchestrator model is V4 Pro (or equivalent frontier) for planning\n- [ ] Worker model is V4 Flash (or cheapest model that handles the task)\n"}, "40bd28f11701abe958c9dbfa7c4fca03321e5dc7": {"t": 1785748529.8716457, "n": 7590, "h": "b7fd8676756cd527", "c": "---\nname: deepswarm\ndescription: Use when running parallel AI workers for any long-running or multi-turn batch API task. Auto-calculates optimal workers + stagger. Supports tiered delegation (V4 Pro orchestrator → V4 Flash workers). 99.95% API success rate at scale.\nversion: 2.0.0\nauthor: Hermes Agent\nlicense: MIT\nmetadata:\n hermes:\n tags: [orchestration, parallel, workers, api-generation, swarm, batch, multi-turn, delegation, tiered-models]\n related_skills: [tmux-agent-orchestrator, hermes-agent]\n---\n\n# DeepSwarm — Task-Agnostic Parallel Worker Orchestration\n\nSpawn N parallel API workers for **any** long-running or multi-turn batch task. Auto-calculates optimal worker count and stagger delay. Supports tiered model delegation: orchestrator plans with a frontier model (V4 Pro), workers execute with a cheaper model (V4 Flash).\n\n## Overview\n\nDeepSwarm 2.0 generalizes the proven orchestration pattern from the 19,331-trace generation project to any batch API task. You define a task — translations, reasoning traces, code reviews, summarization — and DeepSwarm parallelizes it across optimal workers with the right stagger for your API.\n\n**The core insight:** API rate limits are a function of simultaneous connections, not total volume. Auto-calculated stagger + worker count = 99.95% success.\n\n## When to Use\n\n- **Any** batch API task: generation, translation, summarization, extraction, classification\n- Long-running individual calls (30s+) that benefit from parallelization\n- Multi-turn tasks where each worker loops through conversation turns\n- Cost optimization via tiered delegation (orchestrator ≠ worker model)\n- Crash-resilient batch processing (checkpointed, idempotent)\n\n**Don't use for:**\n- Quick calls under 10s (overhead not worth it — just loop)\n- Tasks requiring inter-worker coordination (use `delegate_task`)\n- Real-time interactive sessions (use tmux-agent-orchestrator)\n\n## Quick Start\n\n```bash\n# Install\nhermes skills tap add amanning3390/deepswarm\n\n# Define your task (task.yaml)\n# Generate seeds\npython3 scripts/seed.py --task task.yaml\n\n# Launch — auto-optimizes workers, stagger, model routing\npython3 scripts/swarm.py --task task.yaml --total 1000\n```\n\n## Task Definition (task.yaml)\n\n```yaml\n# What to do\ntask_type: generation # generation | translation | summarization | custom\nprompt_template: |\n You are an AI assistant. {{seed}}\n\n# Model routing (tiered delegation)\norchestrator_model: deepseek-v4-pro # Plans, monitors, handles errors\nworker_model: deepseek-v4-flash # Executes batches (cheaper!)\nworker_api_base: https://api.deepseek.com/v1/chat/completions\nworker_max_tokens: 4096\n\n# Execution control\nmulti_turn: true # Workers loop through conversation turns\nmax_turns: 20 # Max turns per worker conversation\nseeds_file: seeds.jsonl # Pre-generated task seeds\n\n# Worker optimization (auto-calculated if omitted)\nworkers: auto # auto | N\nstagger: auto # auto | seconds\nbatch_size: auto # auto | tasks per worker\n\n# Output\noutput_dir: output/\noutput_format: jsonl # jsonl | json | parquet\ncheckpoint_every: 10 # Save progress every N tasks\n\n# Optional: custom worker logic\nworker_script: custom_worker.py # Override default worker behavior\n```\n\n## Tiered Model Delegation\n\nOrchestrator (V4 Pro) and workers (V4 Flash) can use different models:\n\n```\nUser Task → V4 Pro (plans, monitors)\n ├─ V4 Flash Worker 0 → API → output/\n ├─ V4 Flash Worker 1 → API → output/\n ├─ V4 Flash Worker 2 → API → output/\n └─ ...\n```\n\n**Why tiered delegation matters:**\n- V4 Pro costs ~3× V4 Flash per token\n- Orchestrator only plans + monitors (few calls)\n- Workers make thousands of calls — use the cheapest model that works\n- Typical savings: 60-70% vs using V4 Pro for everything\n\n**When to use same model for both:**\n- Task quality requires frontier reasoning at every step\n- Worker model doesn't support the required format\n- Budget allows it and quality is paramount\n\n## Auto-Optimization\n\nWhen `workers: auto` and `stagger: auto`:\n\n1. DeepSwarm runs a single calibration call to measure call duration\n2. Calculates optimal workers: `min(8, floor(rate_limit / call_duration))`\n3. Sets stagger: `call_duration / workers × 2`\n4. Adjusts batch_size: `total / workers`\n\n**Calibration table (pre-computed):**\n\n| Call Duration | Workers | Stagger | Success | Throughput |\n|--------------|---------|---------|---------|------------|\n| <10s | 16 | 1s | 99.9% | ~5,760/hr |\n| 10-30s | 12 | 2s | 99.9% | ~1,440/hr |\n| 30-60s | 8 | 5s | 99.95% | ~440/hr |\n| 60-90s | 6 | 10s | 99.9% | ~240/hr |\n| >90s | 4 | 15s | 99.9% | ~96/hr |\n\n## Multi-Turn Task Support\n\nFor tasks requiring conversation loops (generation, debugging, interactive work):\n\n```\nWorker loop:\n for each seed:\n messages = [system_prompt, user_task]\n for turn in range(max_turns):\n response = api_call(messages, model=worker_model)\n messages.append({\"role\": \"assistant\", \"content\": response})\n if task_complete(response):\n break\n if needs_tool_call(response):\n messages.append(simulate_tool_response(response))\n```\n\nEach turn is an independent API call. Multi-turn tasks benefit most from parallelization because per-task latency is high.\n\n## Task-Agnostic Worker Design\n\nThe worker (`worker.py`) accepts a YAML task definition and executes any pipeline:\n\n```python\ndef run_task(seed, config):\n messages = build_messages(seed, config)\n for turn in range(config[\"max_turns\"]):\n response = call_api(messages, config)\n if is_complete(response, config):\n return finish(response, messages)\n if needs_continuation(response, config):\n messages = append_turn(messages, response, config)\n return messages\n```\n\nBuilt-in task types:\n- `generation` — Generate content from seed (the trace generation pattern)\n- `translation` — Translate each seed text\n- `summarization` — Summarize each seed document\n- `classification` — Classify each seed input\n- `custom` — Uses `worker_script` for completely custom logic\n\n## Common Pitfalls\n\n1. **`workers: auto` choosing too many.** If calibration call was fast but actual calls are slow, override manually.\n2. **Forgetting to stagger between workers.** Even 2 workers at same millisecond can trigger rate limits on slow APIs.\n3. **Mixing models without checking format compatibility.** Worker model must support the same prompt format as orchestrator.\n4. **Not checkpointing.** Worker dies at 120/125 = lost work. Checkpoint every 10.\n5. **Shell `&` without `wait`.** Without `wait`, shell exits early and kills child workers.\n6. **Using V4 Pro for workers when V4 Flash would work.** Check quality on 10-sample test before committing to expensive model.\n7. **Not deleting error outputs before restarting.** Error files consume indices and inflate disk.\n\n## Verification Checklist\n\n- [ ] Task YAML has valid model names and API base URL\n- [ ] API key exported for both orchestrator and worker models\n- [ ] `workers: auto` or manual count ≤ 8 per batch\n- [ ] `stagger: auto` or manual ≥ call_duration / workers × 2\n- [ ] Worker model tested on 5-sample run before full batch\n- [ ] Output directory exists and is writable\n- [ ] Seeds file exists with correct format\n- [ ] Checkpointing enabled for runs >100 tasks\n- [ ] Orchestrator model is V4 Pro (or equivalent frontier) for planning\n- [ ] Worker model is V4 Flash (or cheapest model that handles the task)\n"}, "cf98835fadfa0a0b81f3303933e1ad8f2b98b550": {"t": 1785748531.0661657, "n": 5030, "h": "c86dcea27f2404bc", "c": "---\nname: deepswarm\ndescription: Use when spawning parallel AI workers for batch API generation tasks. Orchestrates N workers with configurable stagger to avoid rate limits. Packaged from the DeepSeek Hermes Reasoning Traces project — 96 workers, 99.95% success rate.\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nmetadata:\n hermes:\n tags: [orchestration, parallel, workers, api-generation, swarm, batch, deepseek]\n related_skills: [tmux-agent-orchestrator, hermes-agent]\n---\n\n# DeepSwarm — Parallel AI Worker Orchestration\n\nSpawn and coordinate multiple parallel API workers for batch generation tasks. Learned from generating 19,331 DeepSeek V4 Pro reasoning traces with 99.95% success across 96 workers.\n\n## Overview\n\nDeepSwarm takes a batch generation task (traces, translations, summaries, any API-callable work) and parallelizes it across N workers with staggered launch timing. Each worker runs independently, writes results to disk immediately, and checkpoints progress for crash resilience.\n\nThe core insight: API rate limits are a function of simultaneous connections, not total volume. **8 workers per batch + 5-second stagger = 99.95% success at any scale.**\n\n## When to Use\n\n- Generating large datasets via API calls (reasoning traces, synthetic data, translations)\n- Any task where you need to make thousands of independent API calls\n- Workloads that benefit from parallelization but risk rate limiting\n- Batch processing that must survive interruptions (checkpointed, idempotent)\n\n**Don't use for:**\n- Tasks requiring inter-worker coordination (use `delegate_task` or tmux agents)\n- Quick subtasks under 5 minutes (just use `delegate_task`)\n- Interactive sessions needing mid-flight input (use tmux-agent-orchestrator)\n\n## Quick Start\n\n```bash\n# Install the skill\nhermes skills tap add ning423/deepswarm\n\n# Set your API key (DeepSeek, OpenAI, etc.)\nexport DEEPSEEK_API_KEY=sk-...\n\n# Launch a swarm — 8 workers, 1000 traces, 5s stagger\ncd scripts && python3 swarm.py --total 1000 --workers 8 --stagger 5\n```\n\n## Architecture\n\n```\nOrchestrator (swarm.py)\n └─ Worker 0: python3 worker.py --start 0 --total 125\n └─ Worker 1: python3 worker.py --start 125 --total 125 (launched +5s)\n └─ Worker 2: python3 worker.py --start 250 --total 125 (launched +10s)\n └─ ...\n └─ Worker 7: python3 worker.py --start 875 --total 125 (launched +35s)\n```\n\nEach worker:\n1. Generates seeds from templates\n2. Calls API in a loop (multi-turn if needed)\n3. Writes each result to disk immediately (`output/trace_XXXXXX.json`)\n4. Checkpoints progress every 10 traces\n5. Exits when batch complete\n\n## Stagger Formula\n\nThe stagger delay needed depends on your API call duration:\n\n| Call Duration | Workers/Batch | Stagger | Success Rate |\n|--------------|---------------|---------|-------------|\n| <10s | 16 | 1s | 99.9% |\n| 30-60s | 8 | 5s | 99.95% |\n| 60-90s | 8 | 10s | 99.9% |\n| >90s | 4 | 15s | 99.9% |\n\n**Rule of thumb:** stagger ≥ call_duration / workers × 2\n\n## Prompt Templates\n\nWorkers use template files with weighted category distributions. See `templates/prompts.py`.\n\nv1 (original): 20% Hermes-specific, mixed simple+multi-step\nv2 (improved): 30% Hermes, 100% multi-step, explicit length instructions\n\nv2 increased standard+spark yield from 3.8% to 6.4%.\n\n## JSON Repair Filter\n\nAPI-generated tool calls contain ~17% JSON errors (unescaped newlines, wrong format). The filter handles:\n\n- `{\"terminal\", ...}` → `{\"name\": \"terminal\", ...}` (missing name key)\n- Unescaped newlines in string values → escaped `\\n`\n- Unbalanced `<tool_call>` / `</tool_call>` tags → rejected\n\nPass rate: 28% without repair → 62% with repair.\n\n## Common Pitfalls\n\n1. **144 workers = guaranteed API flood (402/429 errors).** Stay at 8/batch with stagger.\n\n2. **Forgetting to stagger between workers.** Even 2 workers launching at the same millisecond can trigger rate limits on slow APIs.\n\n3. **Using tiktoken for ChatML format tokens.** cl100k_base underestimates Qwen tokens by 2-3x. Use `Qwen/Qwen3.6-27B` tokenizer for accurate counts.\n\n4. **Not checkpointing progress.** If a worker dies after 120/125 traces, you lose them all without checkpoints. Checkpoint every 10 traces.\n\n5. **Shell `&` without `wait`.** Without `wait`, the shell exits early and kills child workers. Always: `for ... & done; wait`\n\n6. **max_tokens=4096 produces mostly nano traces.** Bump to 8192 for longer multi-turn traces.\n\n7. **Not deleting error traces before restarting.** Error traces take up indices. Delete them before reusing indices.\n\n## Verification Checklist\n\n- [ ] API key set in environment (`DEEPSEEK_API_KEY` or equivalent)\n- [ ] Worker count ≤ 8 per batch\n- [ ] Stagger delay ≥ 5 seconds for 30-60s calls\n- [ ] Output directory exists and is writable\n- [ ] Progress files have unique names per worker\n- [ ] Templates have weighted category distribution\n- [ ] max_tokens set appropriately (4096 for quick, 8192 for deep)\n- [ ] Error traces deleted before reusing indices\n- [ ] `wait` command after all background workers in shell script\n"}}}} |