1 line
12 KiB
JSON
1 line
12 KiB
JSON
{"repo": "1xianl/airgroundcosys", "files": {"features/factory_task_generation/README.md": {"72d11ab40425143db5c6f9e063826e28e4e57328": {"t": 1785747254.892857, "n": 5907, "h": "7d289e24ca91b344", "c": "# Factory Task Generation Pipeline\n\nThis standalone feature automates task-dataset generation for a UE-based factory simulation.\n\nThe pipeline supports four stages:\n1. export actor and asset metadata from Unreal\n2. capture random environment screenshots\n3. build an LLM-friendly environment summary\n4. generate grounded task JSON for `uav`, `ugv`, and `uav_ugv`\n\n## Structure\n\n- `task_generation_pipeline/`: Python orchestration package\n- `ue_scripts/`: Unreal Python scripts for editor-side export and capture\n- `prompts/`: prompt templates kept separate from code\n- `configs/example_config.json`: example configuration\n- `scripts/run_pipeline.py`: runnable CLI entrypoint\n- `outputs/`: generated assets, screenshots, and task JSON\n\n## Workflow\n\nThe Python runner tries to reuse existing local outputs by default:\n- skip asset export if `asset_metadata.json` and `asset_summary.txt` already exist\n\nDefault runtime assumption:\n- UE is already launched manually\n- the target map is already loaded\n- Play is already running\n- the pipeline does not auto-start Unreal by default\n\nOutput write policy:\n- files under `outputs/assets/` are reused by default and are not rewritten when they already exist\n- files outside `outputs/assets/` are written on each corresponding step run\n- if those files do not exist, they are created first\n- if those files already exist, they are overwritten\n\nAsset export is configured for full export by default. In the example config, `max_assets_for_summary: 0` means no summary truncation and all exported asset records are preserved in the LLM-facing summary.\n\nUse `--force-assets`, `--force-screenshots`, or `--force-tasks` to refresh individual stages.\n\n## Install\n\nThis feature uses only the Python standard library for the orchestration layer.\n\n```powershell\ncd E:\\AirGroundCoSys\\features\\factory_task_generation\npython scripts\\run_pipeline.py --help\n```\n\nYou can either:\n- pass `--config configs\\example_config.json`\n- or edit the default config path directly in [run_pipeline.py](/E:/AirGroundCoSys/features/factory_task_generation/scripts/run_pipeline.py) via `DEFAULT_CONFIG_PATH`\n\n## Run the full pipeline\n\n```powershell\ncd E:\\AirGroundCoSys\\features\\factory_task_generation\npython scripts\\run_pipeline.py run-all --config configs\\example_config.json\n```\n\nIf you omit `--config`, the script uses the built-in default config path.\n\n## Example partial runs\n\n```powershell\npython scripts\\run_pipeline.py export-assets --config configs\\example_config.json\npython scripts\\run_pipeline.py capture-screens --config configs\\example_config.json\npython scripts\\run_pipeline.py generate-tasks --config configs\\example_config.json\n```\n\nIf you want fresh tasks from a fresh LLM call, you must force regeneration:\n\n```powershell\npython scripts\\run_pipeline.py generate-tasks --force-tasks\n```\n\n## Screenshot sources\n\nTwo screenshot modes are supported:\n- `airsim_agents`: capture current RGB from the UAV and UGV directly through AirSim RPC\n- `unreal_random`: capture editor-side random viewpoints from Unreal\n\nThe example config now uses `airsim_agents` with:\n- `Car1` on port `41461`\n- `Drone1` on port `41471`\n\nFor `unreal_random`, the Unreal API surface can vary by engine build. Assumptions are isolated in `ue_scripts/capture_random_screenshots.py` and `task_generation_pipeline/unreal_runner.py`.\n\n## Unreal startup mode\n\nThe example config uses:\n- `unreal.python_exec_mode: \"already_running\"`\n\nThis means:\n- the script assumes UE is already open and playing\n- AirSim screenshots are captured from the running simulation\n- asset export can be performed from the running AirSim scene\n- the pipeline will not auto-launch `UnrealEditor-Cmd.exe`\n\nIn `already_running` mode, asset export uses AirSim live scene-object queries.\nThis is convenient during Play, but the exported metadata is coarser than offline Unreal editor export because AirSim does not expose the full asset registry.\n\nIf you need to regenerate asset metadata from Unreal offline, temporarily switch:\n- `unreal.python_exec_mode` to `\"editor_cmd\"`\n\nand then run:\n\n```powershell\npython scripts\\run_pipeline.py export-assets --force-assets\n```\n\n## LLM integration\n\nThe default code supports:\n- `openai_compatible`: REST call to a configurable OpenAI-compatible endpoint\n- `mock`: local heuristic fallback for offline dataset bootstrapping\n\nThe `openai_compatible` mode can attach screenshots as base64 data URLs when enabled.\n\nDefault example config is set for DeepSeek:\n\n```powershell\n$env:DEEPSEEK_API_KEY=\"your_key_here\"\npython scripts\\run_pipeline.py generate-tasks --force-tasks\n```\n\nEach run also writes an LLM call status record to:\n- `outputs/tasks/llm_call_record.json`\n\nThat file tells you whether the model was really called or whether the pipeline fell back to mock generation.\n\n## Main outputs\n\n- `outputs/assets/asset_metadata.json`\n- `outputs/assets/asset_summary.txt`\n- `outputs/screenshots/screenshot_manifest.json`\n- `outputs/tasks/generated_tasks.json`\n\n## Screenshot viewpoint policy\n\nThe screenshot sampler is intentionally constrained to preserve useful local scene information:\n- prefer near top-down views instead of shallow oblique views\n- keep camera height in a low-altitude band rather than very high aerial viewpoints\n- reject duplicate viewpoints using spatial distance thresholds\n\nThe example config uses:\n- `camera_height_range: [140.0, 260.0]`\n- `camera_pitch_range: [-88.0, -72.0]`\n\nThe Unreal capture script also hard-clamps height and pitch into a useful top-down range, so accidental high-altitude settings will not produce low-information screenshots.\n\n## TODO\n\n- Add project-specific semantic mapping rules for your factory asset taxonomy\n- Add stronger region extraction from level geometry or navmesh zones\n- Add richer screenshot deduplication using scene similarity instead of pose distance only\n- Add experiment scripts for repeated dataset generation across multiple maps\n"}, "3ce4cc73ecb84957582a38ee35017a5f0dcfaa6b": {"t": 1785771832.837263, "n": 5907, "h": "7d289e24ca91b344", "c": "# Factory Task Generation Pipeline\n\nThis standalone feature automates task-dataset generation for a UE-based factory simulation.\n\nThe pipeline supports four stages:\n1. export actor and asset metadata from Unreal\n2. capture random environment screenshots\n3. build an LLM-friendly environment summary\n4. generate grounded task JSON for `uav`, `ugv`, and `uav_ugv`\n\n## Structure\n\n- `task_generation_pipeline/`: Python orchestration package\n- `ue_scripts/`: Unreal Python scripts for editor-side export and capture\n- `prompts/`: prompt templates kept separate from code\n- `configs/example_config.json`: example configuration\n- `scripts/run_pipeline.py`: runnable CLI entrypoint\n- `outputs/`: generated assets, screenshots, and task JSON\n\n## Workflow\n\nThe Python runner tries to reuse existing local outputs by default:\n- skip asset export if `asset_metadata.json` and `asset_summary.txt` already exist\n\nDefault runtime assumption:\n- UE is already launched manually\n- the target map is already loaded\n- Play is already running\n- the pipeline does not auto-start Unreal by default\n\nOutput write policy:\n- files under `outputs/assets/` are reused by default and are not rewritten when they already exist\n- files outside `outputs/assets/` are written on each corresponding step run\n- if those files do not exist, they are created first\n- if those files already exist, they are overwritten\n\nAsset export is configured for full export by default. In the example config, `max_assets_for_summary: 0` means no summary truncation and all exported asset records are preserved in the LLM-facing summary.\n\nUse `--force-assets`, `--force-screenshots`, or `--force-tasks` to refresh individual stages.\n\n## Install\n\nThis feature uses only the Python standard library for the orchestration layer.\n\n```powershell\ncd E:\\AirGroundCoSys\\features\\factory_task_generation\npython scripts\\run_pipeline.py --help\n```\n\nYou can either:\n- pass `--config configs\\example_config.json`\n- or edit the default config path directly in [run_pipeline.py](/E:/AirGroundCoSys/features/factory_task_generation/scripts/run_pipeline.py) via `DEFAULT_CONFIG_PATH`\n\n## Run the full pipeline\n\n```powershell\ncd E:\\AirGroundCoSys\\features\\factory_task_generation\npython scripts\\run_pipeline.py run-all --config configs\\example_config.json\n```\n\nIf you omit `--config`, the script uses the built-in default config path.\n\n## Example partial runs\n\n```powershell\npython scripts\\run_pipeline.py export-assets --config configs\\example_config.json\npython scripts\\run_pipeline.py capture-screens --config configs\\example_config.json\npython scripts\\run_pipeline.py generate-tasks --config configs\\example_config.json\n```\n\nIf you want fresh tasks from a fresh LLM call, you must force regeneration:\n\n```powershell\npython scripts\\run_pipeline.py generate-tasks --force-tasks\n```\n\n## Screenshot sources\n\nTwo screenshot modes are supported:\n- `airsim_agents`: capture current RGB from the UAV and UGV directly through AirSim RPC\n- `unreal_random`: capture editor-side random viewpoints from Unreal\n\nThe example config now uses `airsim_agents` with:\n- `Car1` on port `41461`\n- `Drone1` on port `41471`\n\nFor `unreal_random`, the Unreal API surface can vary by engine build. Assumptions are isolated in `ue_scripts/capture_random_screenshots.py` and `task_generation_pipeline/unreal_runner.py`.\n\n## Unreal startup mode\n\nThe example config uses:\n- `unreal.python_exec_mode: \"already_running\"`\n\nThis means:\n- the script assumes UE is already open and playing\n- AirSim screenshots are captured from the running simulation\n- asset export can be performed from the running AirSim scene\n- the pipeline will not auto-launch `UnrealEditor-Cmd.exe`\n\nIn `already_running` mode, asset export uses AirSim live scene-object queries.\nThis is convenient during Play, but the exported metadata is coarser than offline Unreal editor export because AirSim does not expose the full asset registry.\n\nIf you need to regenerate asset metadata from Unreal offline, temporarily switch:\n- `unreal.python_exec_mode` to `\"editor_cmd\"`\n\nand then run:\n\n```powershell\npython scripts\\run_pipeline.py export-assets --force-assets\n```\n\n## LLM integration\n\nThe default code supports:\n- `openai_compatible`: REST call to a configurable OpenAI-compatible endpoint\n- `mock`: local heuristic fallback for offline dataset bootstrapping\n\nThe `openai_compatible` mode can attach screenshots as base64 data URLs when enabled.\n\nDefault example config is set for DeepSeek:\n\n```powershell\n$env:DEEPSEEK_API_KEY=\"your_key_here\"\npython scripts\\run_pipeline.py generate-tasks --force-tasks\n```\n\nEach run also writes an LLM call status record to:\n- `outputs/tasks/llm_call_record.json`\n\nThat file tells you whether the model was really called or whether the pipeline fell back to mock generation.\n\n## Main outputs\n\n- `outputs/assets/asset_metadata.json`\n- `outputs/assets/asset_summary.txt`\n- `outputs/screenshots/screenshot_manifest.json`\n- `outputs/tasks/generated_tasks.json`\n\n## Screenshot viewpoint policy\n\nThe screenshot sampler is intentionally constrained to preserve useful local scene information:\n- prefer near top-down views instead of shallow oblique views\n- keep camera height in a low-altitude band rather than very high aerial viewpoints\n- reject duplicate viewpoints using spatial distance thresholds\n\nThe example config uses:\n- `camera_height_range: [140.0, 260.0]`\n- `camera_pitch_range: [-88.0, -72.0]`\n\nThe Unreal capture script also hard-clamps height and pitch into a useful top-down range, so accidental high-altitude settings will not produce low-information screenshots.\n\n## TODO\n\n- Add project-specific semantic mapping rules for your factory asset taxonomy\n- Add stronger region extraction from level geometry or navmesh zones\n- Add richer screenshot deduplication using scene similarity instead of pose distance only\n- Add experiment scripts for repeated dataset generation across multiple maps\n"}}}} |