1 line
11 KiB
JSON
1 line
11 KiB
JSON
{"repo": "alibaba/proteasome", "files": {"readme.md": {"e7c96469ceb799198a8adc049b152cc77ea7be1a": {"t": 1785746651.8270996, "n": 10787, "h": "dfe1949e2f140916", "c": "# 1. Testing The Framework\n\nThis section is intended for framework developers.\nAfter modifying the framework's core code, you can test it using mocked datasets by executing these commands:\n\n```shell\n# Generate mocked datasets:\n# - Creates \"_mocked_benchmark\" case set in \"data/case_set/\"\n# - Creates \"_mocked_samples\" sample set in \"data/sample_set/\"\n# - Creates \"_mocked_eval_task\" evaluation task in \"data/eval_task/\"\npython mock.py -case -sample -task\n\n# Execute evaluation (mock mode):\n# 1. Tasks prefixed with \"_mocked_\" will run in mock mode\n# 2. Runtime profiles will be generated in \"data/eval_task/task_running/\"\n# 3. Final results will be stored in \"data/result/\"\npython evaluate.py _mocked_eval_task\n\n# Cleanup mocked assets (optional):\n# Note: This removes only the mocked datasets/tasks, not runtime profiles/results\npython mock.py clean -case -sample -task\n```\n\n# 2. Creating The Case Set\n\nBefore evaluating, a case set must be created unless you have a case set already.\nCreate a case set by the following command:\n\n```shell\n# The name \"my_benchmark\" should be replaced with an appropriate name.\n# Note: This command will ask you for the tile and description of the new case set, you can input any text for them.\npython create_case_set.py my_benchmark\n```\n\n# 3. Making A Case\n\nThis section provides an example of how to create cases of creating resource.\n\n## 3.1. Creating Workshop Directory\n\nCreate a directory named `workshop_of_case`.\n\n## 3.2. Initializing Environment\n\nNavigate into the `workshop_of_case` directory and create an HCL source file - e.g., `main.tf` as shown below:\n\n```hcl\n# A provider block is required. You can specify the provider requirement block if needed.\nprovider \"alicloud\" {\n region = \"cn-shanghai\"\n}\n```\n\nNext, run the `init` and `apply` commands to initialize the environment:\n\n```shell\n# The init command downloads the provider plugins and creates the .terraform.lock.hcl file.\ntofu init\n\n# The apply command generates the Terraform state file.\ntofu apply\n```\n\nAt this point, the environment is initialized and ready for resource creation.\n\n## 3.3. Saving The Pre-Operation Environment\n\nRun the `save_env.py` script to capture the environment state before performing any operations:\n\n```shell\npython save_env.py pre_op\n```\n\nThis will create a directory named `pre_op` inside the workshop directory,\ncontaining the necessary files to represent the pre-operation environment state.\n\n## 3.4. Creating An OSS Bucket\n\nUpdate the `main.tf` file as follows:\n\n```hcl\nprovider \"alicloud\" {\n region = \"cn-shanghai\"\n}\n\n# Declare an OSS bucket.\nresource \"alicloud_oss_bucket\" \"demo_bucket\" {\n bucket = \"demo-bucket-2025-0507-1348\"\n acl = \"public-read\"\n}\n```\n\nThen, run the `apply` command to create the OSS bucket:\n\n```shell\ntofu apply\n```\n\n## 3.5. Saving The Post-Operation Environment\n\nRun the `save_env.py` script to capture the environment state after performing any operations:\n\n```shell\npython save_env.py post_op\n```\n\nThis will create a directory named `post_op` inside the workshop directory,\ncontaining the necessary files to represent the post-operation environment state.\n\n## 3.6. Preparing A Manifest File\n\nCreate a file named manifest.yaml in the workshop directory to define the test cases. Below is an example:\n\n```yaml\nresource_type: alicloud_oss_bucket\noperation_type: create\nsingularity: s0\nessentials:\n - >\n count(alicloud_oss_bucket) == 1\n - >\n opt(\"alicloud_oss_bucket_acl\") is None or count(alicloud_oss_bucket_acl) == 1\n - >\n provider_of(first_of(alicloud_oss_bucket)).region == \"cn-shanghai\"\n - >\n opt(\"alicloud_oss_bucket_acl\") is None or\n provider_of(first_of(alicloud_oss_bucket_acl)).region == \"cn-shanghai\"\n - >\n first_of(alicloud_oss_bucket).bucket == \"demo-bucket-2025-0507-1348\"\n - >\n len(list(filter(\n lambda x: x is not None,\n {\n first_of(alicloud_oss_bucket).opt(\"acl\"),\n first_of(opt(\"alicloud_oss_bucket_acl\"))\n }\n ))) == 1\n - >\n first_of(alicloud_oss_bucket).opt(\"acl\") == \"public-read\" or\n opt(\"alicloud_oss_bucket_acl\") is not None and\n first_of(alicloud_oss_bucket_acl).opt(\"acl\") == \"public-read\"\n - >\n opt(\"alicloud_oss_bucket_acl\") is None or first_of(alicloud_oss_bucket_acl).bucket ==\n \"${\" + first_of(alicloud_oss_bucket).self_path() + \".bucket}\"\n - >\n first_of(alicloud_oss_bucket).opt(\"storage_class\") in {None, \"Standard\"}\n - >\n first_of(alicloud_oss_bucket).opt(\"lifecycle_rule\") is None\nactions:\n side_effect: deny\n required:\n - action: create\n pattern: alicloud_oss_bucket.*\n count: 1\n optional:\n - action: create\n pattern: alicloud_oss_bucket_acl.*\n count: 1\n - action: create\n pattern: alicloud_oss_bucket_versioning.*\n count: 1\nsecurity: []\nmisc:\n - >\n alicloud_oss_bucket.opt(\"demo_bucket\") is not None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"versioning\") is None or\n alicloud_oss_bucket.demo_bucket.versioning.opt(\"status\") == \"Suspended\"\n - >\n opt(\"alicloud_oss_bucket_versioning\") is None or\n first_of(alicloud_oss_bucket_versioning).bucket ==\n \"${\" + first_of(alicloud_oss_bucket).self_path() + \".bucket}\" and\n first_of(alicloud_oss_bucket_versioning).opt(\"status\") in {None, \"Suspended\"}\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"cors_rule\") is None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"website\") is None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"logging\") is None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"server_side_encryption_rule\") is None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"transfer_acceleration\") is None\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"redundancy_type\") is None or\n alicloud_oss_bucket.demo_bucket.redundancy_type == \"LRS\"\n - >\n alicloud_oss_bucket.demo_bucket.opt(\"access_monitor\") is None or\n alicloud_oss_bucket.demo_bucket.access_monitor.opt(\"status\") == \"Disabled\"\nuser_inputs:\n i3: >\n Create an Alibaba Cloud OSS bucket in the cn-shanghai region.\n Name the bucket demo-bucket-2025-0507-1348, and set the resource block name to demo_bucket.\n Set the ACL to public-read.\n Do not enable versioning or lifecycle management.\n i2: >\n Create an Alibaba Cloud OSS bucket in the cn-shanghai region.\n Name the bucket demo-bucket-2025-0507-1348.\n Set the ACL to public-read.\n i1: >\n Create an OSS bucket named demo-bucket-2025-0507-1348 in Shanghai that anyone can read.\n i0: >\n Create an OSS bucket.\nclarity: [c1, c0]\n```\n\nEach case is defined by a combination of five dimensions:\n**resource type**, **operation type**, **integrity**, **clarity**, and **singularity**.\nGiven the combinations specified in this `manifest.yaml`, up to 8 unique cases can be described.\n\n## 3.7. Saving The Cases To The Case Set\n\nUse the `save_cases.py` script to save the cases defined in the workshop directory to a specified case set:\n\n```shell\n# \"my_benchmark\" refers to the name of an existing case set.\npython save_cases.py my_benchmark\n```\n\nAfter saving the case, restore the real environment to the state captured in the \"pre_op\" snapshot.\n\nNow that the cases have been created, you can repeat the steps above to add more cases as needed.\n\n# 4. Setting Samples and Evaluation Task\n\nCreate a sample set file in `data/sample_set`. e.g. `samples_testing.yaml`:\n\n```yaml\nkind: sample_set\nsample_set:\n title: Sample For Testing\n description: This is the sample set is for testing.\n samples:\n - [ qwen-max-2025-01-25, builtin_agent.d1.e1 ]\n - [ deepseek-chat, builtin_agent.d0.e1 ]\n```\n\nIn the samples section of the YAML file, each sample is a pair consisting of a model name and a decorated agent name.\n\n- The model name refers to a specific LLM.\n- The decorated agent name includes the agent's name and two feature flags:\n - \"d0\": the agent does not support documentation.\n - \"d1\": the agent does support documentation.\n - \"e0\": the agent is not aware of the IaC environment.\n - \"e1\": the agent is aware of the IaC environment.\n - \"es\": the agent is only aware of the IaC state.\n - \"ec\": the agent is only aware of the IaC code (HCL).\n\nThe `builtin_agent` is a special type of agent that can enable or disable specific features using feature flags.\n\nAt last, create an evaluation task file in `data/eval_task`, e.g. `eval_testing.yaml`:\n\n```yaml\nkind: eval_task\neval_task:\n case_set: my_benchmark\n sample_set: samples_testing\n repeats: 3\n iac_tool:\n name: tofu\n version: 1.9.0\n path: /opt/homebrew/bin/tofu\n```\n\nKey fields explained:\n\n- case_set: The name of the case set to be evaluated.\n- sample_set: The name of the sample set to be used in this evaluation.\n- repeats: The number of times each case will be evaluated.\n- iac_tool: Configuration for the Infrastructure-as-Code (IaC) tool, including:\n - name: The name of the IaC tool (e.g., tofu)\n - version: The version to be used\n - path: The full path to the executable\n\n# 5. Before Evaluating\n\nBefore running the evaluation, make sure the following preparations are complete:\n\n- **Provider credentials**: Store the credentials for the provider plugins in environment variables (e.g., ALICLOUD_ACCESS_KEY, ALICLOUD_SECRET_KEY).\n\n- **Configuration file**: Create or update the config.yaml file. Example:\n ```yaml\n eval_concurrency: 4\n models:\n - name: qwen-max-2025-01-25\n endpoint: https://dashscope.aliyuncs.com\n base_url_path: /compatible-mode/v1\n secret_key_env_var: LLM_SK\n - name: qwen-plus-2025-01-25\n endpoint: https://dashscope.aliyuncs.com\n base_url_path: /compatible-mode/v1\n secret_key_env_var: LLM_SK\n - name: qwen-turbo-2025-02-11\n endpoint: https://dashscope.aliyuncs.com\n base_url_path: /compatible-mode/v1\n secret_key_env_var: LLM_SK\n - name: deepseek-chat\n endpoint: https://api.deepseek.com\n base_url_path: /v1\n secret_key_env_var: DEEPSEEK_SK\n - name: qwq-plus\n endpoint: https://dashscope.aliyuncs.com\n base_url_path: /compatible-mode/v1\n secret_key_env_var: LLM_SK\n builtin_agent:\n default_model: qwen-max-2025-01-25\n informer_agent:\n c1_model: qwq-plus\n c0_model: qwq-plus\n ```\n- **Custom agents**: If you're using agents other than `builtin_agent`, you must implement corresponding adapters and register them in `adapter/factory.py`.\n\n# 6. Evaluating\n\nTo start the evaluation, run `evaluate.py` with the task name:\n\n```shell\npython evaluate.py eval_testing\n```\n\nIf the evaluation is interrupted, you can resume it by running `resume.py` with the same task name:\n\n```shell\npython resume.py eval_testing\n```\n\n# 7. Analyzing The Result\n\nAfter the evaluation completes, a result file will be generated in the `data/result` directory.\nYou can analyze the results as needed based on your specific criteria.\n"}}}} |