{"repo": "alannikos/edg4llm", "files": {"README.md": {"73b996c36c750fec2d99c934bda01a4d4a57954a": {"t": 1785674181.3890872, "n": 13982, "h": "31957ddbc8711339", "c": "# EDG4LLM\n\n\n\n\n\n\n [English](README.md) | [简体中文](README_zh.md)\n\n
\n\n\n\n[📘Documentation](https://github.com/alannikos/edg4llm) |\n[🛠️Quick Start](https://github.com/alannikos/edg4llm) |\n[🤔Reporting Issues](https://github.com/alannikos/edg4llm/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce the initial release of edg4llm [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n- [x] Recording Introduction Video\n- [ ] Support Gemini2\n- [ ] Support local large language models\n- [ ] Support other types of data, such as picture.\n- [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "2a5c7497f6ad9dc5bd17d1df04bee0357e23cd64": {"t": 1785766593.471151, "n": 13984, "h": "6128bf119687bd24", "c": "# EDG4LLM\n\n\n\n\n\n\n [English](README.md) | [简体中文](README_zh.md)\n\n
\n\n\n\n[📘Documentation](https://github.com/alannikos/edg4llm) |\n[🛠️Quick Start](https://github.com/alannikos/edg4llm) |\n[🤔Reporting Issues](https://github.com/alannikos/edg4llm/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce the initial release of edg4llm [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n- [ x ] Recording Introduction Video\n- [ ] Support Gemini2\n- [ ] Support local large language models\n- [ ] Support other types of data, such as picture.\n- [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "369c0bc515632c818d4614522918d257926493ed": {"t": 1785766594.856456, "n": 13997, "h": "8dd999e992d92d09", "c": "# EDG4LLM\n\n\n\n\n\n\n [English](README.md) | [简体中文](README_zh.md)\n\n
\n\n\n\n[📘Documentation](https://github.com/alannikos/edg4llm) |\n[🛠️Quick Start](https://github.com/alannikos/edg4llm) |\n[🤔Reporting Issues](https://github.com/alannikos/edg4llm/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce the initial release of edg4llm [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [×] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "4007fc1c1685f17bce259c45ed290af167fd1577": {"t": 1785766596.199877, "n": 13951, "h": "080a62647301f1e1", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/alannikos/edg4llm) |\n[🛠️Quick Start](https://github.com/alannikos/edg4llm) |\n[🤔Reporting Issues](https://github.com/alannikos/edg4llm/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce the initial release of edg4llm [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [×] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "da74126b488a60cdb809a364ba8f87c184798857": {"t": 1785766597.6626592, "n": 13948, "h": "fba49bdbb470c7f4", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce the initial release of edg4llm [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [×] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "28e2204615df2a6067925c1453996acd52a8fbdc": {"t": 1785766599.190431, "n": 13948, "h": "39534709cf10bea8", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [×] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "4c361f9bdf682b08ad4d4102be2ecda6ea725023": {"t": 1785766600.8121855, "n": 13950, "h": "d8c1d2ba31cc901f", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ × ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "d2c5f5e384ce2053b3ec2743342eb9bc5b6f20b5": {"t": 1785766602.21219, "n": 13948, "h": "39534709cf10bea8", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [×] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "8c9aa0f2cb355e04b3cb6d8add784cd184237ee4": {"t": 1785766603.8098416, "n": 13948, "h": "3b13c57e453226b6", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: the release of edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "bb66de3aaac1ffb0e4cda97159a3e9f76a00859a": {"t": 1785766605.377494, "n": 13948, "h": "6291f0f268a5890e", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: [**the release of edg4llm v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "ce61d02a54f2164eed3e558b0da262aebce21fc7": {"t": 1785766606.8350418, "n": 13970, "h": "7fc096e716522a35", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: [**the release of edg4llm v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "bfe884b28fb7858288814d775243c589e0ea81e8": {"t": 1785766608.5690782, "n": 13965, "h": "ce6eb649cb5a9904", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models: [**the release of edg4llm v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/). Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "db5a5996a0b1c7dcd4dad20872a867a59b6a1207": {"t": 1785766610.1864004, "n": 13885, "h": "a7903842aac3d466", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 We've added support for **DeepSeek-R1** and **InternLM3** models! Stay tuned for more updates!\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "3fb6ff13a564844d1438b342c057e7a753497fcc": {"t": 1785766611.5939236, "n": 13772, "h": "e74f4624393888a1", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact me](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n5. - [ ] preprocess the base_url and api_key\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "f11d362ed5dcae0c69c21538d89930991d73f80e": {"t": 1785766613.0004342, "n": 13727, "h": "875e945edfce594b", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/12] 📢📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "38cfb9612105df64144968a2cb7df9e8bdb8ee5e": {"t": 1785766614.585872, "n": 13726, "h": "1c0ee43537d3f27b", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/12] 📺 We've released a project introduction video! You can find the video on [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/). If you like it, don't forget to give it a thumbs up 👍 and follow us!\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "18d9caebcaf246d5e6a629c2b0655ff2473b2bba": {"t": 1785766615.9187415, "n": 13511, "h": "df82bb2f2d2173ca", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|--------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "a9a714fb7d6e226a73a9ee5ed2fc0cb6dcf33dbc": {"t": 1785766617.5632892, "n": 13453, "h": "f787d5070be1369a", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** |\n|--------------------|------------------|------------------------------------------------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n4. you can use the CLI to list the supported model_providers and model_names\n\n```\nusage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\nView the list of supported models.\n\noptions:\n -h, --help show this help message and exit\n --list-providers List all supported providers.\n --list-models PROVIDER\n View the list of models for a specific provider.\n```\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "14d69526e2f8cc2a7cd7557bc618fecd0e961da3": {"t": 1785766618.86984, "n": 13044, "h": "2540b6d2a799734b", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** |\n|--------------------|------------------|------------------------------------------------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` |\n\n\n## Quick Start\nTo get started with **edg4llm**, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.\n\n### **Attention!**\n\nIf you want to use `question` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\nIf you want to use `answer` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\nIf you want to use `dialogue` mode, please assure that your user_prompt contains the following format:\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "8baec83a55fa39f2af2622edffa3ebeebecd617e": {"t": 1785766620.3157918, "n": 12537, "h": "d908cdadf7fa4003", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** |\n|--------------------|------------------|------------------------------------------------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` |\n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation: Use the generate method with the following parameters:\n - task_type: Defines the type of task.\n - dialogue: generate the question-answer pairs according to the prompt.\n - question: generate the question data according to the prompt.\n - answer: generate the answer data according to the questions and the prompt.\n \n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "5bf1a58d5b4fe9ac4870fd52bd38040413d73b7d": {"t": 1785766621.729706, "n": 12327, "h": "cc7cb78a351d3bfe", "c": "# EDG4LLM\n\n\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** |\n|--------------------|------------------|------------------------------------------------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` |\n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "9d8acec8e7a14fbc7497b1511e555c045bc2c141": {"t": 1785766623.2913697, "n": 11997, "h": "fbebf355ef94cc32", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#license)\n- [Contact us](#contact-me)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n| **Model** | **Free** | **Base URL** |\n|--------------------|------------------|------------------------------------------------------------|\n| **InternLM** | Yes(Partly) | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` |\n| **ChatGLM** | Yes(Partly) | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` |\n| **DeepSeek** | Yes(Free Trial for New Users) | `https://api.deepseek.com/chat/completions` |\n| **OpenAI ChatGPT** | No (Paid Plans) | `https://api.openai.com/v1/chat/completions` |\n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Contact Me\nThank you for using **EDG4LLM**! Your support and feedback are invaluable in making this project better.\n\nIf you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:\n- Submit an Issue: Visit the [Issues Page](https://github.com/Alannikos/edg4llm/issues) and describe the problem or suggestion.\n- Email Me: You can also reach out directly via email at alannikos768@outlook.com. I'll do my best to respond promptly.\n\nYour contributions and feedback are greatly appreciated. Thank you for helping improve this tool!\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "ce7adedf76f20e1ee453d8d1e5146c31c52e3fc8": {"t": 1785766624.9566712, "n": 10737, "h": "e152fcb39e4bdd24", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**Easy Data Generation For Large Language Model(abbreviated as EDG4LLM)**, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#disclaimer)\n- [Star History](#star-history)\n\n## Introduction\n**edg4llm** is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith **edg4llm**, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "637a2547c6da81a1637ceabe322bca3addaaaf4b": {"t": 1785766626.3155289, "n": 10725, "h": "6bdb06a988ddbd1f", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\nEasy Data Generation For Large Language Model(abbreviated as EDG4LLM), A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#disclaimer)\n- [Star History](#star-history)\n\n## Introduction\nedg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith edg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "775c5bc13c509e1f0ebb8acafc3da0633667c0b0": {"t": 1785766627.726175, "n": 10725, "h": "4166cab555dc75ad", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\nEasy Data Generation For Large Language Model(abbreviated as EDG4LLM), A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#disclaimer)\n- [Star History](#star-history)\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Introduction\nedg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\nWith edg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\nEDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\n4. **Compatibility**: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n5. **Transparent Debugging**: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### Supported LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n\n- [**ChatGLM**](https://github.com/THUDM/)\n - Developer: Jointly developed by Tsinghua University and Zhipu AI.\n - Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - Developer: Developed by the DeepSeek team.\n - Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - Developer: Developed by OpenAI.\n - Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses. \n- [**InternLM**](https://github.com/InternLM)\n - Developer: Developed by the Shanghai Artificial Intelligence Laboratory.\n - Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities. \n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example(Chinese Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n### Code Example(English Version)\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# Set the test data\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\nThis project has **minimal dependencies**, requiring only the requests library. Make sure to have the following version installed:\n\n- requests>=2.32.3\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "cbc5122cce8516e473425f71f7261a1a52dba017": {"t": 1785766629.0760424, "n": 7365, "h": "74f7380c6e5184b6", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n\nEasy Data Generation For Large Language Model, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [License](#disclaimer)\n- [Star History](#star-history)\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 We are excited to announce [**the initial release of edg4llm v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/), marking the completion of its core functionalities. \n\n \n\n## Introduction\n$\\quad$ edg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\n$\\quad$ With edg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **High Efficiency**: Utilizes an optimized generation mechanism to quickly produce large volumes of high-quality fine-tuning data.\n4. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\nCompatibility: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n\n## Installation\nTo install **edg4llm**, simply run the following command in your terminal:\n\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- **Supported Python Versions**: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.\n\n### LLM Provider\nThe current version of edg4llm supports the following large language model providers:\n\n- **ChatGLM**: A versatile LLM optimized for various text generation tasks.\n- **DeepSeek**: A powerful and efficient LLM for fine-tuned applications.\n- **OpenAI ChatGPT**: The widely-used conversational AI from OpenAI.\n- **InternLM**: A robust LLM tailored for enterprise and advanced use cases.\n\nMore providers will be added in future updates to extend compatibility and functionality. \n\n\n## Quick Start\n\nTo get started with **edg4llm**, follow the steps below. This example demonstrates how to use the library to generate dialogue data based on a specific prompt.\n\n### Prerequisites\n\n1. Install the **edg4llm** package:\n```bash\n pip install edg4llm\n```\n\n2. Ensure you have Python version 3.8 or higher.\n3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).\n\n### Code Example\n```python\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n\n### Explanation\n\n1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.__version__).\n\n2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.\n\n3. Prompts:\n - system_prompt defines the behavior or role of the assistant.\n - user_prompt provides specific instructions for generating data.\n4. Data Generation:\nUse the generate method with the following parameters:\n - task_type: Defines the type of task (e.g., dialogue, question-answering).\n - system_prompt and user_prompt: Provide context and task-specific instructions.\n - num_samples: Specifies how many samples to generate.\n5. Output: The generated data is returned as a JSON object in the specified format.\n\n## Requirements\n\n- Only require requests\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Future Development Plans\n1. - [ ] Recording Introduction Video\n2. - [ ] Support Gemini2\n3. - [ ] Support local large language models\n4. - [ ] Support other types of data, such as picture.\n\n## Acknowledgments\n| Project | Description |\n|---|---|\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "df5247f6190d691a1bd164508be3312b0e5b5b44": {"t": 1785766630.601463, "n": 4857, "h": "e093aea29e9c5ce9", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n\nEasy Data Generation For Large Language Model, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [Disclaimer](#disclaimer)\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 \n\n \n\n## Introduction\n$\\quad$ edg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\n$\\quad$ With edg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **High Efficiency**: Utilizes an optimized generation mechanism to quickly produce large volumes of high-quality fine-tuning data.\n4. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\nCompatibility: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n\n## Installation\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- Python >= 3.8\n\n### LLM Provider\n\n- ChatGLM\n- DeepSeek\n- OpenAI ChatGPT\n- InternLM\n\n## Quick Start\n\n```python\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n\n## Requirements\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Future Development Plans\n\n## Acknowledgments\n\n## Disclaimer\n\n\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}, "6bb5470211d19b17d6178463af091416e2a81c76": {"t": 1785766631.9105356, "n": 4695, "h": "21819f8a268eb387", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n\nEasy Data Generation For Large Language Model, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Latest News](#latest-news)\n- [Introduction](#introduction)\n- [Features](#features)\n- [Installation](#installation)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [License](#license)\n- [Future Development Plans](#future-development-plans)\n- [Acknowledgments](#acknowledgments)\n- [Disclaimer](#disclaimer)\n\n## Latest News\n\n\n2025
\n\n- [2025/01/11] 👋👋 \n\n \n\n## Introduction\n$\\quad$ edg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:\n- **Question data**\n- **Answer data**\n- **Dialogue data**\n\n$\\quad$ With edg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.\n## Features\n\n1. **Simple to Use**: Provides a straightforward interface that allows users to get started without complex configurations.\n2. **Lightweight**: Minimal dependencies and low resource consumption make it efficient and easy to use.\n3. **High Efficiency**: Utilizes an optimized generation mechanism to quickly produce large volumes of high-quality fine-tuning data.\n4. **Flexibility**: Supports a variety of data formats and generation options, allowing customization to meet specific needs.\nCompatibility: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.\n\n## Installation\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- Python >= 3.8\n\n### LLM Provider\n\n- ChatGLM\n- DeepSeek\n- OpenAI ChatGPT\n- InternLM\n\n## Quick Start\n\n```python\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n\n## Requirements\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Future Development Plans\n\n## Acknowledgments\n\n## Disclaimer"}, "c638b3361cc0137757441f424c63a9e03115d247": {"t": 1785766633.494459, "n": 2547, "h": "aff8884b5a2cb1b6", "c": "# EDG4LLM\n\n\n\n\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n\nEasy Data Generation For Large Language Model, A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Features](#features)\n- [Installation](#installation)\n- [Library Scope](#library-scope)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Development Setup](#development-setup)\n- [Testing](#testing)\n- [License](#license)\n- [Security](#security)\n\n## Features\n\n\n## Installation\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- Python >= 3.8\n\n### LLM Provider Dependencies\n\n\n## Library Scope\n\n\n### OpenAPI Specification Validation\n\n\n## Quick Start\n\n\n## Requirements\n\n\n## Development Setup\n\n## Testing\n\n# Integration tests\n\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Security\n\n## Author\nAlannikos (alannikos768@outlook.com)"}, "29148289d3675303501a1c80aa9e1b588a267e8b": {"t": 1785766635.1787076, "n": 2821, "h": "7788ebca61d2f7c5", "c": "# EDG4LLM\n\n\n\n\n\n```\n __ __ __ __ ___ __ __ __ __ \n| | |_ | / / \\ |\\/| |_ | / \\ |_ | \\ / _ |__| | | |\\/| \n|/\\| |__ |__ \\__ \\__/ | | |__ | \\__/ |__ |__/ \\__) | |__ |__ | | \n \n```\n\n
\n\n\n\n[📘Documentation](https://github.com/Alannikos/FunGPT) |\n[🛠️Quick Start](https://github.com/Alannikos/FunGPT) |\n[🤔Reporting Issues](https://github.com/Alannikos/FunGPT/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n\nA unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.\n\n\n## Table of Contents\n- [Features](#features)\n- [Installation](#installation)\n- [Library Scope](#library-scope)\n- [Quick Start](#quick-start)\n- [Requirements](#requirements)\n- [Development Setup](#development-setup)\n- [Testing](#testing)\n- [License](#license)\n- [Security](#security)\n\n## Features\n\n\n## Installation\n\n```bash\npip install edg4llm\n```\n\n### Supported Python Versions\n- Python >= 3.8\n\n### LLM Provider Dependencies\n\n\n## Library Scope\n\n\n### OpenAPI Specification Validation\n\n\n## Quick Start\n\n\n## Requirements\n\n\n## Development Setup\n\n## Testing\n\n# Integration tests\n\n\n## License\n\nMIT License - See [LICENSE](LICENSE) for details.\n\n## Security\n\n## Author\nAlannikos (alannikos768@outlook.com)"}}, "README_zh.md": {"73b996c36c750fec2d99c934bda01a4d4a57954a": {"t": 1785674181.5184095, "n": 9885, "h": "985e0d5a50c35f58", "c": "# EDG4LLM\n\n\n\n\n\n\n [English](README_en.md) | [简体中文](README_zh.md)\n
\n\n\n\n[📘Documentation](https://github.com/alannikos/edg4llm) |\n[🛠️Quick Start](https://github.com/alannikos/edg4llm) |\n[🤔Reporting Issues](https://github.com/alannikos/edg4llm/issues) \n\n
\n\n\n\n\n[](https://github.com/Alannikos/edg4llm/issues)\n[](https://github.com/Alannikos/edg4llm/forks)\n\n\n[](https://discord.com/channels/1327445853388144681/)\n[](https://space.bilibili.com/3494365446015137)\n[](https://pypi.org/project/edg4llm/)\n[](https://pypistats.org/packages/edg4llm)\n[](https://pypi.org/project/edg4llm/)\n
\n\n\n**EDG4LLM（Easy Data Generation For Large Language Model）** ，一个用于生成大型语言模型微调数据集的统一工具，现阶段支持生成问题、答案和对话。\n\n## 最新动态\n\n\n2025
\n\n- [2025/01/28] 🚀🎉 新增对 **DeepSeek-R1** 和 **InternLM3** 模型的支持，edg4llm [**v1.0.18**](https://pypi.org/project/edg4llm/1.0.18/) 版本现已发布。敬请期待更多更新！\n- [2025/01/12] 📢📺 发布项目介绍视频！你可以在 [Bilibili](https://www.bilibili.com/video/BV1MccVeyEHp/) 上找到该视频。如果你喜欢的话，别忘了点个赞 👍 并关注**Alannikos**哦！\n- [2025/01/11] 👋👋 **edg4llm** 的初始版本 [**v1.0.12**](https://pypi.org/project/edg4llm/1.0.12/) 正式发布，标志着其核心功能已经完成。\n\n \n\n## 目录\n- [最新动态](#最新动态)\n- [项目简介](#项目简介)\n- [项目特点](#项目特点)\n- [Python库安装](#Python库安装)\n- [快速上手](#快速上手)\n- [Requirements](#requirements)\n- [未来开发计划](#未来开发计划)\n- [致谢](#致谢)\n- [License](#license)\n- [联系方式](#联系方式)\n- [Star History](#star-history)\n\n## 项目简介\n**edg4llm** 是一个专为利用大型语言模型生成微调数据而设计的 Python 库。该工具旨在帮助用户高效创建高质量的训练数据集。在现阶段，它主要支持文本数据的生成。生成的数据包括但不限于：\n- **问题数据**\n- **答案数据**\n- **对话数据**\n\n通过使用 **edg4llm**，用户可以轻松生成针对微调需求的多样化数据集，从而显著提升大型语言模型在特定任务中的性能。\n## 项目特点\n\nEDG4LLM 是一款旨在简化和加速大型语言模型微调数据集创建的统一工具。具有易用性、高效性和适应性的特性，它提供了一系列功能以满足多样化的开发需求，同时确保无缝集成和强大的调试支持。\n\n1. **简单易用**：提供简单直观的接口，用户无需复杂配置即可快速上手。\n2. **轻量化**：依赖极少，资源消耗低，使用高效便捷。\n3. **灵活性**：支持多种数据格式和生成选项，可根据特定需求进行定制。\n4. **兼容性**：与主流大型语言模型无缝集成，适用于各种开发场景。\n5. **透明调试**：提供清晰详尽的日志输出，便于有效调试和问题追踪。\n\n## Python库安装\n\n要安装 **edg4llm**，只需在终端中运行以下命令：\n\n\n```bash\npip install edg4llm\n```\n\n### 支持的 Python 版本\n- **支持的 Python 版本**：需要 Python 3.8 或更高版本才能与此库兼容。请确保您的环境满足此版本要求。\n\n### 支持的大型语言模型提供商\n当前版本的 edg4llm 支持以下大型语言模型提供商：\n\n- [**InternLM**](https://github.com/InternLM)\n - 开发者：由上海人工智能实验室开发。\n - 优势：InternLM 是一系列开源大型语言模型，具有出色的推理能力、长文本处理能力和工具使用能力。\n\n- [**ChatGLM**](https://github.com/THUDM/)\n - 开发者：由清华大学和智谱 AI 联合开发。\n - 优势：ChatGLM 是一个基于通用语言模型（GLM）架构的开源双语对话语言模型。它在大量中英文文本上进行了训练，能够生成自然且上下文相关的响应，效果极佳。\n\n- [**DeepSeek**](https://github.com/deepseek-ai/)\n - 开发者：由 DeepSeek 团队开发。\n - 优势：DeepSeek-V3 是一款性能强大且经济高效的开源大型语言模型，在语言生成、问答和对话系统等任务中表现出色。\n\n- [**OpenAI ChatGPT**](https://chatgpt.com/)\n - 开发者：由 OpenAI 开发。\n - 优势：OpenAI 的 ChatGPT 是一款高度先进的语言模型，以其强大的文本生成能力而闻名。它经过海量数据训练，能够生成高质量且上下文相关的响应。\n\n未来更新中将添加更多提供商，以扩展兼容性和功能性。\n\n| **模型** | **是否免费** | **base_url** | **Model Provider** |\n|--------------------|------------------|------------------------------------------------------------|----------------|\n| **InternLM** | 部分免费 | `https://internlm-chat.intern-ai.org.cn/puyu/api/v1/chat/completions` | `internlm` |\n| **ChatGLM** | 部分免费 | `https://open.bigmodel.cn/api/paas/v4/chat/completions/` | `chatglm` |\n| **DeepSeek** | 新用户免费试用 | `https://api.deepseek.com/chat/completions` | `deepseek` |\n| **OpenAI ChatGPT** | 付费（无免费）| `https://api.openai.com/v1/chat/completions` | `chatgpt` |\n\n## 快速上手\n\n要开始使用 **edg4llm**，请按照以下步骤操作。这些示例演示了如何使用该库根据特定提示生成微调数据。\n\n### **注意！**\n\n如果您想使用 `question` 模式，请确保您的 `user_prompt` 包含以下格式：\n\n```json\n[\n {\n \"question\": \"AAA\"\n }\n]\n```\n\n如果您想使用 `answer` 模式，请确保您的 `user_prompt` 包含以下格式：\n\n```json\n[\n {\n \"answer\": \"AAA\"\n }\n]\n```\n\n如果您想使用 `dialogue` 模式，请确保您的 `user_prompt` 包含以下格式：\n\n```json\n[\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n]\n```\n\n### 前置条件\n\n1. 安装 **edg4llm** 包：\n```bash\n pip install edg4llm\n```\n\n2. 确保您的 Python 版本为 3.8 或更高。\n\n3. 获取所选模型提供商（例如 ChatGLM）所需的 API 密钥和基础 URL。\n\n4. 您可以使用 CLI 列出支持的模型提供商和模型名称：\n\n```\n用法: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]\n\n查看支持的模型列表。\n\n选项:\n -h, --help 显示帮助信息并退出\n --list-providers 列出所有支持的提供商。\n --list-models PROVIDER\n 查看特定提供商的模型列表。\n```\n\n### 示例代码（中文版本）\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n# 设置测试数据\nsystem_prompt = \"\"\"你是一个精通中国古代诗词的古文学大师\"\"\"\n\nuser_prompt = \"\"\"\n 目标: 1. 请生成过年为场景的连续多轮对话记录\n 2. 提出的问题要多样化。\n 3. 要符合人类的说话习惯。\n 4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n 其中input字段表示一个人的话语, output字段表示专家的话语\n\"\"\"\nnum_samples = 1 # 只生成一个对话样本\n\n# 调用 generate 方法生成对话\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n\n### 示例代码（英文版本）\n```python\n# chatglm_demo.py\n\nimport edg4llm\nprint(edg4llm.__version__)\n\nfrom edg4llm import EDG4LLM\n\napi_key = \"xxx\"\nbase_url = \"https://open.bigmodel.cn/api/paas/v4/chat/completions\"\n\nedg = EDG4LLM(model_provider='chatglm', model_name=\"glm-4-flash\", base_url=base_url, api_key=api_key)\n\n# 设置测试数据\nsystem_prompt = \"\"\"You are a master of ancient Chinese literature, specializing in classical poetry.\"\"\"\n\nuser_prompt = \"\"\"\n Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.\n 2. The questions should be diverse.\n 3. The dialogue should align with natural human conversational habits.\n 4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template: \n [\n {{\n \"input\":\"AAA\",\"output\":\"BBB\" \n }}\n ]\n Where the input field represents a person's dialogue, and the output field represents the expert's response.\n\"\"\"\nnum_samples = 1 # Generate only one dialogue sample\n\n# Call the generate method to generate the dialogue\ndata_dialogue = edg.generate(\n task_type=\"dialogue\",\n system_prompt=system_prompt,\n user_prompt=user_prompt,\n num_samples=num_samples\n)\n```\n\n### 说明\n\n1. **导入库**：导入 `edg4llm` 库，并通过 `print(edg4llm.__version__)` 验证版本。\n\n2. **初始化**：使用 `EDG4LLM` 初始化库，传入正确的model_provider、model_name、base_url 和 api_key。\n\n3. **prompt参数**：\n - `system_prompt` 定义助手的行为或角色。\n - `user_prompt` 提供生成数据的具体指令。\n\n4. **数据生成**：使用 `generate` 方法，参数如下：\n - `task_type`：定义任务类型。\n - `dialogue`：根据提示生成问答对。\n - `question`：根据提示生成问题数据。\n - `answer`：根据问题和提示生成答案数据。\n - `system_prompt` 和 `user_prompt`：提供上下文和任务特定的指令。\n - `num_samples`(可选)：指定生成的样本数量。\n\n5. **输出**：生成的数据以指定的 JSON 格式返回。\n\n## Requirements\n\n该项目的**依赖项极少**，仅需安装 `requests` 库。请确保安装了以下版本：\n\n- requests>=2.32.3\n\n## 未来开发计划\n\n- [x] 录制介绍视频 \n- [ ] 支持 Gemini2 \n- [ ] 支持本地大型语言模型 \n- [ ] 支持其他类型的数据，例如图片 \n- [ ] 预处理 `base_url` 和 `api_key`\n\n## 致谢\n| Project | Description |\n|---|---|\n| [FunGPT](https://github.com/Alannikos/FunGPT) | An open-source Role-Play project |\n| [InternLM](https://github.com/InternLM/InternLM) | A series of advanced open-source large language models |\n| [ChatGLM](https://github.com/THUDM/) | A bilingual dialog language model based on the General Language Model (GLM) architecture, jointly developed by Tsinghua University and Zhipu AI. |\n| [DeepSeek](https://github.com/deepseek-ai/) | A powerful and cost-effective open-source large language model, excelling in tasks such as language generation, question answering, and dialog systems. |\n| [ChatGPT](https://openai.com/chatgpt/) | A highly advanced language model developed by OpenAI, known for its robust text generation capabilities. |\n\n## License\nMIT License - See [LICENSE](LICENSE) for details.\n\n## 联系方式\n感谢您使用 **EDG4LLM**！您的支持和反馈对于改进这个项目至关重要。\n\n如果您遇到任何问题、有建议，或者只是想分享您的想法，请随时：\n- 提交问题：访问 [issues页面](https://github.com/Alannikos/edg4llm/issues) 并描述您遇到的问题或建议。\n- 发送邮件：您也可以通过电子邮件直接联系我，邮箱地址为 alannikos768@outlook.com。我会尽力及时回复您。\n\n您的贡献和反馈对我们非常重要。感谢您帮助改进这个工具！\n\n## Star History\n\n[](https://star-history.com/#Alannikos/edg4llm&Date)"}}}}