hack/tools/scripts/llm-key-hunter/results/_cache/content/austinw1995_magnific-llm-evals.json

{"repo": "austinw1995/magnific-llm-evals", "files": {"README.md": {"e21e8e81c6c360c5e7d56b278c4c6237f3196243": {"t": 1785746503.91857, "n": 12945, "h": "52087e1ad51ad65c", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\npip install -e .\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. service_provider takes in the service agent's LLMConfig object wrapped in a provider that supports its LLM model. customer_provider takes in the customer agent's LLMConfig object wrapped in a provider that supports its LLM model.\n2. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n3. first_message is the first utterance of the caller.\n4. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination, this is especially useful when end_call_enabled is set to False.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder.\n"}, "f110bb3a0a34c62a69864eb05821ed84c6cde490": {"t": 1785765028.673587, "n": 12945, "h": "52087e1ad51ad65c", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\npip install -e .\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. service_provider takes in the service agent's LLMConfig object wrapped in a provider that supports its LLM model. customer_provider takes in the customer agent's LLMConfig object wrapped in a provider that supports its LLM model.\n2. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n3. first_message is the first utterance of the caller.\n4. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination, this is especially useful when end_call_enabled is set to False.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder.\n"}, "99457fc90ab81ee514786117e73e37fc64ab6475": {"t": 1785765029.9264376, "n": 12927, "h": "9ae8ad6ce5120b4f", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. service_provider takes in the service agent's LLMConfig object wrapped in a provider that supports its LLM model. customer_provider takes in the customer agent's LLMConfig object wrapped in a provider that supports its LLM model.\n2. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n3. first_message is the first utterance of the caller.\n4. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination, this is especially useful when end_call_enabled is set to False.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "2c6cdc9ad9e1515ba3ea711011be2d51fba576d3": {"t": 1785765031.515415, "n": 12929, "h": "8d9bef668dbdbb4a", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. service_provider takes in the service agent's LLMConfig object wrapped in a provider that suppports its LLM model. customer_provider takes in the customer agent's LLMConfig object wrapped in a provider that suppports its LLM model.\n2. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n3. first_message is the first utterance of the caller.\n4. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination, this is especially useful when end_call_enabled is set to False.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "58c6cb5218e2c2d10250d356ef3b0fff689f9e5f": {"t": 1785765033.0645874, "n": 12694, "h": "478b2b985ae1195e", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n2. first_message is the first utterance of the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination, this is especially useful when end_call_enabled is set to False.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "f68b42f6ce0e6e0ce5d1660cc4737fdacd5c4bb2": {"t": 1785765034.6005242, "n": 12629, "h": "1102e02aab0cac26", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\n## Quick Start\n\nTo get up and running, run the single_test.py example.\n```\ncd examples\n#Fill in your API key with os.environ[\"OPENAI_API_KEY\"] = \"...\"\npython3 single_test.py\n```\n\n## Documentation\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n2. first_message is the first utterance of the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "070a06ee17b8d66516353dff36f3118866106213": {"t": 1785765036.4633427, "n": 12432, "h": "4bd6538f1c90fc02", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" (customer calling in) or \"outbound\" (service agent calling out).\n2. first_message is the first utterance of the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class. max_turns is the maximum number of turns the conversation can have until forced termination.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations, max_turns=20)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "55ab0518b7e80227a4d3bdcd274b76e9294ee9fe": {"t": 1785765037.98326, "n": 12375, "h": "041f0b0c38af346d", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configurations for a service or customer agent (like temperature, max tokens, model, etc.), define a LLMConfig object.\n1. params is a dictionary of any parameter supported by the LLM model provider (can be found in the provider's website/developer documentations).\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "4fa09069dd77974a7747d2c895e930d29951dd36": {"t": 1785765039.8753479, "n": 12292, "h": "4e7ccf4108765f39", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebras, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "8c84c2b6982b50321e31ce545361ea351d7c47fa": {"t": 1785765041.3472533, "n": 12292, "h": "d7cb2597ec714598", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias, specifically for multi-turn conversations under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "6123c54bb8735bdf44d7db1caa816840fcff71d9": {"t": 1785765042.7807038, "n": 12279, "h": "fcf1d55e98a08e56", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | grok-2-1212 |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "ee371b720dae640baa5cba5a52834d99fc0bd36c": {"t": 1785765044.2321808, "n": 12282, "h": "87a7ee400d607a3e", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | xgrok-2-latest |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items are displayed in a structured format, with each item listed on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu, answered all questions regarding the pizzas, sizes, prices, and specials, and offered additional options for sides and drinks, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu is displayed in a structured format with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided necessary information and confirmed the order without excessive detail. However, there were moments where the responses could have been slightly more concise, particularly in the explanation of the menu and specials.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.2,\n        \"max_tokens\": 90\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"llama-3.3-70b-versatile\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 50\n      },\n      \"system_prompt\": \"You are a cheerful customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated empathy by being patient and understanding throughout the conversation, acknowledging the customer's excitement and indecisiveness. However, there could have been more explicit expressions of understanding regarding the customer's hunger or excitement.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and satisfaction throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet. ... IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"gemini-2.0-flash\"\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food. ... IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "fed315509593d4c91538f4789759b26d51d784cb": {"t": 1785765045.862245, "n": 11483, "h": "2e136d6a1af08162", "c": "# Magnific-Evals\n\nA python package for testing LLMs with custom evaluation criterias specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | xgrok-2-latest |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeri...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "cbdd76d1f4d6d9df6adbcb78ba2e7e799b9d798e": {"t": 1785765047.273299, "n": 11450, "h": "d1b18e84b57fbaa5", "c": "# Magnific-Evals\n\nA python package for testing LLMs specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | xgrok-2-latest |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the test result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The transcript, as well as the LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeri...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "ff30831ef0a552c100840f60927320d02b8b15d7": {"t": 1785765048.6971405, "n": 11418, "h": "b5dde0d48190a794", "c": "# Magnific-Evals\n\nA python package for testing LLMs specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | xgrok-2-latest |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call(). (Note: IMPORTANT texts are highlighted in the system prompt to ensure function call reliability).\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo instantiate a list of conversations for testing, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeri...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "e80f71bf35324d99969c8cfc0c4b0d87e47388d6": {"t": 1785765050.2885, "n": 11625, "h": "1b5d2ffbfbd50e7f", "c": "# Magnific-Evals\n\nA python package for testing LLMs specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/austinw1995/magnific-llm-evals\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios by customizing the service and customer agents\n- Define custom evaluation criteria for a successful conversation\n- Run multiple tests in parallel and record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nWe support a variety of LLMs through 7 providers, which are OpenAI, Anthropic, TogetherAI, Groq, DeepSeek, Cerebrad, XAI, and Google Gemini:\nYou can access different models through different providers with\n```\nfrom magnific import (\n    OpenAIProvider,\n    AnthropicProvider,\n    TogetherAIProvider,\n    GroqProvider,\n    DeepSeekProvider,\n    CerebrasProvider,\n    XAIProvider,\n    GeminiProvider,\n)\n```\nFor each provider, the following models are supported:\n| Provider              | Models |\n|-----------------------|--------|\n| **OpenAIProvider**    | gpt-4o<br>gpt-4o-mini<br>gpt-3.5-turbo-0125<br>o1<br>o1-mini<br>o3-mini |\n| **AnthropicProvider** | claude-3-5-sonnet-20241022<br>claude-3-5-haiku-20241022<br>claude-3-opus-20240229<br>claude-3-sonnet-20240229<br>claude-3-haiku-20240307 |\n| **TogetherAIProvider**| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo<br>meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo<br>meta-llama/Llama-3.3-70B-Instruct-Turbo<br>mistralai/Mixtral-8x7B-Instruct-v0.1<br>mistralai/Mistral-7B-Instruct-v0.1<br>Qwen/Qwen2.5-7B-Instruct-Turbo<br>Qwen/Qwen2.5-72B-Instruct-Turbo |\n| **GroqProvider**      | qwen-2.5-32b<br>deepseek-r1-distill-qwen-32b<br>deepseek-r1-distill-llama-70b<br>llama-3.3-70b-versatile<br>llama-3.1-8b-instant<br>mixtral-8x7b-32768<br>gemma2-9b-it |\n| **DeepSeekProvider**  | deepseek-chat<br>deepseek-reasoner |\n| **CerebrasProvider**  | llama3.1-8b<br>llama-3.3-70b<br>DeepSeek-R1-Distill-Llama-70B |\n| **XAIProvider**       | xgrok-2-latest |\n| **GeminiProvider**    | gemini-2.0-flash<br>gemini-2.0-flash-lite-preview-02-05<br>gemini-1.5-flash<br>gemini-1.5-flash-8b<br>gemini-1.5-pro |\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of any parameter supported by the LLM model provider.\n2. system_prompt is the instruction task for the LLM agent to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the function call end_call().\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.''\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet...IMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food...IMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo initialize the providers, use the LLM Provider classes.\n```\nservice_provider_1 = OpenAIProvider(config=service_config_1)\nservice_provider_2 = AnthropicProvider(config=service_config_2)\ncustomer_provider_1 = GeminiProvider(config=customer_config_1)\ncustomer_provider_2 = GroqProvider(config=customer_config_2)\n```\n\nTo instantiate a list of conversations, use the LLMConversation class.\n1. type is the type of conversation, either \"inbound\" or \"outbound\", inbound is the customer calling in and outbound is the service agent calling out.\n2. first_message is the first message in the conversation by the caller.\n3. evaluations is a list of user defined evaluations to be performed on the conversation, customize the name of the evaluation and prompt, which specifies the evaluation criteria.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=AnthropicProvider(config=service_config_2),\n            customer_provider=GroqProvider(config=customer_config_2),\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=OpenAIProvider(config=service_config_1),\n            customer_provider=GeminiProvider(config=customer_config_1),\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nFinally, to run the tests in parallel, with the evaluation criterias (for eval models we only support openai for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the result as the value.\nAn example result based on the conversations above is shown below, where the evaluations output scores and reasons for passing or failing. The LLM configurations of service and customer agents are also included in the result for prompt management purposes.\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeri...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria...\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food...\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "e13efe5c73c9ed0d03e2a60e0e88684439adf39f": {"t": 1785765051.693442, "n": 12808, "h": "101a29d921d69bb4", "c": "# Magnific-Evals\n\nA python package for testing LLMs specifically for voice agents under customer service scenarios.\n\n## Installation\n\n```bash\npip install magnific-llm-evals\n```\n\nThis will automatically install all required dependencies.\n\nAlternatively, if installing from source:\n```bash\ngit clone https://github.com/yourusername/magnific-llm-evals.git\ncd magnific-llm-evals\npip install -r requirements.txt\n```\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios in parallel\n- Define custom evaluation criteria\n- Record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of parameters for the LLM, which can be any parameter supported by the LLM provider.\n2. system_prompt is a string of instructions for the LLM to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the tool use/function call end_call().\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\nKeep responses short and simple. Do not end the call until the customer says bye.\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet.\nYour job is to take the order of customers calling in. The menu has only 3 types of items: burgers, sides, and milkshakes.\nKeep responses short and simple. Do not end the call until the customer says bye.\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food.\nYour tone is casual and excited.\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food.\nYour tone is cheerful and excited.\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo initialize the providers, use the LLMProvider class.\n\n```\nservice_provider_1 = OpenAIProvider(config=service_config_1)\nservice_provider_2 = AnthropicProvider(config=service_config_2)\ncustomer_provider_1 = GeminiProvider(config=customer_config_1)\ncustomer_provider_2 = GroqProvider(config=customer_config_2)\n```\n\nFor each provider, the following models are supported:\n\n- OpenAIProvider: gpt-4o, gpt-4o-mini, gpt-3.5-turbo-0125, o1, o1-mini, o3-mini\n- AnthropicProvider: claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022, claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307\n- TogetherAIProvider: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo, meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo, meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo, meta-llama/Llama-3.3-70B-Instruct-Turbo, mistralai/Mixtral-8x7B-Instruct-v0.1, mistralai/Mistral-7B-Instruct-v0.1, Qwen/Qwen2.5-7B-Instruct-Turbo, Qwen/Qwen2.5-72B-Instruct-Turbo\n- GroqProvider: qwen-2.5-32b, deepseek-r1-distill-qwen-32b, deepseek-r1-distill-llama-70b, llama-3.3-70b-versatile, llama-3.1-8b-instant, mixtral-8x7b-32768, gemma2-9b-it\n- DeepSeekProvider: deepseek-chat, deepseek-reasoner\n- CerebrasProvider: llama3.1-8b, llama-3.3-70b, DeepSeek-R1-Distill-Llama-70B\n- XAIProvider: xgrok-2-latest\n- GeminiProvider: gemini-2.0-flash, gemini-2.0-flash-lite-preview-02-05, gemini-1.5-flash, gemini-1.5-flash-8b, gemini-1.5-pro\n\nTo instantiate a list of conversations, use the LLMConversation class.\n1. service_provider is the provider for the service agent.\n2. customer_provider is the provider for the customer agent.\n3. type is the type of conversation, either \"inbound\" or \"outbound\", where inbound is the customer calling in and outbound is the service agent calling out.\n4. first_message is the first message in the conversation by the caller.\n5. evaluations is a list of custom evaluations to be performed on the conversation, where name is the name of the evaluation and prompt is the prompt/criteria for the evaluation.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=service_provider_1,\n            customer_provider=customer_provider_1,\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=service_provider_2,\n            customer_provider=customer_provider_2,\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=service_provider_1,\n            customer_provider=customer_provider_1,\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nTo run the tests in parallel, with a specific llm-as-a-judge evaluation model (we only support openai models for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the result as the value.\nAn example result based on the conversations above is:\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}, "08365dd7a5b9b6c4301934404e1b7baaeb51303e": {"t": 1785765053.3375401, "n": 12645, "h": "6d89e0987be76e72", "c": "# Magnific-Evals\n\nA python package for testing LLMs specifically for voice agents under customer service scenarios.\n\n## Installation\n\npip install openai\npip install anthropic\npip install groq\npip install cerebras_cloud_sdk\npip install google-genai\n\n## Features\n\n- Evaluate multiple LLMs against various customer conversation scenarios in parallel\n- Define custom evaluation criteria\n- Record conversation transcripts\n- Generate detailed evaluation reports\n\n## Usage\n\nFirst, set up your API keys depending on what LLMs you want to test.\n```\nos.environ[\"OPENAI_API_KEY\"] = \"...\"\nos.environ[\"ANTHROPIC_API_KEY\"] = \"...\"\nos.environ[\"TOGETHER_API_KEY\"] = \"...\"\nos.environ[\"GROQ_API_KEY\"] = \"...\"\nos.environ[\"DEEPSEEK_API_KEY\"] = \"...\"\nos.environ[\"XAI_API_KEY\"] = \"...\"\nos.environ[\"GEMINI_API_KEY\"] = \"...\"\n```\n\nTo set the configuration for a service or customer agent, create a LLMConfig object with the desired parameters.\n1. params is a dictionary of parameters for the LLM, which can be any parameter supported by the LLM provider.\n2. system_prompt is a string of instructions for the LLM to follow.\n3. end_call_enabled is a boolean that determines if the LLM should be able to end the call with the tool use/function call end_call().\n\n```\nservice_config_1 = LLMConfig(\n        params={\n            \"model\": \"gpt-4o-mini\",\n            \"temperature\": 0.7,\n            \"max_tokens\": 150,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\nKeep responses short and simple. Do not end the call until the customer says bye.\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\nservice_config_2 = LLMConfig(\n        params={\n            \"model\": \"claude-3-5-sonnet-20241022\",\n            \"temperature\": 0.2,\n            \"max_tokens\": 90,\n        },\n        system_prompt=\"\"\"You are a voice assistant for Vappy's Burgers, a burger shop located on the Internet.\nYour job is to take the order of customers calling in. The menu has only 3 types of items: burgers, sides, and milkshakes.\nKeep responses short and simple. Do not end the call until the customer says bye.\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \"bye\" or \"see you.'\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_1 = LLMConfig(\n        params={\n            \"model\": \"gemini-2.0-flash\"\n        },\n        system_prompt=\"\"\"You are a hungry customer who wants to order food.\nYour tone is casual and excited.\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n\ncustomer_config_2 = LLMConfig(\n        params={\n            \"model\": \"llama-3.3-70b-versatile\",\n            \"temperature\": 0.9,\n            \"max_tokens\": 50,\n        },\n        system_prompt=\"\"\"You are a cheerful customer who wants to order food.\nYour tone is cheerful and excited.\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\n\"\"\",\n        end_call_enabled=True\n    )\n```\n\nTo initialize the providers, use the LLMProvider class.\n\n```\nservice_provider_1 = OpenAIProvider(config=service_config_1)\nservice_provider_2 = AnthropicProvider(config=service_config_2)\ncustomer_provider_1 = GeminiProvider(config=customer_config_1)\ncustomer_provider_2 = GroqProvider(config=customer_config_2)\n```\n\nFor each provider, the following models are supported:\n\n- OpenAIProvider: gpt-4o, gpt-4o-mini, gpt-3.5-turbo-0125, o1, o1-mini, o3-mini\n- AnthropicProvider: claude-3-5-sonnet-20241022, claude-3-5-haiku-20241022, claude-3-opus-20240229, claude-3-sonnet-20240229, claude-3-haiku-20240307\n- TogetherAIProvider: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo, meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo, meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo, meta-llama/Llama-3.3-70B-Instruct-Turbo, mistralai/Mixtral-8x7B-Instruct-v0.1, mistralai/Mistral-7B-Instruct-v0.1, Qwen/Qwen2.5-7B-Instruct-Turbo, Qwen/Qwen2.5-72B-Instruct-Turbo\n- GroqProvider: qwen-2.5-32b, deepseek-r1-distill-qwen-32b, deepseek-r1-distill-llama-70b, llama-3.3-70b-versatile, llama-3.1-8b-instant, mixtral-8x7b-32768, gemma2-9b-it\n- DeepSeekProvider: deepseek-chat, deepseek-reasoner\n- CerebrasProvider: llama3.1-8b, llama-3.3-70b, DeepSeek-R1-Distill-Llama-70B\n- XAIProvider: xgrok-2-latest\n- GeminiProvider: gemini-2.0-flash, gemini-2.0-flash-lite-preview-02-05, gemini-1.5-flash, gemini-1.5-flash-8b, gemini-1.5-pro\n\nTo instantiate a list of conversations, use the LLMConversation class.\n1. service_provider is the provider for the service agent.\n2. customer_provider is the provider for the customer agent.\n3. type is the type of conversation, either \"inbound\" or \"outbound\", where inbound is the customer calling in and outbound is the service agent calling out.\n4. first_message is the first message in the conversation by the caller.\n5. evaluations is a list of custom evaluations to be performed on the conversation, where name is the name of the evaluation and prompt is the prompt/criteria for the evaluation.\n\n```\nconversations = [\n        LLMConversation(\n            service_provider=service_provider_1,\n            customer_provider=customer_provider_1,\n            type=\"inbound\",\n            first_message=\"Hi, what's on the menu today?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"helpfulness\", prompt=\"The service agent should be helpful and answer all questions.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=service_provider_2,\n            customer_provider=customer_provider_2,\n            type=\"outbound\",\n            first_message=\"Hi, what would you like to order?\",\n            evaluations=[\n                Evaluation(name=\"Menu\", prompt=\"The menu should be displayed in a structured format, with each item on a new line.\"),\n                Evaluation(name=\"conciseness\", prompt=\"The service agent should be concise.\")\n            ]\n        ),\n        LLMConversation(\n            service_provider=service_provider_1,\n            customer_provider=customer_provider_1,\n            type=\"inbound\",\n            first_message=\"Hi, I'm so hungry\",\n            evaluations=[\n                Evaluation(name=\"empathy\", prompt=\"The service agent should be empathetic and show understanding of the customer's situation.\"),\n                Evaluation(name=\"frustration\", prompt=\"The customer should not be frustrated or annoyed.\")\n            ]\n        )\n    ]\n```\n\nTo run the tests in parallel, with a specific llm-as-a-judge evaluation model (we only support openai models for now), use the TestRunner class.\n\n```\nrunner = TestRunner(eval_model=\"gpt-4o-mini\")\nresults = await runner.run_tests(conversations)\n```\n\nThe results will be a dictionary with the test_id as the key and the result as the value.\nAn example result based on the conversations above is:\n\n```\n{\n  \"1\": {\n    \"test_id\": 1,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": false,\n        \"score\": 0.2,\n        \"reason\": \"The menu items are mentioned in a conversational format rather than a structured format with each item on a new line. The response does not meet the requirement for clear presentation.\"\n      },\n      {\n        \"name\": \"helpfulness\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The service agent provided detailed information about the menu items, answered all questions asked by the customer, and offered additional options, demonstrating a high level of helpfulness.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"2\": {\n    \"test_id\": 2,\n    \"call_type\": \"outbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"Menu\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The menu items were clearly listed in a structured format, with each item on a new line, making it easy to read and understand.\"\n      },\n      {\n        \"name\": \"conciseness\",\n        \"passed\": true,\n        \"score\": 0.8,\n        \"reason\": \"The service agent provided clear and relevant information without unnecessary elaboration. However, there were moments where the responses could have been slightly more succinct, particularly in confirming the order.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  },\n  \"3\": {\n    \"test_id\": 3,\n    \"call_type\": \"inbound\",\n    \"transcript\": \"...\",\n    \"evaluation_results\": [\n      {\n        \"name\": \"empathy\",\n        \"passed\": true,\n        \"score\": 0.9,\n        \"reason\": \"The service agent demonstrated a good level of empathy by responding positively to the customer's excitement about food and acknowledging their hunger. However, there could have been more explicit expressions of understanding or concern for the customer's situation.\"\n      },\n      {\n        \"name\": \"frustration\",\n        \"passed\": true,\n        \"score\": 1.0,\n        \"reason\": \"The customer expressed excitement and eagerness throughout the conversation, showing no signs of frustration or annoyance.\"\n      }\n    ],\n    \"service_config\": {\n      \"params\": {\n        \"model\": \"gpt-4o-mini\",\n        \"temperature\": 0.7,\n        \"max_tokens\": 150\n      },\n      \"system_prompt\": \"You are a voice assistant for Vappy's Pizzeria, a pizza shop located on the Internet.\\nYour job is to take the order of customers calling in. The menu has only 3 types of items: pizza, sides, and drinks.\\nKeep responses short and simple. Do not end the call until the customer says bye.\\nIMPORTANT: Do not use tool end_call() until all of the customer's questions are answered and they say something like \\\"bye\\\" or \\\"see you.\\\"\",\n      \"end_call_enabled\": true\n    },\n    \"customer_config\": {\n      \"params\": {\n        \"model\": \"claude-3-5-sonnet-20241022\",\n        \"temperature\": 0.9,\n        \"max_tokens\": 100\n      },\n      \"system_prompt\": \"You are a hungry customer who wants to order food.\\nYour tone is casual and excited.\\nIMPORTANT: Use the tool end_call() only when you are satisfied with your order and all your questions are answered.\",\n      \"end_call_enabled\": true\n    }\n  }\n}\n```\n\nMore examples can be found in the examples folder."}}}}