hack/tools/scripts/llm-key-hunter/results/_cache/content/atifassari_mcp-skills-universe.json

{"repo": "atifassari/mcp-skills-universe", "files": {"skills_library/batch_11/deepeval.md": {"f209670231297f1b75c11764bba06c6d9ec2355d": {"t": 1786068106.2180245, "n": 13817, "h": "6bc0fe7ee1c0d9fb", "c": "---\ntitle: deepeval\nurl: https://skills.sh/sammcj/agentic-coding/deepeval\n---\n\n# deepeval\n\nskills/sammcj/agentic-coding/deepeval\ndeepeval\nInstallation\n$ npx skills add https://github.com/sammcj/agentic-coding --skill deepeval\nSKILL.md\nDeepEval\nOverview\n\nDeepEval is a pytest-based framework for testing LLM applications. It provides 50+ evaluation metrics covering RAG pipelines, conversational AI, agents, safety, and custom criteria. DeepEval integrates into development workflows through pytest, supports multiple LLM providers, and includes component-level tracing with the @observe decorator.\n\nRepository: https://github.com/confident-ai/deepeval Documentation: https://deepeval.com\n\nInstallation\npip install -U deepeval\n\n\nRequires Python 3.9+.\n\nQuick Start\nBasic pytest test\nimport pytest\nfrom deepeval import assert_test\nfrom deepeval.test_case import LLMTestCase\nfrom deepeval.metrics import AnswerRelevancyMetric\n\ndef test_chatbot():\n    metric = AnswerRelevancyMetric(threshold=0.7, model=\"athropic-claude-sonnet-4-5\")\n    test_case = LLMTestCase(\n        input=\"What if these shoes don't fit?\",\n        actual_output=\"You have 30 days for full refund\"\n    )\n    assert_test(test_case, [metric])\n\n\nRun with: deepeval test run test_chatbot.py\n\nEnvironment setup\n\nDeepEval automatically loads .env.local then .env:\n\n# .env\nOPENAI_API_KEY=\"sk-...\"\n\nCore Workflows\nRAG Evaluation\n\nEvaluate both retrieval and generation phases:\n\nfrom deepeval.metrics import (\n    ContextualPrecisionMetric,\n    ContextualRecallMetric,\n    ContextualRelevancyMetric,\n    AnswerRelevancyMetric,\n    FaithfulnessMetric\n)\n\n# Retrieval metrics\ncontextual_precision = ContextualPrecisionMetric(threshold=0.7)\ncontextual_recall = ContextualRecallMetric(threshold=0.7)\ncontextual_relevancy = ContextualRelevancyMetric(threshold=0.7)\n\n# Generation metrics\nanswer_relevancy = AnswerRelevancyMetric(threshold=0.7)\nfaithfulness = FaithfulnessMetric(threshold=0.8)\n\ntest_case = LLMTestCase(\n    input=\"What are the side effects of aspirin?\",\n    actual_output=\"Common side effects include stomach upset and nausea.\",\n    expected_output=\"Aspirin side effects include gastrointestinal issues.\",\n    retrieval_context=[\n        \"Aspirin common side effects: stomach upset, nausea, vomiting.\",\n        \"Serious aspirin side effects: gastrointestinal bleeding.\",\n    ]\n)\n\nevaluate(test_cases=[test_case], metrics=[\n    contextual_precision, contextual_recall, contextual_relevancy,\n    answer_relevancy, faithfulness\n])\n\n\nComponent-level tracing:\n\nfrom deepeval.tracing import observe, update_current_span\n\n@observe(metrics=[contextual_relevancy])\ndef retriever(query: str):\n    chunks = your_vector_db.search(query)\n    update_current_span(\n        test_case=LLMTestCase(input=query, retrieval_context=chunks)\n    )\n    return chunks\n\n@observe(metrics=[answer_relevancy, faithfulness])\ndef generator(query: str, chunks: list):\n    response = your_llm.generate(query, chunks)\n    update_current_span(\n        test_case=LLMTestCase(\n            input=query,\n            actual_output=response,\n            retrieval_context=chunks\n        )\n    )\n    return response\n\n@observe\ndef rag_pipeline(query: str):\n    chunks = retriever(query)\n    return generator(query, chunks)\n\nConversational AI Evaluation\n\nTest multi-turn dialogues:\n\nfrom deepeval.test_case import Turn, ConversationalTestCase\nfrom deepeval.metrics import (\n    RoleAdherenceMetric,\n    KnowledgeRetentionMetric,\n    ConversationCompletenessMetric,\n    TurnRelevancyMetric\n)\n\nconvo_test_case = ConversationalTestCase(\n    chatbot_role=\"professional, empathetic medical assistant\",\n    turns=[\n        Turn(role=\"user\", content=\"I have a persistent cough\"),\n        Turn(role=\"assistant\", content=\"How long have you had this cough?\"),\n        Turn(role=\"user\", content=\"About a week now\"),\n        Turn(role=\"assistant\", content=\"A week-long cough should be evaluated.\")\n    ]\n)\n\nmetrics = [\n    RoleAdherenceMetric(threshold=0.7),\n    KnowledgeRetentionMetric(threshold=0.7),\n    ConversationCompletenessMetric(threshold=0.6),\n    TurnRelevancyMetric(threshold=0.7)\n]\n\nevaluate(test_cases=[convo_test_case], metrics=metrics)\n\nAgent Evaluation\n\nTest tool usage and task completion:\n\nfrom deepeval.test_case import ToolCall\nfrom deepeval.metrics import (\n    TaskCompletionMetric,\n    ToolUseMetric,\n    ArgumentCorrectnessMetric\n)\n\nagent_test_case = ConversationalTestCase(\n    turns=[\n        Turn(role=\"user\", content=\"When did Trump first raise tariffs?\"),\n        Turn(\n            role=\"assistant\",\n            content=\"Let me search for that information.\",\n            tools_called=[\n                ToolCall(\n                    name=\"WebSearch\",\n                    arguments={\"query\": \"Trump first raised tariffs year\"}\n                )\n            ]\n        ),\n        Turn(role=\"assistant\", content=\"Trump first raised tariffs in 2018.\")\n    ]\n)\n\nevaluate(\n    test_cases=[agent_test_case],\n    metrics=[\n        TaskCompletionMetric(threshold=0.7),\n        ToolUseMetric(threshold=0.7),\n        ArgumentCorrectnessMetric(threshold=0.7)\n    ]\n)\n\nSafety Evaluation\n\nCheck for harmful content:\n\nfrom deepeval.metrics import (\n    ToxicityMetric,\n    BiasMetric,\n    PIILeakageMetric,\n    HallucinationMetric\n)\n\ndef safety_gate(output: str, input: str) -> tuple[bool, list]:\n    \"\"\"Returns (passed, reasons) tuple\"\"\"\n    test_case = LLMTestCase(input=input, actual_output=output)\n\n    safety_metrics = [\n        ToxicityMetric(threshold=0.5),\n        BiasMetric(threshold=0.5),\n        PIILeakageMetric(threshold=0.5)\n    ]\n\n    failures = []\n    for metric in safety_metrics:\n        metric.measure(test_case)\n        if not metric.is_successful():\n            failures.append(f\"{metric.name}: {metric.reason}\")\n\n    return len(failures) == 0, failures\n\nMetric Selection Guide\nRAG Metrics\n\nRetrieval Phase:\n\nContextualPrecisionMetric - Relevant chunks ranked higher than irrelevant ones\nContextualRecallMetric - All necessary information retrieved\nContextualRelevancyMetric - Retrieved chunks relevant to input\n\nGeneration Phase:\n\nAnswerRelevancyMetric - Output addresses the input query\nFaithfulnessMetric - Output grounded in retrieval context\nConversational Metrics\nTurnRelevancyMetric - Each turn relevant to conversation\nKnowledgeRetentionMetric - Information retained across turns\nConversationCompletenessMetric - All aspects addressed\nRoleAdherenceMetric - Chatbot maintains assigned role\nTopicAdherenceMetric - Conversation stays on topic\nAgent Metrics\nTaskCompletionMetric - Task successfully completed\nToolUseMetric - Correct tools selected\nArgumentCorrectnessMetric - Tool arguments correct\nMCPUseMetric - MCP correctly used\nSafety Metrics\nToxicityMetric - Harmful content detection\nBiasMetric - Biased outputs identification\nHallucinationMetric - Fabricated information\nPIILeakageMetric - Personal information leakage\nCustom Metrics\n\nG-Eval (LLM-based):\n\nfrom deepeval.metrics import GEval\nfrom deepeval.test_case import LLMTestCaseParams\n\ncustom_metric = GEval(\n    name=\"Professional Tone\",\n    criteria=\"Determine if response maintains professional, empathetic tone\",\n    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],\n    threshold=0.7,\n    model=\"anthropic-claude-sonnet-4-5\"\n)\n\n\nBaseMetric subclass:\n\nSee references/custom_metrics.md for complete guide on creating custom metrics with BaseMetric subclassing and deterministic scorers (ROUGE, BLEU, BERTScore).\n\nConfiguration\nLLM Provider Setup\n\nDeepEval supports OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, and 100+ providers via LiteLLM. Anthropic models are preferred.\n\nCLI configuration (global):\n\ndeepeval set-azure-openai --openai-endpoint=... --openai-api-key=... --deployment-name=...\ndeepeval set-ollama deepseek-r1:1.5b\n\n\nPython configuration (per-metric):\n\nfrom deepeval.models import AnthropicModel, OllamaModel\n\nanthropic_model = AnthropicModel(\n    model_id=settings.anthropic_model_id,\n    client_args={\"api_key\": settings.anthropic_api_key},\n    temperature=settings.agent_temperature\n)\n\nmetric = AnswerRelevancyMetric(model=anthropic_model)\n\n\nSee references/model_providers.md for complete provider configuration guide.\n\nPerformance Optimisation\n\nAsync mode is enabled by default. Configure with AsyncConfig and CacheConfig:\n\nfrom deepeval import evaluate, AsyncConfig, CacheConfig\n\nevaluate(\n    test_cases=[...],\n    metrics=[...],\n    async_config=AsyncConfig(\n        run_async=True,\n        max_concurrent=20,    # Reduce if rate limited\n        throttle_value=0      # Delay between test cases (seconds)\n    ),\n    cache_config=CacheConfig(\n        use_cache=True,       # Read from cache\n        write_cache=True      # Write to cache\n    )\n)\n\n\nCLI parallelisation:\n\ndeepeval test run -n 4 -c -i  # 4 processes, cached, ignore errors\n\n\nBest practices:\n\nLimit to 5 metrics maximum (2-3 generic + 1-2 custom)\nUse the latest available Anthropic Claude Sonnet or Haiku models\nReduce max_concurrent to 5 if hitting rate limits\nUse evaluate() function over individual measure() calls\n\nSee references/async_performance.md for detailed performance optimisation guide.\n\nDataset Management\nLoading datasets\nfrom deepeval.dataset import EvaluationDataset, Golden\n\ndataset = EvaluationDataset()\n\n# From CSV\ndataset.add_goldens_from_csv_file(\n    file_path=\"./test_data.csv\",\n    input_col_name=\"question\",\n    expected_output_col_name=\"answer\",\n    context_col_name=\"context\",\n    context_col_delimiter=\"|\"\n)\n\n# From JSON\ndataset.add_goldens_from_json_file(\n    file_path=\"./test_data.json\",\n    input_key_name=\"query\",\n    expected_output_key_name=\"response\"\n)\n\nSynthetic generation\nfrom deepeval.synthesizer import Synthesizer\n\nsynthesizer = Synthesizer()\n\n# From documents\ngoldens = synthesizer.generate_goldens_from_docs(\n    document_paths=[\"./docs/knowledge_base.pdf\"],\n    max_goldens_per_document=10,\n    evolution_types=[\"REASONING\", \"MULTICONTEXT\", \"COMPARATIVE\"]\n)\n\n# From scratch\ngoldens = synthesizer.generate_goldens_from_scratch(\n    subject=\"customer support for SaaS product\",\n    task=\"answer user questions about billing\",\n    max_goldens=20\n)\n\n\nEvolution types: REASONING, MULTICONTEXT, CONCRETISING, CONSTRAINED, COMPARATIVE, HYPOTHETICAL, IN_BREADTH\n\nSee references/dataset_management.md for complete dataset guide including versioning and cloud integration.\n\nTest Case Types\nSingle-turn (LLMTestCase)\nfrom deepeval.test_case import LLMTestCase\n\ntest_case = LLMTestCase(\n    input=\"What if these shoes don't fit?\",\n    actual_output=\"You have 30 days for full refund\",\n    expected_output=\"We offer 30-day full refund\",\n    retrieval_context=[\"All customers eligible for 30 day refund\"],\n    tools_called=[ToolCall(name=\"...\", arguments={\"...\": \"...\"})]\n)\n\nMulti-turn (ConversationalTestCase)\nfrom deepeval.test_case import Turn, ConversationalTestCase\n\nconvo_test_case = ConversationalTestCase(\n    chatbot_role=\"helpful customer service agent\",\n    turns=[\n        Turn(role=\"user\", content=\"I need help with my order\"),\n        Turn(role=\"assistant\", content=\"I'd be happy to help\"),\n        Turn(role=\"user\", content=\"It hasn't arrived yet\")\n    ]\n)\n\nMultimodal (MLLMTestCase)\nfrom deepeval.test_case import MLLMTestCase, MLLMImage\n\nm_test_case = MLLMTestCase(\n    input=[\"Describe this image\", MLLMImage(url=\"./photo.png\", local=True)],\n    actual_output=[\"A red bicycle leaning against a wall\"]\n)\n\nCI/CD Integration\n# .github/workflows/test.yml\nname: LLM Tests\non: [push, pull_request]\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v5\n      - name: Install dependencies\n        run: pip install deepeval\n      - name: Run evaluations\n        env:\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n        run: deepeval test run tests/\n\nReferences\n\nDetailed implementation guides:\n\nreferences/model_providers.md - Complete guide for configuring OpenAI, Anthropic, Gemini, Bedrock, and local models. Includes provider-specific considerations, cost analysis, and troubleshooting.\n\nreferences/custom_metrics.md - Complete guide for creating custom metrics by subclassing BaseMetric. Includes deterministic scorers (ROUGE, BLEU, BERTScore) and LLM-based evaluation patterns.\n\nreferences/async_performance.md - Complete guide for optimising evaluation performance with async mode, caching, concurrency tuning, and rate limit handling.\n\nreferences/dataset_management.md - Complete guide for dataset loading, saving, synthetic generation, versioning, and cloud integration with Confident AI.\n\nBest Practices\nMetric Selection\nMatch metrics to use case (RAG systems need retrieval + generation metrics)\nStart with 2-3 essential metrics, expand as needed\nUse appropriate thresholds (0.7-0.8 for production, 0.5-0.6 for development)\nCombine complementary metrics (answer relevancy + faithfulness)\nTest Case Design\nCreate representative examples covering common queries and edge cases\nInclude context when needed (retrieval_context for RAG, expected_output for G-Eval)\nUse datasets for scale testing\nVersion test cases over time\nEvaluation Workflow\nComponent-level first - Use @observe for individual parts\nEnd-to-end validation before deployment\nAutomate in CI/CD with deepeval test run\nTrack results over time with Confident AI cloud\nTesting Anti-Patterns\n\nAvoid:\n\nTesting only happy paths\nUsing unrealistic inputs\nIgnoring metric reasons\nSetting thresholds too high initially\nRunning full test suite on every change\n\nDo:\n\nTest edge cases and failure modes\nUse real user queries as test inputs\nRead and analyse metric reasons\nAdjust thresholds based on empirical results\nUse component-level tests during development\nSeparate config and eval content from code\nWeekly Installs\n74\nRepository\nsammcj/agentic-coding\nGitHub Stars\n125\nFirst Seen\nJan 30, 2026\nSecurity Audits\nGen Agent Trust HubPass\nSocketPass\nSnykFail"}}}}