1 line
14 KiB
JSON
1 line
14 KiB
JSON
{"repo": "alexklashitsky/ai-powered-anomaly-detection-in-github-actions-ci-cd-pipelines", "files": {"QUICK_REFERENCE.md": {"3a237c428acc8b662b1ea9768200bed797e2e44c": {"t": 1785596448.4506001, "n": 13052, "h": "1cbb74704c5b99ed", "c": "# Quick Reference Guide - AI Anomaly Detection System\n\n## 🚀 Common Operations\n\n### Initial Setup (One-time)\n\n```powershell\n# 1. Login to Azure\naz login\n\n# 2. Set variables\n$APP_NAME = \"github-actions-flask-app\"\n$SUBSCRIPTION_ID = (az account show --query id -o tsv)\n$TENANT_ID = (az account show --query tenantId -o tsv)\n\n# 3. Create App Registration\naz ad app create --display-name $APP_NAME\n$APP_ID = (az ad app list --display-name $APP_NAME --query \"[0].appId\" -o tsv)\n\n# 4. Create Service Principal\naz ad sp create --id $APP_ID\n\n# 5. Assign Contributor role\naz role assignment create `\n --assignee $APP_ID `\n --role \"Contributor\" `\n --scope \"/subscriptions/$SUBSCRIPTION_ID\"\n\n# 6. Create Terraform state storage\naz group create --name terraform-state-rg --location westeurope\n\naz storage account create `\n --name tfstatedevopsdelk `\n --resource-group terraform-state-rg `\n --location westeurope `\n --sku Standard_LRS\n\n$ACCOUNT_KEY = (az storage account keys list `\n --resource-group terraform-state-rg `\n --account-name tfstatedevopsdelk `\n --query '[0].value' -o tsv)\n\naz storage container create `\n --name tfstate `\n --account-name tfstatedevopsdelk `\n --account-key $ACCOUNT_KEY\n\n# 7. Configure GitHub Secrets (manually in GitHub UI)\n# Add these secrets to your repository:\n# - AZURE_CLIENT_ID: $APP_ID\n# - AZURE_TENANT_ID: $TENANT_ID\n# - AZURE_SUBSCRIPTION_ID: $SUBSCRIPTION_ID\n```\n\n### Deploy Infrastructure\n\n```powershell\n# Via GitHub Actions (Recommended)\ngit add terraform/\ngit commit -m \"Update infrastructure\"\ngit push origin main\n\n# Or locally\ncd terraform\nterraform init `\n -backend-config=\"resource_group_name=terraform-state-rg\" `\n -backend-config=\"storage_account_name=tfstatedevopsdelk\" `\n -backend-config=\"container_name=tfstate\" `\n -backend-config=\"key=flask-app.terraform.tfstate\"\n\nterraform plan\nterraform apply\n```\n\n### Train ML Model\n\n```powershell\n# Via GitHub Actions (Recommended)\ngh workflow run train-ml-model.yml\n\n# Or locally with Azure authentication\n$env:AZURE_CLIENT_ID = \"your-client-id\"\n$env:AZURE_TENANT_ID = \"your-tenant-id\"\n$env:AZURE_SUBSCRIPTION_ID = \"your-subscription-id\"\n\npython train_anomaly_detection.py\n```\n\n### Deploy Azure Function\n\n```powershell\n# Via GitHub Actions (Recommended)\ngh workflow run deploy-function.yml\n\n# Or locally\ncd .\nfunc azure functionapp publish <function-app-name>\n```\n\n### Run Full CI/CD Pipeline\n\n```powershell\n# Simply push to main branch\ngit add .\ngit commit -m \"Deploy application\"\ngit push origin main\n\n# This will automatically:\n# 1. Build and test the app\n# 2. Deploy to Azure App Service\n# 3. Run anomaly detection monitoring\n```\n\n## 🔍 Monitoring & Debugging\n\n### Check Pipeline Metrics\n\n```powershell\n# View latest GitHub Actions run\ngh run list --workflow=ci-cd.yml --limit 1\n\n# View run details\ngh run view <run-id>\n\n# Download anomaly detection artifacts\ngh run download <run-id> --name anomaly-detection-report\n```\n\n### View Azure Function Logs\n\n```powershell\n# Stream logs in real-time\naz webapp log tail --name <function-app-name> --resource-group flask-app-rg\n\n# Download recent logs\naz webapp log download --name <function-app-name> --resource-group flask-app-rg\n```\n\n### Query Application Insights\n\n```powershell\n# Open Application Insights in browser\naz portal appinsights show --name <workspace-name>-insights\n\n# Query with Azure CLI (requires app insights extension)\naz monitor app-insights query `\n --app <app-insights-id> `\n --analytics-query \"traces | where message contains 'anomaly' | take 10\"\n```\n\n### Check Azure ML Workspace\n\n```powershell\n# List registered models\naz ml model list --workspace-name <workspace-name> --resource-group flask-app-rg\n\n# View model details\naz ml model show --name pipeline-anomaly-detector --workspace-name <workspace-name> --resource-group flask-app-rg\n```\n\n## 🛠️ Troubleshooting\n\n### Anomaly Detection Not Running\n\n```powershell\n# 1. Check if Azure Function is deployed\naz functionapp list --resource-group flask-app-rg --query \"[].name\"\n\n# 2. Verify function is running\naz functionapp show --name <function-app-name> --resource-group flask-app-rg --query state\n\n# 3. Check function app settings\naz functionapp config appsettings list --name <function-app-name> --resource-group flask-app-rg\n\n# 4. Test function manually\ncurl -X POST \"https://<function-app-name>.azurewebsites.net/api/detect_anomalies\" `\n -H \"Content-Type: application/json\" `\n -d '{\"data\":[{\"build_id\":\"test\",\"duration\":300,\"failure_rate\":0.0}]}'\n```\n\n### GitHub Actions Workflow Failing\n\n```powershell\n# 1. Check OIDC authentication\n# Verify federated credentials in Azure Portal:\n# Azure AD → App registrations → Your app → Certificates & secrets → Federated credentials\n\n# 2. Verify GitHub secrets are set\ngh secret list\n\n# 3. Check workflow logs\ngh run view <run-id> --log\n\n# 4. Validate Terraform state access\naz storage blob list `\n --account-name tfstatedevopsdelk `\n --container-name tfstate `\n --auth-mode key\n```\n\n### ML Model Not Found\n\n```powershell\n# 1. Check if model is registered\naz ml model list --workspace-name <workspace-name> --resource-group flask-app-rg\n\n# 2. Retrain and register model\npython train_anomaly_detection.py\n\n# 3. Verify model in Azure ML Studio\n# Browse to: https://ml.azure.com → Models\n```\n\n### Alerts Not Sending\n\n```powershell\n# 1. Check Azure Function environment variables\naz functionapp config appsettings show `\n --name <function-app-name> `\n --resource-group flask-app-rg `\n --settings TEAMS_WEBHOOK_URL SENDGRID_API_KEY\n\n# 2. Test Teams webhook\nInvoke-WebRequest -Uri \"YOUR_TEAMS_WEBHOOK_URL\" `\n -Method POST `\n -ContentType \"application/json\" `\n -Body '{\"text\":\"Test message\"}'\n\n# 3. Verify SendGrid API key\ncurl -X GET \"https://api.sendgrid.com/v3/scopes\" `\n -H \"Authorization: Bearer YOUR_SENDGRID_KEY\"\n```\n\n## 📊 Useful Queries\n\n### GitHub Issues for Anomalies\n\n```powershell\n# List all anomaly issues\ngh issue list --label anomaly\n\n# View specific anomaly issue\ngh issue view <issue-number>\n\n# Close resolved anomaly\ngh issue close <issue-number> --comment \"Issue resolved\"\n```\n\n### Terraform State Management\n\n```powershell\n# View current state\ncd terraform\nterraform show\n\n# List all resources\nterraform state list\n\n# View specific resource\nterraform state show azurerm_machine_learning_workspace.ml_workspace\n\n# Refresh state from Azure\nterraform refresh\n```\n\n### Azure Resource Management\n\n```powershell\n# List all resources in resource group\naz resource list --resource-group flask-app-rg --output table\n\n# Check App Service status\naz webapp show --name <app-service-name> --resource-group flask-app-rg --query state\n\n# View App Service URL\naz webapp show --name <app-service-name> --resource-group flask-app-rg --query defaultHostName\n\n# Restart services\naz webapp restart --name <app-service-name> --resource-group flask-app-rg\naz functionapp restart --name <function-app-name> --resource-group flask-app-rg\n```\n\n## 🎯 Testing\n\n### Test Flask Application Locally\n\n```powershell\n# Install dependencies\npip install -r requirements.txt\n\n# Run tests\npytest --cov=. --cov-report=term\n\n# Start application\npython app.py\n\n# Test health endpoint\ncurl http://localhost:8000/health\n```\n\n### Test Azure Function Locally\n\n```powershell\n# Install Azure Functions Core Tools\n# Download from: https://docs.microsoft.com/azure/azure-functions/functions-run-local\n\n# Install dependencies\npip install -r azure_function_requirements.txt\n\n# Start function locally\nfunc start\n\n# Test HTTP trigger\nInvoke-WebRequest -Uri \"http://localhost:7071/api/detect_anomalies\" `\n -Method POST `\n -ContentType \"application/json\" `\n -Body '{\"data\":[{\"build_id\":\"test\",\"duration\":300,\"failure_rate\":0.0}]}'\n```\n\n### Test ML Model Locally\n\n```powershell\n# Install dependencies\npip install -r requirements.txt\npip install scikit-learn pandas joblib azure-ai-ml\n\n# Run training script\npython train_anomaly_detection.py\n\n# Test scoring\npython -c \"\nimport joblib\nimport numpy as np\n\n# Load model (if saved locally)\nmodel = joblib.load('model/pipeline_anomaly_detector.pkl')\n\n# Test prediction\nX = np.array([[300, 0.0]]) # duration, failure_rate\nprediction = model.predict(X)\nprint(f'Prediction: {prediction[0]}') # -1 = anomaly, 1 = normal\n\"\n```\n\n## 🔧 Configuration Updates\n\n### Update Terraform Variables\n\n```powershell\n# Edit terraform.tfvars\nnotepad terraform\\terraform.tfvars\n\n# Apply changes\ncd terraform\nterraform plan\nterraform apply\n```\n\n### Update Azure Function Settings\n\n```powershell\n# Set single setting\naz functionapp config appsettings set `\n --name <function-app-name> `\n --resource-group flask-app-rg `\n --settings \"TEAMS_WEBHOOK_URL=https://outlook.office.com/webhook/...\"\n\n# Set multiple settings\naz functionapp config appsettings set `\n --name <function-app-name> `\n --resource-group flask-app-rg `\n --settings `\n \"ML_MODEL_NAME=pipeline-anomaly-detector\" `\n \"TEAMS_WEBHOOK_URL=https://outlook.office.com/webhook/...\" `\n \"SENDGRID_API_KEY=SG.xxx\"\n```\n\n### Update GitHub Secrets\n\n```powershell\n# Update secret\ngh secret set AZURE_CLIENT_ID --body \"new-client-id\"\n\n# List all secrets\ngh secret list\n\n# Delete secret\ngh secret delete SECRET_NAME\n```\n\n## 📈 Performance Optimization\n\n### Optimize Terraform Apply Time\n\n```powershell\n# Use targeted apply for specific resources\nterraform apply -target=azurerm_app_service.app_service\n\n# Use parallelism flag\nterraform apply -parallelism=10\n```\n\n### Optimize Azure Function Performance\n\n```powershell\n# Scale up function app\naz functionapp plan update `\n --name <plan-name> `\n --resource-group flask-app-rg `\n --sku P1V2\n\n# Enable always on\naz functionapp config set `\n --name <function-app-name> `\n --resource-group flask-app-rg `\n --always-on true\n```\n\n### Optimize ML Model Training\n\n```python\n# In train_anomaly_detection.py\n# Adjust these parameters for faster training:\n\nmodel = IsolationForest(\n n_estimators=50, # Reduce from 100\n max_samples=256, # Limit samples per tree\n n_jobs=-1, # Use all CPU cores\n random_state=42\n)\n```\n\n## 🔄 Maintenance Tasks\n\n### Monthly Checklist\n\n```powershell\n# 1. Review anomaly detection accuracy\ngh issue list --label anomaly --state closed --limit 20\n\n# 2. Retrain ML model with new data\ngh workflow run train-ml-model.yml\n\n# 3. Update dependencies\npip list --outdated\npip install --upgrade -r requirements.txt\n\n# 4. Review Azure costs\naz consumption usage list --start-date 2024-01-01 --end-date 2024-01-31\n\n# 5. Clean up old artifacts\ngh run list --status completed --limit 100 | ForEach-Object { gh run delete $_.id }\n```\n\n### Backup & Recovery\n\n```powershell\n# Backup Terraform state\naz storage blob download `\n --account-name tfstatedevopsdelk `\n --container-name tfstate `\n --name flask-app.terraform.tfstate `\n --file backup-terraform.tfstate\n\n# Backup ML models (download from Azure ML)\naz ml model download `\n --name pipeline-anomaly-detector `\n --version 1 `\n --workspace-name <workspace-name> `\n --resource-group flask-app-rg\n\n# Restore from backup\naz storage blob upload `\n --account-name tfstatedevopsdelk `\n --container-name tfstate `\n --name flask-app.terraform.tfstate `\n --file backup-terraform.tfstate `\n --overwrite\n```\n\n## 📚 Additional Resources\n\n- [Main README](./README.md) - Project overview\n- [AI Overview](./AI_ANOMALY_DETECTION_OVERVIEW.md) - Detailed system architecture\n- [ML Guide](./ML_ANOMALY_DETECTION_GUIDE.md) - ML model documentation\n- [Function Guide](./AZURE_FUNCTION_README.md) - Azure Function details\n\n## 💡 Pro Tips\n\n1. **Use aliases for common commands:**\n ```powershell\n function tf { terraform $args }\n function tfp { terraform plan }\n function tfa { terraform apply -auto-approve }\n ```\n\n2. **Set up auto-completion:**\n ```powershell\n # Add to PowerShell profile\n Register-ArgumentCompleter -Native -CommandName gh -ScriptBlock {\n param($wordToComplete, $commandAst, $cursorPosition)\n gh completion -s powershell | Out-String | Invoke-Expression\n }\n ```\n\n3. **Monitor costs:**\n ```powershell\n # Check daily costs\n az consumption usage list `\n --start-date (Get-Date).AddDays(-1).ToString(\"yyyy-MM-dd\") `\n --end-date (Get-Date).ToString(\"yyyy-MM-dd\")\n ```\n\n4. **Quick health check:**\n ```powershell\n # Create a health check script\n function Check-SystemHealth {\n Write-Host \"🏥 System Health Check\" -ForegroundColor Cyan\n \n Write-Host \"`n1. App Service Status:\" -ForegroundColor Yellow\n az webapp show --name <app-service-name> --resource-group flask-app-rg --query state\n \n Write-Host \"`n2. Function App Status:\" -ForegroundColor Yellow\n az functionapp show --name <function-app-name> --resource-group flask-app-rg --query state\n \n Write-Host \"`n3. Latest Anomaly Detection:\" -ForegroundColor Yellow\n gh issue list --label anomaly --limit 1\n \n Write-Host \"`n4. Recent Pipeline Runs:\" -ForegroundColor Yellow\n gh run list --workflow=ci-cd.yml --limit 3\n }\n ```\n\n---\n\n**Need help?** Open an issue on GitHub or check the detailed documentation files.\n"}}}} |