Play 17
AI Observability
Medium🔧 Skeleton
Monitor AI workloads with KQL, quality alerts, and interactive workbooks.
Instrument your AI applications with Application Insights, query logs with KQL in Log Analytics, set up quality alerts (latency, error rate, token usage, groundedness scores), and build interactive Azure Workbooks dashboards. Distributed tracing tracks requests across AI Search → OpenAI → your app.
Architecture Pattern
KQL dashboards, quality metrics, alerting, APM, distributed tracing
Azure Services
Application InsightsLog AnalyticsAzure MonitorWorkbooks
DevKit (.github Agentic OS)
- agent.md — root orchestrator with builder→reviewer→tuner handoffs
- 3 agents — Observability Builder (gpt-4o), Reviewer (gpt-4o-mini), Tuner (gpt-4o-mini)
- 3 skills — deploy (128 lines), evaluate (110 lines), tune (120 lines)
- 4 prompts — /deploy, /test, /review, /evaluate with agent routing
- .vscode/mcp.json — FrootAI MCP with App Insights + Log Analytics inputs + envFile
TuneKit (AI Config)
- config/monitoring.json — KQL queries, alert thresholds, dashboards
- config/metrics.json — quality KPIs
- infra/ — workbook templates
Tuning Parameters
KQL queriesAlert thresholdsQuality metrics definitionsDashboard layouts
Estimated Cost
Dev/Test
$30–80/mo
Production
$200–1K/mo