Play 17
AI Observability
Monitor AI workloads with KQL, quality alerts, and interactive workbooks.
Instrument your AI applications with Application Insights, query logs with KQL in Log Analytics, set up quality alerts (latency, error rate, token usage, groundedness scores), and build interactive Azure Workbooks dashboards. Distributed tracing tracks requests across AI Search → OpenAI → your app.
Architecture Pattern
KQL dashboards, quality metrics, alerting, APM, distributed tracing
Azure Services
DevKit (.github Agentic OS)
- agent.md — root orchestrator with builder→reviewer→tuner handoffs
- 3 agents — Observability Builder (gpt-4o), Reviewer (gpt-4o-mini), Tuner (gpt-4o-mini)
- 3 skills — deploy (128 lines), evaluate (110 lines), tune (120 lines)
- 4 prompts — /deploy, /test, /review, /evaluate with agent routing
- .vscode/mcp.json — FrootAI MCP with App Insights + Log Analytics inputs + envFile
TuneKit (AI Config)
- config/monitoring.json — KQL queries, alert thresholds, dashboards
- config/metrics.json — quality KPIs
- infra/ — workbook templates
Tuning Parameters
Machine evidence
Certified runtime lifecycle
Maturity is calculated from contiguous, content-bound evidence. Missing or expired evidence demotes automatically; catalog claims cannot promote a play.
This play currently has design evidence only. A runnable scenario, endpoint evaluation, and build receipts are the next contiguous gates.
Repo Intelligence
v1A no-clone, revision-pinned map for agents and humans. Observed evidence is separated from inferred flow so the output stays useful without pretending to be a full call graph.