FAI learning resource

FAI End-to-End Workshop

Build a complete solution play from idea to deployment — 8 steps, 45 minutes, the full FAI lifecycle.

L15·30 min read·High

Workshop Overview

This workshop walks you through building a complete FAI solution play from scratch. By the end, you will have a production-ready play with a manifest, wired primitives, Azure infrastructure, evaluation pipeline, and marketplace listing. Estimated total time: 45 minutes.

Step	Task	Time
1	Choose use case	3 min
2	Create fai-manifest.json	5 min
3	Initialize DevKit primitives	8 min
4	Configure TuneKit	5 min
5	Set model parameters	3 min
6	Deploy to Azure	10 min
7	Run evaluation	6 min
8	Ship to marketplace	5 min

Step 1: Choose Your Use Case (3 min)

Pick a scenario for your solution play. For this workshop, we will build an IT Helpdesk Knowledge Bot — a RAG-powered chatbot that answers IT support questions from internal documentation.

Use Case

IT staff ask questions like “How do I reset MFA?” or “What's the VPN setup for remote work?” The bot retrieves answers from the IT knowledge base.

Architecture

Azure AI Search (vector index) + Azure OpenAI (GPT-4o) + Azure Container Apps (API host). Classic RAG pattern.

WAF Pillars

Security (Managed Identity, no API keys), Reliability (retry + circuit breaker), Cost Optimization (model routing).

Step 2: Create fai-manifest.json (5 min)

The manifest is the heart of every play — it declares context, primitives, infrastructure, and guardrails:

solution-plays/69-it-helpdesk-bot/fai-manifest.json

{  "play": "69-it-helpdesk-bot",  "version": "1.0.0",  "description": "RAG-powered IT helpdesk knowledge bot",  "context": {    "knowledge": ["R2", "O4"],    "waf": [      "security",      "reliability",      "cost-optimization"    ],    "compatible-plays": ["01-enterprise-rag"]  },  "primitives": {    "agents": [      "fai-rag-architect",      "fai-azure-ai-search-expert",      "fai-code-reviewer"    ],    "instructions": [      "typescript-waf",      "python-waf"    ],    "skills": [      "fai-play-initializer",      "fai-rag-evaluator"    ]  },  "infrastructure": {    "services": [      "azure-openai",      "azure-ai-search",      "azure-container-apps",      "azure-key-vault"    ],    "region": "eastus2",    "iac": "bicep"  },  "toolkit": {    "tunekit": {      "openai": "config/openai.json",      "guardrails": "config/guardrails.json"    }  },  "guardrails": {    "groundedness": 0.8,    "relevance": 0.7,    "safety": 0.9  }}

Step 3: Initialize DevKit Primitives (8 min)

Create the builder/reviewer/tuner agent chain and supporting primitives for your play:

Scaffold Agents

# Create the play directorymkdir -p solution-plays/69-it-helpdesk-bot
# Scaffold domain-specific agentsnode scripts/scaffold-primitive.js agent# Name: fai-it-helpdesk-builder# Description: "Builds IT helpdesk RAG pipeline components"# WAF: security, reliability# Plays: 69-it-helpdesk-bot
node scripts/scaffold-primitive.js agent# Name: fai-it-helpdesk-reviewer# Description: "Reviews IT helpdesk bot for quality and security"# WAF: security, responsible-ai# Plays: 69-it-helpdesk-bot

Create Play-Specific Instruction

# .github/instructions/it-helpdesk-rag.instructions.md---description: "IT helpdesk RAG patterns — chunking, search config, grounding"applyTo: "solution-plays/69-it-helpdesk-bot/**/*.{ts,py,json}"waf: [security, reliability]---
## IT Helpdesk RAG Guidelines
- Chunk IT documentation by section headers (H2/H3 boundaries)- Use hybrid search: keyword for error codes, vector for concepts- System prompt must include: "Answer ONLY from the provided context"- PII filter: strip employee IDs, emails, phone numbers from responses- Max response length: 500 tokens (concise IT answers)

Validate All Primitives

# Verify everything is wired correctlynode scripts/validate-primitives.js --verbose
# Expected: 0 errors, 0 warnings

Step 4: Configure TuneKit (5 min)

TuneKit holds the runtime configuration — model parameters and guardrail thresholds:

config/guardrails.json

{  "content_safety": {    "enabled": true,    "categories": {      "hate":       { "threshold": "medium", "action": "block" },      "violence":   { "threshold": "medium", "action": "block" },      "self_harm":  { "threshold": "low",    "action": "block" },      "sexual":     { "threshold": "medium", "action": "block" }    }  },  "pii_detection": {    "enabled": true,    "categories": ["email", "phone", "ssn", "employee_id"],    "action": "redact"  },  "prompt_injection": {    "enabled": true,    "detection_model": "azure-content-safety",    "action": "block_and_log"  },  "evaluation": {    "metrics": {      "groundedness": { "threshold": 4.0 },      "relevance":    { "threshold": 3.5 },      "coherence":    { "threshold": 4.0 },      "safety":       { "threshold": 1.0 }    }  }}

Step 5: Set Model Parameters (3 min)

Configure the OpenAI model settings for your IT helpdesk scenario — low temperature for factual answers:

config/openai.json

{  "model": "gpt-4o",  "api_version": "2024-12-01-preview",  "temperature": 0.1,  "max_tokens": 512,  "top_p": 0.95,  "frequency_penalty": 0,  "presence_penalty": 0,  "system_prompt": "You are an IT helpdesk assistant. Answer questions using ONLY the provided context from the IT knowledge base. If the context does not contain the answer, respond: 'I don't have information about that. Please contact the IT helpdesk at ext. 4357.' Never speculate or provide information not in the context.",  "routing": {    "simple_queries": "gpt-4o-mini",    "complex_queries": "gpt-4o",    "complexity_threshold": 0.6  }}

Key decisions: temperature: 0.1 keeps answers factual, max_tokens: 512 keeps IT answers concise, and model routing saves ~60% on costs by sending simple queries to GPT-4o-mini.

Step 6: Deploy to Azure (10 min)

Deploy the infrastructure and application using Bicep and the Azure CLI:

Deploy Infrastructure

# Set variables$RG = "rg-it-helpdesk-bot"$LOCATION = "eastus2"
# Create resource groupaz group create --name $RG --location $LOCATION
# Deploy Bicep template (creates OpenAI, Search, Container Apps, Key Vault)az deployment group create \  --resource-group $RG \  --template-file infra/main.bicep \  --parameters location=$LOCATION \               projectName="it-helpdesk"
# Verify deploymentaz deployment group show \  --resource-group $RG \  --name "main" \  --query "properties.provisioningState"

Index Documents

# Upload IT knowledge base documents to the search indexpython scripts/index-documents.py \  --source "./data/it-knowledge-base/" \  --index-name "it-helpdesk-index" \  --chunk-strategy "section-headers" \  --embedding-model "text-embedding-3-large"
# Expected output:# Indexed 342 documents → 1,284 chunks# Embedding dimensions: 3072# Index size: 48 MB

Deploy Application

# Build and deploy the containeraz containerapp up \  --name "it-helpdesk-api" \  --resource-group $RG \  --source . \  --ingress external \  --target-port 8080
# Verify health endpointcurl https://it-helpdesk-api.<region>.azurecontainerapps.io/health# Expected: { "status": "healthy" }

Step 7: Run Evaluation (6 min)

Run the FAI Engine evaluation pipeline against your deployed play:

Run Evaluation

# Run full evaluation suitenode engine/index.js \  solution-plays/69-it-helpdesk-bot/fai-manifest.json \  --eval \  --test-set "evaluation/it-helpdesk-queries.jsonl" \  --output "evaluation/results.json"
# Expected output:# ┌─────────────────┬───────┬───────────┬────────┐# │ Metric          │ Score │ Threshold │ Status │# ├─────────────────┼───────┼───────────┼────────┤# │ Groundedness    │ 4.3   │ 4.0       │ ✓ PASS │# │ Relevance       │ 4.1   │ 3.5       │ ✓ PASS │# │ Coherence       │ 4.5   │ 4.0       │ ✓ PASS │# │ Safety          │ 1.0   │ 1.0       │ ✓ PASS │# └─────────────────┴───────┴───────────┴────────┘# Overall: PASS — all metrics above threshold

If Evaluation Fails

Groundedness low? → Add stronger grounding instructions to system prompt. Reduce chunk size for more precise retrieval.

Relevance low? → Tune search k-value (try k=3 instead of k=5). Enable reranking. Check embedding model quality.

Safety failing? → Enable content safety filters in guardrails.json. Check for prompt injection vulnerabilities.

Step 8: Ship to Marketplace (5 min)

Package your play as a plugin and register it in the FAI Marketplace:

Create Plugin Package

# Create the plugin structuremkdir -p plugins/it-helpdesk-bot
# Create plugin.jsoncat > plugins/it-helpdesk-bot/plugin.json << 'EOF'{  "name": "it-helpdesk-bot",  "description": "RAG-powered IT helpdesk with hybrid search and model routing",  "version": "1.0.0",  "author": {    "name": "Your Name",    "url": "https://github.com/yourname"  },  "license": "MIT",  "play": "69-it-helpdesk-bot",  "primitives": {    "agents": 2,    "instructions": 1,    "skills": 2  },  "tags": ["rag", "helpdesk", "it-support", "azure"]}EOF

Generate & Validate

# Regenerate marketplace indexnode scripts/generate-marketplace.js
# Run the full build pipelinenode scripts/validate-primitives.jsnode scripts/generate-website-data.jsnode scripts/update-readme.js
# Verify plugin appearsnpx frootai list | grep "it-helpdesk"# Expected: it-helpdesk-bot  RAG-powered IT helpdesk...

Publish

# Commit and taggit add .git commit -m "feat: add play 69 — IT helpdesk knowledge bot"git tag v2.1.0
# Push (triggers CI/CD pipeline)git push origin main --tags
# The FAI Factory automatically:# 1. Validates all primitives# 2. Generates marketplace listing# 3. Publishes npm package# 4. Updates website data

Workshop Complete

You have built a complete FAI solution play from scratch. Here is what you created:

fai-manifest.jsonPlay wiring — context, primitives, infrastructure, guardrails

2 agentsBuilder and reviewer for the IT helpdesk domain

1 instructionRAG-specific patterns auto-attached to play files

openai.jsonModel config — GPT-4o with low temperature and model routing

guardrails.jsonSafety config — content safety, PII redaction, prompt injection protection

Bicep templatesInfrastructure as code — OpenAI, Search, Container Apps, Key Vault

Evaluation resultsAll quality metrics passing: groundedness 4.3, relevance 4.1

Marketplace pluginPackaged and published — discoverable via npx frootai list

Learning Hub Browse Primitives Back to FrootAI