Azure AI Foundry Setup
Step-by-step guides for standing up Azure AI Foundry, organized by
{a}OS stratum. Each section shows what Foundry provides, what you
must build, and how to set it up.
Glossary — Abbreviations Used Below
- GA
- Generally Available (production-ready)
- Preview
- Public preview (not SLA-backed)
- PTU
- Provisioned Throughput Units (reserved capacity)
- CMK
- Customer-Managed Keys (encryption)
- SFT
- Supervised Fine-Tuning
- DPO
- Direct Preference Optimization
- RFT
- Reinforcement Fine-Tuning
- RAG
- Retrieval-Augmented Generation
- BYO
- Bring Your Own (storage / resources)
- MCP
- Model Context Protocol
- RBAC
- Role-Based Access Control
- APIM
- Azure API Management
- VNet
- Virtual Network (Azure networking)
- IaC
- Infrastructure as Code
- PII
- Personally Identifiable Information
- PaaS
- Platform as a Service
- FIPS
- Federal Information Processing Standards
- Entra ID
- Microsoft's identity platform (formerly Azure AD)
- Bicep
- Azure's declarative IaC language
- Rego
- Policy language for Open Policy Agent
Getting Started
Azure AI Foundry (rebranded to "Microsoft Foundry" in 2026) is
Microsoft's unified
PaaS for AI — built on
Microsoft.CognitiveServices. It bundles model hosting,
agent frameworks, content safety, evaluation, and a management
portal into a single control plane.
Two resource models coexist
The "classic" model (based on
Microsoft.MachineLearningServices) and the "new" model
(based on Microsoft.CognitiveServices) are both active.
The new model is where Microsoft invests, but the classic model has
more mature network isolation. Check which you are on:
az provider show -n Microsoft.CognitiveServices
Prerequisites:
- Azure subscription (pay-as-you-go is fine for learning)
- Azure CLI installed —
az --version
-
Python 3.10+ and
pip install azure-ai-projects azure-identity
-
Register the provider:
az provider register -n Microsoft.CognitiveServices
Foundry coverage across the {a}OS stack:
| Stratum |
Coverage |
Key Takeaway |
| L7 Experience |
Partial |
Portal + publishing, but new portal lacks
VNet support
|
| L6 Governance |
Partial |
Content Safety +
RBAC. No
approval gates, no cost governance
|
| L5 Observability |
Partial |
Tracing + evaluators. Breaks with private App Insights |
| L4 Orchestration |
Partial |
Prompt agents
GA, workflow agents preview
|
| L3 Execution |
Good |
Function calling,
MCP, Code
Interpreter all GA
|
| L2 Knowledge |
Partial |
AI Search GA, Foundry
IQ preview. No cross-agent state
|
| L1 Models |
Strong |
Model catalog,
PTU,
fine-tuning, CMK —
all GA
|
7-Day Learning Path
A structured curriculum that builds bottom-up through the strata.
Each day: ~2-4 hours. See the
full curriculum
for detailed instructions.
D1
L1 — Models
Create Foundry resource, deploy model, first inference. Posts
1-2.
D2
L2+L3 — Data & Tools
RAG pipeline, AI Search, tool calling, MCP. Posts 5, 8.
D3
L4 — Agents
Prompt agent, identity, versioning, maturity matrix. Post 7.
D4
L5 — Observability
App Insights, evaluators, CI/CD gates, private tracing gap. Post
10.
D5
L6+L7 — Governance
RBAC hardening, Content Safety, approval architecture. Posts 3,
4, 6.
D6
Cross-Cutting
Network isolation, VNet injection, APIM cost governance. Posts
5, 9.
D7
Integration
Bicep deployment, gap analysis, {a}OS classification. Posts 11,
12.
L1: Models & Infrastructure
L1
Models & Infrastructure
Is the model/runtime/compute/network foundation available and
behaving correctly?
What Foundry Provides
| Capability |
Feature |
Status |
| Model hosting |
Model catalog (11K+ models, ~50-100 that matter) |
GA |
| Deployment types |
Standard,
PTU, Global, DataZone, Batch
|
GA |
| Fine-tuning |
SFT/DPO
for GPT-4o/4.1;
RFT for o4-mini
|
GA |
| Encryption |
CMK via Azure Key
Vault (FIPS
140-2)
|
GA |
| Edge inference |
Foundry Local |
GA |
What You Build
-
Custom model training (requires Azure ML — separate service)
- Cross-region disaster recovery (manual architecture)
- Gov cloud model gap mitigation (model abstraction layer)
- Hybrid cloud patterns for classified environments
Setup Walkthrough
-
Create a Foundry resource:
az cognitiveservices account create --kind AIServices --sku
S0
- Create a project within the resource
-
Deploy a model:
az cognitiveservices account deployment create --model-name
gpt-4.1 --sku-name GlobalStandard
- Test inference via Python SDK or REST API
-
Enable CMK encryption via Key Vault if handling sensitive data
Key Decisions
| Decision |
Option A |
Option B |
Recommendation |
| Deployment type |
Pay-as-you-go |
PTU (reserved) |
Start with pay-as-you-go; PTU when you have predictable load
|
| Scope |
Global deployment |
Regional |
Regional for data residency; Global for cost optimization
|
| Model portability |
Hardcode model names |
Abstraction layer |
Always abstract — build for the model you might lose access to
|
Related: LinkedIn Post 2 (What Foundry Is), Post 6 (Models &
Money) — Learning Day 1
L2: Knowledge & Memory
L2
Knowledge & Memory
What context did the system know, retrieve, remember, or forget?
What Foundry Provides
| Capability |
Feature |
Status |
|
RAG engine
|
Foundry IQ |
Preview |
| Vector search |
Azure AI Search |
GA |
| Agent memory |
Conversation memory per agent |
GA |
| File processing |
File Search tool |
GA (no VNet) |
| Document processing |
Document Intelligence |
GA |
What You Build
- Cross-agent shared state (Cosmos DB)
- Memory compaction and lifecycle management
- Production RAG governance and access controls
- Context window management strategies
Critical: Basic vs. Standard Storage
"Basic" agent storage is Microsoft-managed multitenant. If you
handle sensitive data,
PII,
financial records, or any controlled data, you need "Standard" with
BYO storage — and you need to
know this before your first deployment, not after.
Setup Walkthrough
-
Create Azure AI Search:
az search service create --sku standard
- Configure private endpoint for AI Search
- Upload documents and create a vector index
- Connect AI Search as an agent tool
-
Configure BYO storage (Standard agent setup) —
do not use Basic for production
Related: LinkedIn Post 8 (RAG & Data) — Learning Day 2
L3: Execution & Interfaces
L3
Execution & Interfaces
What external action was invoked, against which interface, and
what happened?
What Foundry Provides
| Capability |
Feature |
Status |
VNet |
| Function calling |
Native function/tool calling |
GA |
Works |
| MCP support |
MCP server integration |
GA |
Works |
| Code execution |
Code Interpreter |
GA |
Works |
| Enterprise connectors |
1,400+ via Logic Apps |
GA |
Partial |
| Web search |
Bing Grounding |
GA |
PUBLIC |
| Browser |
Browser Automation |
Preview |
Broken |
Warning: Public Internet Tools
Bing Grounding and Web Search route over public internet even in
"isolated" environments. Block via Azure Policy on day one:
az policy assignment create --name block-bing --policy "deny Bing
grounding"
Setup Walkthrough
-
Add a function tool to your agent — test with a simple calculator
function
- Set up MCP server integration for your own services
- Test Code Interpreter with a data analysis task
- Block Bing/Web Search via Azure Policy in production
- Build your personal VNet tool compatibility checklist
Related: LinkedIn Post 5 (Network Isolation), Post 7 (Agent
Maturity) — Learning Day 2
L4: Orchestration & Decisioning
L4
Orchestration & Decisioning
What happens next, in what order, with which agent, under which
stop conditions?
What Foundry Provides
| Agent Type |
Status |
VNet |
Production Ready |
| Prompt Agents |
GA |
Yes |
Yes |
| Workflow Agents |
Preview |
Partial |
No |
| Hosted Agents |
Preview |
No |
No |
Start with Prompt Agents
They are GA and production-safe. Workflow and hosted agents are
preview — build your first value on what is stable, not what is
exciting. Complex orchestration? Use Semantic Kernel or LangGraph in
code.
Setup Walkthrough
- Create a prompt agent with a detailed system prompt
- Attach tools from L3 (function calling, MCP)
- Configure per-agent Entra managed identity
- Test agent in the playground
- Create a second version — test versioning and rollback
-
Explore workflow agents (preview) — understand their limitations
before committing
Related: LinkedIn Post 7 (Agent Maturity) — Learning Day 3
L5: Observability & Evaluation
L5
Observability & Evaluation
What actually happened, how well did it perform, and where did
it fail?
What Foundry Provides
| Capability |
Feature |
Status |
| Tracing |
OpenTelemetry via App Insights |
GA (prompt agents) |
| Evaluators |
Quality, safety, task adherence |
GA |
| Custom evaluation |
Configurable pipelines |
GA |
| Continuous eval |
Scheduled eval against live data |
GA |
| CI/CD |
GitHub Actions evaluation gates |
GA |
Critical Gap: Private App Insights
Tracing does NOT work with private Application Insights. If you need
both network isolation and observability — which is most serious
deployments — you must architect a workaround: non-private App
Insights in a peered network with restricted access, or export
traces to Log Analytics.
Setup Walkthrough
-
Create Application Insights:
az monitor app-insights component create
- Connect to your Foundry project
- Run your agent — inspect traces in the portal
- Run a quality evaluator against agent output
- Run a safety evaluator — verify Content Safety scoring
- Set up a GitHub Actions workflow with an evaluation gate
-
If using private networking: test tracing (it may break) and plan
the workaround
Related: LinkedIn Post 10 (Observability Paradox) — Learning Day
4
L6: Governance & Trust
L6
Governance & Trust
Who is allowed to do this, under what policy, with what risk and
budget?
What Foundry Provides
| Capability |
Feature |
Status |
| Content filtering |
Content Safety (always on, configurable) |
GA |
| Prompt protection |
Prompt Shields — jailbreak detection |
GA |
| RBAC |
Entra ID
+ 4 built-in roles + custom
|
GA |
| Infra policy |
Azure Policy integration |
GA |
| Threat detection |
Microsoft Defender |
GA |
What You Build (This Is the Big One)
Governance is not a feature — it's an architecture
Foundry has no approval workflows, no governance dashboard, no
aggregated audit trail, no compliance reporting, no agent behavior
policy engine, and no cost governance. Everything in this list is
your responsibility.
RBAC Roles
| Role |
Use For |
Risk Level |
| Azure AI User |
Developers (least privilege) |
Low — default for all devs |
| Azure AI Project Manager |
Team leads |
Medium |
| Azure AI Account Owner |
Resource creation only |
Medium — no data actions |
| Azure AI Owner |
Never without justification |
High — loaded gun in prod |
Setup Walkthrough
- Switch to Entra ID authentication — remove all API keys
-
Assign
Azure AI User to developers (least privilege)
-
Verify
Azure AI Owner is NOT assigned to any
developer accounts
- Configure Content Safety severity thresholds
- Test Prompt Shields — attempt a jailbreak, verify detection
-
Create Azure Policy assignments to block dangerous tools in
production
-
Design your approval gate architecture (Logic Apps or custom
workflow — Foundry does not provide this)
Related: LinkedIn Post 3 ({a}OS debut), Post 4 (Identity) —
Learning Day 5
L7: Experience & Intent
L7
Experience & Intent
What is the user asking, seeing, approving, or rejecting?
What Foundry Provides
| Capability |
Feature |
Status |
Limitation |
| Portal |
Foundry Portal (classic + new) |
GA |
New portal lacks VNet support |
| Agent builder |
Visual agent configuration |
GA |
— |
| Publishing |
Teams, M365, BizChat |
GA |
Requires public endpoints |
| API surface |
REST + Python/JS SDKs |
GA |
— |
| Playground |
Agent testing playground |
GA |
— |
Most of L7 is yours to build
The Foundry Portal is a management surface, not an end-user
experience. For most production deployments, the L7 surface — the
thing your actual users see and interact with — will be entirely
custom-built. Agent publishing to Teams requires public endpoints,
which is often a non-starter for security-conscious environments.
Setup Walkthrough
-
Navigate the Foundry Portal — toggle between classic and new
portals
-
Use the classic portal or CLI for network-isolated management (the
new portal doesn't support it)
-
Test agent publishing to Teams (note the public endpoint
requirement)
- Design your L7 surface: what will end users actually see?
-
Plan: approval surfaces, escalation flows, intent disambiguation —
none of these come from Foundry
Related: LinkedIn Post 3 (Governance debut) — Learning Day 5
Network Isolation
Three isolation dimensions: inbound (private
endpoints), outbound from resource (VNet
injection), and outbound from agents (tool-level).
What Works in VNet
| Tool / Feature |
VNet Status |
| Private endpoints (inbound) |
Works |
| VNet injection (agent compute) |
Works (/27+ subnet) |
| Function calling |
Works |
| MCP (private) |
Works |
| Code Interpreter |
Works |
| AI Search (private endpoint) |
Works |
What Breaks in VNet
| Tool / Feature |
Issue |
| File Search |
Not supported in VNet isolation |
| OpenAPI Tool |
Not supported in VNet isolation |
| Azure Functions (tool) |
Not supported in VNet isolation |
| Browser Automation |
Not supported in VNet isolation |
| Hosted Agents |
No VNet support at all |
| Workflow Agents |
Inbound only |
| Tracing (private App Insights) |
Breaks entirely |
| New Foundry Portal |
Must use classic, SDK, or CLI |
What Goes Over Public Internet
| Tool |
Risk |
| Bing Grounding |
Public internet even in "isolated" environments |
| Web Search |
Public internet even in "isolated" environments |
| SharePoint Grounding |
Public internet even in "isolated" environments |
Block these via Azure Policy in production
The marketing says "end-to-end network isolation." The reality is an
exceptions list. Know it before your security team finds it for you.
Related: LinkedIn Post 5 (Network Isolation Exceptions Table) —
Learning Day 6
Cost Governance
The platform itself is free to explore. Costs come from model
consumption, storage, search, networking, and compute.
Native vs. Custom
| Capability |
Source |
| Budget alerts |
Native — Azure Cost Management |
| Per-resource tracking |
Native — Azure Cost Management |
| Per-team chargeback |
Custom — requires APIM gateway |
| Per-user token budgets |
Custom — requires APIM + logic |
| Real-time spend enforcement |
Custom — requires APIM kill switch
|
| Cross-service attribution |
Custom — Log Analytics queries |
Deploy APIM as an AI
Gateway from Day One
Azure API Management sits between your consumers and Foundry. It
provides token metering, rate limiting, and per-team chargeback that
Foundry lacks natively. Build it before you need it — not after an
agent loops and spends your monthly budget overnight.
Related: LinkedIn Post 9 (Cost Governance) — Learning Day 6
IaC & CI/CD
Deploy via
Bicep/CLI, not the portal.
The new portal does not support network isolation workflows.
IaC is not optional —
it's the only reliable path for production.
Bicep Templates
00-basic — Simple Foundry resource + project
-
19-hybrid-private-resources-agent-setup —
Network-isolated agent setup with BYO storage
Fork the azure-ai-foundry/foundry-samples Bicep
templates. Customize for your policies and compliance. Maintain as
an internal module.
Phase 1 Architecture (Minimum Viable Platform)
- Foundry resource + project
- Private endpoints
- One model deployment (pay-as-you-go)
- One prompt agent (GA)
- Application Insights (with tracing workaround if private)
- APIM AI Gateway (cost metering from day one)
- Azure Cost Management budget with alerts
- Entra-only authentication (no API keys)
CI/CD Pattern
- Bicep deployment via GitHub Actions
- Agent evaluation gate (quality + safety evaluators)
- Cost check gate (budget threshold)
-
Promote: dev → staging → prod (separate Foundry resources per
environment)
Projects are NOT environments
Projects within a single Foundry resource share networking and
cannot serve as environment boundaries. Use separate Foundry
resources per environment (dev, staging, prod).
Related: LinkedIn Post 11 (Phase 1) — Learning Day 7