Deploying and Scaling AI Models on Azure Container Apps

Running AI models in production means dealing with cold starts, GPU availability, and unpredictable traffic patterns. Azure Container Apps (ACA) with GPU-enabled workload profiles solves this without the overhead of full Kubernetes. This post covers the end-to-end workflow — from Docker image to auto-scaled, production-ready inference endpoint.

Why Container Apps over AKS for AI Inference

AKS is the right choice when you need multi-node GPU clusters with custom scheduling. But for many inference workloads — text embeddings, smaller language models, classification pipelines — ACA’s serverless GPU profiles strike a better balance. You get scale-to-zero during idle periods, event-driven scaling via KEDA, and managed TLS termination without maintaining a cluster.

GPU Workload Profiles

ACA supports dedicated GPU workload profiles with NVIDIA A10 and T4 GPUs. You assign a workload profile to your container app environment at creation time.

az containerapp env create \
  --name ai-inference-env \
  --resource-group ai-production \
  --location westeurope \
  --enable-workload-profiles

Then add a GPU profile:

az containerapp env workload-profile add \
  --name ai-inference-env \
  --resource-group ai-production \
  --workload-profile-type NC4as_T4_v3 \
  --workload-profile-name gpu-t4 \
  --min-nodes 0 \
  --max-nodes 5

The workload profile type determines the GPU model and vCPU/memory allocation. NC4as_T4_v3 gives you a single T4 GPU with 4 vCPUs and 28 GB RAM — solid for a 7B-parameter model.

Dockerizing the Inference Server

Package your model server with the model weights baked in or mounted from Azure Files. For smaller models (< 3 GB), embedding weights in the image avoids cold-start downloads.

FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04

RUN apt-get update && apt-get install -y python3.11 python3-pip && \
    pip3 install vllm fastapi uvicorn

WORKDIR /app
COPY ./model /app/model
COPY ./serve.py /app/

ENV MODEL_PATH=/app/model
ENV CUDA_VISIBLE_DEVICES=0

EXPOSE 8000

CMD ["python3", "-m", "uvicorn", "serve:app", "--host", "0.0.0.0", "--port", "8000"]

For larger models, use an init container to download weights from Azure Blob Storage or a model registry before the main container starts.

Deploying to Azure Container Apps

Push your image to Azure Container Registry and deploy with GPU allocation.

az acr build --registry aiRegistry --image inference-server:v1 .

az containerapp create \
  --name phi4-inference \
  --resource-group ai-production \
  --environment ai-inference-env \
  --image aiRegistry.azurecr.io/inference-server:v1 \
  --target-port 8000 \
  --ingress external \
  --workload-profile-name gpu-t4 \
  --cpu 2.0 \
  --memory 14.0Gi \
  --gpu 1 \
  --min-replicas 1 \
  --max-replicas 10 \
  --scale-rule-name http-scale \
  --scale-rule-type http \
  --scale-rule-http-concurrency 5 \
  --env-vars MODEL_NAME=phi-4 AZURE_STORAGE_CONNECTION=secretref:storage-conn

The --gpu 1 flag allocates one full GPU to each replica. ACA stripes GPU requests in increments of 1 — you can’t request fractional GPUs.

Autoscaling with KEDA

ACA uses KEDA under the hood. For inference workloads, HTTP concurrency scaling works well, but you can also scale on custom metrics like queue depth or request latency.

az containerapp update \
  --name phi4-inference \
  --resource-group ai-production \
  --scale-rule-name gpu-queue \
  --scale-rule-type azure-queue \
  --scale-rule-metadata \
    queueName=inference-requests \
    queueLength=10 \
  --scale-rule-auth connection=queue-conn

Scale-to-zero is powerful for cost control — a T4 GPU costs approximately $0.50/hour. If your inference endpoint sees traffic only 8 hours a day, scale-to-zero saves 66% over a 24/7 deployment.

Model Caching Strategy

Cold starts are the biggest UX problem with scale-to-zero. Mitigate this with a layered caching approach.

import os
import hashlib
from pathlib import Path

CACHE_DIR = Path("/mnt/model-cache")

def get_cached_model(model_name: str, version: str):
    cache_key = hashlib.sha256(f"{model_name}@{version}".encode()).hexdigest()[:16]
    cache_path = CACHE_DIR / cache_key

    if cache_path.exists():
        return str(cache_path)

    # Download only if miss — use Azure Blob with high throughput
    download_from_blob(model_name, version, cache_path)
    return str(cache_path)

Mount an Azure Files share at /mnt/model-cache across all replicas. The first replica warms the cache; subsequent replicas and scale-out instances read from it.

az containerapp env storage set \
  --name ai-inference-env \
  --resource-group ai-production \
  --storage-name modelcache \
  --azure-file-account-name aistorageacct \
  --azure-file-share-name model-cache \
  --access-mode ReadWrite

Health Probes and Graceful Shutdown

ACA supports liveness, readiness, and startup probes. For GPU workloads, startup probes are critical — model loading can take 30–120 seconds.

az containerapp update \
  --name phi4-inference \
  --resource-group ai-production \
  --probe-type startup \
  --probe-http-path /health \
  --probe-initial-delay 10 \
  --probe-period 30 \
  --probe-timeout 5 \
  --probe-failure-threshold 20

The startup probe gives the container up to initialDelay + (period × failureThreshold) = 10 + (30 × 20) = 610 seconds to become ready before ACA considers it failed.

Cost Optimization

Combining scale-to-zero with a spot workload profile on non-critical inference endpoints can reduce GPU costs by up to 80% compared to always-on, pay-as-you-go pricing.

Azure Container Apps with GPU support brings serverless economics to AI inference. Start with a single T4 for your prototype, layer in autoscaling as traffic grows, and optimize with scale-to-zero and spot instances once you understand your usage patterns. The result is a production inference pipeline that costs nothing when idle and scales to handle bursts — without a single kubectl command.