ModelDeployment and operator configuration reference for Foundry Local

This article is a reference for the ModelDeployment CRD spec and status fields, the Model CRD spec and status fields, and the inference operator configuration settings.

ModelDeployment spec fields

For catalog deployments, copy the full assetId of the required version and variant from GET /api/v1/models. Runtime and compute can be omitted; if supplied, they must match the selected catalog entry. New manifests must use model.assetId, not model.catalog name/version selectors.

For a registered BYO model, set model.assetId to its oci:// asset ID, or set model.ref to the Model resource's name. The deployment inherits compute and, for generative workloads, runtime from the registration, so you don't need to repeat these settings. Explicitly supplied values must match the registration.

For inline generative BYO deployments using model.custom, set both compute and runtime explicitly. For predictive BYO deployments, set workloadType: predictive; inline predictive deployments also require compute. Omit runtime from predictive ModelDeployments.

Chat, text embeddings and speech recognition use workloadType: generative. Predictive models use workloadType: predictive with ONNX on CPU or GPU and require BYO models. Set model.custom.task for an inline ONNX BYO model or capabilities.task on a named Model. Catalog models provide their task metadata. For vLLM, use a compatible model for the selected task; image and audio input in chat use the chat completions API. ARM64 model-serving support is available in preview. On ARM64, ONNX is supported only for predictive BYO; use vLLM for generative GPU deployments. See ARM64 workload support (preview).

Thor memory defaults

These defaults apply to the NVIDIA Jetson Thor GPU deployment path, which is available in preview.

For platform: jetson with compute: gpu, when both memory request and limit are omitted, the operator defaults generative workloads to 16 GiB request / 24 GiB limit and predictive workloads to 4 GiB request / 6 GiB limit. These are starting defaults, not measured model minimums. User memory settings take precedence; if either is supplied, the operator does not fill in the other with the Thor pair. Thor unified CPU/GPU memory counts against the pod limit. Generic 256-MiB/1-GiB defaults in the field table are not Thor GPU defaults.

Kubernetes service account token roles

GeneralInferenceUser is currently the only supported SAT role. For setup examples, see Configure Kubernetes service account token authentication.

Annotation migration

The nginx Ingress annotations that earlier releases honored on endpoint.annotations are no longer applied. Replace them with the Istio Gateway API equivalents on endpoint.gatewayAnnotations, the per-deployment HTTPRoute spec, or the cluster-wide external Gateway:

GPU configuration examples

GPU with skip GPU resource:

spec:
  compute: gpu
  skipGpuResource: true
  nodeSelector:
    kubernetes.io/gpu-partition: "1g.5gb"

GPU with node selector and tolerations:

spec:
  compute: gpu
  nodeSelector:
    accelerator: nvidia-a100
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

Configure model cache storage

The model cache persistent volume claim (PVC) defaults to 100 GiB. Large models, such as magistral, can exceed this size, so set spec.vllm.modelCacheStorageGi to a higher value when you deploy them. This field applies only to deployments that use runtime: vllm.

spec:
  compute: gpu
  runtime: vllm
  vllm:
    modelCacheStorageGi: 200

ModelDeployment status fields

Model CRD spec fields

The Model CRD registers BYO (custom) models and their artifact address, credentials reference, metadata, runtime, and compute. Registration alone does not download artifacts or start an inference endpoint. Catalog models do not use Model resources; deploy them through spec.model.assetId in a ModelDeployment.

Model CRD status fields

Available means the BYO registration is available for deployment. It does not mean model artifacts have been downloaded or an inference endpoint is running.

Inference operator configuration

The inference operator reads its configuration from a ConfigMap mounted at /etc/inference-operator/config.yaml.

Configuration file example

# Container registry for inference images
registry: "myregistry.azurecr.io"

# Container images for different workload types
images:
  generative_cpu:
    repository: generative-cpu
    tag: "latest"
  generative_gpu:
    repository: generative-gpu
    tag: "latest"
  predictive_cpu_oras:
    repository: predictive-cpu-byo
    tag: "latest"
  predictive_gpu_oras:
    repository: predictive-gpu-byo
    tag: "latest"
  Vllm_gpu:
    repository: vllm-server
    tag: "latest"
# Gateway API networking defaults 
networking: 
  internalGateway: 
    gatewayName: inference-internal-gateway 
    gatewayClassName: istio 
    listenerName: https 
  externalGateway: 
    gatewayName: inference-external-gateway 
    gatewayClassName: istio 
    serviceType: LoadBalancer 
    listenerName: https 
    port: 443 
    # Annotations forwarded to the auto-created LoadBalancer Service. 
    # Default points the Azure LB health probe at Istio's /healthz/ready 
    # on port 15021 so the gateway's "404 on GET /" does not mark 
    # backends unhealthy. 
    annotations: 
      networking.istio.io/service-annotations: | 
        { 
          "service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path": "/healthz/ready", 
          "service.beta.kubernetes.io/port_443_health-probe_port": "15021", 
          "service.beta.kubernetes.io/port_443_health-probe_protocol": "http" 
        } 
    tls: 
      # Customer-supplied TLS secret for external traffic termination. 
      # Leave empty to auto-issue from the internal CA. 
      secretName: "" 
  # EPP (Endpoint Picker) — shared infrastructure config used by every 
  # vLLM deployment that opts in via spec.vllm.epp.enabled. 
  epp: 
    image: 
      repository: mcr.microsoft.com/oss/v2/gateway-api-inference-extension/epp 
      tag: v1.3.1 
    replicas: 1 
    resources: 
      requests: { cpu: 250m, memory: 512Mi } 
      limits:   { cpu: "2",  memory: 2Gi } 
    targetPort: 8443 
    metricsDataSource: 
      scheme: https 
      insecureSkipVerify: true 
# Catalog settings
catalog:
  configmapName: "foundry-local-catalog"
  configmapNamespace: "foundry-local-operator"

Configuration fields

Kubernetes service account token deployment settings

These settings apply to the Foundry Local installation. They aren't ModelDeployment fields.

Related content

• Inference operator and model lifecycle
• Deploy Foundry Local as an Azure Arc extension
• Inference API endpoints and payload reference