Skip to main content

Container Deployment

Configurations for SCR deployment​

For the creation of the SAS Container Runtime publishing destination it is recommended to use the SAS Viya CLI. The command detailed below is used to create a publishing destination that is using a Azure Container Registry.

Kubernetes Deployment​

mdb deploy <model_id> --registry <registry> --host <ingress-host> renders a ready-to-apply manifest from the templates in the SCR-LLM-Deployment-YAML folder (see Model Definition Builder): deploy-modelName-template.yaml for hosted-API models, deploy-modelName-env-template.yaml for Azure definitions that read their resource, key and deployment from the container environment, and deploy-modelName-PV-template.yaml plus llm-weights-pvc-template.yaml / stage-weights-job-template.yaml for self-hosted models with staged weights (Serving Open-Weight Models). The templates were written for Azure Kubernetes Service (spot-node affinity and tolerations) and are easily adapted. The following assumptions are made:

  • Namespace called llm in which the SCR containers will be deployed - every object of a template names it; mdb deploy --namespace <name> renders the Deployment, Service and Ingress into another one. llm-weights-pvc-template.yaml and stage-weights-job-template.yaml are not rendered by mdb deploy and keep llm: for another namespace change their namespace: lines before you apply them (the command reminds you when it renders the PV variant).
  • The Ingress serves its host with the TLS secret llm-tls-certs; mdb deploy --tls-secret <name> names another one. The Secret has to exist in the namespace of the Ingress - an Ingress reads no Secret of another namespace -, so the certificate Secret of your SAS Viya ingress, which lives in the namespace of SAS Viya, is copied there first (how).
  • A definition that takes its provider key from the container's environment (--auth env) gets that variable from a Kubernetes Secret: mdb deploy adds a secretKeyRef to the Deployment (Secret llm-api-keys, or --key-secret <name>, one entry per variable name) and prints the kubectl create secret generic command for it, with the variable names only. Create the Secret in the namespace before the pod starts; a missing Secret or entry keeps the pod in CreateContainerConfigError. To add a key to a Secret that exists, use the kubectl patch line the command prints - kubectl apply of a Secret that lists only this run's variables removes the others. Any other hosted definition takes its key per call, from the caller.
  • The Ingress has no authentication. A container that holds its provider key (--auth env, and azure-foundry-env) answers anyone who can reach host/llm/model_name with the organisation's key, without a credential-domain check: restrict who can reach the path - an allowlist of source addresses on the Ingress (nginx.ingress.kubernetes.io/whitelist-source-range with ingress-nginx), a NetworkPolicy, or no Ingress where only callers inside the cluster use the model - and set a spending limit for the key at the provider.
  • Each model container requests 1000m CPU and 4000Mi of memory and has no limit, and the pods go to nodes labelled and tainted app=llm (spot nodes preferred) - that is why we recommend having a separate node pool for this workload; in non production environments it is recommended to use a spot-instance with a lot of available CPU and RAM, e.g. Standard_D64s_v5. mdb deploy --requests CPU,MEMORY, --limit-memory and --pull-policy change the model container, --node <name> pins the pods to one node, and --node-class <class> tolerates the SAS node taint workload.sas.com/class=<class>:NoSchedule in place of the app=llm affinity, so the models can share a node pool of SAS Viya (placing the pods). --requests and --limit-memory apply to every model of one render: small values suit models that call a provider's API, not those that run their own weights (the Hugging Face transformers LLMs, the sentence-transformers embedding models, von_1_2) - render those in a second run with larger values and keep the router of the --all run.
  • The URL endpoint schema looks like this host/llm/model_name here the container will be reached, that means the full address for a container is host/llm/model_name/model_name.
  • The Ingress lets a request take 300 seconds (proxy-read-timeout and proxy-send-timeout); the NGINX default of 60 would end the call of a model that reasons for minutes before it answers. The scorer itself waits runtime.timeout_s of its definition, see How long a call may take.
  • The Ingress of each model is one for the NGINX Ingress controller: it rewrites host/llm/model_name/model_name to /model_name, the only path the Python SCR runtime answers. On a cluster whose ingress controller is Contour, as on current SAS Viya deployments, such an Ingress reaches no model. Render with mdb deploy --all --ingress contour: the models get no Ingress, and llm-router.yaml adds one NGINX router (nginx-unprivileged, resolving each model's Service through the cluster DNS) behind one Contour Ingress for /llm, with the same URLs - see A cluster with Contour.
  • The templates set SAS_SCR_LOG_LEVEL_SCR_IO to TRACE, which is what the log parser reads - at that level every request and response, prompts included, is written to the container's standard output. Lower it if your logs must not carry prompt text, and accept that the usage report then has no token counts.
  • At TRACE the container also logs secrets. The options and headers of each request reach its standard output as they arrive. A model that takes its key per call writes the provider key in clear text, as API_KEY:<key>. A call from the browser (the Prompt Builder) writes the user's Cookie header, and an Authorization or X-CSRF-TOKEN header where it has one. Restrict who can read the pod logs of the namespace, and keep the namespace out of shared log shipping. The accelerator's own tools strip the secrets before they write: the collection script replaces the API_KEY value and the Cookie, Authorization and X-CSRF-TOKEN headers with REDACTED in its file, and the log parser writes the options of a call to LLM_LOGS without their keys and keeps of the headers only the X-Request-Id and the kind of client. Neither changes what kubectl logs or a cluster log pipeline reads (The containers also log secrets).
  • A definition made with --auth env keeps its provider key out of that log. The container reads the key from a Kubernetes Secret, and the call carries no API_KEY option, so the key never reaches the TRACE line. Its limits:
    • The route has no authentication of its own: anyone who reaches it uses the organisation's key, so it needs an allowlist or a NetworkPolicy (see above).
    • Every caller uses the one key of the container; there is no key per user and no credential-domain check.
    • A caller that still sends a non-empty API_KEY option wins over the container's key, and that key is logged as before. Remove the option from flows and manifested models built before the switch.
    • The cookies of browser calls are still logged.
    • A new key in the Secret reaches the container only when its pod is recreated.
  • A republished image with the same :latest tag is only picked up when the pod is recreated. On a node pool without spare CPU a kubectl rollout restart can deadlock (the new pod cannot schedule while the old one holds the resources); scaling the deployment to 0 and back to 1 always works.

Azure Container Apps/Instances​

If you want to deploy the LLM containers as Azure Container Apps or Azure Container Instances than please make sure that when you follow the Deploying the Builder UIs you set the object's deploymentType to aca (the value recorded in your llm-prompt-builder.json).

Please also note that the attribute SCREndpoint contains the value randomString.region.azurecontainerapps.io from the https://model.randomString.region.azurecontainerapps.io/model URL of your Azure Container App.