Logging

The logs from the LLM containers is standardized across all of the different models. This has been done in order to be able to parse all of the relevant information from the logs and create monitoring on top of them.

In order to make use of these logs, which are written to the standard out of each container, you must collect the logs. If you do not have an established logging and monitoring process, you can use the simplified logging script provided below. Otherwise, if you do have an established way of collecting logs, ensure that you can export them to a folder that can be accessed from SAS Studio as a .log file (as that it is the input for the parsing utilities).

For more information on how to parse the log and load it into SAS please take a look at SAS-Viya-Integrations/Logging-Monitoring.

What the containers log

The shipped deployment templates set SAS_SCR_LOG_LEVEL_SCR_IO=TRACE, and the parser depends on it: the Request: POST / Request Data: lines carry the system prompt, the user prompt and the options of every call, and the response: lines the full answer. That is what makes the usage report possible, and it also means prompt text and responses land in your log pipeline. Decide who may read the collected log file and the LLM_LOGS table accordingly, or lower the level and forgo the report.

Simplified Logging via a Script

Only use this script if you do not have a more robust process in place.

Create a script—for example collect_llm_logs.sh—and ensure that it is executable. Use the code below and change both the namespace and the log path. The namespace should be the namespace into which you have deployed the models, and the path needs to be reachable from inside of SAS Studio. You can also change the logging interval; note that any changes you make here will have to be reflected in the entry for the crontab. The default here also replaces the log file as the assumption is that you pick up the new entries using the parsing script and append (though a full replacement is also supported).

# Set the script variables for your environment
llm_namespace="llm"
llm_log_path="/viya-share/pvs/sasdata/data/llm/llms.log"
llm_logging_interval="1h"
# Remove the next line to append to the log file instead
rm $llm_log_path
for pod in $(kubectl get pods -n $llm_namespace -o name); do
  kubectl logs -n $llm_namespace $pod --all-containers --since=$llm_logging_interval >> $llm_log_path
done

Next add the following entry to your crontab (crontab -e), adjusting your path to the script:

0 * * * * /viya-share/pvs/sasdata/data/llm/collect_llm_logs.sh

This entry will run the log collection script at the top of every hour.

The LLM Usage Report

The repository ships a ready-made SAS Visual Analytics report, SAS-Viya-Integrations/Logging-Monitoring/LLM Usage Report.json, as a transfer package. It is the recommended template for monitoring LLM usage and prompt experimentation in your environment — import it and use it as-is, or as the starting point for your own dashboards.

Prerequisites — the tables it reads

The report builds on four CAS tables, all expected in the Public caslib by default. Load them (promoted and saved) before or after importing so the report has data to bind to:

Table	Produced by
LLM_LOGS	Parsing the collected container logs — Log-Parser-Code.sas or the LLM - Log Parser SAS Studio custom step
LLM_FACT_SHEET / EMBEDDING_FACT_SHEET	mdb load-facts (or Load-Fact-Sheets.sas) — see Model Definition Builder
PROMPT_EXPERIMENTS	Get-All-Prompts.sas (the LLM Prompt Builder experiment tracker)

Import the report

Import the transfer package exactly as for the Prompt Builder package — through the SAS Environment Manager → Content → Import page, or with the sas-viya CLI transfer plugin:

sas-viya transfer packages upload --file "LLM Usage Report.json"   # prints the package id
sas-viya transfer packages import --id <package-id>

(On older transfer plugin versions the same commands are sas-viya transfer upload / sas-viya transfer import — they still work, marked deprecated.)

The CLI must be installed with the transfer plugin and a signed-in profile — see Introduction — SAS Viya CLI Setup. After import the report appears under SAS Content > SAS Agentic AI Accelerator > Logging and Monitoring > LLM Usage Report.

Change the CAS library (default: Public)

The report binds each of its four tables to server=cas-shared-default;library=Public. If your tables live in a different CAS library, repoint them one of two ways:

• SAS Viya CLI — before uploading, replace the library in the package (every binding uses the same library=Public token, so a single substitution is safe), then upload/import as above:

  # bash / Linux / macOS
  sed -i 's/library=Public/library=MyLib/g' "LLM Usage Report.json"

  # PowerShell / Windows
  (Get-Content "LLM Usage Report.json") -replace 'library=Public','library=MyLib' |
    Set-Content "LLM Usage Report.json"

  (If your CAS server is not cas-shared-default, substitute server=cas-shared-default the same way.)

• SAS Environment Manager / SAS Visual Analytics — import the package as-is, then open LLM Usage Report in SAS Visual Analytics and, in the Data pane, use Edit → Change data source on each of the four tables to point to the same table in your library.

note

The imported report keeps its identifier /reports/reports/1f08db8d-d6c6-4ed1-9684-e7d11c4ec50c — the same URI shown as the example SAS_LLM_MODEL_CARD_REPORT_URI and the Prompt Builder Model card report URI — so once imported it can double as the model-card custom chart for your registered models and manifested prompts.