Prompt Builder

The LLM Prompt Builder is a no-code tool for prompt engineers. You test a prompt across several LLMs at once, compare the responses side by side, let an LLM judge which one is best, keep a versioned history of every experiment, and turn the winning prompt into a scoreable model you can use elsewhere on the platform — for example in SAS Intelligent Decisioning. Everything is saved in SAS Model Manager, so your work is governed and shareable.

This guide walks through using the tool. For installing and embedding it in a SAS Visual Analytics report (and the one-time configuration a report author does), see Deploying the LLM Prompt Builder.

The workflow in one minute

You work top to bottom through the page:

1. Pick a project and a prompt-test — where this experiment history is stored in Model Manager.
2. Choose the LLMs you want to compare and tune their options.
3. Write your prompt — a system prompt and a user prompt, with optional variables.
4. Run the experiment — every selected model answers in parallel and the responses appear side by side.
5. Judge the responses — optionally have a judge model rank them and highlight the strongest.
6. Mark the best response and save — the run is versioned into Model Manager.
7. Manifest the best prompt — turn it into a scoreable model for the rest of the platform.

Nothing is sent to SAS Viya until you have the configuration in place, and no paid model call happens until you press Run Experiments.

Pick a project and a prompt-test

At the top, choose an existing project or create one. A project is a Model Manager container that groups related prompt-tests; the Prompt Builder tags the ones it creates so they are easy to find. Inside a project you choose or create a prompt-test — this is the thing whose experiment history you are building. Both lists have a name filter and a "created/modified by" filter, so long lists stay manageable.

Use Create a new Project / Create a new Prompt to add them without leaving the tool. Delete Prompt and Delete Project remove them again; before deleting, the tool checks whether any SAS Intelligent Decisioning decisions still use the prompt and warns you, so you do not break a running decision by accident.

Selecting a prompt-test loads its saved experiment runs into the Prompt Experiment Tracker further down the page, and brings the most recent run you marked as best straight into the workbench so you can carry on where you left off.

Choose the LLMs and tune their options

The LLM list shows every model available in your environment's LLM project. Tick each model you want to include in the comparison. When you tick one, its options appear — temperature, top-p, top-k, maximum length/tokens and any model-specific settings. Each option has an ℹ️ info icon; hover or focus it for an explanation of what the setting does and a sensible range.

You can select as many models as you like; they all run against the same prompt so you get a true side-by-side comparison. Models an administrator has marked as deprecated are hidden from the list.

Write your prompt

The workbench has two boxes:

• System prompt — how the model should behave: its role, tone, task and any rules. This is the mostly-static part.
• User prompt — the variable input the system prompt acts on.

Variables

Real prompts rarely use fixed text. Define variables above the prompt boxes — each has a name, an optional description, a type (string or decimal) and a value — then reference them anywhere in either prompt with the {{variableName}} syntax. Right-click inside a prompt box to insert a defined variable at the cursor.

When you run an experiment, the current values are filled in before the prompts are sent. Variables are more than a convenience: when you later manifest the best prompt, they become the documented inputs of the resulting model, so the model's callers know exactly what to supply.

Run the experiment

Press Run Experiments. The tool sends the resolved prompts to every selected model in parallel and, when they return, shows each model's result in the tracker: the response (rendered as Markdown), the time to respond, and the input/output token counts. Two mechanical flags are added automatically for the run — a ⚡ icon on the fastest response and a ⌄ icon on the one with the fewest output tokens. Each response is a starting point; speed and length are not quality, which is where judging comes in.

Judge the responses

Speed and token count do not tell you which answer is actually best. The Judge the responses section lets another LLM do that for you.

1. Pick a judge model. A deployment can set a default, but you can always override it here. Choose a model that is not one of the ones you are comparing where possible — a model tends to over-rate its own style (self-preference bias).
2. Optionally tick Auto-judge when the experiment finishes so judging runs by itself after every experiment, or leave it off and judge on demand.
3. Press Judge this run on any run in the tracker.

The judge sees all of the run's responses at once, in a shuffled order and under anonymous labels (so position and brand cannot sway it), reasons about them step by step, and returns a ranking. The result appears as a banner on the run: the winner, the full ranking, a confidence level, and the judge's reasoning behind a "Show reasoning" toggle. The best-ranked response also gets a 🏆 icon next to the ⚡ and ⌄ icons, and each response shows its judge rank.

The judge is advisory

Judging never changes your Best Response selection — that stays a decision you make. The judge rank is a signal to help you decide, alongside the responses themselves and the run metrics.

Include the judge's own response. By default, if your judge model is also one of the models being compared, its own response is left out of the ranking to avoid self-preference bias. The Include the judge's own response toggle overrides that if you want it ranked anyway — its ℹ️ icon explains the trade-off. Judging needs at least two responses to compare, so if excluding the judge would leave only one, either include it or pick a different judge.

The judge model, the include-self setting, the auto-judge setting and the full verdict (including the reasoning) are saved with the run, so loading a run later restores everything exactly as it was judged.

Mark the best response and save

Tick Best Response on the model result you consider best. This is the choice that matters: it is what a manifested model is built from and what downstream monitoring reports on. You can change it at any time.

Press Save Experiments to persist the run history to the selected prompt-test in Model Manager. Saving creates a new model version, so you keep a full, auditable trail of how a prompt evolved. Open in SAS Model Manager jumps straight to the prompt's files there.

The experiment tracker

Every run you save is listed in the tracker, newest first. For each run you can:

• Load it back into the workbench — prompts, variables, the selected LLMs and their options, and the judge configuration are all restored.
• Delete it — the remaining runs renumber cleanly.

Manifest the best prompt

Once a run has a Best Response, Manifest the Best Prompt turns it into a scoreable model in SAS Model Manager, ready to be consumed elsewhere — most commonly by the Call LLM node in SAS Intelligent Decisioning. The variables you defined become the model's input variables (with their descriptions), and the prompt text is baked into the score code.

Two options shape what you get:

• Include the LLM call in the manifested model. Off by default: the model returns the request body and endpoint (llmBody, llmURL) for the Call LLM node to execute. On: the model calls the LLM container itself and returns the response and metrics directly.
• Parse the LLM response into output variables. If your prompt asks the model to reply with JSON, define the output variables you expect; the model then reads each one from the response, falling back to a default, and adds a parse_status output that reports whether everything was extracted.

From here the manifested model behaves like any other model on the platform — see Deployment of Decisions for using it in a decision flow.

Where your experiments show up for reporting

Everything you save lands in Model Manager as a Prompt-Experiment-Tracker.json on the prompt-test. Administrators can roll all of it up into a single PROMPT_EXPERIMENTS table for Visual Analytics reporting — including the option values used, the per-run metrics, your best-response choices and the judge's verdict (rank, chosen model and confidence). The tracker files are the source of truth; the report is a convenience view over them.

A note on API keys

Hosted models (OpenAI, Anthropic, Google, …) need an API key. Keys are not typed into the tool or the URL — they are supplied to the object through governed data by whoever sets up the report, so they never end up in a shareable link and access to running paid model calls can be restricted. If a model call fails with an authorization error, that key is missing or not readable for you; ask your administrator. See Deploying the LLM Prompt Builder for the details.

Where to go next

• Deploying the LLM Prompt Builder — embedding it in a Visual Analytics report and the one-time configuration
• Model Definition Builder — how the LLMs you test with get added to the environment
• Deployment of Decisions — using a manifested prompt in SAS Intelligent Decisioning