Stack9 AI Studio
Stack9 AI Studio is the native interface for operating Stack9 AI Agents over their lifecycle. It is where agents are configured, evaluated against agreed benchmarks, replayed step by step, compared, and promoted to live or rolled back. It sits over the same Amazon Bedrock AgentCore observability used across the platform.
This is a concept page. It describes what Stack9 AI Studio does and how it fits into the agent lifecycle. It does not document screens or configuration formats.
What is Stack9 AI Studio?
AI Studio gives the people accountable for an agent's behaviour a place to change that behaviour safely, without code and without waiting on an April9 release, while keeping every change evaluated and under change control.
Who uses it
AI Studio is used by AI Agent configurators: typically the client's own nominated business or assurance staff, working alongside April9. Configurators own agent behaviour (prompts, tool access, skills, guardrails, model selection) and the evaluation evidence behind each release. April9 remains responsible for deeper platform, infrastructure and model-provider changes.
What it provides
| Activity | What AI Studio provides |
|---|---|
| Configure | Tune each agent's system prompt, tool access, skills, guardrails and model selection without code, and author the standard phrases an agent reuses (greeting, clarifying question, handover, "I can't answer that") |
| Evaluate | Build, extend and run evaluation datasets (benchmark questions with predefined correct outcomes) and score an agent version against them before release. Real anonymised sessions can be loaded as new test cases, with approval |
| Replay | Open a complete conversation and follow it step by step: what the agent was asked, every function and tool call, the identity each call ran under, the sources retrieved, the context the agent had at each step, and what it did with it |
| Trace to effect | See how an agent's output flowed into the rest of the solution: which record it wrote to, which workflow step it triggered, which notification it caused |
| Compare | Compare versions side by side on performance, cost and accuracy, so a change can be attributed to exactly what moved |
| Promote and roll back | An authorised configurator promotes a version to live, or rolls back to a prior one, in seconds. Every promotion still passes the evaluation gate, and every change is captured under change control |
| Score per agent | Each specialist agent is scored on how often it resolves without escalating, its accuracy against the agreed answer set, and the feedback it receives. An agent trending low is queued for review ahead of its next release |
Why a dedicated studio
Agent behaviour changes more often than platform code, and the people best placed to judge it are usually the client's subject-matter experts, not developers. Without a dedicated tool, two things go wrong: changes are gatekept behind engineering releases, or they are made informally with no evidence of what they did. AI Studio resolves both by pairing self-service configuration with an evaluation gate and a versioned history.
How it works
Versions and snapshots
An agent is a versioned artefact (see What is a Stack9 AI Agent?). In AI Studio a change to any element (system prompt, tools, skills, memory, retrieval scope, guardrails or model) produces a new version snapshot rather than mutating the running one. The live version keeps serving users while the new one is tested. Because earlier versions are retained, rollback is a promotion of a prior snapshot, not a rebuild.
Versions are compared side by side, so a reviewer can see what changed and what it did to accuracy, cost and performance.
Evaluation datasets and the release gate
Evaluation is the single most important assurance mechanism, and AI Studio puts it on the release path.
- The evaluation set is a design-phase deliverable. The client and April9 co-author a dataset of scenarios, each carrying the correct outcome and the source that should be cited. The client approves the set and the pass threshold before the first agent is configured, so "ready to release" is a number agreed in advance.
- Four things are scored: whether the outcome is correct; whether the right source is cited at the right version; whether the agent correctly declines and escalates where the content does not support an answer; and whether materially equivalent questions reach the same outcome.
- Scoring runs on Amazon Bedrock AgentCore Evaluations, the same managed tooling the agents run on.
- A version below threshold is not promoted. Re-runs are automatic on every agent change and every content refresh.
- Whole conversations are tested, not only answers. A synthetic user with a persona and a goal holds a full conversation with the assistant, and an AI judge scores the conversation against the expected outcome.
The evaluation tooling is native. The benchmark questions and correct outcomes are not: they are authored per client, with the client's subject-matter experts, and they are part of the delivery effort.
The flowchart below shows the authoring loop a configurator follows. The claim question is illustrative.
Real sessions feed tuning, not training
Anonymised production sessions feed AI Studio so configurators tune against real interactions, and, with the client's approval, add them to the evaluation set as new cases. This happens within the client's own instance only. Sessions are never used to train the underlying model: what is tuned is agent behaviour (prompts, tool access, skills, retrieval scope, guardrails), not model weights. See AI governance.
How much is configurable
In one scoping, around 80% of the AI layer was estimated to be configurable through Stack9 AI Studio: agent setup, system prompts, tool and guardrail configuration, and model selection. Treat that figure as indicative, not a platform constant. The remaining effort is development:
| Configured in AI Studio | Development effort |
|---|---|
| Agent setup and system prompts | Authoring the evaluation datasets themselves |
| Tool and guardrail configuration | Function-calling development for complex, client-specific tool integrations |
| Model selection | First-time connection of a new data source or tool |
| Standard phrases and tone rules | Conversational UI components in the Web App Template |
What stays with April9
AI Studio hands agent behaviour to configurators. It does not hand over the platform: April9 remains responsible for deeper platform, infrastructure and model-provider changes. A new model version, for example, is regression-scored against the baseline sets before adoption rather than switched on silently (see AI governance).
Limits
- The ~80% configurability figure is indicative, from one scoping.
- Evaluation datasets must be authored per client; they are not pre-built.
- First-time connection of a new data source or tool, and complex function-calling, need a developer.
- Promotion is gated. Self-service promotion does not bypass the evaluation threshold or change control.
- No fine-tuning on client data, by design.
- AI prompt-to-app is roadmap only.
Related
- Stack9 AI Agents: what an agent is and how agents work together
- AI governance: change control, capture and the lifecycle loops
- MCP Server: the Stack9 Core tools agents are granted