Roast

Company: Shopify
Approach type: Platform
Work supported: Coding, Code review, On-call, Research
Interfaces: Cli
Deployment stage: Deployed
Evidence strength: Detailed primary
Entry reviewed: 2026-09-21

Purpose

Roast is the workflow orchestration framework Shopify's Augmented Engineering Developer Experience team extracted from its internal AI tools and open-sourced; Shopify runs Roast workflows internally to interleave AI steps with deterministic code.

Check the reviewed sources and claim details.

Capabilities and architecture

Model: An OpenAI API key is a prerequisite, or OpenRouter for other models; the Raix library abstracts AI providers and adds retry, caching and structured output; the built-in CodingAgent tool is powered by Claude Code.The post names the API-key prerequisite, the Raix abstraction and Claude Code behind the CodingAgent; it names no model for a plain AI step.Observation date: 2025-06
Harness: As released in 2025: a Ruby command-line framework that reads a workflow.yml file and matching markdown prompt files, interprets each step from its structure, and can be used with any programming language.The claim describes the 2025 release; the repository captured in 2026 documents a Ruby DSL of cogs instead.Observation date: 2025-06
Sandbox: Not reportedThe reviewed sources document WriteFile security restrictions and configurable Cmd restrictions, not an execution or isolation boundary; the unknown claim stays in research details.
Tool access: Built-in ReadFile, WriteFile, UpdateFiles, Grep, SearchFile, Cmd, Bash and CodingAgent tools; WriteFile carries security restrictions and Cmd carries configurable restrictions.
Knowledge: Markdown prompt files use ERB templating to read the workflow context; complex logic goes in custom Ruby step classes that inherit from BaseStep.
Context management: Steps in a workflow share their conversation transcript, so a later step references earlier discoveries without extra configuration; every execution is saved so a run can resume from any step.
Credentials: An OpenAI or OpenRouter API key is required; Raix initializers in the .roast/initializers directory can add custom authentication schemes.The post names the provider API key and custom authentication schemes through Raix initializers; it names no employee sign-in or secret store.
Interfaces: cliThe post names a command line tool distributed as a Ruby gem; the Slack channels the SRE workflow reads are a data source, not an invocation interface.

Mechanisms

Declarative workflow definition

A workflow.yml file and matching prompt.md files define the steps, and Roast interprets each step from its structure.

Mixed deterministic and AI steps

Directory prompts, shell commands wrapped in $(), inline prompts, custom Ruby step classes and nested-array parallel steps run in one workflow.

Advanced control flow

Workflows iterate over collections, execute steps conditionally and branch with case statements.

Built-in tool set

ReadFile, WriteFile, UpdateFiles, Grep, SearchFile, Cmd and Bash are supplied out of the box.

CodingAgent step

A Claude Code integration runs as a step and iterates within the guardrails and objectives the workflow author sets.

Shared transcript

Steps share their conversation transcript so later steps build on the context earlier steps produced.

Session replay

Roast saves every workflow execution so development resumes from any step instead of rerunning expensive AI operations.

Documented uses

Documented use example: Boba workflow: untyped Ruby test file → deterministic cleanup and Sorbet autocorrect → coding-agent iteration until tests and type checking pass. Research details

Clean up the test file

Deterministic sed steps clean up the code before any AI step runs.

Bump the file to strict typing

The workflow raises the test file to strict Sorbet typing.

Run Sorbet autocorrect

Sorbet's autocorrect resolves the type errors it can fix on its own.

Hand remaining issues to the coding agent

The CodingAgent iteratively fixes the type errors the deterministic steps left behind.

Access and controls

See documented credential and access boundaries in the architecture above. Scoped human-review assessments for individual uses remain in research details.

Reliability and validation

Confirm the file passes

The workflow runs the tests and ensures that both the tests and type checking pass.

Adoption and operating evidence

The catalog records what the sources report, with the scope and the denominator of every figure. A qualification below limits the figure it sits under.

Implementation scale · Estimate · Test files analyzed with Roast workflows inside Shopify

Key observation

Shopify engineers analyzed thousands of test files with Roast, identifying and fixing common antipatterns and increasing test coverage; the post reports no count, period or baseline.

Reported by: Shopify
Scope: Test files analyzed with Roast workflows inside Shopify

The source does not report the denominator of this figure.

Observed in June 2025

Adoption output · Estimate · Engineers who had contributed features and workflows to the public Roast repository

Key observation

A dozen engineers had contributed features and workflows to the public Roast repository by publication.

Reported by: Shopify
Scope: Engineers who had contributed features and workflows to the public Roast repository as of publication

The source does not report the denominator of this figure.

Observed in June 2025

Lessons

Reported opinion: Shopify says AI agents need help staying on track and work much better when a complicated prompt is broken into discrete steps; letting AI roam free around millions of lines of code did not work well for its unit-test work.

For type annotation, Shopify runs deterministic cleanup and Sorbet autocorrect first and hands only the remaining issues to the coding agent.

Reported opinion: Shopify says saving every workflow execution and resuming from any step speeds up workflow development because expensive AI operations are not rerun.

Reported opinion: Shopify describes using an AI step as a placeholder for a part of a workflow the author does not yet understand, then replacing it with deterministic code once the problem is clearer.

Sources and research details

Citations link to the original publisher. Each source also keeps a preserved copy in the repository, so a changed or removed page stays checkable.

Introducing Roast: Structured AI workflows made easyhttps://shopify.engineering/introducing-roastEngineering blog · First party · Last source verification: 2026-09-21
Preserved copy
Shopify/roast: Structured AI workflows made easyhttps://github.com/Shopify/roastRepository · First party · Last source verification: 2026-09-21
Preserved copy

Question coverage and scope

purpose: Reported
workflow: Reported: The post breaks out one internal workflow step by step, the Boba type-annotation workflow; the other internal uses it names are described only by purpose.
human involvement: Not applicable: Roast is a framework, so no single review boundary covers its workflows; the scoped Boba operating-model claim stays in research details and its boundary is unknown.
implementation: Reported
validation: Reported
observations: Reported
lessons: Reported

Research details for every claim on this page

Summary
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 12, 16, 170: the DX team built and open-sourced Roast, extracted it from internal AI tools, and reports internal deployment
Sandbox
unknown
Statement type
Inference
Provenance
Catalog judgment
Confidence
Medium
Confidence reason
The reviewed sources describe WriteFile security restrictions and configurable Cmd restrictions but no execution or isolation boundary; unknown does not mean absent.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 70-77: the tool list names file and command restrictions, not an execution boundary
Harness
Statement type
Fact
Provenance
Reported
Confidence
Medium
Confidence reason
The 2025 post describes the YAML-and-markdown design in detail; the repository captured in 2026 documents a Ruby DSL instead, so the claim is scoped to the release the post describes.
Observation date
2025-06
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 16, 26-64: convention-oriented framework, workflow.yml plus prompt files, command-line tool usable from any language
- ContextualizesShopify/roast: Structured AI workflows made easyPreserved content.md, lines 12-66: the repository now documents a Ruby DSL built from cogs and run with bin/roast execute, a later redesign of the 2025 YAML configuration
Model
Statement type
Fact
Provenance
Reported
Confidence
Medium
Confidence reason
The post names the API-key prerequisite, Raix, and Claude Code behind the CodingAgent, but never names the model a plain AI step uses; the repository captured in 2026 lists a different provider set.
Observation date
2025-06
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 50, 146-154: OpenAI API key or OpenRouter prerequisite, Raix provider abstraction, CodingAgent powered by Claude Code
- ContextualizesShopify/roast: Structured AI workflows made easyPreserved content.md, lines 60-140: the repository now documents OpenAI, Anthropic, Perplexity and Gemini chat providers and Pi or Claude Code agent CLIs
Tool access
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 68-77: the built-in toolkit list with its restriction notes
Knowledge
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 34-40, 54-58: prompt.md files with ERB templating and BaseStep Ruby step classes
Credentials
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 146-150, 156-162: the API key prerequisite and Raix initializers adding custom authentication schemes
Context management
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 96-98, 118-124: shared conversation transcript and saved executions that resume from any step
Interfaces
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 16, 142-144: Roast is a command line tool distributed as a Ruby gem
Clean up the test file
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 174: Boba performs cleanup with sed
Bump the file to strict typing
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 174: Boba bumps tests to strict typing
Run Sorbet autocorrect
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 94, 174: Boba runs Sorbet autocorrect
Hand remaining issues to the coding agent
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 94, 174: remaining issues are fed to the coding agent, which iteratively fixes type errors
Confirm the file passes
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 94, 174: the agent runs tests and ensures both tests and type checking pass
Declarative workflow definition
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 24-36: convention over configuration, workflow.yml and directory-based prompt.md steps
Mixed deterministic and AI steps
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 30-64: the five step formats Roast interprets by structure
Advanced control flow
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 102-116: iteration over collections, conditional execution and case statements
Built-in tool set
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 66-76: the built-in toolkit list
CodingAgent step
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 77, 79-92: the CodingAgent integrates Claude Code and has autonomy within the guardrails the author sets
Shared transcript
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 96-98: steps share their conversation transcript
Session replay
Statement type
Fact
Provenance
Reported
Confidence
High
Confidence reason
A linked first-party source states the claim.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 118-124: session replay saves every execution and resumes from any step
Key observation
Statement type
Metric
Provenance
Reported
Confidence
Low
Confidence reason
Shopify reported an approximate scale in its own engineering blog with no count, measurement window, coverage baseline or method.
Reported by
Shopify
Scope
Test files analyzed with Roast workflows inside Shopify
Denominator
Not reported
Method
Not reported
Observation date
2025-06
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 172: engineers have analyzed thousands of test files
Key observation
Statement type
Metric
Provenance
Reported
Confidence
Low
Confidence reason
Shopify reported an approximate contributor count for its public repository without a period, and without saying whether the contributors are Shopify employees or outside developers.
Reported by
Shopify
Scope
Engineers who had contributed features and workflows to the public Roast repository as of publication
Denominator
Not reported
Method
Not reported
Observation date
2025-06
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 20: a dozen Engineers have already contributed features and workflows on GitHub
Lesson
Statement type
Opinion
Provenance
Reported
Confidence
Medium
Confidence reason
The opening section states this as what the team learned while building AI workflows for unit-test grading; it is Shopify's account of its own experience, with no measurement behind it.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 10: AI agents need help staying on track; allowing AI to roam free around millions of lines of code did not work very well
Lesson
Statement type
Fact
Provenance
Reported
Confidence
Medium
Confidence reason
The CodingAgent section and the internal-impact section both describe this step order for Boba; the post does not compare it against an all-agent alternative, so it stays a description of Shopify's own workflow.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 94, 174: deterministic steps clean up and autocorrect, then remaining issues go to the CodingAgent
Lesson
Statement type
Opinion
Provenance
Reported
Confidence
Medium
Confidence reason
The session-replay section asserts the speed-up for workflow authors; the post reports no development-time measurement.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 120-124: session replay eliminates the need to rerun expensive AI operations
Lesson
Statement type
Opinion
Provenance
Reported
Confidence
Low
Confidence reason
The closing section presents this as the post author's reading of a quoted Shopify engineer; the Steel capture preserves the author's paragraph but drops the quotation it interprets, so the engineer's own wording was not reviewed from the capture.
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, line 190: use AI as a placeholder to keep the workflow moving, then replace it with deterministic code
Operating model assessment
Unclassified for Boba workflow: untyped Ruby test file → deterministic cleanup and Sorbet autocorrect → coding-agent iteration until tests and type checking pass; human attention boundary: unknown.
Statement type
Inference
Provenance
Catalog judgment
Confidence
Unverified
Confidence reason
The post follows the Boba workflow to a file where tests and type checking pass and says the team pursued unit-test work with minimal human intervention, but it names no point at which a person reviews the result.
Observation date
2025-06
- SupportsIntroducing Roast: Structured AI workflows made easyPreserved content.md, lines 94, 174: the Boba workflow runs deterministic cleanup and autocorrect, then the CodingAgent iterates until tests and type checking pass

Back to infrastructure