Check Run Agents [beta]

Custom agentic check runs that enforce your rules and take action on every PR

Check Run Agents are in beta. Contact [e-mail] to get access.

Check Run Agents are fully customizable AI agents that trigger on every pull request open, push, and manual check rerun. They have access to your codebase, git history, and connected integrations. You define what to check, how to format the output, and what actions to take.

Macroscope automatically runs two built-in check runs on every PR: Correctness (catches runtime bugs and logic errors) and Approvability (evaluates merge readiness). Your custom agents appear alongside them in the Checks tab.

Getting Started

1. Create a .md file in .macroscope/check-run-agents/ in your repo root (e.g. .macroscope/check-run-agents/web-review.md)
2. Configure the frontmatter fields you need (all optional) and write your instructions:

---
title: Security Review          # Name in GitHub Checks UI (max 60 chars)
model: claude-opus-4-6          # claude-opus-4-5, claude-opus-4-6, gpt-5-2
reasoning: low                  # off, low, medium, high
effort: low                     # low, medium, high
input: full_diff                # full_diff or code_object
tools:                          # Omit to use defaults
  - browse_code
  - git_tools
include:                        # Glob patterns for files to review
  - "src/**"
conclusion: neutral             # neutral (non-blocking) or failure (blocking)
---

Your instructions here.

3. Commit and push to your default branch.

Macroscope reads check run agent files from your default branch. Changes won’t take effect on new PRs until they’re merged.

Scope and Organization

Each .md file in .macroscope/check-run-agents/ becomes its own agent, scoped to the repo it lives in. We recommend starting with one file and splitting only when you need different configurations (e.g. different tools or input modes).

A typical repo structure:

.macroscope/
├── check-run-agents/
│   ├── security-review.md
│   └── web-review.md
├── approvability.md
└── ignore.md

The filenames approvability.md and ignore are reserved — they cannot be used as check run agent definitions. approvability.md configures custom approvability rules, and ignore controls which files are excluded from code review and Check Run Agents. These files always live in the .macroscope/ root, not in the check-run-agents/ subfolder.

Existing setup? If your check run agent files are in .macroscope/ instead of .macroscope/check-run-agents/, they will continue to work. We recommend moving them to the subfolder when convenient — the root location will stop being read in a future release. See Migrating to the subfolder for a one-line move.

If you already use a CLAUDE.md to guide how AI writes code, you can reference it from your check run agent’s instructions to keep coding conventions and review standards in sync.

File Format

The file has two parts: an optional YAML frontmatter block for settings and a markdown body with your instructions. Every frontmatter field is optional. If you omit frontmatter entirely, defaults apply and the filename becomes the title.

Field	Default	Options	Description
title	Derived from filename	Any string (max 60 chars)	Name in GitHub Checks UI
model	claude-opus-4-6	claude-opus-4-5, claude-opus-4-6, gpt-5-2	Which model powers the agent
reasoning	low	off, low, medium, high	Extended thinking depth
effort	low	low, medium, high	How deep the agent digs
input	full_diff	full_diff, code_object	How the PR diff is fed to the agent
tools	[browse_code, git_tools, github_api_read_only, modify_pr]	See Tools	Which tools the agent can use
include	none	Glob patterns	Only review files matching these patterns (e.g. "*.go", "src/**")
exclude	none	Glob patterns	Files to skip (e.g. "*.lock", "vendor/**")
conclusion	neutral	neutral, failure	Maximum severity the agent can report

include and exclude: Scoping by File

Use include to review only files matching your patterns, or exclude to skip files you don’t care about.

Configuration	Behavior
Neither set	All changed files are reviewed
Only include	Only files matching at least one pattern are reviewed
Only exclude	All files except those matching a pattern are reviewed
Both set	include wins — exclude is ignored (a configuration warning is emitted)

Don’t set both include and exclude. If you do, exclude is silently ignored and Macroscope emits a warning. Pick one.

Both fields accept the same glob syntax: "*.go", "src/**", "services/auth/**/*.go", etc.

.macroscope/ignore: Repository-Wide Exclusions

To exclude files from all check runs at once — not just a single agent — add patterns to your repository’s .macroscope/ignore file. These patterns are additive with include/exclude: a file is out of scope if it matches either the agent’s front matter patterns or the repository-wide ignore patterns.

How filtering behaves when every changed file is excluded:

Filtering source	Behavior
include / exclude only	Check run is not created (invisible in Checks tab)
.macroscope/ignore only	Check run is created but skipped (visible in Checks tab)
Both combined	Check run is created but skipped

When some files are in scope and the agent runs, both pattern sets shape what happens:

• In full_diff mode, the agent sees the entire PR diff for context. It is instructed to respect include/exclude and .macroscope/ignore patterns when posting comments or reporting issues — it won’t flag out-of-scope files unless your instructions explicitly override this.
• In code_object mode, out-of-scope code objects are filtered out before the agent runs — the agent never sees them.

Use include/exclude for per-agent scoping (e.g. “only Go files”) and .macroscope/ignore for repo-wide exclusions that apply to all checks (e.g. generated code, vendored dependencies).

input: Full Diff vs Code Object

Mode	What happens	When to use	Cost
full_diff (default)	One agent processes the entire PR diff	Enforcement that needs PR-level context (“new function added without tests”)	Lower
code_object	Up to 20 agents in parallel, one per changed code object	Per-unit enforcement (“don’t log PII in any changed file”)	Higher

conclusion: Blocking vs Non-Blocking

By default, the conclusion is capped at neutral: issues show up in the Checks tab but never block merging. Set conclusion: failure to let the agent block PRs when it finds issues.

Even with failure, the check still reports success when nothing is wrong. The setting only raises the ceiling on severity, not the floor.

See .macroscope/ignore: Repository-Wide Exclusions for details on how include/exclude and .macroscope/ignore interact when all files are filtered out.

Tools

Included by Default

Tool	What it does
browse_code	Explore the file tree, read files, search by filename or content
git_tools	Git log, blame, diff, grep
github_api_read_only	Read-only GitHub API: issues, labels, PR metadata, commit statuses
modify_pr	Update the PR (title, description, labels, assignees, reviewers) and post line-level review comments

Specifying tools: in frontmatter overrides the defaults. To keep defaults and add more, list them all.

Additional Tools

Tool	Requires	What it does
web_tools	No connection needed	Search the web, fetch URLs
slack	Slack connected	Post messages to channels, look up users
sentry	Sentry connected	Search error issues, view event history
posthog	PostHog connected	Query analytics, feature flags, session recordings
launchdarkly	LaunchDarkly connected	Query feature flags, targeting rules
bigquery	BigQuery connected	Run read-only SQL queries
amplitude	Amplitude connected	Event segmentation, funnels, retention
gcp_cloud_logging	GCP connected	Query log entries by severity, resource, timestamp
issue_tracking_tools	Jira or Linear connected	Read issues and projects
image_gen	No connection needed	Generate images from text prompts and upload them to GitHub or Slack
mcp	MCP server connected	Invoke tools from any connected MCP server

Connected tools require the integration in Settings > Connections. Missing connections are silently disabled.

Output and Instructions

Where Results Appear

Results appear in three places:

1. Check run details. Click into the check in the Checks tab to see the full report: title, summary, and detailed findings. Your instructions influence how this output is structured and formatted.
2. Inline PR comments. The agent posts comments directly on specific lines in the PR diff, attributed to the check name.
3. Issue comments. The agent can also post top-level comments on the PR itself for broader findings or summaries.

Controlling Output Format

You can control formatting in your instructions. Some examples:

• “Use a markdown table with columns: file, line, issue, severity”
• “Group findings by priority, critical first”
• “Use 🔴 🟡 🟢 emoji for severity levels”
• “Start with a one-line summary, then list details”
• “If no issues found, just say ‘All clear’ with no extra detail”
• “Format as a checklist so the reviewer can tick items off”
• “Be concise. Each bullet point should be under 20 words”

Writing Good Instructions

• Be specific. “Review for quality” is too vague. “Flag any function over 50 lines without a doc comment” is actionable.
• Define severity. Spell out what critical vs minor means for your team.
• Don’t replicate the Correctness check run. It already catches runtime bugs. Focus on your team’s conventions and workflows.
• Scope with include or exclude. If your check only applies to Go files, use include: ["*.go"]. If it applies to everything except lock files, use exclude: ["*.lock"].
• Give the agent permission to do nothing. “If nothing applies, report that no issues were found” prevents invented findings.
• Use sections in your instructions. Markdown headings (##) in the body help the agent organize its work and output.
• Reference specific paths. “Check files in services/auth/” is better than “check auth code.”
• Tell it what not to flag. “Ignore test files” or “don’t flag TODOs in draft PRs” reduces noise.

Example

We recommend consolidating related rules into a single check rather than splitting across many files. Here’s a web team example that handles several concerns in one agent — saved as .macroscope/check-run-agents/web-review.md:

---
title: Web Review
model: claude-opus-4-6
effort: medium
input: full_diff
tools:
  - browse_code
  - git_tools
  - modify_pr
  - slack
  - sentry
include:
  - "targets/app/**"
  - "web-etc/**"
---

Review this PR against our web team's standards:

## Event Tracking
If this PR touches payment flows, signup funnels, analytics calls,
CTA buttons, or redirect logic, check whether it could break event
tracking. Rate each issue: 🔴 will stop firing, 🟡 may fire incorrectly,
🟢 low risk.

## Accessibility
Check new or modified React components for basic accessibility:
- Images must have alt text
- Buttons and links must have accessible labels
- Form inputs must have associated labels

## Production Errors
For each file modified, check Sentry for unresolved issues. If any
active errors exist, list them with frequency and last seen date.

## Labels
Add labels to this PR based on what changed:
- "frontend" if any UI components are modified
- "styles" if CSS or styled-components changed
- "docs" if only markdown files changed

## Notifications
If any 🔴/🟡 event tracking issues or accessibility violations are
found, post a summary to #eng on Slack with the PR link.

If nothing noteworthy is found, report that all checks passed.

Migrating to the Subfolder

If you have check run agent files in the root .macroscope/ directory, move them into the check-run-agents/ subfolder:

mkdir -p .macroscope/check-run-agents
mv .macroscope/web-review.md .macroscope/check-run-agents/
# Repeat for each check run agent file. Do not move approvability.md or ignore.

No other changes are needed — the file format and frontmatter are identical. Commit and merge to your default branch. New PRs will use the updated location automatically.

approvability.md and ignore are not check run agent files — leave them in the .macroscope/ root.

Migrating from Custom Rules

Check Run Agents replace Custom Rules. If you have an existing macroscope.md file, move your rules into .macroscope/check-run-agents/my-rules.md instead. Your existing rules become the instructions body, and you can optionally add frontmatter for model, tools, and input mode.

Check Run Agents go further: agents can browse the codebase, query git history, post to Slack, check Sentry, and more. You also get structured output in the Checks tab instead of just inline comments.

Billing

Check Run Agents are free during beta. We will announce pricing changes in advance.

To keep costs down: use include/exclude or .macroscope/ignore to scope to relevant files, prefer full_diff over code_object, and use lower effort for simple checks.