AGENTS.md in CI/CD: Automated Diff Review, Validation, and Drift Detection

AGENTS.md has a specific failure mode that doesn’t happen to most config files: it goes stale without breaking anything. A wrong package.json causes a build failure. A stale AGENTS.md just makes your AI agents work from incorrect assumptions — and you won’t know until someone complains that the agent keeps using the old test command or keeps touching files it was told to leave alone.

CI/CD is the right place to solve this. Not because you need to run AGENTS.md through a linter every push, but because CI is where you catch drift: when your pnpm-lock.yaml changes, did someone update the AGENTS.md install command? When a new protected directory appears, is it mentioned? When the test runner changes from Jest to Vitest, did the AGENTS.md catch up?

This guide covers practical GitHub Actions workflows, pre-commit hooks, and a lightweight review bot setup for keeping AGENTS.md in sync with the actual codebase.

The Core Problem: Config Files Change, AGENTS.md Doesn’t

Here’s the drift pattern in practice. Your team migrates from Jest to Vitest over two sprints. Someone updates package.json, vitest.config.ts, all the test files. The PR passes CI. But AGENTS.md still says npx jest --testPathPattern= and your AI agents inherit that bad information.

The fix is a CI check that flags when key config files change without a corresponding AGENTS.md update. You’re not validating AGENTS.md content — you’re detecting when it probably needs attention.

GitHub Actions: Drift Detection

The most useful check is surprisingly simple. When a PR modifies any file on a watchlist, verify that AGENTS.md was also touched. If it wasn’t, post a comment requiring the author to either update AGENTS.md or explicitly confirm it doesn’t need changes.

# .github/workflows/agents-md-check.yml
name: AGENTS.md drift check

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  agents-md-drift:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Check for drift-triggering changes
        id: drift-check
        run: |
          BASE=${{ github.event.pull_request.base.sha }}
          HEAD=${{ github.event.pull_request.head.sha }}

          # Files that should trigger an AGENTS.md review when changed
          WATCHED_PATTERNS=(
            "package.json"
            "pnpm-lock.yaml"
            "package-lock.json"
            "yarn.lock"
            "Makefile"
            "*.config.ts"
            "*.config.js"
            "docker-compose*.yml"
            ".env.example"
            ".env.ci"
          )

          # Build grep pattern
          PATTERN=$(IFS="|"; echo "${WATCHED_PATTERNS[*]}")
          
          CHANGED=$(git diff --name-only "$BASE" "$HEAD")
          AGENTS_CHANGED=$(echo "$CHANGED" | grep -c "AGENTS.md" || true)
          CONFIG_CHANGED=$(echo "$CHANGED" | grep -E "$PATTERN" | grep -v "node_modules" || true)

          if [ -n "$CONFIG_CHANGED" ] && [ "$AGENTS_CHANGED" -eq 0 ]; then
            echo "drift_detected=true" >> $GITHUB_OUTPUT
            echo "changed_files<<EOF" >> $GITHUB_OUTPUT
            echo "$CONFIG_CHANGED" >> $GITHUB_OUTPUT
            echo "EOF" >> $GITHUB_OUTPUT
          else
            echo "drift_detected=false" >> $GITHUB_OUTPUT
          fi

      - name: Comment on PR if drift detected
        if: steps.drift-check.outputs.drift_detected == 'true'
        uses: actions/github-script@v7
        with:
          script: |
            const changedFiles = `${{ steps.drift-check.outputs.changed_files }}`;
            
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## AGENTS.md Review Required
            
            The following config files changed in this PR, but \`AGENTS.md\` was not updated:
            
            \`\`\`
            ${changedFiles}
            \`\`\`
            
            **Action required:** Either update \`AGENTS.md\` to reflect any changed commands, paths, or conventions — or add a comment explaining why no update is needed.
            
            <details>
            <summary>Common reasons to update AGENTS.md</summary>
            
            - Package manager or lock file changed → update install command
            - \`*.config.ts\` changed → check if build/test/lint commands changed
            - \`docker-compose.yml\` changed → check if dev setup instructions changed
            - \`.env.example\` changed → check if environment variable docs need updating
            </details>`
            });

This gives you a comment-based nudge rather than a hard failure — which is the right call for AGENTS.md, because the relationship between config files and AGENTS.md content is semantic, not syntactic. A human needs to decide if an update is warranted.

Pre-commit Hook: Syntax and Structure Validation

For structure validation, a pre-commit hook that runs fast and fails loudly is more useful than CI. This checks that AGENTS.md has the expected sections before the commit lands.

#!/bin/bash
# .git/hooks/pre-commit (or via pre-commit framework)
# Also useful as scripts/validate-agents-md.sh

set -e

AGENTS_FILE="AGENTS.md"

if [ ! -f "$AGENTS_FILE" ]; then
  echo "ERROR: AGENTS.md not found at repo root"
  exit 1
fi

# Check for required sections
REQUIRED_SECTIONS=(
  "## Commands"
  "## Code Style"
)

MISSING=()
for section in "${REQUIRED_SECTIONS[@]}"; do
  if ! grep -q "^${section}$" "$AGENTS_FILE"; then
    MISSING+=("$section")
  fi
done

if [ ${#MISSING[@]} -gt 0 ]; then
  echo "ERROR: AGENTS.md is missing required sections:"
  printf '  %s\n' "${MISSING[@]}"
  echo ""
  echo "Add these sections or update REQUIRED_SECTIONS in .git/hooks/pre-commit"
  exit 1
fi

# Check for obvious stale markers
STALE_PATTERNS=(
  "npm install"   # if you're using pnpm
  "yarn add"      # if you're using pnpm
  "TODO:"
  "FIXME:"
)

for pattern in "${STALE_PATTERNS[@]}"; do
  if grep -qi "$pattern" "$AGENTS_FILE"; then
    echo "WARNING: AGENTS.md contains potentially stale content: '$pattern'"
    echo "Review and update or remove before committing."
    # Warning only — don't block commit for this
  fi
done

echo "AGENTS.md validation passed"
exit 0

To use this with the pre-commit framework (which is more manageable than raw hooks):

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: agents-md-validate
        name: Validate AGENTS.md structure
        entry: scripts/validate-agents-md.sh
        language: script
        files: AGENTS.md
        pass_filenames: false

CI: Command Verification

The most reliable AGENTS.md check is also the most direct: actually run the commands it claims are correct.

# .github/workflows/verify-agents-md-commands.yml
name: Verify AGENTS.md commands

on:
  push:
    paths:
      - 'AGENTS.md'
  schedule:
    - cron: '0 6 * * 1'  # Weekly on Monday — catches drift from other merges

jobs:
  verify-commands:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: pnpm/action-setup@v4
        with:
          version: 9

      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'pnpm'

      - name: Parse and verify build command from AGENTS.md
        run: |
          # Extract build command from AGENTS.md
          BUILD_CMD=$(grep -A1 "^\- \*\*Build\*\*:" AGENTS.md | head -1 | sed 's/.*`\(.*\)`.*/\1/')
          
          if [ -z "$BUILD_CMD" ]; then
            echo "Could not parse build command from AGENTS.md"
            exit 1
          fi
          
          echo "Testing build command from AGENTS.md: $BUILD_CMD"
          pnpm install --frozen-lockfile
          eval "$BUILD_CMD"

      - name: Parse and verify test command from AGENTS.md
        run: |
          TEST_CMD=$(grep -A1 "^\- \*\*Test\*\*:" AGENTS.md | head -1 | sed 's/.*`\(.*\)`.*/\1/')
          
          if [ -z "$TEST_CMD" ]; then
            echo "Could not parse test command from AGENTS.md"
            exit 1
          fi
          
          echo "Testing test command from AGENTS.md: $TEST_CMD"
          eval "$TEST_CMD"

This approach requires that you document commands in AGENTS.md with a consistent format (- **Build**: `cmd`), which is a good practice anyway. The weekly schedule catches cases where the codebase changed but AGENTS.md wasn’t updated in the same PR.

Protected Path Verification

If your AGENTS.md lists directories that agents should never touch, verify those paths actually exist:

#!/bin/bash
# scripts/verify-agents-md-paths.sh

AGENTS_FILE="AGENTS.md"

# Extract lines starting with "Never modify" or "Do not touch" patterns
PROTECTED_PATHS=$(grep -oE '`[^`]+/[^`]+`' "$AGENTS_FILE" | tr -d '`' | grep "/" | sort -u)

MISSING_PATHS=()
for path in $PROTECTED_PATHS; do
  # Skip if it looks like a command pattern
  if [[ "$path" == *"$"* ]] || [[ "$path" == *"*"* ]]; then
    continue
  fi
  
  if [ ! -e "$path" ]; then
    MISSING_PATHS+=("$path")
  fi
done

if [ ${#MISSING_PATHS[@]} -gt 0 ]; then
  echo "WARNING: AGENTS.md references paths that don't exist:"
  printf '  %s\n' "${MISSING_PATHS[@]}"
  echo ""
  echo "These may be stale references. Review AGENTS.md and remove or update them."
  exit 1
fi

echo "All paths referenced in AGENTS.md exist."

The Scheduled Review: Weekly Freshness Check

Beyond PR-time checks, a weekly scheduled workflow that runs command verification catches the drift that happens when multiple small PRs accumulate. Each individual PR touches only one config file and doesn’t seem to need an AGENTS.md update, but after four PRs the document is noticeably wrong.

# .github/workflows/agents-md-weekly-review.yml
name: Weekly AGENTS.md health check

on:
  schedule:
    - cron: '0 9 * * 1'  # Monday 9am UTC
  workflow_dispatch:

jobs:
  health-check:
    runs-on: ubuntu-latest
    permissions:
      issues: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Run full AGENTS.md verification
        id: verify
        run: |
          bash scripts/validate-agents-md.sh 2>&1 | tee verify-output.txt
          echo "exit_code=$?" >> $GITHUB_OUTPUT

      - name: Create issue if verification fails
        if: steps.verify.outputs.exit_code != '0'
        uses: actions/github-script@v7
        with:
          script: |
            const output = require('fs').readFileSync('verify-output.txt', 'utf8');
            
            // Check if there's already an open issue for this
            const issues = await github.rest.issues.listForRepo({
              owner: context.repo.owner,
              repo: context.repo.repo,
              state: 'open',
              labels: 'agents-md-stale'
            });
            
            if (issues.data.length > 0) {
              console.log('Stale AGENTS.md issue already open, skipping creation');
              return;
            }
            
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: 'AGENTS.md may be stale — weekly check failed',
              body: `The weekly AGENTS.md health check failed.\n\n\`\`\`\n${output}\n\`\`\`\n\nReview AGENTS.md and update as needed.`,
              labels: ['agents-md-stale', 'maintenance']
            });

Deciding What to Automate

Not everything about AGENTS.md maintenance should be automated. The check table:

The automation handles mechanical correctness. Content quality is still human work. The goal is removing the “I forgot to update AGENTS.md” failure mode — not replacing judgment about what the file should say.

Related Articles

• AGENTS.md in Monorepos: Per-Package Instructions Without the Chaos
• Writing Effective AGENTS.md: The Definitive Guide
• AGENTS.md vs CLAUDE.md: What’s the Difference