fly-e2e-test

Deploy a fresh dspy-cli project to Fly.io using your local code changes, run full integration tests (health, auth, LLM execution), and guarantee cleanup regardless of success or failure.

⚠️ CRITICAL RULES

1. NEVER commit directly to main - Always create a side branch first, even for small changes
2. ALWAYS clean up - Destroy Fly apps and delete temp branches, even if tests fail
3. Use temp branches - Name them e2e-test/{timestamp}-{random} for easy identification

Prerequisites

1. fly CLI: Installed and authenticated (fly auth whoami)
2. OPENAI_API_KEY: In environment or .env file
3. Git: Clean working directory (stash uncommitted changes first)
4. Git push access: Ability to push to origin

Quick Start

Run each phase in a tmux session to enable output capture and cleanup tracking.

Phase 1: Setup Environment

# Create tmux session
tmux new-session -d -s e2e-fly -c /Users/isaac/projects/dspy-cli

# Set variables
tmux send-keys -t e2e-fly 'export DSPY_CLI_DIR="/Users/isaac/projects/dspy-cli"' C-m
tmux send-keys -t e2e-fly 'export TIMESTAMP=$(date +%s)' C-m
tmux send-keys -t e2e-fly 'export RANDOM_SUFFIX=$(head -c 4 /dev/urandom | xxd -p)' C-m
tmux send-keys -t e2e-fly 'export FLY_APP_NAME="dspy-e2e-${RANDOM_SUFFIX}"' C-m
tmux send-keys -t e2e-fly 'export TEMP_BRANCH="e2e-test/${TIMESTAMP}-${RANDOM_SUFFIX}"' C-m

# Source .env for OPENAI_API_KEY
tmux send-keys -t e2e-fly 'set -a && source .env && set +a' C-m

# Verify setup
tmux send-keys -t e2e-fly 'echo "App: $FLY_APP_NAME Branch: $TEMP_BRANCH"' C-m

Phase 2: Pre-flight Checks

# Verify fly CLI
tmux send-keys -t e2e-fly 'fly version && fly auth whoami' C-m

# Check for uncommitted changes (stash if needed)
tmux send-keys -t e2e-fly 'git status --porcelain' C-m

# Clean up any orphaned e2e resources
tmux send-keys -t e2e-fly 'fly apps list 2>/dev/null | grep "dspy-e2e" || echo "No orphaned apps"' C-m

Phase 3: Create and Push Temp Branch

tmux send-keys -t e2e-fly 'git checkout -b "$TEMP_BRANCH"' C-m
tmux send-keys -t e2e-fly 'git push -u origin "$TEMP_BRANCH"' C-m

Phase 4: Create Test Project

# Create temp directory
tmux send-keys -t e2e-fly 'export TEST_DIR=$(mktemp -d) && echo "TEST_DIR=$TEST_DIR"' C-m

# Create project (will prompt for API key confirmation - send Y)
tmux send-keys -t e2e-fly 'uv run --directory "$DSPY_CLI_DIR" dspy-cli new fly-e2e-test --program-name qa_module --signature "question:str -> answer:str" --module-type Predict --model openai/gpt-4o-mini' C-m

# When prompted "Proceed with this API key? [Y/n]:", send:
tmux send-keys -t e2e-fly 'Y' C-m

# Move project to temp dir (dspy-cli creates in current dir)
tmux send-keys -t e2e-fly 'mv "$DSPY_CLI_DIR/fly-e2e-test" "$TEST_DIR/" && cd "$TEST_DIR/fly-e2e-test"' C-m

Phase 5: Modify for Git-Based dspy-cli

# Update pyproject.toml to install dspy-cli from temp branch
tmux send-keys -t e2e-fly 'sed -i.bak "s|\"dspy-cli\"|\"dspy-cli @ git+https://github.com/cmpnd-ai/dspy-cli.git@$TEMP_BRANCH\"|" pyproject.toml' C-m

# IMPORTANT: Update Dockerfile to include git (required for git-based deps)
# NOTE: This is an example dockerfile. There may be specific changes in a newer version of dspy-cli. Check the current Dockerfile and add the Git install line
tmux send-keys -t e2e-fly 'cat > Dockerfile << '"'"'EOF'"'"'
FROM python:3.11-slim

ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV XDG_CACHE_HOME=/tmp/.cache

# Install git for fetching dspy-cli from git URL
RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*

COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

COPY . .
RUN uv sync --no-dev

EXPOSE 8000

CMD ["uv", "run", "dspy-cli", "serve", "--host", "0.0.0.0", "--port", "8000", "--auth", "--no-reload"]
EOF' C-m

Phase 6: Create fly.toml and Deploy

# Create fly.toml
tmux send-keys -t e2e-fly 'cat > fly.toml << EOF
app = '"'"'$FLY_APP_NAME'"'"'
primary_region = '"'"'ewr'"'"'

[build]

[http_service]
  internal_port = 8000
  force_https = true
  auto_stop_machines = true
  auto_start_machines = true
  min_machines_running = 0
  processes = ['"'"'app'"'"']

[[vm]]
  memory = '"'"'512mb'"'"'
  cpu_kind = '"'"'shared'"'"'
  cpus = 1
EOF' C-m

# Create app
tmux send-keys -t e2e-fly 'fly apps create "$FLY_APP_NAME" --org personal' C-m

# Generate a random API key for testing
tmux send-keys -t e2e-fly 'export DSPY_API_KEY_VALUE="test-e2e-$(head -c 8 /dev/urandom | xxd -p)"' C-m

# Set secrets using fly secrets (required env vars for your app)
# Add any additional env vars your project needs here
tmux send-keys -t e2e-fly 'fly secrets set OPENAI_API_KEY="$OPENAI_API_KEY" DSPY_API_KEY="$DSPY_API_KEY_VALUE" --app "$FLY_APP_NAME"' C-m

# Deploy (takes ~2-3 minutes)
tmux send-keys -t e2e-fly 'fly deploy --app "$FLY_APP_NAME" --wait-timeout 300' C-m

Phase 7: Run Integration Tests

tmux send-keys -t e2e-fly 'export FLY_APP_URL="https://$FLY_APP_NAME.fly.dev"' C-m

# Test 1: Health Check
tmux send-keys -t e2e-fly 'echo "=== Test 1: Health Check ===" && curl -s "$FLY_APP_URL/health"' C-m
# Expected: {"status":"ok"}

# Test 2: Auth Redirect (unauthenticated)
tmux send-keys -t e2e-fly 'echo "=== Test 2: Auth Redirect ===" && curl -s -o /dev/null -w "HTTP: %{http_code}\n" "$FLY_APP_URL/programs"' C-m
# Expected: HTTP: 303

# Test 3: Auth Success (authenticated)
tmux send-keys -t e2e-fly 'echo "=== Test 3: Auth Success ===" && curl -s -H "Authorization: Bearer $DSPY_API_KEY_VALUE" "$FLY_APP_URL/programs"' C-m
# Expected: {"programs":[{"name":"QaModulePredict",...}]}

# Test 4: LLM Module Execution
tmux send-keys -t e2e-fly 'echo "=== Test 4: LLM Execution ===" && curl -s -X POST -H "Authorization: Bearer $DSPY_API_KEY_VALUE" -H "Content-Type: application/json" -d '"'"'{"question": "What is 2+2?"}'"'"' "$FLY_APP_URL/QaModulePredict"' C-m
# Expected: {"answer":"4"} (or similar)

Phase 8: Guaranteed Cleanup

ALWAYS run cleanup, even if tests fail:

# Destroy Fly app
tmux send-keys -t e2e-fly 'fly apps destroy "$FLY_APP_NAME" --yes' C-m

# Delete remote branch
tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" push origin --delete "$TEMP_BRANCH"' C-m

# Return to main and delete local branch
tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" checkout main' C-m
tmux send-keys -t e2e-fly 'git -C "$DSPY_CLI_DIR" branch -D "$TEMP_BRANCH"' C-m

# Remove temp directory
tmux send-keys -t e2e-fly 'rm -rf "$TEST_DIR"' C-m

# Kill tmux session
tmux kill-session -t e2e-fly

Verification Checklist

| Test | Expected Result | |------|-----------------| | Health Check | {"status":"ok"} | | Auth Redirect (no auth) | HTTP 303 | | Auth Success (Bearer token) | JSON with QaModulePredict | | LLM Execution | JSON with "answer" field |

Cleanup Verification

After running cleanup, verify:

# No orphaned Fly apps
fly apps list | grep "dspy-e2e" || echo "Clean"

# No orphaned branches
git branch -r | grep "e2e-test/" || echo "Clean"

Troubleshooting

Deploy fails with "Git executable not found"

The Dockerfile must include git installation. Ensure the Dockerfile has:

RUN apt-get update && apt-get install -y git && rm -rf /var/lib/apt/lists/*

pyproject.toml sed command doesn't expand variables

Use double quotes for the sed command, not single quotes:

sed -i.bak "s|...|...@$TEMP_BRANCH\"|" pyproject.toml  # Correct
sed -i.bak 's|...|...|' pyproject.toml  # Won't expand $TEMP_BRANCH

Project created in wrong directory

dspy-cli new creates projects relative to the current working directory, not where it's run from. Move the project after creation:

mv "$DSPY_CLI_DIR/fly-e2e-test" "$TEST_DIR/"

Cleanup fails

If any cleanup step fails, run them individually:

fly apps destroy "dspy-e2e-XXXX" --yes
git push origin --delete "e2e-test/XXXX"
git checkout main
git branch -D "e2e-test/XXXX"

App crashes due to missing environment variables

Use fly secrets to set any required env vars. Check the app logs to see which vars are missing:

# View logs to find missing env vars
fly logs --app "$FLY_APP_NAME" --no-tail

# Set additional secrets as needed
fly secrets set VAR_NAME="value" ANOTHER_VAR="value" --app "$FLY_APP_NAME"

# List current secrets
fly secrets list --app "$FLY_APP_NAME"

Common env vars that might be needed:

• OPENAI_API_KEY - Required for OpenAI models
• DSPY_API_KEY - Required when --auth is enabled
• Project-specific vars (check your gateway's setup() method)

Multi-Layer Cleanup Protection

1. Unique naming: dspy-e2e-{random} prevents conflicts
2. Pre-test orphan cleanup: Removes stale resources before starting
3. tmux session: Enables output capture and manual recovery
4. Explicit cleanup phase: Always runs after tests
5. Verification commands: Confirm cleanup succeeded