root
/
gemini-cli
mirror of https://github.com/google-gemini/gemini-cli.git
1
0
Fork 0
Code Issues Actions 7 Packages Projects Releases Wiki Activity
gemini-cli/docs/release-confidence.md

5.5 KiB
Raw Permalink Blame History

Release confidence strategy

This document outlines the strategy for gaining confidence in every release of the Gemini CLI. It serves as a checklist and quality gate for release manager to ensure we are shipping a high-quality product.

The goal

To answer the question, "Is this release truly ready for our users?" with a high degree of confidence, based on a holistic evaluation of automated signals, manual verification, and data.

Level 1: Automated gates (must pass)

These are the baseline requirements. If any of these fail, the release is a no-go.

1. CI/CD health

All workflows in .github/workflows/ci.yml must pass on the main branch (for nightly) or the release branch (for preview/stable).

  • Platforms: Tests must pass on Linux and macOS.
    • Note: Windows tests currently run with continue-on-error: true. While a failure here doesn't block the release technically, it should be investigated.
  • Checks:
    • Linting: No linting errors (ESLint, Prettier, etc.).
    • Typechecking: No TypeScript errors.
    • Unit Tests: All unit tests in packages/core and packages/cli must pass.
    • Build: The project must build and bundle successfully.

2. End-to-end (E2E) tests

All workflows in .github/workflows/chained_e2e.yml must pass.

  • Platforms: Linux, macOS and Windows.
  • Sandboxing: Tests must pass with both sandbox:none and sandbox:docker on Linux.

3. Post-deployment smoke tests

After a release is published to npm, the smoke-test.yml workflow runs. This must pass to confirm the package is installable and the binary is executable.

  • Command: npx -y @google/gemini-cli@<tag> --version must return the correct version without error.
  • Platform: Currently runs on ubuntu-latest.

Level 2: Manual verification and dogfooding

Automated tests cannot catch everything, especially UX issues.

1. Dogfooding via preview tag

The weekly release cadence promotes code from main -> nightly -> preview -> stable.

  • Requirement: The preview release must be used by maintainers for at least one week before being promoted to stable.
  • Action: Maintainers should install the preview version locally: 
    npm install -g @google/gemini-cli@preview
  • Goal: To catch regressions and UX issues in day-to-day usage before they reach the broad user base.

2. Critical user journey (CUJ) checklist

Before promoting a preview release to stable, a release manager must manually run through this checklist.

  • Setup:

    • Uninstall any existing global version: npm uninstall -g @google/gemini-cli
    • Clear npx cache (optional but recommended): npm cache clean --force
    • Install the preview version: npm install -g @google/gemini-cli@preview
    • Verify version: gemini --version

  • Authentication:

    • In interactive mode run /auth and verify all login flows work:
      • Login With Google
      • API Key
      • Vertex AI

  • Basic prompting:

    • Run gemini "Tell me a joke" and verify a sensible response.
    • Run in interactive mode: gemini. Ask a follow-up question to test context.

  • Piped input:

    • Run echo "Summarize this" | gemini and verify it processes stdin.

  • Context management:

    • In interactive mode, use @file to add a local file to context. Ask a question about it.

  • Settings:

    • In interactive mode run /settings and make modifications
    • Validate that setting is changed

  • Function calling:

    • In interactive mode, ask gemini to "create a file named hello.md with the content 'hello world'" and verify the file is created correctly.

If any of these CUJs fail, the release is a no-go until a patch is applied to the preview channel.

3. Pre-Launch bug bash (tier 1 and 2 launches)

For high-impact releases, an organized bug bash is required to ensure a higher level of quality and to catch issues across a wider range of environments and use cases.

Definition of tiers:

  • Tier 1: Industry-Moving News 🚀
  • Tier 2: Important News for Our Users 📣
  • Tier 3: Relevant, but Not Life-Changing 💡
  • Tier 4: Bug Fixes ⚒️

Requirement:

A bug bash must be scheduled at least 72 hours in advance of any Tier 1 or Tier 2 launch.

Rule of thumb:

A bug bash should be considered for any release that involves:

  • A blog post
  • Coordinated social media announcements
  • Media relations or press outreach
  • A "Turbo" launch event

Level 3: Telemetry and data review

Dashboard health

  • Go to go/gemini-cli-dash.
  • Navigate to the "Tool Call" tab.
  • Validate that there are no spikes in errors for the release you would like to promote.

Model evaluation

  • Navigate to go/gemini-cli-offline-evals-dash.
  • Make sure that the release you want to promote's recurring run is within average eval runs.

The "go/no-go" decision

Before triggering the Release: Promote workflow to move preview to stable:

  1. Level 1: CI and E2E workflows are green for the commit corresponding to the current preview tag.
  2. Level 2: The preview version has been out for one week, and the CUJ checklist has been completed successfully by a release manager. No blocking issues have been reported.
  3. Level 3: Dashboard Health and Model Evaluation checks have been completed and show no regressions.

If all checks pass, proceed with the promotion.