grafana/pkg/operators/provisioning

Provisioning Controllers

Git sync has two different controllers: the jobs controller and the repo controller.

Jobs Controller

Behavior

• Watches provisioning Jobs and emits notifications on job creation.

• Optionally cleans up HistoricJobs after a configurable expiration. Disable when job history is stored in Loki.

• Queueing and claiming:

  • Creating a Job enqueues work. Drivers “claim” one job at a time under a time-bound lease so only one worker processes it at once.
  • If a driver crashes or loses its lease, cleanup makes the job eligible to be claimed again. This yields at-least-once processing.
  • New job notifications reduce latency; periodic ticks ensure progress even without notifications.

• Processing and status:

  • A supporting worker processes the job, renewing the lease in the background. If lease renewal fails or expires, processing aborts.
  • Status updates are persisted with conflict-aware retries. Progress is throttled to avoid excessive writes while still providing timely feedback.
  • When processing finishes, the job is marked complete and a copy is written to history.

• Historic jobs role:

  • Historic jobs are a read-only audit trail and UX surface for recent job outcomes, progress summaries, errors, and reference URLs.

• Retention is implementation-dependent: this controller can prune old history objects periodically, or history can be stored in Loki; when using Loki, disable local cleanup with --history-expiration=0.

This binary currently wires informers and emits job-create notifications. In the full setup, concurrent drivers consume notifications and execute workers to process jobs using the behavior above.

Flags

• --token (string): Token to use for authentication against the provisioning API.
• --token-exchange-url (string): Token exchange endpoint used to mint the access token for the provisioning API.
• --provisioning-server-url (string): Base URL to the provisioning API server (e.g., https://localhost:6446).
• --history-expiration (duration): If greater than zero, enables HistoricJobs cleanup and sets the retention window (e.g., 30s, 15m, 24h). If 0, cleanup is disabled.

TLS Configuration

• --tls-insecure (bool): Skip TLS certificate verification. Default: true (for development/testing).
• --tls-cert-file (string): Path to TLS client certificate file for mutual TLS authentication.
• --tls-key-file (string): Path to TLS client private key file for mutual TLS authentication.
• --tls-ca-file (string): Path to TLS CA certificate file for server certificate verification.

How to run

1. Build grafana:
   • make build
2. Ensure the following services are running locally: provisioning API server, secrets service API server, repository controller, unified storage, and auth.
3. Create a operator.ini file:

[database]
ensure_default_org_and_user = false
skip_migrations = true
[operator]
provisioning_server_url = https://localhost:6446
tls_insecure = true

[grpc_client_authentication]
token = ProvisioningAdminToken
token_exchange_url = http://localhost:6481/sign/access-token
# Uncomment to enable history cleanup via Loki. First ensure the Provisioning API is configured with Loki for job history (see `createJobHistoryConfigFromSettings` in `pkg/registry/apis/provisioning/register.go`).
# history_expiration = 24h  

3. Start the controller:

• GF_DEFAULT_TARGET=operator GF_OPERATOR_NAME=provisioning-jobs ./bin/darwin-arm64/grafana server target --config=conf/operator.ini

TLS Configuration Examples

• Production with proper TLS verification:

[operator]
provisioning_server_url = https://localhost:6446
tls_insecure = false
tls_ca_file = /path/to/ca-cert.pem

[grpc_client_authentication]
token = ProvisioningAdminToken
token_exchange_url = http://localhost:6481/sign/access-token

• Mutual TLS authentication:

[operator]
provisioning_server_url = https://localhost:6446
tls_insecure = false
tls_ca_file = /path/to/ca-cert.pem
tls_cert_file = /path/to/client-cert.pem 
tls_key_file = /path/to/client-key.pem

[grpc_client_authentication]
token = ProvisioningAdminToken
token_exchange_url = http://localhost:6481/sign/access-token

• Development with self-signed certificates (insecure):

[operator]
provisioning_server_url = https://localhost:6446
tls_insecure = true

[grpc_client_authentication]
token = ProvisioningAdminToken
token_exchange_url = http://localhost:6481/sign/access-token

Expected behavior

1. Create a repository and enqueue a job (note that the repository must be marked as healthy):

export ACCESS_TOKEN=$(curl -X POST http://localhost:6481/sign/access-token \
  -H "X-Realms: [{\"type\":\"system\",\"identifier\":\"system\"}]" \
  -H "X-Org-ID: 0" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ProvisioningAdminToken" \
  -d '{
    "namespace": "*",
    "audiences": ["provisioning.grafana.app"]
  }' | jq -r '.data.token')

curl -X POST https://localhost:6446/apis/provisioning.grafana.app/v0alpha1/namespaces/default/repositories/test6/jobs \
  -H "Content-Type: application/json" --insecure \
  -H "X-Access-Token: Bearer $ACCESS_TOKEN" \
  -d '{
    "action": "pull",
    "pull": {
      "incremental": false
    }
  }'

2. The controller emits a notification on job creation.

➜  job-controller git:(feature/standalone-job-controller) ✗ ./bin/job-controller --token-exchange-url=http://localhost:6481/sign/access-token --token=ProvisioningAdminToken --provisioning-server-url=https://localhost:6446
{"time":"2025-08-21T14:27:03.789337+02:00","level":"INFO","msg":"job create notification received","logger":"provisioning-job-controller"}

3. In a full setup with the concurrent driver, workers claim and process jobs, updating status and writing history.
4. Entries move to HistoricJobs; if cleanup is enabled, older entries are pruned based on --history-expiration.

[WIP] Repository Controller

This controller is responsible for watching repositories. It will eventually do health checks, queue sync jobs, and create/delete github hooks.

To run locally, run GF_DEFAULT_TARGET=operator GF_OPERATOR_NAME=provisioning-repo ./bin/darwin-arm64/grafana server target --config=conf/operator.ini