Secrets Management #

This runbook covers how to view, update, and manage secrets and environment variables using Doppler across different environments.

When to Use This Runbook #

Need to view configuration for debugging
Updating API keys or credentials
Rotating secrets after security incidents
Adding new environment variables for features
Troubleshooting missing or incorrect configuration

Overview #

BonsAI uses Doppler for centralized secrets management across all environments. Secrets are synced from Doppler to:

Local development - via .env files or Doppler CLI
Kubernetes - via External Secrets Operator
GitHub Actions - via Doppler service tokens

Doppler Project Structure #

Project: bonsai
├── dev_local     (Local development)
├── dev_aws       (Development environment)
└── prod          (Production environment)

Prerequisites #

Installing Doppler CLI #

# macOS
brew install dopplerhq/cli/doppler

# Linux
curl -sLf --retry 3 --tlsv1.2 --proto "=https" 'https://packages.doppler.com/public/cli/install.sh' | sudo bash

# Verify installation
doppler --version

Logging In #

# Initial login (opens browser for auth)
doppler login

# Verify login
doppler me

Setting Up Project Locally #

# From the bonsai repository root
cd ~/bonsai

# Run setup script
mise run doppler-setup

# Or manually:
doppler setup --project bonsai --config dev_local

Viewing Secrets #

List All Secrets #

# View all secrets for dev_local
doppler secrets --project bonsai --config dev_local

# View secrets in table format
doppler secrets --project bonsai --config dev_local --table

# View only production secrets (requires appropriate access)
doppler secrets --project bonsai --config prod

Get Specific Secret #

# Get single secret value
doppler secrets get DATABASE_URL --project bonsai --config dev_local --plain

# Get multiple secrets
doppler secrets get DATABASE_URL REDIS_HOST --project bonsai --config dev_local

Download All Secrets #

# Download as .env file
doppler secrets download --project bonsai --config dev_local --no-file --format env > .env

# Download as JSON
doppler secrets download --project bonsai --config dev_local --no-file --format json > secrets.json

WARNING: Be careful with downloaded secret files. Never commit them to git!

Updating Secrets #

Update Single Secret #

# Set new value
doppler secrets set API_KEY="new-api-key-value" --project bonsai --config dev_local

# Update from stdin (for complex values)
echo "my-secret-value" | doppler secrets set SECRET_NAME --project bonsai --config dev_local

Update Multiple Secrets #

# From .env file
doppler secrets upload .env --project bonsai --config dev_local

# Interactive mode
doppler secrets set --project bonsai --config dev_local
# Then enter: KEY=value

Bulk Updates via Doppler Dashboard #

Go to Doppler Dashboard
Select project: bonsai
Select config: dev_local, dev, or prod
Click Add Secret or edit existing values
Changes are immediately available

Rotating Secrets #

Emergency Secret Rotation #

When a secret is compromised:

Assess Impact
- Which secret was compromised?
- Which environments are affected?
- Which services use this secret?

Update Secret in Doppler

# Production example (requires approval)
doppler secrets set CLERK_SECRET_KEY="new-secret-key" \
  --project bonsai --config prod

Sync to Kubernetes

# Force External Secrets sync
kubectl delete externalsecret bonsai-external-secret
kubectl apply -f deployment/resources/manifests/external_secret.yaml

# Verify secret updated
kubectl get secret bonsai-secret -o yaml

Restart Affected Services

# Restart deployment to pick up new secret
kubectl rollout restart deployment/bonsapi-deployment
kubectl rollout restart deployment/webapp-deployment

Verify Services
- Check pod logs for errors
- Test application functionality
- Monitor error rates in Datadog

Planned Secret Rotation #

For regular credential rotation:

Create new credentials in the external service (AWS, Stripe, etc.)
Test new credentials in development first
Update Doppler for production
Deploy and verify services are working
Revoke old credentials in external service
Document the rotation in incident/change log

Environment-Specific Operations #

Development Environment #

# View dev secrets
doppler secrets --project bonsai --config dev

# Update dev secret
doppler secrets set DEBUG_MODE="true" --project bonsai --config dev

After updating dev secrets, trigger a deployment or restart pods:

# Force sync in Kubernetes
kubectl delete externalsecret bonsai-external-secret
kubectl apply -f deployment/resources/manifests/external_secret.yaml

# Restart services
kubectl rollout restart deployment/bonsapi-deployment

Production Environment #

IMPORTANT: Production secret changes require:

Approval from team lead or senior engineer
Documentation of change reason
Coordination with team (avoid during business hours if possible)

# View prod secrets (read-only unless you have full_access role)
doppler secrets --project bonsai --config prod

# Update prod secret (requires appropriate permissions)
doppler secrets set API_KEY="new-value" --project bonsai --config prod

Production Deployment After Secret Change:

Follow the deployment process in Deployment Monitoring or manually trigger GitHub Actions workflow.

Local Development #

# Update local secrets
doppler secrets set DATABASE_URL="postgresql://localhost:5432/bonsai" \
  --project bonsai --config dev_local

# Verify by running app
mise run dev

Managing Secrets in Kubernetes #

External Secrets Operator #

BonsAI uses External Secrets Operator to sync secrets from Doppler to Kubernetes.

Architecture:

Doppler (source of truth)
    ↓
External Secrets Operator (sync every 1 hour)
    ↓
Kubernetes Secret (bonsai-secret)
    ↓
Pods (environment variables)

Verifying Secret Sync #

# Check External Secret status
kubectl get externalsecret bonsai-external-secret

# Should show:
NAME                     STORE                   STATUS   READY   AGE
bonsai-external-secret   doppler-secret-store   Synced   True    24h

# If status is not Synced, check logs
kubectl describe externalsecret bonsai-external-secret

# Check External Secrets operator logs
kubectl logs -l app.kubernetes.io/name=external-secrets -n default

Manual Secret Sync #

If secrets aren’t syncing automatically:

# Delete and recreate external secret
kubectl delete externalsecret bonsai-external-secret

# Recreate from manifest
kubectl apply -f deployment/resources/manifests/external_secret.yaml

# Wait for sync (up to 1 minute)
kubectl get externalsecret bonsai-external-secret --watch

Forcing Pod Secret Refresh #

After secrets are synced to Kubernetes:

# Rolling restart to pick up new secrets
kubectl rollout restart deployment/bonsapi-deployment
kubectl rollout restart deployment/webapp-deployment
kubectl rollout restart deployment/bonsai-invoice-deployment
kubectl rollout restart deployment/bonsai-knowledge-deployment

# Monitor rollout
kubectl rollout status deployment/bonsapi-deployment

Common Secret Categories #

Database Credentials #

# View database connection string
doppler secrets get DATABASE_URL --project bonsai --config prod --plain

# Format: postgresql://username:password@host:5432/database

Related Secrets:

DATABASE_URL - Full connection string
DATABASE_USERNAME - Username only
DATABASE_PASSWORD - Password only
DATABASE_HOST - Host only
DATABASE_PORT - Port only

Redis Configuration #

# View Redis settings
doppler secrets get REDIS_HOST REDIS_PORT --project bonsai --config prod

Related Secrets:

REDIS_HOST - Redis hostname
REDIS_PORT - Redis port (usually 6379)
REDIS_PASSWORD - Password (if authentication enabled)

RabbitMQ Configuration #

# View RabbitMQ settings
doppler secrets get RABBITMQ_HOST RABBITMQ_PORT RABBITMQ_USER RABBITMQ_PASSWORD \
  --project bonsai --config prod

Related Secrets:

RABBITMQ_PROTOCOL - Protocol (amqps)
RABBITMQ_HOST - RabbitMQ hostname
RABBITMQ_PORT - Port (5671 for amqps)
RABBITMQ_USER - Username
RABBITMQ_PASSWORD - Password

Authentication Services #

# Clerk (authentication)
doppler secrets get CLERK_SECRET_KEY CLERK_WEBHOOK_SIGNING_SECRET \
  --project bonsai --config prod

# Stripe (payments)
doppler secrets get STRIPE_SECRET_KEY STRIPE_WEBHOOK_SIGNING_SECRET \
  --project bonsai --config prod

Third-Party Integrations #

# Integration credentials
doppler secrets get XERO_CLIENT_ID XERO_CLIENT_SECRET \
  SHAREPOINT_CLIENT_ID SHAREPOINT_CLIENT_SECRET \
  --project bonsai --config prod

AWS Credentials #

# AWS access keys
doppler secrets get AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_REGION \
  --project bonsai --config prod

Troubleshooting #

Secret Not Updating in Kubernetes #

Symptoms: Changed secret in Doppler but pods still using old value

Investigation:

Check External Secret sync

kubectl describe externalsecret bonsai-external-secret

Verify Kubernetes secret updated

kubectl get secret bonsai-secret -o jsonpath='{.data.YOUR_SECRET}' | base64 -d

Check sync interval External Secrets syncs every 1 hour by default. Force sync by deleting and recreating.

Solution:

# Force immediate sync
kubectl delete externalsecret bonsai-external-secret
kubectl apply -f deployment/resources/manifests/external_secret.yaml

# Restart pods to pick up new values
kubectl rollout restart deployment/<deployment-name>

Doppler Access Denied #

Symptoms: Cannot view or update secrets

Causes:

Not logged in
Wrong project/config
Insufficient permissions

Solutions:

Re-authenticate
```
doppler logout
doppler login
```
Verify project access
```
doppler projects
```
Request access from team lead if project not listed

Missing Secret in Application #

Symptoms: Application errors about missing environment variable

Investigation:

Check if secret exists in Doppler

doppler secrets get SECRET_NAME --project bonsai --config prod

Verify secret in Kubernetes

kubectl get secret bonsai-secret -o yaml | grep SECRET_NAME

Check deployment manifest

kubectl get deployment <deployment-name> -o yaml | grep -A 5 SECRET_NAME

Solutions:

Add missing secret to Doppler
Update deployment manifest to reference the secret
Restart deployment after changes

External Secrets Operator Issues #

Symptoms: External Secret status shows errors

Common Errors:

“Secret not found in Doppler”
- Secret name mismatch between K8s and Doppler
- Check exact secret name (case-sensitive)

“Authentication failed”

Doppler service token invalid or expired

Update token:

kubectl create secret generic doppler-token-secret \
  --from-literal=dopplerToken=<new-token> \
  --dry-run=client -o yaml | kubectl apply -f -

“Sync interval too long”
- Adjust sync interval in external secret manifest
- Or force manual sync (delete and recreate)

Best Practices #

Security #

Never log secrets - Don’t print or echo secret values
Use principle of least privilege - Only access secrets you need
Rotate regularly - Establish rotation schedule for sensitive credentials
Audit changes - Doppler tracks all secret changes
Separate environments - Never use prod secrets in dev

Operations #

Test in dev first - Always test secret changes in dev before prod
Document changes - Note reason for secret updates
Coordinate deployments - Communicate secret changes to team
Have rollback plan - Keep old credentials available during rotation
Monitor after changes - Watch error rates and logs

Development Workflow #

# 1. Setup Doppler locally (one-time)
mise run doppler-setup

# 2. Run services with Doppler
doppler run -- mise run dev

# 3. Override individual secrets for testing
doppler secrets set DEBUG_MODE="true" --project bonsai --config dev_local

# 4. Reset to defaults
doppler secrets delete DEBUG_MODE --project bonsai --config dev_local

Emergency Procedures #

Suspected Secret Leak #

Immediately rotate the compromised secret in Doppler
Verify rotation in all environments
Restart all services to pick up new secrets
Check logs for unauthorized access
Document the incident and root cause
Review access logs in Doppler dashboard

Lost Access to Doppler #

Contact team lead for access restoration
Use emergency access (if configured)
Check GitHub Actions for service tokens (read-only)
Never share personal tokens as workaround

GitHub Actions Integration #

Secrets are injected into GitHub Actions via Doppler service tokens.

Configuration:

# .github/workflows/deploy.yaml
env:
  DOPPLER_TOKEN: ${{ secrets.DOPPLER_TOKEN }}

# Actions can then access secrets
doppler secrets download --no-file --format env

Updating Service Tokens:

Generate new service token in Doppler dashboard
Update GitHub secret DOPPLER_TOKEN
Test in development workflow first

Secrets Management #

When to Use This Runbook #

Overview #

Doppler Project Structure #

Prerequisites #

Installing Doppler CLI #

Logging In #

Setting Up Project Locally #

Viewing Secrets #

List All Secrets #

Get Specific Secret #

Download All Secrets #

Updating Secrets #

Update Single Secret #

Update Multiple Secrets #

Bulk Updates via Doppler Dashboard #

Rotating Secrets #

Emergency Secret Rotation #

Planned Secret Rotation #

Environment-Specific Operations #

Development Environment #

Production Environment #

Local Development #

Managing Secrets in Kubernetes #

External Secrets Operator #

Verifying Secret Sync #

Manual Secret Sync #

Forcing Pod Secret Refresh #

Common Secret Categories #

Database Credentials #

Redis Configuration #

RabbitMQ Configuration #

Authentication Services #

Third-Party Integrations #

AWS Credentials #

Troubleshooting #

Secret Not Updating in Kubernetes #

Doppler Access Denied #

Missing Secret in Application #

External Secrets Operator Issues #

Best Practices #

Security #

Operations #

Development Workflow #

Emergency Procedures #

Suspected Secret Leak #

Lost Access to Doppler #

GitHub Actions Integration #

See Also #