Grafana

​

Overview

1. Grafana Cloud / self-hosted setup
2. Local Grafana Setup (Minikube example)

​

Grafana Cloud / self-hosted setup

​

Prerequisites

• Grafana instance URL (Cloud stack URL or self-hosted origin)
• Service account token with read access — see Grafana service account tokens

​

Option 1: Interactive CLI

opensre integrations setup grafana

​

Option 2: Environment variables

GRAFANA_INSTANCE_URL=https://your-stack.grafana.net
GRAFANA_READ_TOKEN=glsa_your_service_account_token
GRAFANA_VERIFY_SSL=true                    # optional — set false only for local/lab
GRAFANA_CA_BUNDLE=/path/to/internal-ca.pem # optional — internal CA

​

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "grafana-prod",
      "service": "grafana",
      "status": "active",
      "credentials": {
        "endpoint": "https://your-stack.grafana.net",
        "api_key": "glsa_your_token",
        "verify_ssl": true
      }
    }
  ]
}

​

Option 4: Hosted web app (OpenSRE Cloud)

1. In app.tracer.cloud, go to Integrations → Grafana
2. Enter a name, instance URL, and service account token
3. Click Save

​

Verify

opensre integrations verify grafana

​

Self-signed or internal CA certificates

• Verify SSL certificate? — answer No to skip TLS verification entirely (only recommended for local/lab instances).
• Path to CA bundle for SSL verification — point this at a PEM file containing your internal CA certificate to keep full TLS verification while trusting your organization’s CA. This is the recommended option for a real internal Grafana deployment, since it still validates the certificate chain and hostname.

​

Local Grafana Setup (Minikube example)

​

Steps

1. Start minikube:
   minikube start
   
2. Add prometheus community and podinfo helm repositories:
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo add podinfo https://stefanprodan.github.io/podinfo
   helm repo update
   
3. Create monitoring namespace and install kube-prometheus stack:
   kubectl create namespace monitoring
   helm install kube-stack prometheus-community/kube-prometheus-stack -n monitoring
   
4. Create podinfo namespace and install podinfo application:
   kubectl create namespace podinfo
   helm install podinfo podinfo/podinfo -n podinfo --set serviceMonitor.enabled=true
   
5. (Optional) Check the status of the pods in the podinfo and monitoring namespaces:
   kubectl -n podinfo get pods
   kubectl -n monitoring get pods
   
6. (Run in separate terminals) Port forward the podinfo service:
   kubectl -n podinfo port-forward deploy/podinfo 8080:9898
   
7. (Run in separate terminals) Port forward the prometheus service:
   kubectl -n monitoring port-forward svc/kube-stack-kube-prometheus-prometheus 9090:9090
   
8. (Run in separate terminals) Port forward the grafana service and expose it on all network interfaces:
   kubectl -n monitoring port-forward --address 0.0.0.0 svc/kube-stack-grafana 3000:80
   
9. Patch the kube-stack-prometheus service to allow it to scrape the podinfo service:
   kubectl patch prometheus kube-stack-kube-prometheus-prometheus \
   -n monitoring \
   --type=merge \
   -p '{"spec":{"serviceMonitorSelector":{},"serviceMonitorNamespaceSelector":{}}}'
   

​

Credentials for Grafana

kubectl get secret kube-stack-grafana --namespace monitoring -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

​

Access

• podinfo microservices application at http://localhost:8080
• the prometheus dashboard at http://localhost:9090
• the grafana dashboard at http://localhost:3000.

​

Simulate load on the podinfo application

while true; do curl -s http://localhost:8080/status/500 >/dev/null; sleep 1; done &
while true; do curl -s http://localhost:8080/delay/5 >/dev/null; sleep 1; done &

​

Grafana Dashboard

• For the number of requests to the podinfo application:

sum(rate(http_requests_total{job="podinfo"}[1m])) by (status)

• For the average response time of the podinfo application:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="podinfo"}[1m])) by (le))

​

Setting up Prometheus Alerts

kubectl apply -f - <<'YAML'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: podinfo-alerts
  namespace: monitoring
  labels:
    release: kube-stack
spec:
  groups:
  - name: podinfo.rules
    rules:
    - alert: PodinfoHighErrorRate
      expr: increase(http_requests_total{status="500"}[5m]) > 5
      for: 30s
      labels:
        severity: critical
      annotations:
        summary: "Podinfo error rate is high"
YAML

​

Integrating with OpenSRE

1. Get the LAN IP address of your machine:
   hostname -I | awk '{print $1}' #linux
   ipconfig getifaddr en0 #macOS
   (Get-NetIPAddress -InterfaceAlias "Wi-Fi" -AddressFamily IPv4).IPAddress #Windows PowerShell
   # Remember to replace "Wi-Fi" with the name of your network interface if it's different.
   
2. Create a service account token in Grafana following the official documentation: How to add a token to a service account in Grafana
3. Run opensre integration add grafana and follow the prompts to enter the required information:
   • Instance URL: Enter the endpoint URL for your Grafana instance (e.g., http://<LAN_IP_ADDRESS>:3000)
   • Service account token: Paste the Service Account token you created in Grafana