Scale Applications Using SAP BTP Cloud Logging Service Metrics
Configure KEDA to autoscale a Kubernetes workload using application metrics stored in SAP BTP Cloud Logging Service (OpenSearch) as the scaling signal.
Overview
You will learn
- How to create a Cloud Logging Service instance and expose its OpenSearch API
- How to deploy a demo application that exposes a custom Prometheus metric
- How to configure the Kyma Telemetry module to scrape and forward metrics to CLS
- How to create a KEDA ScaledObject that queries CLS (OpenSearch) to drive autoscaling
- How to observe your workload scaling in response to metric changes
Prerequisites
Prerequisites
- Enable SAP BTP, Kyma Runtime
- The Keda, Telemetry, and SAP BTP Operator modules are enabled in your Kyma cluster. See Enable and Disable a Kyma Module.
- Your subaccount has an entitlement for Cloud Logging Service with the
standardplan. See Configure Entitlements and Quotas for Subaccounts. kubectlis installed and configured to access your Kyma cluster.
Steps
Intro
SAP BTP Cloud Logging Service (CLS) is a managed observability backend built on OpenSearch. If your application metrics already flow into CLS through the Kyma Telemetry module, you can use them as autoscaling signals. No separate metrics store is needed.
KEDA 2.20 ships a native opensearch scaler that queries CLS directly using an inline query. This tutorial covers deploying a demo app that emits a queue_depth metric, setting up Telemetry scraping, and creating a ScaledObject that uses that metric as the scaling signal.
In the SAP BTP cockpit, go to Services โ Instances and Subscriptions and choose Create.
Configure the service instance and choose Next:
- Service: Cloud Logging
- Plan: standard
- Runtime Environment: Other
- Instance Name: Enter a name, for example
cloud-logging
In the Parameters field, add the following parameters to the default JSON and choose Create:
JSON{ "backend": { "api_enabled": true, "max_data_nodes": 2 }, "ingest_otlp": { "enabled": true } }Wait until the instance status changes to Created.
In the instance row, choose the … (Actions) menu and select Create Service Binding. Enter a name for the binding, for example
cloud-logging-binding, and choose Create.After the binding is created, choose View Credentials and record the following values:
Key Description backend-endpoint OpenSearch REST API endpoint backend-username Username for OpenSearch REST API authentication backend-password Password for OpenSearch REST API authentication ingest-otlp-endpoint OTLP ingest endpoint for the Telemetry module ingest-otlp-cert Client certificate for mTLS ingest-otlp-key Client key for mTLS Create a namespace in your SAP BTP, Kyma cluster for the CLS resources:
Shellkubectl create namespace clsCreate a Kubernetes Secret with the credentials:
Shellkubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: cloud-logging-binding namespace: cls type: Opaque stringData: backend-endpoint: "<BACKEND_ENDPOINT>" backend-username: "<BACKEND_USERNAME>" backend-password: "<BACKEND_PASSWORD>" ingest-otlp-endpoint: "<INGEST_OTLP_ENDPOINT>" ingest-otlp-cert: | <INGEST_OTLP_CERT> ingest-otlp-key: | <INGEST_OTLP_KEY> EOFReplace each placeholder with the corresponding value from the cockpit. For
ingest-otlp-certandingest-otlp-key, paste the full PEM content including the-----BEGIN ...-----and-----END ...-----lines.
Create a namespace in your SAP BTP, Kyma cluster for the CLS resources:
Shellkubectl create namespace clsCreate a service instance for Cloud Logging:
Shellkubectl apply -f - <<EOF apiVersion: services.cloud.sap.com/v1 kind: ServiceInstance metadata: name: cloud-logging namespace: cls spec: serviceOfferingName: cloud-logging servicePlanName: standard parameters: backend: api_enabled: true max_data_nodes: 2 ingest_otlp: enabled: true EOFWait until the instance is ready:
Shellkubectl get serviceinstance cloud-logging -n cls -wThe output looks similar to this example:
CodeNAME OFFERING PLAN STATUS AGE cloud-logging cloud-logging standard Created 2mCreate a service binding to generate the credentials Secret:
Shellkubectl apply -f - <<EOF apiVersion: services.cloud.sap.com/v1 kind: ServiceBinding metadata: name: cloud-logging-binding namespace: cls spec: serviceInstanceName: cloud-logging EOFVerify that the binding Secret was created and contains the required keys:
Shellkubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data}' | jq 'keys'The Secret must contain backend-endpoint, backend-username, backend-password, ingest-otlp-endpoint, ingest-otlp-cert, and ingest-otlp-key.
To reuse a single CLS instance across multiple Kyma clusters in the same global account, use SAP Service Manager instance sharing. This lets you create a pointer instance in each cluster that references the shared instance, without affecting its lifecycle when the pointer is deleted.
Mark your CLS instance as shareable. In the SAP BTP cockpit, choose Share Instance from the … menu for your CLS instance. Alternatively, use the btp CLI:
Shellbtp share services/instance <instance-id> --subaccount <subaccount-id>In each Kyma cluster that needs access, create a service instance with the
reference-instanceplan pointing to the shared instance:Shellkubectl apply -f - <<EOF apiVersion: services.cloud.sap.com/v1 kind: ServiceInstance metadata: name: cloud-logging-pointer namespace: cls spec: serviceOfferingName: cloud-logging servicePlanName: reference-instance parameters: instance_name_selector: "<SHARED_INSTANCE_NAME>" EOFCreate a service binding on the pointer instance as described in the SAP BTP Operator option above.
For cross-subaccount sharing, see Working with Multiple Subaccounts.
The demo application exposes a Prometheus-format queue_depth gauge metric at the /metrics endpoint. KEDA uses this value (as stored in CLS) to determine the desired replica count.
The QUEUE_DEPTH value is set by an init container at Pod startup. To change the value, update the environment variable and restart the Pod.
Deploy the demo application:
Shellcat <<'EOF' | kubectl apply -f - apiVersion: v1 kind: Namespace metadata: name: keda-cls-demo --- apiVersion: v1 kind: ConfigMap metadata: name: fake-metrics-nginx-config namespace: keda-cls-demo data: default.conf.template: | server { listen 8080; root /usr/share/nginx/html; location /metrics { default_type "text/plain; version=0.0.4; charset=utf-8"; try_files /metrics.txt =404; } location /health { default_type text/plain; return 200 "OK"; } location / { return 404; } } --- apiVersion: apps/v1 kind: Deployment metadata: name: fake-metrics namespace: keda-cls-demo labels: app: fake-metrics spec: replicas: 1 selector: matchLabels: app: fake-metrics template: metadata: labels: app: fake-metrics spec: initContainers: - name: generate-metrics image: busybox:1.36 command: - sh - -c - printf '# HELP queue_depth The current depth of the queue\n# TYPE queue_depth gauge\nqueue_depth %d\n' "$QUEUE_DEPTH" > /data/metrics.txt env: - name: QUEUE_DEPTH value: "10" volumeMounts: - name: metrics-data mountPath: /data containers: - name: fake-metrics image: nginx:alpine ports: - containerPort: 8080 resources: requests: memory: "64Mi" cpu: "100m" limits: memory: "128Mi" cpu: "200m" volumeMounts: - name: nginx-config mountPath: /etc/nginx/templates - name: metrics-data mountPath: /usr/share/nginx/html livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 10 readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 5 volumes: - name: nginx-config configMap: name: fake-metrics-nginx-config - name: metrics-data emptyDir: {} --- apiVersion: v1 kind: Service metadata: name: fake-metrics namespace: keda-cls-demo annotations: prometheus.io/scrape: "true" prometheus.io/port: "8080" prometheus.io/path: "/metrics" spec: selector: app: fake-metrics ports: - port: 8080 protocol: TCP targetPort: 8080 type: ClusterIP EOFVerify that the Pod is running:
Shellkubectl get pods -n keda-cls-demoThe output looks similar to this example:
CodeNAME READY STATUS RESTARTS AGE fake-metrics-<hash> 1/1 Running 0 30sConfirm the
/metricsendpoint is reachable:Shellkubectl run curl-test --image=curlimages/curl --rm -it --restart=Never --quiet \ -- curl -s http://fake-metrics.keda-cls-demo.svc.cluster.local:8080/metricsThe output looks similar to this example:
Code# HELP queue_depth The current depth of the queue # TYPE queue_depth gauge queue_depth 10
The Kyma Telemetry module scrapes Prometheus metrics from annotated Services and forwards them to your CLS instance. The demo application manifest already includes the Prometheus scraping annotations.
Create a MetricPipeline resource that sends the scraped metrics to CLS using the OTLP credentials from the binding Secret:
Shellkubectl apply -f - <<EOF apiVersion: telemetry.kyma-project.io/v1beta1 kind: MetricPipeline metadata: name: cls-metric-pipeline spec: input: prometheus: enabled: true namespaces: include: - keda-cls-demo istio: enabled: false runtime: enabled: false otlp: enabled: true output: otlp: endpoint: valueFrom: secretKeyRef: name: cloud-logging-binding namespace: cls key: ingest-otlp-endpoint tls: cert: valueFrom: secretKeyRef: name: cloud-logging-binding namespace: cls key: ingest-otlp-cert key: valueFrom: secretKeyRef: name: cloud-logging-binding namespace: cls key: ingest-otlp-key EOFVerify the pipeline is ready:
Shellkubectl get metricpipeline cls-metric-pipelineThe output looks similar to this example:
CodeNAME CONFIGURATION GENERATED GATEWAY HEALTHY AGENT HEALTHY FLOW HEALTHY AGE cls-metric-pipeline True True True True 2m
Find the
dashboards-endpointvalue under View Credentials for your CLS binding in the SAP BTP cockpit.Open the URL in your browser and log in with the Dashboards credentials from your CLS binding.
In the navigation menu, go to Discover, select the
metrics-otel-v1-*index pattern, and filter for documents withname: queue_depth. The metric appears within 1-2 minutes after the MetricPipeline becomes healthy.
- Run the following command to get the Dashboards URL:
echo "https://$(kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.dashboards-endpoint}' | base64 -d)"- To retrieve the Dashboards credentials, run:
kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.dashboards-username}' | base64 -d && echo
kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.dashboards-password}' | base64 -d && echoOpen the URL in your browser and log in with the credentials.
In the navigation menu, go to Discover, select the
metrics-otel-v1-*index pattern, and filter for documents withname: queue_depth. The metric appears within 1-2 minutes after the MetricPipeline becomes healthy.
KEDA must authenticate with the CLS OpenSearch REST API to run queries. Store the CLS credentials in a Kubernetes Secret and reference them from a TriggerAuthentication.
Create a Secret with your CLS OpenSearch credentials:
Shellkubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: cls-keda-auth namespace: keda-cls-demo type: Opaque stringData: username: "$(kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.backend-username}' | base64 -d)" password: "$(kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.backend-password}' | base64 -d)" EOFCreate a TriggerAuthentication that references the Secret:
Shellkubectl apply -f - <<EOF apiVersion: keda.sh/v1alpha1 kind: TriggerAuthentication metadata: name: cls-trigger-auth namespace: keda-cls-demo spec: secretTargetRef: - parameter: username name: cls-keda-auth key: username - parameter: password name: cls-keda-auth key: password EOF
The ScaledObject tells KEDA to query CLS for the latest queue_depth value and scale the demo application accordingly.
Export the OpenSearch endpoint and username from your CLS service binding Secret:
Shellexport CLS_OPENSEARCH_ENDPOINT=https://$(kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.backend-endpoint}' | base64 -d) export CLS_OPENSEARCH_USERNAME=$(kubectl get secret cloud-logging-binding -n cls -o jsonpath='{.data.backend-username}' | base64 -d)Create the ScaledObject:
Shellcat <<EOF | kubectl apply -f - apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: cls-queue-depth-scaler namespace: keda-cls-demo spec: scaleTargetRef: name: fake-metrics minReplicaCount: 1 maxReplicaCount: 10 triggers: - type: opensearch metadata: addresses: "${CLS_OPENSEARCH_ENDPOINT}" username: "${CLS_OPENSEARCH_USERNAME}" index: "metrics-otel-v1-*" query: | { "size": 0, "query": { "bool": { "filter": [ { "term": { "name": "queue_depth" } }, { "range": { "time": { "gte": "now-1m" } } } ] } }, "aggs": { "latest_value": { "max": { "field": "value" } } } } valueLocation: "aggregations.latest_value.value" targetValue: "10" skipTLSVerify: "false" authenticationRef: name: cls-trigger-auth EOFThe
targetValueof10means KEDA targets one replica per 10 units ofqueue_depth. With aqueue_depthof 42, KEDA targets 5 replicas (ceil(42/10)).Verify that KEDA has picked up the scaler:
Shellkubectl get scaledobject cls-queue-depth-scaler -n keda-cls-demoThe output looks similar to this example:
CodeNAME SCALETARGETKIND SCALETARGETNAME MIN MAX READY ACTIVE FALLBACK PAUSED TRIGGERS AUTHENTICATIONS AGE cls-queue-depth-scaler apps/v1.Deployment fake-metrics 1 10 True True False False opensearch cls-trigger-auth 2m
Check the KEDA-managed HPA:
Shellkubectl get hpa -n keda-cls-demoThe output looks similar to this example:
CodeNAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE keda-hpa-cls-queue-depth-scaler Deployment/fake-metrics 10/10 1 10 1 2mSimulate a metric spike by updating the
QUEUE_DEPTHenvironment variable and restarting the Pod:Shellkubectl set env deployment/fake-metrics QUEUE_DEPTH=80 -n keda-cls-demo kubectl rollout restart deployment/fake-metrics -n keda-cls-demoAfter the next Telemetry scrape and CLS ingestion cycle (within 1-2 minutes), KEDA queries CLS and adjusts the replica count.
Watch the Pods scale up:
Shellkubectl get pods -n keda-cls-demo -wSet the metric back to a lower value to observe scale-down:
Shellkubectl set env deployment/fake-metrics QUEUE_DEPTH=5 -n keda-cls-demo kubectl rollout restart deployment/fake-metrics -n keda-cls-demoAfter the cooldown period, the replica count returns to the minimum of 1. This may take up to 5 minutes.
- To remove the resources created in this tutorial, run:
kubectl delete namespace keda-cls-demo
kubectl delete metricpipeline cls-metric-pipeline
kubectl delete namespace cls- Delete the CLS instance in the cockpit under Services โ Instances and Subscriptions.
To remove the resources created in this tutorial, run:
kubectl delete namespace keda-cls-demo
kubectl delete metricpipeline cls-metric-pipeline
kubectl delete namespace clsResources
Discussion
Share feedback on this tutorial or join the conversation in SAP Community.