DevFW-CICD/stacks-instances
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(observability): 🐛 disable false-positive control-plane alerts and fix empty cluster_environment label

Browse source 
Hub defaultRules groups kubernetesSystemControllerManager, kubeScheduler, and
kubernetesSystemScheduler used wrong key 'enabled: false' — chart expects 'create: false'.
This caused KubeControllerManagerDown/KubeSchedulerDown to fire as false positives
because OTC CCE managed k8s does not expose control plane for scraping.

Dev local vmagent had empty externalLabels, so backup-alert rules evaluated by local
vmalert had no cluster_environment label on kube_job_status_failed metrics. Added
cluster_environment=dev to match what the vm-client-stack vmagent adds for hub shipping.
This commit is contained in:
Daniel Sy 2026-06-19 12:42:04 +02:00
parent 32e998df5b
commit 0316eefa43
Signed by untrusted user: danielsy
GPG key ID: 1F39A8BBCD2EE3D3
2 changed files with 5 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

3
otc/dev.t09.de/stacks/observability/victoria-k8s-stack/values.yaml
View file

@ -708,7 +708,8 @@ vmagent:
port: "8429"
selectAllByDefault: true
scrapeInterval: 20s
externalLabels: {}
externalLabels:
cluster_environment: "dev"
# For multi-cluster setups it is useful to use "cluster" label to identify the metrics source.
# For example:
# cluster: cluster-name