K8s Crashback loop

chris

Hi

I am currently going through the Kubernetes Chapter and I am running into an issue with the deployment. The prometheus container keeps going into a crashbackloop.

  Normal   Pulled     10m (x5 over 11m)   kubelet, vagrant   Container image "prom/prometheus:v2.1.0" already present on machine
  Normal   Created    10m (x5 over 11m)   kubelet, vagrant   Created container prometheus
  Warning  BackOff    83s (x49 over 11m)  kubelet, vagrant   Back-off restarting failed container

It comes and then all the services shutdown

level=info ts=2019-04-25T12:59:17.96088709Z caller=web.go:383 component=web msg="Start listening for connections" address=:9090
level=info ts=2019-04-25T12:59:17.960921987Z caller=main.go:499 msg="Starting TSDB ..."
level=info ts=2019-04-25T12:59:17.961378121Z caller=main.go:386 msg="Stopping scrape discovery manager..."
level=info ts=2019-04-25T12:59:17.961462142Z caller=main.go:400 msg="Stopping notify discovery manager..."
level=info ts=2019-04-25T12:59:17.961534296Z caller=main.go:424 msg="Stopping scrape manager..."
level=info ts=2019-04-25T12:59:17.961598612Z caller=manager.go:460 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-04-25T12:59:17.961702813Z caller=manager.go:466 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-04-25T12:59:17.961739804Z caller=notifier.go:493 component=notifier msg="Stopping notification manager..."
level=info ts=2019-04-25T12:59:17.961767336Z caller=main.go:382 msg="Scrape discovery manager stopped"
level=info ts=2019-04-25T12:59:17.961870772Z caller=main.go:396 msg="Notify discovery manager stopped"
level=info ts=2019-04-25T12:59:17.961972292Z caller=manager.go:59 component="scrape manager" msg="Starting scrape manager..."
level=info ts=2019-04-25T12:59:17.962012646Z caller=main.go:418 msg="Scrape manager stopped"
level=info ts=2019-04-25T12:59:17.962037506Z caller=main.go:570 msg="Notifier manager stopped"
level=error ts=2019-04-25T12:59:17.96406764Z caller=main.go:579 err="Opening storage failed mkdir data/: permission denied"
level=info ts=2019-04-25T12:59:17.964102554Z caller=main.go:581 msg="See you next time!"

Anyone else bump into this issue?

Running K8s 1.14.1, with containerd as my runtime.

chris

In case anyone else runs into this issue, I managed to sort it out by applying the following SecurityContext

securityContext:
  runAsUser: 0

Found the answer here, https://github.com/prometheus/prometheus/issues/2939#issuecomment-315046699