Spinnaker installation

jsu

Hi there!

Great course, but concerning the last part with Spinnaker, I ran into an issue.

Indeed, when trying to install it with the script, I had a CrashLoopBackOff from the minio container. This issue was raised on github and I managed to fix it by directly editing the replicaset with a tagged image.

But even after doing so, connecting to the address gave me an error 404.

I don't know if it's related to the fact that I have way less containers running than Gourav on the video.

Any idea what's happening and/or how to fix it?

Thanks in advance for your help!

luisviveropena

Hi @jsu ,

I'm not aware of the issue. Can you provide the Chapter and Section, and the exact command and output you are running/getting, please?

Regards,
Luis.

jsu

Hi @luisviveropena,

It is on Chapter 13: Continuous Deployment with Spinnaker, Section Setting Up Spinnaker on GCP.

The minnaker installation with the script doesn't work. I used the command that was provided in the course (and on https://github.com/armory/minnaker)

./scripts/install.sh -o -P $EXTERNAL_IP

and I had a CrashLoopBackOff with minio. I don't have the exact logs right now as actually I forgot to send this message last Thursday and I moved on to the next course.

luisviveropena

Hi @jsu, I was able to reproduce the issue. We are investigating about this, and will provide you an update as soon as we have news.

Regards,
Luis.

zinebhalib

I have the same issue. @jsu please let me know if you found the solution.

zinebhalib

I found the solution here https://github.com/armory/minnaker/issues/102
"What worked for me was editing spinsvc/infrastructure/minio.yml, changing the path from /usr/bin/minio to /opt/bin/minio and then running ./deploy.sh, then deleting the old minio-0 pod."

luisviveropena

Hi @zinebhalib , I opened that bug few days ago, hehehe. I haven't tried the workaround, I'll try it asap.

Thanks for the update!

Luis.

rknmit

Hello,

After doing the same steps or fix as suggested in https://github.com/armory/minnaker/issues/102. The pod is running but still getting the 404 error

pod logs:
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
API: http://10.42.0.16:9000 http://127.0.0.1:9000

Console: http://10.42.0.16:35287 http://127.0.0.1:35287

Documentation: https://docs.min.io

WARNING: Console endpoint is listening on a dynamic port (35287), please use --console-address ":PORT" to choose a static port.

Regards,
Ravi

fredlev

Hi, I have ran into the same issue.

I am wondering if using spinnaker for anything still makes sense since it seems that the open source version is falling behind the Armory enterprise version.

I've tried to configure halyard to use a dev deployment on a single VM but the bintray repository for spinnaker packages is KO.

CF:

https://stackoverflow.com/questions/68851795/spinnaker-installation-issue-due-to-dl-bintray-com-decommission

I've also tried spinnaker the hardway but I am hitting a bunch of java compilation errors when trying to run the services:

https://github.com/indrayam/spinnaker-the-hard-way/blob/master/LocalGitInstallation.md

All of this makes me question the validity of spinnaker as a deployment tool going forward.

IMO the course should have provided tagged images/versions of the tools and images used by those tools.

For example: here minnaker will pull the latest image of minio-o by default instead of a TAGGED release hence the path issue in the command field.

lbono

@fredlev said:
Hi, I have ran into the same issue.

I am wondering if using spinnaker for anything still makes sense since it seems that the open source version is falling behind the Armory enterprise version.

I've tried to configure halyard to use a dev deployment on a single VM but the bintray repository for spinnaker packages is KO.

CF:

https://stackoverflow.com/questions/68851795/spinnaker-installation-issue-due-to-dl-bintray-com-decommission

I've also tried spinnaker the hardway but I am hitting a bunch of java compilation errors when trying to run the services:

https://github.com/indrayam/spinnaker-the-hard-way/blob/master/LocalGitInstallation.md

All of this makes me question the validity of spinnaker as a deployment tool going forward.

IMO the course should have provided tagged images/versions of the tools and images used by those tools.

For example: here minnaker will pull the latest image of minio-o by default instead of a TAGGED release hence the path issue in the command field.

Agree. Same problem here. The WA to fix minio-0 deployment doesn't work either.
And worse, my free Google cloud credit is finishing.
Any Solution? Some Ubuntu server 4 cores/8Gigs to borrow me?

Regards.-

luisviveropena

Hi @lbono,

We are currently working on this issue, and we'll provide you an update as soon as we have news.

Regards,
Luis.

aderock

@luisviveropena, I am having similar issues. What is the ETA on the fix?. Thanks

redrom35

@luisviveropena, I'm facing the same issue.
Any update on this?

luisviveropena

Hi @aderock and @redrom35 , the workarounds suggested in the bug are not working for me. So we are still looking at this issue.

Regards,
Luis.

gouravshah

@jsu @zinebhalib @rknmit @redrom35 @fredlev @lbono @luisviveropena finally I have found some fixes which I am sharing here for you to try. And this goes beyond minio issue reported here.

While I recreated the same environment I found the following issues,

1. Minio Issue - minio was going into crashloopbackoff error.
2. Spinnaker Version Issue - After fixing minio, operator could come up ( in spinnaker-operator namespace) but could not launch spinnaker (only in case of open source version) as it was unable to find the correct version of spinnaker defined.

Root Cause and Solutions :

1. Minio Issue - this was very clear as reported by some of you above. This is a classic problem when using latest as image tag. In the deployment for minio, had a specific version been mentioned, this would have never happend.

Whats interesting is, this was fixed in the master branch used by armory (tag : RELEASE.2021-08-20T18-32-01Z) as you could find it here https://github.com/armory/spinnaker-kustomize-patches/blob/master/infrastructure/minio.yml however, they just forgot about the minnaker branch used in our installation of open source minnaker here https://github.com/armory/spinnaker-kustomize-patches/blob/minnaker/infrastructure/minio.yml. (look at scripts/install.sh for default branch).

This cleanest and easiest fix for this is to switch the branch while installing as,

./scripts/install.sh -o --branch master -P <PUBLIC_IP>

2. Version Issue - This took me the longest and multiple iterations of troubleshoooting. Everytime I would setup the open source version of spinnaker it would get stuck at the ,"error":"You must pick a version of Spinnaker to deploy." ( spinnaker-operator logs)

You could check the logs by listing all pods, finding the operator and using kubectl logs command,

for example , start with kubectl get pods -A

kube-system          svclb-traefik-vgvhm                         2/2     Running     0          104s
kube-system          traefik-5dd496474-rp27r                     1/1     Running     0          104s
spinnaker-operator   spinnaker-operator-7b755b8789-mlwfl         2/2     Running     0          2m5s
spinnaker            svclb-spin-agent-clouddriver-m6vnb          1/1     Running     0          50s
spi

and then

kubectl logs -f  -n spinnaker-operator -c spinnaker-operator spinnaker-operator-7b755b8789-mlwfl

the error always was,

"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"You must pick a version of Spinnaker to deploy.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\....

Finding the symptom above was fairly straight forward. The root cause analysis was what took really long time to figure out. Mainly due to very little documentation, no information about this error reported anywhere, and most importantly, bad code practice ( the code shows this version as being defined as param which is a good practice, but then completely bypasses this param and does its own hard coded thing, very bad).

Finally after many attempts, recreating the env a few times, comparing with commercial version vs open source and diving into the code after, I found a very silly thing in the install script (script/install.sh) where it was using the hard coded version of code to fetch the version

  VERSION=$(curl -s https://spinnaker.io/community/releases/versions/ | grep 'id="version-' | head -1 | sed -e 's/\(<[^<][^<]*>\)//g; /^$/d' | cut -d' ' -f2)

This is the broken code . try running ,

curl -s https://spinnaker.io/community/releases/versions/ 

which now redirects, versus observe this for the armory version

curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml

which returns the correct version info. This was the problem.

A quick fix was to commenting line number 123 with previous VERSION being defined and to set the version manually as,

After this if you clean up and install, it works ! At least the installation completes. Havent tested further by creating application pipelines yet.

How to apply this fix

As mentioned, edit file: scripts/install.sh and comment line 123, set version manually as shown in the section just above this.

And then reinstall from within minnaker-0.1.3 directory by running the following

./scripts/uninstall-k3s.sh
rm -rf spinsvc/

and then install again as,

./scripts/install.sh -o -P PUBLIC_IP --branch master

This should fix both issues, one with minio, also with version.

manuel.nhiuana

Right in time @gouravshah, I'll test your solution later today.

manuel.nhiuana

@manuel.nhiuana said:
Right in time @gouravshah, I'll test your solution later today.

@gouravshah said:
@jsu @zinebhalib @rknmit @redrom35 @fredlev @lbono @luisviveropena finally I have found some fixes which I am sharing here for you to try. And this goes beyond minio issue reported here.

While I recreated the same environment I found the following issues,

1. Minio Issue - minio was going into crashloopbackoff error.
2. Spinnaker Version Issue - After fixing minio, operator could come up ( in spinnaker-operator namespace) but could not launch spinnaker (only in case of open source version) as it was unable to find the correct version of spinnaker defined.

Root Cause and Solutions :

1. Minio Issue - this was very clear as reported by some of you above. This is a classic problem when using latest as image tag. In the deployment for minio, had a specific version been mentioned, this would have never happend.

Whats interesting is, this was fixed in the master branch used by armory (tag : RELEASE.2021-08-20T18-32-01Z) as you could find it here https://github.com/armory/spinnaker-kustomize-patches/blob/master/infrastructure/minio.yml however, they just forgot about the minnaker branch used in our installation of open source minnaker here https://github.com/armory/spinnaker-kustomize-patches/blob/minnaker/infrastructure/minio.yml. (look at scripts/install.sh for default branch).

This cleanest and easiest fix for this is to switch the branch while installing as,

./scripts/install.sh -o --branch master -P <PUBLIC_IP>

2. Version Issue - This took me the longest and multiple iterations of troubleshoooting. Everytime I would setup the open source version of spinnaker it would get stuck at the ,"error":"You must pick a version of Spinnaker to deploy." ( spinnaker-operator logs)

You could check the logs by listing all pods, finding the operator and using kubectl logs command,

for example , start with kubectl get pods -A

kube-system          svclb-traefik-vgvhm                         2/2     Running     0          104s
kube-system          traefik-5dd496474-rp27r                     1/1     Running     0          104s
spinnaker-operator   spinnaker-operator-7b755b8789-mlwfl         2/2     Running     0          2m5s
spinnaker            svclb-spin-agent-clouddriver-m6vnb          1/1     Running     0          50s
spi

and then

kubectl logs -f  -n spinnaker-operator -c spinnaker-operator spinnaker-operator-7b755b8789-mlwfl

the error always was,

"Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"You must pick a version of Spinnaker to deploy.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\....

Finding the symptom above was fairly straight forward. The root cause analysis was what took really long time to figure out. Mainly due to very little documentation, no information about this error reported anywhere, and most importantly, bad code practice ( the code shows this version as being defined as param which is a good practice, but then completely bypasses this param and does its own hard coded thing, very bad).

Finally after many attempts, recreating the env a few times, comparing with commercial version vs open source and diving into the code after, I found a very silly thing in the install script (script/install.sh) where it was using the hard coded version of code to fetch the version

  VERSION=$(curl -s https://spinnaker.io/community/releases/versions/ | grep 'id="version-' | head -1 | sed -e 's/\(<[^<][^<]*>\)//g; /^$/d' | cut -d' ' -f2)

This is the broken code . try running ,

curl -s https://spinnaker.io/community/releases/versions/ 

which now redirects, versus observe this for the armory version

curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml

which returns the correct version info. This was the problem.

A quick fix was to commenting line number 123 with previous VERSION being defined and to set the version manually as,

After this if you clean up and install, it works ! At least the installation completes. Havent tested further by creating application pipelines yet.

How to apply this fix

As mentioned, edit file: scripts/install.sh and comment line 123, set version manually as shown in the section just above this.

And then reinstall from within minnaker-0.1.3 directory by running the following

./scripts/uninstall-k3s.sh
rm -rf spinsvc/

and then install again as,

./scripts/install.sh -o -P PUBLIC_IP --branch master

This should fix both issues, one with minio, also with version.

I tried commenting the line 123, and now I'm having error deploying the operator, seems like the script is unable to parse and get the version of spinnaker-operator.
the url " https://github.com/armory/spinnaker-operator/releases/download/v/manifests.tgz" doesn't exist

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s
[INFO ] --- END K3s ---
[INFO ] Setting Kubernetes context to Spinnaker namespace
Context "default" modified.
[INFO ] Spinnaker Operator Version:
[INFO ] Spinnaker flavor: oss
[INFO ] Changing spinnaker flavor...Done
[INFO ] Resolved operator namespace: spinnaker-operator
[INFO ] Deploying oss operator...
[INFO ] Downloading operator from https://github.com/armory/spinnaker-operator/releases/download/v/manifests.tgz
[KUBE ] kubectl apply -f /root/minnaker-0.1.3/spinsvc/operator/deploy/crds/
[ERROR] Error executing command:
error: the path "/root/minnaker-0.1.3/spinsvc/operator/deploy/crds/" does not exist[KUBE ] kubectl apply -f https://engineering.armory.io/manifests/pacrd-1.0.1.yaml -n spinnaker
[ERROR] Error executing command:
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
Error from server (NotFound): error when creating "https://engineering.armory.io/manifests/pacrd-1.0.1.yaml": namespaces "spinnaker" not foun

manuel.nhiuana

also tried set the version like this:

if [[ ${OPEN_SOURCE} == 1 ]]; then
info "Using OSS Spinnaker"
SPIN_FLAVOR=oss
VERSION="1.26.7"
else
info "Using Armory Spinnaker"
SPIN_FLAVOR=armory
VERSION=$(curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml | grep 'version: ' | awk '{print $NF}' | sort | tail -1)
fi

the error is
[KUBE ] kubectl apply -f /root/minnaker-0.1.3/min/minnaker-0.1.3/spinsvc/operator/deploy/crds/
[ERROR] Error executing command:
error: the path "/root/minnaker-0.1.3/min/minnaker-0.1.3/spinsvc/operator/deploy/crds/" does not exist[KUBE ] kubectl apply -f https://engineering.armory.io/manifests/pacrd-1.0.1.yaml -n spinnaker
[ERROR] Error executing command:
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unav

gouravshah

Try setting the version by commenting #123 and then just manually setting

VERSION=1.26.7

donquijotedelasnubes

hi ... this is still the "state of the art" workaround for this problem??

luisviveropena

Hi @donquijotedelasnubes, we are working on updates on this topic.

Regards,
Luis.