Welcome to the Linux Foundation Forum!

Spinnaker installation

Hi there!

Great course, but concerning the last part with Spinnaker, I ran into an issue.

Indeed, when trying to install it with the script, I had a CrashLoopBackOff from the minio container. This issue was raised on github and I managed to fix it by directly editing the replicaset with a tagged image.

But even after doing so, connecting to the address gave me an error 404.

I don't know if it's related to the fact that I have way less containers running than Gourav on the video.

Any idea what's happening and/or how to fix it?

Thanks in advance for your help!

Comments

  • Hi @jsu ,

    I'm not aware of the issue. Can you provide the Chapter and Section, and the exact command and output you are running/getting, please?

    Regards,
    Luis.

  • jsu
    jsu Posts: 5

    Hi @luisviveropena,

    It is on Chapter 13: Continuous Deployment with Spinnaker, Section Setting Up Spinnaker on GCP.

    The minnaker installation with the script doesn't work. I used the command that was provided in the course (and on https://github.com/armory/minnaker)

    ./scripts/install.sh -o -P $EXTERNAL_IP
    

    and I had a CrashLoopBackOff with minio. I don't have the exact logs right now as actually I forgot to send this message last Thursday and I moved on to the next course.

  • Hi @jsu, I was able to reproduce the issue. We are investigating about this, and will provide you an update as soon as we have news.

    Regards,
    Luis.

  • I have the same issue. @jsu please let me know if you found the solution.

  • zinebhalib
    zinebhalib Posts: 2
    edited November 2021

    I found the solution here https://github.com/armory/minnaker/issues/102
    "What worked for me was editing spinsvc/infrastructure/minio.yml, changing the path from /usr/bin/minio to /opt/bin/minio and then running ./deploy.sh, then deleting the old minio-0 pod."

  • Hi @zinebhalib , I opened that bug few days ago, hehehe. I haven't tried the workaround, I'll try it asap.

    Thanks for the update!

    Luis.

  • rknmit
    rknmit Posts: 5
    edited January 2022

    Hello,

    After doing the same steps or fix as suggested in https://github.com/armory/minnaker/issues/102. The pod is running but still getting the 404 error

    pod logs:
    WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
    Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
    API: http://10.42.0.16:9000 http://127.0.0.1:9000

    Console: http://10.42.0.16:35287 http://127.0.0.1:35287

    Documentation: https://docs.min.io

    WARNING: Console endpoint is listening on a dynamic port (35287), please use --console-address ":PORT" to choose a static port.

    Regards,
    Ravi

  • fredlev
    fredlev Posts: 4
    edited February 2022

    Hi, I have ran into the same issue.

    I am wondering if using spinnaker for anything still makes sense since it seems that the open source version is falling behind the Armory enterprise version.

    I've tried to configure halyard to use a dev deployment on a single VM but the bintray repository for spinnaker packages is KO.

    CF:

    https://stackoverflow.com/questions/68851795/spinnaker-installation-issue-due-to-dl-bintray-com-decommission

    I've also tried spinnaker the hardway but I am hitting a bunch of java compilation errors when trying to run the services:

    https://github.com/indrayam/spinnaker-the-hard-way/blob/master/LocalGitInstallation.md

    All of this makes me question the validity of spinnaker as a deployment tool going forward.

    IMO the course should have provided tagged images/versions of the tools and images used by those tools.

    For example: here minnaker will pull the latest image of minio-o by default instead of a TAGGED release hence the path issue in the command field.

  • lbono
    lbono Posts: 4

    @fredlev said:
    Hi, I have ran into the same issue.

    I am wondering if using spinnaker for anything still makes sense since it seems that the open source version is falling behind the Armory enterprise version.

    I've tried to configure halyard to use a dev deployment on a single VM but the bintray repository for spinnaker packages is KO.

    CF:

    https://stackoverflow.com/questions/68851795/spinnaker-installation-issue-due-to-dl-bintray-com-decommission

    I've also tried spinnaker the hardway but I am hitting a bunch of java compilation errors when trying to run the services:

    https://github.com/indrayam/spinnaker-the-hard-way/blob/master/LocalGitInstallation.md

    All of this makes me question the validity of spinnaker as a deployment tool going forward.

    IMO the course should have provided tagged images/versions of the tools and images used by those tools.

    For example: here minnaker will pull the latest image of minio-o by default instead of a TAGGED release hence the path issue in the command field.

    Agree. Same problem here. The WA to fix minio-0 deployment doesn't work either.
    And worse, my free Google cloud credit is finishing.
    Any Solution? Some Ubuntu server 4 cores/8Gigs to borrow me? :smile:

    Regards.-

  • Hi @lbono,

    We are currently working on this issue, and we'll provide you an update as soon as we have news.

    Regards,
    Luis.

  • @luisviveropena, I am having similar issues. What is the ETA on the fix?. Thanks

  • redrom35
    redrom35 Posts: 5

    @luisviveropena, I'm facing the same issue.
    Any update on this?

  • luisviveropena
    luisviveropena Posts: 1,144

    Hi @aderock and @redrom35 , the workarounds suggested in the bug are not working for me. So we are still looking at this issue.

    Regards,
    Luis.

  • gouravshah
    gouravshah Posts: 139

    @jsu @zinebhalib @rknmit @redrom35 @fredlev @lbono @luisviveropena finally I have found some fixes which I am sharing here for you to try. And this goes beyond minio issue reported here.

    While I recreated the same environment I found the following issues,

    1. Minio Issue - minio was going into crashloopbackoff error.
    2. Spinnaker Version Issue - After fixing minio, operator could come up ( in spinnaker-operator namespace) but could not launch spinnaker (only in case of open source version) as it was unable to find the correct version of spinnaker defined.

    Root Cause and Solutions :

    1. Minio Issue - this was very clear as reported by some of you above. This is a classic problem when using latest as image tag. In the deployment for minio, had a specific version been mentioned, this would have never happend.

    Whats interesting is, this was fixed in the master branch used by armory (tag : RELEASE.2021-08-20T18-32-01Z) as you could find it here https://github.com/armory/spinnaker-kustomize-patches/blob/master/infrastructure/minio.yml however, they just forgot about the minnaker branch used in our installation of open source minnaker here https://github.com/armory/spinnaker-kustomize-patches/blob/minnaker/infrastructure/minio.yml. (look at scripts/install.sh for default branch).

    This cleanest and easiest fix for this is to switch the branch while installing as,

    ./scripts/install.sh -o --branch master -P <PUBLIC_IP>
    

    2. Version Issue - This took me the longest and multiple iterations of troubleshoooting. Everytime I would setup the open source version of spinnaker it would get stuck at the ,"error":"You must pick a version of Spinnaker to deploy." ( spinnaker-operator logs)

    You could check the logs by listing all pods, finding the operator and using kubectl logs command,

    for example , start with kubectl get pods -A

    kube-system          svclb-traefik-vgvhm                         2/2     Running     0          104s
    kube-system          traefik-5dd496474-rp27r                     1/1     Running     0          104s
    spinnaker-operator   spinnaker-operator-7b755b8789-mlwfl         2/2     Running     0          2m5s
    spinnaker            svclb-spin-agent-clouddriver-m6vnb          1/1     Running     0          50s
    spi
    

    and then

    kubectl logs -f  -n spinnaker-operator -c spinnaker-operator spinnaker-operator-7b755b8789-mlwfl
    

    the error always was,

    "Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"You must pick a version of Spinnaker to deploy.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\....
    

    Finding the symptom above was fairly straight forward. The root cause analysis was what took really long time to figure out. Mainly due to very little documentation, no information about this error reported anywhere, and most importantly, bad code practice ( the code shows this version as being defined as param which is a good practice, but then completely bypasses this param and does its own hard coded thing, very bad).

    Finally after many attempts, recreating the env a few times, comparing with commercial version vs open source and diving into the code after, I found a very silly thing in the install script (script/install.sh) where it was using the hard coded version of code to fetch the version

      VERSION=$(curl -s https://spinnaker.io/community/releases/versions/ | grep 'id="version-' | head -1 | sed -e 's/\(<[^<][^<]*>\)//g; /^$/d' | cut -d' ' -f2)
    

    This is the broken code . try running ,

    curl -s https://spinnaker.io/community/releases/versions/ 
    
    

    which now redirects, versus observe this for the armory version

    curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml
    

    which returns the correct version info. This was the problem.

    A quick fix was to commenting line number 123 with previous VERSION being defined and to set the version manually as,


    After this if you clean up and install, it works ! At least the installation completes. Havent tested further by creating application pipelines yet.

    How to apply this fix

    As mentioned, edit file: scripts/install.sh and comment line 123, set version manually as shown in the section just above this.

    And then reinstall from within minnaker-0.1.3 directory by running the following

    ./scripts/uninstall-k3s.sh
    rm -rf spinsvc/
    

    and then install again as,

    ./scripts/install.sh -o -P PUBLIC_IP --branch master
    

    This should fix both issues, one with minio, also with version.

  • Right in time @gouravshah, I'll test your solution later today.

  • @manuel.nhiuana said:
    Right in time @gouravshah, I'll test your solution later today.

    @gouravshah said:
    @jsu @zinebhalib @rknmit @redrom35 @fredlev @lbono @luisviveropena finally I have found some fixes which I am sharing here for you to try. And this goes beyond minio issue reported here.

    While I recreated the same environment I found the following issues,

    1. Minio Issue - minio was going into crashloopbackoff error.
    2. Spinnaker Version Issue - After fixing minio, operator could come up ( in spinnaker-operator namespace) but could not launch spinnaker (only in case of open source version) as it was unable to find the correct version of spinnaker defined.

    Root Cause and Solutions :

    1. Minio Issue - this was very clear as reported by some of you above. This is a classic problem when using latest as image tag. In the deployment for minio, had a specific version been mentioned, this would have never happend.

    Whats interesting is, this was fixed in the master branch used by armory (tag : RELEASE.2021-08-20T18-32-01Z) as you could find it here https://github.com/armory/spinnaker-kustomize-patches/blob/master/infrastructure/minio.yml however, they just forgot about the minnaker branch used in our installation of open source minnaker here https://github.com/armory/spinnaker-kustomize-patches/blob/minnaker/infrastructure/minio.yml. (look at scripts/install.sh for default branch).

    This cleanest and easiest fix for this is to switch the branch while installing as,

    ./scripts/install.sh -o --branch master -P <PUBLIC_IP>
    

    2. Version Issue - This took me the longest and multiple iterations of troubleshoooting. Everytime I would setup the open source version of spinnaker it would get stuck at the ,"error":"You must pick a version of Spinnaker to deploy." ( spinnaker-operator logs)

    You could check the logs by listing all pods, finding the operator and using kubectl logs command,

    for example , start with kubectl get pods -A

    kube-system          svclb-traefik-vgvhm                         2/2     Running     0          104s
    kube-system          traefik-5dd496474-rp27r                     1/1     Running     0          104s
    spinnaker-operator   spinnaker-operator-7b755b8789-mlwfl         2/2     Running     0          2m5s
    spinnaker            svclb-spin-agent-clouddriver-m6vnb          1/1     Running     0          50s
    spi
    

    and then

    kubectl logs -f  -n spinnaker-operator -c spinnaker-operator spinnaker-operator-7b755b8789-mlwfl
    

    the error always was,

    "Reconciler error","controller":"spinnakerservice-controller","request":"spinnaker/spinnaker","error":"You must pick a version of Spinnaker to deploy.","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\....
    

    Finding the symptom above was fairly straight forward. The root cause analysis was what took really long time to figure out. Mainly due to very little documentation, no information about this error reported anywhere, and most importantly, bad code practice ( the code shows this version as being defined as param which is a good practice, but then completely bypasses this param and does its own hard coded thing, very bad).

    Finally after many attempts, recreating the env a few times, comparing with commercial version vs open source and diving into the code after, I found a very silly thing in the install script (script/install.sh) where it was using the hard coded version of code to fetch the version

      VERSION=$(curl -s https://spinnaker.io/community/releases/versions/ | grep 'id="version-' | head -1 | sed -e 's/\(<[^<][^<]*>\)//g; /^$/d' | cut -d' ' -f2)
    

    This is the broken code . try running ,

    curl -s https://spinnaker.io/community/releases/versions/ 
    
    

    which now redirects, versus observe this for the armory version

    curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml
    

    which returns the correct version info. This was the problem.

    A quick fix was to commenting line number 123 with previous VERSION being defined and to set the version manually as,


    After this if you clean up and install, it works ! At least the installation completes. Havent tested further by creating application pipelines yet.

    How to apply this fix

    As mentioned, edit file: scripts/install.sh and comment line 123, set version manually as shown in the section just above this.

    And then reinstall from within minnaker-0.1.3 directory by running the following

    ./scripts/uninstall-k3s.sh
    rm -rf spinsvc/
    

    and then install again as,

    ./scripts/install.sh -o -P PUBLIC_IP --branch master
    

    This should fix both issues, one with minio, also with version.

    I tried commenting the line 123, and now I'm having error deploying the operator, seems like the script is unable to parse and get the version of spinnaker-operator.
    the url " https://github.com/armory/spinnaker-operator/releases/download/v/manifests.tgz" doesn't exist

    Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
    [INFO] systemd: Starting k3s
    [INFO ] --- END K3s ---
    [INFO ] Setting Kubernetes context to Spinnaker namespace
    Context "default" modified.
    [INFO ] Spinnaker Operator Version:
    [INFO ] Spinnaker flavor: oss
    [INFO ] Changing spinnaker flavor...Done
    [INFO ] Resolved operator namespace: spinnaker-operator
    [INFO ] Deploying oss operator...
    [INFO ] Downloading operator from https://github.com/armory/spinnaker-operator/releases/download/v/manifests.tgz
    [KUBE ] kubectl apply -f /root/minnaker-0.1.3/spinsvc/operator/deploy/crds/
    [ERROR] Error executing command:
    error: the path "/root/minnaker-0.1.3/spinsvc/operator/deploy/crds/" does not exist[KUBE ] kubectl apply -f https://engineering.armory.io/manifests/pacrd-1.0.1.yaml -n spinnaker
    [ERROR] Error executing command:
    Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
    Error from server (NotFound): error when creating "https://engineering.armory.io/manifests/pacrd-1.0.1.yaml": namespaces "spinnaker" not foun

  • also tried set the version like this:

    if [[ ${OPEN_SOURCE} == 1 ]]; then
    info "Using OSS Spinnaker"
    SPIN_FLAVOR=oss
    VERSION="1.26.7"
    else
    info "Using Armory Spinnaker"
    SPIN_FLAVOR=armory
    VERSION=$(curl -sL https://halconfig.s3-us-west-2.amazonaws.com/versions.yml | grep 'version: ' | awk '{print $NF}' | sort | tail -1)
    fi

    the error is
    [KUBE ] kubectl apply -f /root/minnaker-0.1.3/min/minnaker-0.1.3/spinsvc/operator/deploy/crds/
    [ERROR] Error executing command:
    error: the path "/root/minnaker-0.1.3/min/minnaker-0.1.3/spinsvc/operator/deploy/crds/" does not exist[KUBE ] kubectl apply -f https://engineering.armory.io/manifests/pacrd-1.0.1.yaml -n spinnaker
    [ERROR] Error executing command:
    Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unav

  • gouravshah
    gouravshah Posts: 139

    Try setting the version by commenting #123 and then just manually setting

    VERSION=1.26.7

  • hi ... this is still the "state of the art" workaround for this problem??

  • luisviveropena
    luisviveropena Posts: 1,144
    edited October 2022

    Hi @donquijotedelasnubes, we are working on updates on this topic.

    Regards,
    Luis.

Categories

Upcoming Training