download.docker.com repository signature verification problem

cn20230504

I ran into problems with Lab 3.1 following the instructions to add the "download.docker.com" repository and installing containerd. I am using the posted lab instructions using Ubuntu 20.04 nodes hosted on AWS (I first tried 22.04, but encountered the cgroup error reported in "Still unable to join worker node to master node, please help").

In steps 12 and 13 the instructions describe downloading a GPG from "download.docker.com", installing "download.docker.com" as a source repository and using the apt-get update command to update the apt cache. When I did, I got the following output:

Hit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease   
Hit:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease 
Get:4 https://download.docker.com/linux/ubuntu jammy InRelease [48.9 kB]       
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease               
Hit:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease    
Err:4 https://download.docker.com/linux/ubuntu jammy InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
Reading package lists... Done
W: GPG error: https://download.docker.com/linux/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EA0A9C3F273FCD8
E: The repository 'https://download.docker.com/linux/ubuntu jammy InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://apt.kubernetes.io/dists/kubernetes-xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Subsequent apt updates and installs fail after this making it impossible to complete the cluster setup.

Here is a short bash script replicating this problem copying commands from the Lab 3.1 PDF document:

#! /bin/bash
sudo mkdir -p /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt-get update

I eventually worked around the problem when I realized that as of Ubuntu 20.04 (Focal Fossa) packages for containerd (package name containerd) which obviated the need to download a key and add "download.docker.com" as a source repository.

So I suggest the command on step 7 simply add the package "containerd" like:

apt install curl apt-transport-https vim git wget gnupg2 \
software-properties-common lsb-release ca-certificates uidmap containerd -y

remove step 12 and remove the "apt-get update" command from step 13 in the Lab 3.1 instructions.

chrispokorni

Hi @cn20230504,

The containerd package from the Ubuntu 20.04 LTS repo is typically an older release, while installing the containerd.io package from docker repo is the latest.

Regards,
-Chris

cn20230504

Thank you @chrispokorni for the response. Thank you also for pointing out that the version released from the docker repo is more up-to-date. That said, if the repo signature doesn't verify, it blocks ALL installs and I cannot complete the cluster setup.

Is there a way to resolve the signature verification problem? Alternatively, is using the older release of containerd packaged by Ubuntu acceptable to complete the course?

cn20230504

As a follow-up to my message I'd like to point out that the signature problem may be due to a change in the Docker repo key. The instructions worked for me the first time I ran through them and I managed to successfully build a cluster and completed labs 3.1-3.5. However, when I tried to build the cluster from scratch a second time I encountered the signature problem. It appears that between these two build attempts the docker gpg key had changed (timestamp on the key file reads 2023-05-08 13:45:50) as one can see on the following screenshot:

I have now attempted to obtain containerd from the Docker repository multiple times over nearly 24 hours, so the issue seems to be sticking around. I don't know how to contact any of the repository maintainers or whether others are encountering this problem, so I don't know whether this is a transient glitch or whether something more permanent is happening. I'd agree that the lab instructions are fine if this is simply a matter of a maintainer slip-up to be fixed shortly but if something else is going on alternatives might be valuable. Am I the only one encountering this problem?

chrispokorni

Hi @cn20230504,

The discussion was moved to the correct Forum - LFS258, as the description did not match lab exercises 3.1-3.5 of LFD259.

Regards,
-Chris

chrispokorni

Hi @cn20230504,

I attempted to reproduce the issue reported in this discussion - with no luck.

I followed the Infra provisioning videos from the intro chapter of the course, and provisioned two EC2 instances on AWS, of the recommended sizes, recommended OS - Ubuntu 20.04 LTS, recommended VPC network and SG rule.

Every single step of exercise 3.1 was run as presented in the lab guide, and extra validation steps were completed to ensure that all the desired config files were set into place and properly populated with all the necessary data.

It all worked as expected. There are no issues with the docker gpg key, no issues with the docker repo, no issues downloading the containerd.io package.

Please review all sources provided for you in the course material, and follow them closely to provision the infrastructure according to the recommended specs.

Deviating from these instructions will not only prevent your Kubernetes cluster from running as expected, but also impact the communication between application components running on different nodes (as you reported in a comment on another discussion thread).

Regards,
-Chris

cn20230504

@chrispokorni,

Thank you for taking the time to attempt to replicate my problem. I finally got it working for myself again. I was puzzled why it did not work after my first successful build attempt, but I traced the problem down to a permissions issue. The first time I went through the procedure I made the /etc/apt/keyrings directory by hand, and got containerd to install from Docker just fine. But for my subsequent attempts, I automated the procedure, and out of an abundance of caution I slapped 0700 permissions on the /etc/apt/keyrings directory which I (mistakenly) assumed would be innocuous but safe. It turns out that the 0700 permissions were preventing the key from getting read by apt-update, thus subsequent update attempts failed. When I changed the permissions on /etc/apt/keyrings to 0755, everything worked just fine once again. The timestamp change on the gpg key was evidently a red herring that threw me off and sent me down a pointless rabbit hole.

Henceforth I shall follow the lab instructions, which as they pull from the most up-to-date source is what I should have been doing all along.

Thank you once again for your help with this problem.

Best Regards,
Ralph Santos