Exercise 3.2: Configure A Local Docker Repo - Unable to Push to Local Docker Repo
Hi all,
My Kubernetes cluster is setup behind a proxy, and I have had to create NO_PROXY entries for cluster nodes and services in the /etc/systemd/system/docker.service.d/http-proxy.conf file.
Environment="NO_PROXY="localhost,127.0.0.1,10.100.113.117.....
where "10.100.113.117" is the service/registry ClusterIP
All looks good with the Local docker repo setup on this Exercise 3.2, till the point of pushing the docker image to the local repo.
It fails with error : Client.Timeout exceeded while awaiting headers
root@kubemaster:~# sudo docker push 10.100.113.117:5000/tagtest
The push refers to repository [10.100.113.117:5000/tagtest]
Get http://10.100.113.117:5000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
The registry and nginx pods are running, as below;
root@kubemaster:~# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-f875c68bb-mtxhv 1/1 Running 0 3h20m
registry-7ccd695dc7-4d8px 1/1 Running 0 3h20m
curl http://10.100.113.117:5000/v2/
{}root@kubemaster
sudo vim /etc/docker/daemon.json
{ "insecure-registries":["10.100.113.117:5000"] }sudo systemctl status docker.service | grep Active
Active: active (running) since Sun 2020-05-10 13:27:36 UTC; 15min ago
Anybody experienced similar issue with pushing docker images to a local repo? Can anyone please with this?
Answers
-
Hi @eromskiee,
While the NO_PROXY seems to be a common fix when running behind a proxy, there may be other settings specific to your own environment that should also be examined. Your hypervisor, cloud provider, operating system, and possibly others play a key role in figuring out what causes the timeout. I would start by looking at firewalls, both at your infra level and OS level as most often they are the reason users experience timeouts.
Regards,
-Chris0 -
I have the same issue, but my question is: how could I access to a ClusterIP service from a Node (not from a Pod) without creating a proxy? Shouldn't it be a NodePort type service to be accessible from nodes?
In the Exercise 3.2: Configure A Local Repo there is no mention about proxy..0 -
Hello Simone,
Calico allows the nodes to contact a ClusterIP. Assuming there isn't a misconfiguration in firewall or networking each node should be able to connect to each ClusterIP. The ClusterIP is the persistent IP within the cluster. A NodePort would be used for traffic outside of the cluster.
Regards,
0 -
Thanks @serewicz for clarify! You're right, indeed from the same worker node where the service is running, I'm able to reach the ClusterIP service, only from the control plane I'm not!
0 -
Hello,
Please share the output of kubectl get pods --all-namespaces, so we can see if calico is having any issues. Also are you sure there is no firewall between nodes? If using GCE ensure that the VPC allows ALL traffic.
Regards,
0 -
Hi, this is the output:
root@ip-172-31-2-67:~# k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 1 (22h ago) 22h
default dep1-5749cf8c94-7b29l 1/1 Running 0 39h
default dep1-5749cf8c94-gwjl9 1/1 Running 0 39h
default dep1-5749cf8c94-kq4xp 1/1 Running 0 21h
default dep1-5749cf8c94-wsv4k 1/1 Running 0 39h
default nginx-b68dd9f75-9fs86 1/1 Running 0 22h
default probe-pod 1/1 Running 19 (22m ago) 19h
default registry-6b5bb79c4-4gkzk 1/1 Running 0 22h
default sleep3for5--1-ls62m 0/1 Completed 0 18h
default sleep3for5--1-r887m 0/1 Completed 0 18h
default test 1/1 Running 0 40h
kube-system calico-kube-controllers-5d995d45d6-m6n2h 1/1 Running 0 9d
kube-system calico-node-xd6cg 1/1 Running 1 41h
kube-system calico-node-zrs64 1/1 Running 0 9d
kube-system coredns-78fcd69978-m4ztt 1/1 Running 0 9d
kube-system coredns-78fcd69978-qqgzg 1/1 Running 0 9d
kube-system etcd-cp 1/1 Running 0 9d
kube-system kube-apiserver-cp 1/1 Running 0 9d
kube-system kube-controller-manager-cp 1/1 Running 0 9d
kube-system kube-proxy-5d68h 1/1 Running 0 9d
kube-system kube-proxy-j5s66 1/1 Running 2 9d
kube-system kube-scheduler-cp 1/1 Running 0 9dAbout the network, the instances (AWS EC2) are in the same security group and I allowed all traffic (from their private ip) between them.
0 -
Hi @SimoneZennaro,
I see two issues here:
the instances (AWS EC2) are in the same security group
Since SG names are unique, it seems that each instance is in its own SG, and you have 2 separate SGs defined. I would correct this first.
allowed all traffic
It seems that only TCP traffic is allowed, while all other protocols will be blocked. The earlier recommendation to allow ALL traffic, implied all protocols, all sources. Kubernetes and its plugins may use other protocols to communicate in addition to TCP to manage the cluster.
The course includes a demo video guiding the set up process for AWS EC2 instances, including VPC and SG configuration. I would recommend watching the video as it may clarify the overall set up.
Regards,
-Chris0 -
thanks @chrispokorni , changing All TCP with All Traffic it worked!
0
Categories
- All Categories
- 61 LFX Mentorship
- 114 LFX Mentorship: Linux Kernel
- 618 Linux Foundation IT Professional Programs
- 322 Cloud Engineer IT Professional Program
- 141 Advanced Cloud Engineer IT Professional Program
- 56 DevOps Engineer IT Professional Program
- 67 Cloud Native Developer IT Professional Program
- 6 Express Training Courses
- 6 Express Courses - Discussion Forum
- 2.3K Training Courses
- 19 LFC110 Class Forum - Discontinued
- 9 LFC131 Class Forum
- 31 LFD102 Class Forum
- 178 LFD103 Class Forum
- LFD110 Class Forum
- 24 LFD121 Class Forum
- LFD133 Class Forum
- 2 LFD137 Class Forum
- 62 LFD201 Class Forum
- 2 LFD210 Class Forum
- 1 LFD210-CN Class Forum
- 1 LFD213 Class Forum - Discontinued
- 128 LFD232 Class Forum - Discontinued
- LFD233 Class Forum
- LFD237 Class Forum
- 23 LFD254 Class Forum
- 659 LFD259 Class Forum
- 108 LFD272 Class Forum
- 1 LFD272-JP クラス フォーラム
- 4 LFD273 Class Forum
- LFS101 Class Forum
- LFS116 Class Forum
- 2 LFS145 Class Forum
- 28 LFS200 Class Forum
- 740 LFS201 Class Forum - Discontinued
- 1 LFS201-JP クラス フォーラム
- 13 LFS203 Class Forum
- 98 LFS207 Class Forum
- 301 LFS211 Class Forum
- 54 LFS216 Class Forum
- 47 LFS241 Class Forum
- 41 LFS242 Class Forum
- 38 LFS243 Class Forum
- 12 LFS244 Class Forum
- LFS245 Class Forum
- 41 LFS250 Class Forum
- 1 LFS250-JP クラス フォーラム
- LFS251 Class Forum
- 143 LFS253 Class Forum
- LFS254 Class Forum
- LFS255 Class Forum
- 1 LFS256 Class Forum
- LFS257 Class Forum
- 1.2K LFS258 Class Forum
- 10 LFS258-JP クラス フォーラム
- 108 LFS260 Class Forum
- 147 LFS261 Class Forum
- 39 LFS262 Class Forum
- 83 LFS263 Class Forum - Discontinued
- 15 LFS264 Class Forum - Discontinued
- 11 LFS266 Class Forum - Discontinued
- 21 LFS267 Class Forum
- 18 LFS268 Class Forum
- 26 LFS269 Class Forum
- 204 LFS272 Class Forum
- 1 LFS272-JP クラス フォーラム
- LFS274 Class Forum
- 3 LFS281 Class Forum
- 258 LFW211 Class Forum
- 178 LFW212 Class Forum
- 9 SKF100 Class Forum
- SKF200 Class Forum
- 908 Hardware
- 221 Drivers
- 74 I/O Devices
- 44 Monitors
- 116 Multimedia
- 210 Networking
- 102 Printers & Scanners
- 86 Storage
- 765 Linux Distributions
- 88 Debian
- 66 Fedora
- 15 Linux Mint
- 13 Mageia
- 24 openSUSE
- 144 Red Hat Enterprise
- 33 Slackware
- 13 SUSE Enterprise
- 357 Ubuntu
- 483 Linux System Administration
- 40 Cloud Computing
- 70 Command Line/Scripting
- Github systems admin projects
- 95 Linux Security
- 80 Network Management
- 108 System Management
- 52 Web Management
- 74 Mobile Computing
- 25 Android
- 34 Development
- 1.2K New to Linux
- 1.1K Getting Started with Linux
- 544 Off Topic
- 131 Introductions
- 223 Small Talk
- 22 Study Material
- 835 Programming and Development
- 284 Kernel Development
- 517 Software Development
- 975 Software
- 261 Applications
- 185 Command Line
- 3 Compiling/Installing
- 119 Games
- 318 Installation
- 65 All In Program
- 65 All In Forum
Upcoming Training
-
August 20, 2018
Kubernetes Administration (LFS458)
-
August 20, 2018
Linux System Administration (LFS301)
-
August 27, 2018
Open Source Virtualization (LFS462)
-
August 27, 2018
Linux Kernel Debugging and Security (LFD440)