Exercise 15.2: DevDan doesn't work

nlp

Exercise 15.2: Authentication and Authorization

Summary: I can't work with resources associated with the user DevDan, the context DevDan-context, etc.

Details --

Problem symptoms:

$kubectl config use-context kubernetes-admin@kubernetes
Switched to context "kubernetes-admin@kubernetes"

$ kubectl --context=DevDan-context get pods
error: You must be logged in to the server (Unauthorized)

$ sudo kubectl --context=DevDan-context get pods
Error in configuration: context was not found for specified context: DevDan-context

$ kubectl config use-context DevDan-context
Switched to context "DevDan-context"

$ sudo kubectl get pods
The connection to the server localhost:8080 was refused - did you specify the right host or port?

I'm not going to walk through every single step I took - I followed the directions exactly, with the exection of naming the various DevDan auth files "dd.key/.crt/.csr" for brevity.

Here's what I'm working with, with extraneous results removed --

System: Ubuntu 20.04

• Config details:

$ kubectl version --short
Client Version: v1.24.0
Kustomize Version: v4.5.4
Server Version: v1.24.0

$ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://k8scp:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: development
    user: DevDan
  name: DevDan-context
- context:
    cluster: kubernetes
    namespace: default
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: DevDan
  user:
    client-certificate: /home/nathaniel_lapier/dd.crt
    client-key: /home/nathaniel_lapier/dd.key
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

$ less .kube/config
contexts:
- context:
    cluster: kubernetes
    namespace: development
    user: DevDan
  name: DevDan-context
- context:
    cluster: kubernetes
    namespace: default
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: DevDan-context
kind: Config
preferences: {}
users:
- name: DevDan
  user:
    client-certificate: /home/nathaniel_lapier/dd.crt
    client-key: /home/nathaniel_lapier/dd.key
- name: kubernetes-admin
  user:
    client-certificate-data: [...]
    client-key-data: [...]

$ sudo less /etc/kubernetes/kubelet.conf 
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [...]
    server: https://k8scp:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:k8stest-raw-1
  name: system:node:k8stest-raw-1@kubernetes
current-context: system:node:k8stest-raw-1@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:k8stest-raw-1
  user:
    client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
    client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

• Remaining exercise-specific settings:

$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   14d
development       Active   17h
[...]
production        Active   17h
small             Active   4d21h

$ sudo less /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
[...]
DevDan:x:1001:1002::/home/DevDan:/bin/bash

nathaniel_lapier@k8stest-raw-1:~$ ls -a
dd.crt    dd.csr    dd.key    .rnd     LFS258    

$ kubectl get roles -A
NAMESPACE     NAME                                         
default       kdash-kubernetes-dashboard                       
default       myingress-ingress-nginx                          
development   developer                                        
kube-public   kubeadm:bootstrap-signer-clusterinfo             
[...]             

$ kubectl get rolebindings -A
NAMESPACE     NAME                                         ROLE 
default      myingress-ingress-nginx   Role/myingress-ingress-nginx                         
development  developer-role-binding                    Role/developer
[...]

Is any part of my confuration not what should be expected? I've walked through a slew of Googled tips around users, contexts, permissions etc. This post shows the results of my second attempt at this exercise - after hitting this dead end on my first attempt, I deleted everything from the exercise and started over, with no effect.

pnts

kubectl --context=DevDan-context get pods is correct and should be able to work.

You do not want sudo when using kubectl.
It will look for kubeconfig in /root/.kube/config which does probably not exist.

2 things I would try:

1. kubectl --context=DevDan-context get pods -v=10 Anything to learn from the verbose output?
2. Try embedding the client cert and key in the kubeconfig using --embed-certs: kubectl config set-credentials DevDan --client-key dd.key --client-certificate dd.crt --embed-certs

nlp

@pnts said:
kubectl --context=DevDan-context get pods is correct and should be able to work.

You do not want sudo when using kubectl.
It will look for kubeconfig in /root/.kube/config which does probably not exist.

2 things I would try:

1. kubectl --context=DevDan-context get pods -v=10 Anything to learn from the verbose output?
2. Try embedding the client cert and key in the kubeconfig using --embed-certs: kubectl config set-credentials DevDan --client-key dd.key --client-certificate dd.crt --embed-certs

$ kubectl config set-credentials DevDan --client-key dd.key --client-certificate dd.crt --embed-certs
User "DevDan" set.

$ kubectl --context=DevDan-context get pods -v=10
I1110 23:14:07.097741 2828059 loader.go:372] Config loaded from file:  /home/nathaniel_lapier/.kube/config
[tons of kube/cache jsons...]
 curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.24.0 (linux/amd64) kubernetes/4ce5a89" 'https://k8scp:6443/api/v1/namespaces/development/pods?limit=500'
HTTP Trace: DNS Lookup for k8scp resolved to [{10.2.0.4 }]
 HTTP Trace: Dial to tcp:10.2.0.4:6443 succeed
 GET https://k8scp:6443/api/v1/namespaces/development/pods?limit=500 401 Unauthorized in 13 milliseconds
 HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 9 ms ServerProcessing 2 ms Duration 13 ms
 Response Headers:
     Content-Type: application/json
     Content-Length: 129
     Date: Thu, 10 Nov 2022 23:12:00 GMT
     Audit-Id: 95f6a005-64bf-4b97-a21b-415618577460
     Cache-Control: no-cache, private
 Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I1110 23:12:00.710028 2826896 helpers.go:222] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}]
error: You must be logged in to the server (Unauthorized)

Lots of output, but in the end it's just a 401, even with the certs.

nlp

Hah - I'll eat some crow and hopefully help the next person who gets stuck:

openssl req -new -key DevDan.key \
-out dd.csr -subj "/CN=DevDan/O=development" #make sure this -subj argument exactly matches your user and namespace