Lab 22.1: Cortex "No address found for [eth0 en0]"

donaldsebleung

In Lab 22.1, after I follow the instructions to download the Cortex binary and the single process configuration file single-process-config.yaml, starting Cortex with the given configuration reports that no address has been found for interfaces eth0 and en0:

$ ./cortex-linux-amd64 -config.file=single-process-config.yaml
level=info ts=2022-04-15T01:27:36.793236294Z caller=main.go:188 msg="Starting Cortex" version="(version=1.8.0, branch=release-1.8, revision=51662ea3e)"
level=info ts=2022-04-15T01:27:36.793757267Z caller=server.go:229 http=[::]:9009 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2022-04-15T01:27:36.79576571Z caller=net.go:17 msg="error getting interface" inf=eth0 err="route ip+net: no such network interface"
level=warn ts=2022-04-15T01:27:36.795949987Z caller=net.go:17 msg="error getting interface" inf=en0 err="route ip+net: no such network interface"
level=error ts=2022-04-15T01:27:36.796036815Z caller=log.go:106 msg="error running cortex" err="No address found for [eth0 en0]\nerror initialising module: ingester-service\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/Users/peter/Grafana/cortex/pkg/util/modules/modules.go:105\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/Users/peter/Grafana/cortex/pkg/util/modules/modules.go:75\ngithub.com/cortexproject/cortex/pkg/cortex.(*Cortex).Run\n\t/Users/peter/Grafana/cortex/pkg/cortex/cortex.go:388\nmain.main\n\t/Users/peter/Grafana/cortex/cmd/cortex/main.go:190\nruntime.main\n\t/usr/local/Cellar/go/1.15.5/libexec/src/runtime/proc.go:204\nruntime.goexit\n\t/usr/local/Cellar/go/1.15.5/libexec/src/runtime/asm_amd64.s:1374"

A bit of Googling reveals that this is because Cortex assumes the default network interface names as above are present (at least one of them, anyway) and tries to read the IP address(es) from those interfaces, but fails since none of those interfaces are present (on my machine anyway - the default network interface is ens5). Even after attempting to specify the network interface explicitly as suggested in a comment in cortexproject/cortex#3854, the same error persists and I have been unable to start Cortex successfully. Therefore, I would like to request an update for the lab description using an alternative configuration that works or specifying a workaround for the issue (such as how to specify the interface that Cortex should query). Thanks!

For reference, I am performing the LFS241 labs on a 2-node kubeadm cluster (1 master, 1 worker) on AWS, both m5.large instances, with Calico as the CNI. In particular, I am attempting Lab 22.1 on the master node with the following network interfaces and IP addresses as reported by ip addr show:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 06:6a:f5:93:bd:dc brd ff:ff:ff:ff:ff:ff
    inet 172.31.28.90/20 brd 172.31.31.255 scope global dynamic ens5
       valid_lft 3387sec preferred_lft 3387sec
    inet6 fe80::46a:f5ff:fe93:bddc/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:9d:49:61:44 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 8981 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 192.168.107.192/32 scope global tunl0
       valid_lft forever preferred_lft forever
7: cali2f1bdf57c08@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8981 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
8: califa6e4029763@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8981 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever
9: cali98697e0732b@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8981 qdisc noqueue state UP group default 
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link 
       valid_lft forever preferred_lft forever

zorque

For someone coming across this issue (as it is among the top google hits), the issue seems to be that the representation in the config file and the representation via command line flags for the interface overrides is not aligned in an intuitive way. I fixed this by stepping through the docs myself and building the command line to launch cortex as follows:

./cortex-linux-amd64 -config.file=single-process-config.yaml \
 -distributor.ring.instance-interface-names=enp0s3 \
 -ingester.lifecycler.interface=enp0s3 \
 -frontend.instance-interface-names=enp0s3 \
 -ruler.ring.instance-interface-names=enp0s3 \
 -alertmanager.sharding-ring.instance-interface-names=enp0s3 \
 -compactor.ring.instance-interface-names=enp0s3 \
 -store-gateway.sharding-ring.instance-interface-names=enp0s3