Lab 7.5 Falco and kube-apiserver audit webhook

cyril31

Hello,

I following the CKS course and I have an issue with the chapter 7.5 concerning Falco audit log from kube-apiserver.

My problem is, following the course instructions, I didn’t receive any log on Falco from kube-apiserver (journalctl -u falco -f) after configuring webhook audit log on the kube-apiserver.

Reading the documentation of Falco I found another process using Falco plugin and not embedded webserver of Falco to process audit log :

https://falco.org/docs/event-sources/kubernetes-audit/#what-s-new-in-falco

https://github.com/falcosecurity/charts/tree/master/falco#kubernetes-audit-log

The second link point on the helm chart config, but for the training I used the ubuntu apt package manager to install Falco like in the course, the configuration is the same.

With this plugin, I received the audit log incoming from kube-apiserver.

The main difference between the embedded and plugin based solution is the integrated solution listen on port 8765 and the plugin on 9765.

One thing very import (If i correctly understand, please correct me if I’m wrong) is the embedded solution was removed from Falco version 0.32.0 and when we install the apt package we do not mention a version so the latest stable version is installed → 0.32.2.

https://github.com/falcosecurity/charts/blob/master/falco/README.md#falco-event-sources-and-kubernetes

Since Falco 0.32.0 all the related code to the k8s Audit Logs in Falco was removed and ported in a plugin. At the time being Falco supports different event sources coming from plugins or the drivers (system events).

Thanks,
Cyril

aleksvand

Hey Cyril,

Yes, the course content is outdated.

The Kubernetes team removed the AuditSink (dynamic webhook) in version 1.19 so Falco uses plugins. Starting from v0.31.0 the new plugin system is stable and production ready.

In the scenario when Falco is used with plugins as data sources, then the best option is to deploy it as a k8s deployment. Configuration settings to enable k8s audit are completely different between 0.31 and 0.32. Before it was deployed as a DeamonSet and now it is a Deployment. The Falco service which exposes the pods now by default has a nodePort.

Networking:
port 3007 = nodePort (external to the cluster)
port 8765 = used only for the healthz endpoint
port 9765 = service port (internal to the cluster)

In August 2022 Falco creator Loris Degioanni and core maintainer Leonardo Grasso released a book on Falco published by Oreilly. It is available for free from the Sysdig website.

Kind regards