Compilation results for ml-serve1014.eqiad.wmnet: System changes detected

You can retrieve this result from host.json.

Catalog differences

Summary

Total Resources:	2865
Resources added:	353
Resources removed:	5
Resources modified:	364
Change percentage:	25.20%

Resources only in the new catalog

Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh]
File[/etc/rsyslog.d/00-imfile.conf]
Systemd::Unit[cpupower]
Exec[renew certificate - mlserve__istio-cni]
Class[Profile::Containerd]
Cfssl::Csr[/etc/cfssl/csr/mlserve__calicoctl.csr]
Service[containerd]
Exec[/usr/sbin/dpkg-reconfigure -p critical -f noninteractive wikimedia-lvs-realserver]
File[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]
File[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]
Exec[create chained cert /etc/kubernetes/pki/mlserve__calico-cni.chain.pem]
Service[kubelet]
File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem]
File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]
Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh on intermediate ca change]
File[/etc/calico/calicoctl.cfg]
Package[containerd]
Concat[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
Sysctl::Conffile[kube_proxy_conntrack]
Exec[/sbin/modprobe overlay]
Exec[Generate cert mlserve__rsyslog refresh]
File[/etc/modules-load.d/overlay.conf]
File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.service]
File[/etc/systemd/system/kubelet.service.d]
File[/etc/cni/net.d/istio-kubeconfig]
Exec[systemd daemon-reload for kube-proxy.service (kube-proxy)]
File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]
File[/etc/cni/net.d/calico-kubeconfig]
Class[Profile::Dragonfly::Dfdaemon]
File[/etc/kubernetes/pki/mlserve__system_kube-proxy.csr]
Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)]
File[/etc/ferm/conf.d/10_calico_typha]
Exec[Generate cert mlserve__istio-cni]
Class[Containerd::Nerdctl]
File[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]
Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet]
Package[linux-cpupower]
Exec[apt_pin_apt_pin_linux-6.16-trixie_trixie-bpo]
Service[kube-proxy]
File[/etc/udev/rules.d/70-render.rules]
Cfssl::Cert[mlserve__rsyslog]
Apt::Package_from_bpo[firmware-amd-graphics-trixie-bpo]
Exec[apt_package_from_component_kubernetes131]
Exec[systemd daemon-reload for rsyslog-imfile-remedy.timer (rsyslog-imfile-remedy.timer)]
Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref]
File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem]
Systemd::Syslog[rsyslog-release-deleted-inotify-watches]
File[/etc/cfssl/csr/mlserve__kubelet_server.csr]
Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem]
Systemd::Override[ferm-service-auto-restart]
Class[Calico]
Exec[apt_repository_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
K8s::Package[proxy]
File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.csr]
File[/etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf]
Ferm::Service[dragonfly_dfget]
Node[__node_regexp__ml-serve1001-910-5.eqiad.]
Cfssl::Csr[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]
K8s::Kubeconfig[/etc/kubernetes/kubelet.conf]
Rsyslog::Conf[input-file-kubernetes-json]
Package[kubernetes-node]
Systemd::Timer::Job[rsyslog-imfile-remedy]
File[/etc/ferm/conf.d/10_kubelet-http]
File[/var/lib/kubelet/config.json]
File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem]
Exec[create chained cert /etc/kubernetes/pki/mlserve__calicoctl.chain.pem]
File[/etc/default/kube-proxy]
Cfssl::Csr[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]
Exec[Generate cert mlserve__istio-cni refresh on intermediate ca change]
Rsyslog::Conf[rsyslog-release-deleted-inotify-watches]
Sysctl::Parameters[increase_inotify_limits]
File[/etc/default/cpupower]
Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Concat[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
Concat_file[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
Systemd::Unit[rsyslog-imfile-remedy.timer]
File[/etc/kubernetes/pki/mlserve__calicoctl.chained.pem]
File[/etc/kubernetes/proxy.conf]
File[/etc/cfssl/csr/mlserve__calicoctl.csr]
File[/usr/local/sbin/rsyslog-release-deleted-inotify-watches]
Package[amd-k8s-node-labeller]
K8s::Kubeconfig[/etc/calico/calicoctl-kubeconfig]
Firewall::Service[dragonfly_dfget]
Package[amd-k8s-device-plugin]
Exec[cpupower_reload]
Exec[Generate cert mlserve__calico-cni]
Apt::Package_from_component[kubernetes131]
File[/etc/dragonfly/dfget.yml]
Service[dragonfly-dfdaemon]
File[/etc/kubernetes/pki]
File[/etc/containerd]
File[/etc/systemd/system/ferm.service.d/ferm-service-auto-restart.conf]
Class[K8s::Kubelet]
Exec[create chained cert /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]
Cfssl::Cert[mlserve__system_kube-proxy]
Class[Wmflib::Service::Catalog]
Apt::Package_from_component[calico329]
Exec[Generate cert mlserve__calico-cni refresh]
Class[K8s::Base_dirs]
File[/etc/udev/rules.d/75-kube_proxy_conntrack.rules]
Systemd::Unit[ferm-ferm-service-auto-restart]
Exec[systemd daemon-reload for rsyslog-imfile-remedy.service (rsyslog-imfile-remedy.service)]
File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem]
Class[Profile::Calico::Kubernetes]
File[/etc/ferm/conf.d/10_dragonfly_dfget]
Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.service (rsyslog-release-deleted-inotify-watches.service)]
Exec[Generate cert mlserve__system_kube-proxy refresh]
Sysctl::Parameters[kube_proxy_conntrack]
Package[dragonfly-dfget]
Service[cpupower]
Ferm::Service[kubelet-http]
Ferm::Service[calico_typha]
Exec[systemd daemon-reload for amd-k8s-node-labeller.service (amd-k8s-node-labeller-amd-devplugin-after-labeller)]
File[/etc/default/wikimedia-lvs-realserver]
Concat_file[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
K8s::Kubelet::Cni[calico]
Systemd::Service[cpupower]
File[/etc/calico/pki]
Cfssl::Cert[mlserve__calico-cni]
Cfssl::Cert[mlserve__system_node_ml-serve1014_eqiad_wmnet]
File[/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem]
Exec[apt_repository_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
Apt::Package_from_component[istio115]
Class[Toil::Rsyslog_imfile_remedy]
Kmod::Module[overlay]
Exec[Generate cert mlserve__amdgpu-node-labeller]
Systemd::Service[rsyslog-release-deleted-inotify-watches]
File[/etc/kubernetes/pki/mlserve__calicoctl.pem]
Apt::Repository[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/cfssl/csr/mlserve__calico-cni.csr]
Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Systemd::Override[container-runtime]
Class[Base::Sysctl::Inotify]
Rsyslog::Conf[output_kafka_k8s]
File[/etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref]
Group[kube]
Sysctl::Conffile[ipv6-fowarding-accept-ra]
K8s::Kubeconfig[/etc/kubernetes/proxy.conf]
Package[calicoctl]
File[/etc/kubernetes/pki/mlserve__calico-cni.csr]
Class[Profile::Rsyslog::Kubernetes]
File[/etc/amd]
Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh on intermediate ca change]
Apt::Pin[apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]
Exec[renew certificate - mlserve__kubelet_server]
File[/etc/nerdctl/nerdctl.toml]
File[/etc/systemd/system/kube-proxy.service.d]
Exec[Generate cert mlserve__kubelet_server]
Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet]
Package[wikimedia-lvs-realserver]
Class[Dragonfly::Dfdaemon]
Sysctl::Conffile[kube_proxy_icmp]
K8s::Kubeconfig[/etc/cni/net.d/calico-kubeconfig]
Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo]
File[/etc/kubernetes/pki/mlserve__istio-cni.csr]
File[/etc/cni/net.d]
File[/etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf]
Exec[Generate cert mlserve__kubelet_server refresh]
File[/etc/amd/node-labeller-kubeconfig]
File[/etc/nerdctl]
File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem]
File[/var/run/kubernetes]
Firewall::Service[calico-typha]
File[/etc/kubernetes/pki/mlserve__istio-cni-key.pem]
Cfssl::Csr[/etc/cfssl/csr/mlserve__rsyslog.csr]
Class[K8s::Proxy]
File[/etc/dragonfly]
Cfssl::Cert[discovery__ml-serve1014_eqiad_wmnet]
File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer]
Package[calico-cni]
Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Class[Profile::Rsyslog::Shellbox]
Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem]
Exec[Generate cert mlserve__amdgpu-node-labeller refresh on intermediate ca change]
File[/etc/ferm/conf.d/10_calico-bird]
Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Exec[systemd daemon-reload for kubelet.service (kubelet-container-runtime)]
File[/etc/kubernetes/pki/mlserve__system_kube-proxy.pem]
File[/etc/cni]
Class[Apparmor]
Exec[apt_repository_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/default/kubelet]
File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]
File[/etc/sysctl.d/70-increase_inotify_limits.conf]
File[/etc/kubernetes/pki/mlserve__calico-cni.chained.pem]
Ferm::Service[calico-bird]
Exec[create chained cert /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]
Exec[Generate cert mlserve__calico-cni refresh on intermediate ca change]
Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
Class[Lvs::Realserver]
Systemd::Unit[rsyslog-release-deleted-inotify-watches.service]
Exec[renew certificate - mlserve__calicoctl]
File[/etc/rsyslog.d/35-output-kafka-k8s.conf]
Cfssl::Cert[mlserve__kubelet_server]
Systemd::Unit[kube-proxy]
File[/usr/libexec/cpupower]
Class[Profile::Kubernetes::Node]
Sysctl::Conffile[increase_inotify_limits]
Class[Containerd]
Package[rsyslog-kubernetes]
Exec[apt_package_from_component_istio115]
File[/etc/rsyslog.d/10-kubernetes-node-filters.conf]
Class[Cpufrequtils]
Systemd::Unit[rsyslog-imfile-remedy.service]
Systemd::Unit[kubelet-container-runtime]
Exec[Generate cert mlserve__system_kube-proxy]
File[/etc/cfssl/ssl/mlserve__rsyslog]
File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.csr]
File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem]
Systemd::Override[amd-devplugin-after-labeller]
Exec[Generate cert mlserve__kubelet_server refresh on intermediate ca change]
Exec[Generate cert mlserve__calicoctl]
File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]
Class[K8s::Clusters]
File[/etc/kubernetes/kube-proxy-config.yaml]
File[/etc/kubernetes]
File[/etc/rsyslog.d/09-kubernetes.conf]
Exec[systemd daemon-reload for ferm.service (ferm-ferm-service-auto-restart)]
Package[dragonfly-dfdaemon]
Exec[create chained cert /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]
Class[Profile::Kubernetes::Container_runtime]
Rsyslog::Conf[kubernetes-node-filters]
Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh]
Package[crictl]
Apt::Repository[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]
Exec[Generate cert mlserve__calicoctl refresh]
Exec[renew certificate - mlserve__rsyslog]
Service[rsyslog-release-deleted-inotify-watches.timer]
File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem]
File[/etc/sysctl.d/75-kube_proxy_icmp.conf]
Exec[systemd daemon-reload for cpupower.service (cpupower)]
File[/var/log/rsyslog-release-deleted-inotify-watches]
File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.csr]
K8s::Package[kubelet]
File[/etc/kubernetes/pki/mlserve__kubelet_server-key.pem]
Systemd::Unit[rsyslog-release-deleted-inotify-watches.timer]
Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo]
Class[Role::Ml_k8s::Worker]
User[kube]
Systemd::Timer[rsyslog-release-deleted-inotify-watches]
K8s::Kubeconfig[/etc/cni/net.d/istio-kubeconfig]
File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem]
Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem]
File[/etc/kubernetes/kubelet.conf]
File[/etc/rsyslog.d/20-shellbox.conf]
Logrotate::Conf[rsyslog-release-deleted-inotify-watches]
Exec[apt_package_from_component_calico329]
Exec[Generate cert mlserve__istio-cni refresh]
File[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]
Exec[Generate cert mlserve__rsyslog]
Udev::Rule[kube_proxy_conntrack]
Sysctl::Parameters[ipv6-fowarding-accept-ra]
File[/etc/dragonfly/dfdaemon.yml]
Systemd::Timer[rsyslog-imfile-remedy]
Exec[Generate cert mlserve__amdgpu-node-labeller refresh]
File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem]
Docker::Credentials[/var/lib/kubelet/config.json]
File[/etc/kubernetes/pki/mlserve__calicoctl-key.pem]
Exec[Generate cert mlserve__calicoctl refresh on intermediate ca change]
File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem]
File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem]
File[/etc/kubernetes/pki/mlserve__kubelet_server.pem]
File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]
File[/etc/calico/calicoctl-kubeconfig]
File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]
Package[nerdctl]
File[/etc/systemd/system/amd-k8s-node-labeller.service.d]
Package[linux-image-6.16.3+deb13-amd64]
Apt::Package_from_bpo[linux-6.16-trixie]
File[/etc/kubernetes/kubelet-config.yaml]
Motd::Script[ml_k8s::worker]
Cfssl::Csr[/etc/cfssl/csr/mlserve__istio-cni.csr]
Rsyslog::Conf[kubernetes]
File[/etc/rsyslog.d/08-input-file-kubernetes-json.conf]
Cfssl::Cert[mlserve__calicoctl]
Exec[renew certificate - mlserve__system_kube-proxy]
File[/lib/systemd/system/rsyslog-imfile-remedy.service]
Cfssl::Cert[mlserve__amdgpu-node-labeller]
Class[Profile::Lvs::Realserver]
Exec[Generate cert mlserve__system_kube-proxy refresh on intermediate ca change]
File[/etc/containerd/config.toml]
File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem]
Cfssl::Cert[mlserve__istio-cni]
Concat_file[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
Service[rsyslog-imfile-remedy.timer]
Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
File[/etc/systemd/system/kube-proxy.service.d/puppet-override.conf]
Systemd::Timer::Job[rsyslog-release-deleted-inotify-watches]
File[/etc/cfssl/csr/mlserve__rsyslog.csr]
File[/etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem]
K8s::Kubeconfig[/etc/amd/node-labeller-kubeconfig]
Exec[renew certificate - mlserve__calico-cni]
File[/lib/systemd/system/cpupower.service]
File[/etc/udev/rules.d/70-kfd.rules]
Package[apparmor]
Exec[create chained cert /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]
Class[Containerd::Configuration]
Package[socat]
File[/etc/systemd/system/kubelet.service.d/container-runtime.conf]
File[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]
Cfssl::Csr[/etc/cfssl/csr/mlserve__kubelet_server.csr]
Systemd::Service[kube-proxy]
Class[K8s::Kubelet::Cni::Base]
Sysctl::Parameters[kube_proxy_icmp]
File[/etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf]
File[/lib/systemd/system/rsyslog-imfile-remedy.timer]
File[/etc/calico]
File[/etc/logrotate.d/rsyslog-release-deleted-inotify-watches]
Exec[renew certificate - mlserve__system_node_ml-serve1014_eqiad_wmnet]
File[/etc/cfssl/csr/mlserve__istio-cni.csr]
Apt::Repository[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
Package[istio-cni]
File[/etc/kubernetes/pki/mlserve__calicoctl.csr]
File[/etc/apparmor.d/abstractions]
File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]
File[/etc/sysctl.d/75-kube_proxy_conntrack.conf]
Cfssl::Csr[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]
File[/var/lib/kubelet]
Rsyslog::Conf[imfile]
Rsyslog::Input::File[kubernetes-json]
Systemd::Unit[amd-k8s-node-labeller-amd-devplugin-after-labeller]
File[/etc/update-motd.d/05-ml-k8s--worker]
Cfssl::Csr[/etc/cfssl/csr/mlserve__calico-cni.csr]
File[/etc/kubernetes/pki/mlserve__istio-cni.chained.pem]
File[/etc/kubernetes/pki/mlserve__kubelet_server.csr]
File[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]
File[/etc/cni/net.d/10-calico.conflist]
File[/etc/kubernetes/pki/mlserve__calico-cni.pem]
Exec[renew certificate - discovery__ml-serve1014_eqiad_wmnet]
Exec[create chained cert /etc/kubernetes/pki/mlserve__istio-cni.chain.pem]
Apt::Pin[apt_pin_linux-6.16-trixie_trixie-bpo]
File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem]
Exec[create chained cert /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]
File[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]
Systemd::Service[rsyslog-imfile-remedy]
Exec[apt_pin_apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]
File[/etc/kubernetes/pki/mlserve__istio-cni.pem]
Exec[Generate cert mlserve__rsyslog refresh on intermediate ca change]
Exec[renew certificate - mlserve__amdgpu-node-labeller]
Motd::Message[ml_k8s::worker]
File[/etc/kubernetes/pki/mlserve__calico-cni-key.pem]
Rsyslog::Conf[shellbox]
Concat[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]
Service[apparmor]
Cfssl::Csr[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]
Class[Profile::Lvs::Configuration]
Exec[create chained cert /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

Resources only in the old catalog

Motd::Script[ml_k8s::insetup_gpu]
Class[Role::Ml_k8s::Insetup_gpu]
Motd::Message[ml_k8s::insetup_gpu]
Node[__node_regexp__ml-serve10145.eqiad.]
File[/etc/update-motd.d/05-ml-k8s--insetup-gpu]

Resources modified

Exec[create chained cert /etc/kubernetes/pki/mlserve__calico-cni.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__calico-cni.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__calico-cni.chain.pem]

+    require   => Exec[Generate cert mlserve__calico-cni refresh on intermediate ca change]
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__calico-cni.pem /etc/kubernetes/pki/mlserve__calico-cni.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__calico-cni.chained.pem | sha512sum)"

+    subscribe => ['Exec[renew certificate - mlserve__calico-cni]', 'File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__calico-cni.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__calico-cni.pem /etc/kubernetes/pki/mlserve__calico-cni.chain.pem > /etc/kubernetes/pki/mlserve__calico-cni.chained.pem

File[/etc/systemd/system/kubelet.service.d]

Parameters differences:

--- File[/etc/systemd/system/kubelet.service.d].orig
+++ File[/etc/systemd/system/kubelet.service.d]

+    owner  => root
+    group  => root
+    mode   => 0555
+    ensure => directory

Exec[systemd daemon-reload for kube-proxy.service (kube-proxy)]

Parameters differences:

--- Exec[systemd daemon-reload for kube-proxy.service (kube-proxy)].orig
+++ Exec[systemd daemon-reload for kube-proxy.service (kube-proxy)]

+    notify      => ['Service[kube-proxy]']
+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

File[/etc/kubernetes/pki/mlserve__system_kube-proxy.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_kube-proxy.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__system_kube-proxy.csr]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet].orig
+++ Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]
+    notify      => ['Service[kubelet]']
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']

Package[linux-cpupower]

Parameters differences:

--- Package[linux-cpupower].orig
+++ Package[linux-cpupower]

+    ensure   => installed
+    provider => apt

Exec[apt_pin_apt_pin_linux-6.16-trixie_trixie-bpo]

Parameters differences:

--- Exec[apt_pin_apt_pin_linux-6.16-trixie_trixie-bpo].orig
+++ Exec[apt_pin_apt_pin_linux-6.16-trixie_trixie-bpo]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Class[Calico]

Parameters differences:

--- Class[Calico].orig
+++ Class[Calico]

+    auth_cert          => {'cert': '/etc/kubernetes/pki/mlserve__calicoctl.pem', 'key': '/etc/kubernetes/pki/mlserve__calicoctl-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__calicoctl.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__calicoctl.chained.pem'}
+    calicoctl_username => calicoctl
+    version            => 3.29
+    master_fqdn        => ml-ctrl.svc.eqiad.wmnet

Ferm::Service[dragonfly_dfget]

Parameters differences:

--- Ferm::Service[dragonfly_dfget].orig
+++ Ferm::Service[dragonfly_dfget]

+    proto    => tcp
+    notrack  => False
+    ensure   => present
+    prio     => 10
+    port     => 15001
+    src_sets => ['DOMAIN_NETWORKS']
+    desc     =>

Exec[Generate cert mlserve__istio-cni refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__istio-cni refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__istio-cni refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__istio-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__istio-cni

K8s::Kubelet::Cni[calico]

Parameters differences:

--- K8s::Kubelet::Cni[calico].orig
+++ K8s::Kubelet::Cni[calico]

+    require  => ['Class[K8s::Kubelet::Cni::Base]']
+    priority => 10
+    config   => {'name': 'k8s-pod-network', 'cniVersion': '0.3.1', 'plugins': [{'type': 'calico', 'log_level': 'info', 'datastore_type': 'kubernetes', 'mtu': 1460, 'ipam': {'type': 'calico-ipam', 'assign_ipv4': 'true', 'assign_ipv6': 'true'}, 'policy': {'type': 'k8s'}, 'kubernetes': {'kubeconfig': '/etc/cni/net.d/calico-kubeconfig'}}, {'name': 'istio-cni', 'type': 'istio-cni', 'log_level': 'info', 'kubernetes': {'kubeconfig': '/etc/cni/net.d/istio-kubeconfig', 'cni_bin_dir': '/opt/cni/bin', 'exclude_namespaces': ['istio-system', 'kube-system', 'knative-serving', 'cert-manager', 'kserve']}}]}

Systemd::Override[container-runtime]

Parameters differences:

--- Systemd::Override[container-runtime].orig
+++ Systemd::Override[container-runtime]

+    unit    => kubelet
+    ensure  => present
+    restart => True

Group[kube]

Parameters differences:

--- Group[kube].orig
+++ Group[kube]

+    system => True
+    ensure => present

Exec[renew certificate - mlserve__kubelet_server]

Parameters differences:

--- Exec[renew certificate - mlserve__kubelet_server].orig
+++ Exec[renew certificate - mlserve__kubelet_server]

+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve -profile server /etc/kubernetes/pki/mlserve__kubelet_server.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__kubelet_server

+    require     => Exec[Generate cert mlserve__kubelet_server]
+    notify      => ['Service[kubelet]']
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__kubelet_server.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/ferm/conf.d/10_calico-bird]

Parameters differences:

--- File[/etc/ferm/conf.d/10_calico-bird].orig
+++ File[/etc/ferm/conf.d/10_calico-bird]

+    group   => root
+    mode    => 0400
+    ensure  => present
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    owner   => root
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_calico-bird.orig
+++ /etc/ferm/conf.d/10_calico-bird
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 179, ($NETWORK_INFRA 10.64.155.1));
+
+

File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => root

Exec[renew certificate - mlserve__calicoctl]

Parameters differences:

--- Exec[renew certificate - mlserve__calicoctl].orig
+++ Exec[renew certificate - mlserve__calicoctl]

+    require     => Exec[Generate cert mlserve__calicoctl]
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__calicoctl.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__calicoctl.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calicoctl

Systemd::Unit[kube-proxy]

Parameters differences:

--- Systemd::Unit[kube-proxy].orig
+++ Systemd::Unit[kube-proxy]

+    override          => True
+    ensure            => present
+    override_filename => puppet-override.conf
+    restart           => True
+    require           => ['Class[Systemd]']
+    unit              => kube-proxy

Exec[Generate cert mlserve__calicoctl]

Parameters differences:

--- Exec[Generate cert mlserve__calicoctl].orig
+++ Exec[Generate cert mlserve__calicoctl]

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__calicoctl.csr]
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__calicoctl.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__calicoctl-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calicoctl.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calicoctl

Exec[systemd daemon-reload for cpupower.service (cpupower)]

Parameters differences:

--- Exec[systemd daemon-reload for cpupower.service (cpupower)].orig
+++ Exec[systemd daemon-reload for cpupower.service (cpupower)]

+    notify      => ['Service[cpupower]']
+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

User[kube]

Parameters differences:

--- User[kube].orig
+++ User[kube]

+    shell  => /usr/sbin/nologin
+    system => True
+    home   => /nonexistent
+    ensure => present
+    gid    => kube

File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => root

File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => root

File[/etc/kubernetes/kubelet-config.yaml]

Parameters differences:

--- File[/etc/kubernetes/kubelet-config.yaml].orig
+++ File[/etc/kubernetes/kubelet-config.yaml]

+    group   => kube
+    mode    => 0400
+    ensure  => file
+    require => K8s::Package[kubelet]
+    owner   => kube
+    notify  => Service[kubelet]

Content differences:

--- /etc/kubernetes/kubelet-config.yaml.orig
+++ /etc/kubernetes/kubelet-config.yaml
@@ -0,0 +1,25 @@
+---
+apiVersion: kubelet.config.k8s.io/v1beta1
+kind: KubeletConfiguration
+tlsPrivateKeyFile: "/etc/kubernetes/pki/mlserve__kubelet_server-key.pem"
+tlsCertFile: "/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem"
+clusterDomain: cluster.local
+clusterDNS:
+- 10.67.0.3
+authentication:
+  anonymous:
+    enabled: false
+  webhook:
+    enabled: true
+  x509:
+    clientCAFile: "/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem"
+authorization:
+  mode: Webhook
+cgroupDriver: systemd
+evictionHard:
+  imagefs.available: 15%
+  memory.available: 300M
+  nodefs.available: 10%
+  nodefs.inodesFree: 5%
+containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
+seccompDefault: true

Systemd::Timer::Job[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Systemd::Timer::Job[rsyslog-release-deleted-inotify-watches].orig
+++ Systemd::Timer::Job[rsyslog-release-deleted-inotify-watches]

+    user                      => root
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    send_mail_only_on_error   => True
+    environment               => {}
+    ignore_errors             => False
+    send_mail_to              => root@ml-serve1014.eqiad.wmnet
+    syslog_match_startswith   => True
+    command                   => /usr/local/sbin/rsyslog-release-deleted-inotify-watches
+    logfile_basedir           => /var/log
+    logging_enabled           => True
+    logfile_group             => root
+    logfile_perms             => all
+    syslog_force_stop         => True
+    success_exit_status       => []
+    ensure                    => absent
+    fixed_random_delay        => False
+    private_tmp               => False
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* *:7:00'}
+    send_mail                 => False
+    monitoring_enabled        => False
+    description               => Restart rsyslog to release inotify watches of deleted container logs
+    logfile_name              => syslog.log
+    monitoring_contact_groups => admins

Package[socat]

Parameters differences:

--- Package[socat].orig
+++ Package[socat]

+    ensure   => installed
+    provider => apt

File[/etc/systemd/system/kubelet.service.d/container-runtime.conf]

Parameters differences:

--- File[/etc/systemd/system/kubelet.service.d/container-runtime.conf].orig
+++ File[/etc/systemd/system/kubelet.service.d/container-runtime.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for kubelet.service (kubelet-container-runtime)]

Content differences:

--- /etc/systemd/system/kubelet.service.d/container-runtime.conf.orig
+++ /etc/systemd/system/kubelet.service.d/container-runtime.conf
@@ -0,0 +1,3 @@
+[Unit]
+After=containerd.service
+Requires=containerd.service

Apt::Repository[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Apt::Repository[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Apt::Repository[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    allow_releaseinfo_change => False
+    source                   => True
+    ensure                   => present
+    trust_repo               => False
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    uri                      => http://apt.wikimedia.org/wikimedia
+    dist                     => trixie-wikimedia
+    components               => component/istio115
+    bin                      => True

File[/etc/kubernetes/pki/mlserve__calicoctl.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calicoctl.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__calicoctl.csr]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Rsyslog::Input::File[kubernetes-json]

Parameters differences:

--- Rsyslog::Input::File[kubernetes-json].orig
+++ Rsyslog::Input::File[kubernetes-json]

+    ensure             => present
+    syslog_tag         => kubernetes
+    syslog_tag_prefix  => input-file
+    addmetadata        => on
+    reopen_on_truncate => on
+    priority           => 8
+    path               => /var/log/containers/*.log
+    addceetag          => on

Exec[renew certificate - discovery__ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Exec[renew certificate - discovery__ml-serve1014_eqiad_wmnet].orig
+++ Exec[renew certificate - discovery__ml-serve1014_eqiad_wmnet]

+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label discovery  /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet

+    require     => Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet]
+    notify      => ['Service[dragonfly-dfdaemon]']
+    unless      => /usr/bin/openssl x509 -in /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => root
+    show_diff => False
+    backup    => False

Exec[renew certificate - mlserve__amdgpu-node-labeller]

Parameters differences:

--- Exec[renew certificate - mlserve__amdgpu-node-labeller].orig
+++ Exec[renew certificate - mlserve__amdgpu-node-labeller]

+    require     => Exec[Generate cert mlserve__amdgpu-node-labeller]
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__amdgpu-node-labeller

Service[apparmor]

Parameters differences:

--- Service[apparmor].orig
+++ Service[apparmor]

+    require    => Package[apparmor]
+    hasstatus  => True
+    ensure     => running
+    hasrestart => True

Rsyslog::Conf[shellbox]

Parameters differences:

--- Rsyslog::Conf[shellbox].orig
+++ Rsyslog::Conf[shellbox]

+    mode     => 0444
+    priority => 20
+    source   => puppet:///modules/profile/rsyslog/shellbox.rsyslog.conf
+    ensure   => present

Cfssl::Csr[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

+    ensure      => present
+    common_name => system:node:ml-serve1014.eqiad.wmnet
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => [{'organisation': 'system:nodes'}]

Package[containerd]

Parameters differences:

--- Package[containerd].orig
+++ Package[containerd]

+    ensure   => installed
+    provider => apt

Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)].orig
+++ Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Service[kube-proxy]

Parameters differences:

--- Service[kube-proxy].orig
+++ Service[kube-proxy]

+    enable => True
+    ensure => running

Apt::Package_from_bpo[firmware-amd-graphics-trixie-bpo]

Parameters differences:

--- Apt::Package_from_bpo[firmware-amd-graphics-trixie-bpo].orig
+++ Apt::Package_from_bpo[firmware-amd-graphics-trixie-bpo]

+    priority        => 1001
+    ensure_packages => True
+    distro          => trixie
+    packages        => {'firmware-amd-graphics': '20251021-1~bpo13+1'}

File[/etc/modprobe.d/blacklist-wmf_overlay.conf]

Parameters differences:

--- File[/etc/modprobe.d/blacklist-wmf_overlay.conf].orig
+++ File[/etc/modprobe.d/blacklist-wmf_overlay.conf]

@@
-    ensure => present
+    ensure => absent

Content differences:

--- /etc/modprobe.d/blacklist-wmf_overlay.conf.orig
+++ /etc/modprobe.d/blacklist-wmf_overlay.conf
@@ -1,7 +1,3 @@
 # wmf_overlay - blacklisted kernel modules
 # This file is managed by Puppet
 #
-blacklist overlay
-install overlay /bin/true
-blacklist overlayfs
-install overlayfs /bin/true

File[/etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf]

Parameters differences:

--- File[/etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf].orig
+++ File[/etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for amd-k8s-node-labeller.service (amd-k8s-node-labeller-amd-devplugin-after-labeller)]

Content differences:

--- /etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf.orig
+++ /etc/systemd/system/amd-k8s-node-labeller.service.d/amd-devplugin-after-labeller.conf
@@ -0,0 +1,3 @@
+[Unit]
+After=amd-k8s-device-plugin.service
+Requires=amd-k8s-device-plugin.service

K8s::Kubeconfig[/etc/kubernetes/kubelet.conf]

Parameters differences:

--- K8s::Kubeconfig[/etc/kubernetes/kubelet.conf].orig
+++ K8s::Kubeconfig[/etc/kubernetes/kubelet.conf]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem', 'key': '/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem'}
+    group       => kube
+    mode        => 0400
+    ensure      => present
+    owner       => kube
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['Class[K8s::Base_dirs]']
+    username    => default-auth

Rsyslog::Conf[input-file-kubernetes-json]

Parameters differences:

--- Rsyslog::Conf[input-file-kubernetes-json].orig
+++ Rsyslog::Conf[input-file-kubernetes-json]

+    require  => Rsyslog::Conf[imfile]
+    priority => 8
+    mode     => 0444
+    ensure   => present

File[/etc/ferm/conf.d/10_kubelet-http]

Parameters differences:

--- File[/etc/ferm/conf.d/10_kubelet-http].orig
+++ File[/etc/ferm/conf.d/10_kubelet-http]

+    group   => root
+    mode    => 0400
+    ensure  => present
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    owner   => root
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_kubelet-http.orig
+++ /etc/ferm/conf.d/10_kubelet-http
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 10250, (@resolve((ml-serve-ctrl1001.eqiad.wmnet ml-serve-ctrl1002.eqiad.wmnet)) @resolve((ml-serve-ctrl1001.eqiad.wmnet ml-serve-ctrl1002.eqiad.wmnet), AAAA)));
+
+

Sysctl::Parameters[increase_inotify_limits]

Parameters differences:

--- Sysctl::Parameters[increase_inotify_limits].orig
+++ Sysctl::Parameters[increase_inotify_limits]

+    priority => 70
+    values   => {'fs.inotify.max_user_watches': 32768, 'fs.inotify.max_user_instances': 512}
+    ensure   => present

Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01

File[/etc/dragonfly/dfget.yml]

Parameters differences:

--- File[/etc/dragonfly/dfget.yml].orig
+++ File[/etc/dragonfly/dfget.yml]

+    group  => root
+    mode   => 0644
+    ensure => file
+    owner  => root
+    notify => Service[dragonfly-dfdaemon]

Content differences:

--- /etc/dragonfly/dfget.yml.orig
+++ /etc/dragonfly/dfget.yml
@@ -0,0 +1,5 @@
+# List of supernodes in the format
+# host:port(default:8002)=weight(default:1)
+# FIXME: Figure out how weight is exactly handled, could we use multiple supernodes without split brain?
+nodes:
+  - dragonfly-supernode1001.eqiad.wmnet:8002=1

Service[dragonfly-dfdaemon]

Parameters differences:

--- Service[dragonfly-dfdaemon].orig
+++ Service[dragonfly-dfdaemon]

+    ensure => running

Systemd::Unit[ferm-ferm-service-auto-restart]

Parameters differences:

--- Systemd::Unit[ferm-ferm-service-auto-restart].orig
+++ Systemd::Unit[ferm-ferm-service-auto-restart]

+    source            => puppet:///modules/profile/kubernetes/node/ferm_systemd_override
+    override          => True
+    override_filename => ferm-service-auto-restart
+    ensure            => present
+    restart           => False
+    require           => ['Class[Systemd]']
+    unit              => ferm

Cfssl::Cert[mlserve__system_node_ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Cfssl::Cert[mlserve__system_node_ml-serve1014_eqiad_wmnet].orig
+++ Cfssl::Cert[mlserve__system_node_ml-serve1014_eqiad_wmnet]

+    outdir          => /etc/kubernetes/pki
+    names           => [{'organisation': 'system:nodes'}]
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => kube
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => system:node:ml-serve1014.eqiad.wmnet
+    hosts           => []
+    notify_services => ['kubelet']

Apt::Package_from_component[istio115]

Parameters differences:

--- Apt::Package_from_component[istio115].orig
+++ Apt::Package_from_component[istio115]

+    component       => component/istio115
+    priority        => 1001
+    ensure_packages => True
+    ensure          => present
+    packages        => {'istio-cni': 'present'}
+    uri             => http://apt.wikimedia.org/wikimedia
+    distro          => trixie-wikimedia

Systemd::Service[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Systemd::Service[rsyslog-release-deleted-inotify-watches].orig
+++ Systemd::Service[rsyslog-release-deleted-inotify-watches]

+    override                 => False
+    ensure                   => absent
+    restart                  => False
+    monitoring_critical      => False
+    unit_type                => timer
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_enabled       => False
+    migration_task           => T407130
+    require                  => Systemd::Unit[rsyslog-release-deleted-inotify-watches.service]

Cfssl::Csr[/etc/cfssl/csr/mlserve__rsyslog.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__rsyslog.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__rsyslog.csr]

+    ensure      => present
+    common_name => rsyslog
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => [{'organisation': 'view'}]

Package[calico-cni]

Parameters differences:

--- Package[calico-cni].orig
+++ Package[calico-cni]

+    ensure   => >=3.29 <3.30
+    provider => apt

Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01
+    tag    => _etc_apt_sources.list.d_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01

Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10
+    tag    => _etc_apt_sources.list.d_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

Content differences:

--- component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.orig
+++ component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: trixie-wikimedia
+Components: component/calico329
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Exec[Generate cert mlserve__amdgpu-node-labeller refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__amdgpu-node-labeller refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__amdgpu-node-labeller refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__amdgpu-node-labeller

File[/etc/kubernetes/pki/mlserve__calico-cni.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calico-cni.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calico-cni.chained.pem]

+    owner   => root
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__calico-cni.chain.pem]

Ferm::Service[calico-bird]

Parameters differences:

--- Ferm::Service[calico-bird].orig
+++ Ferm::Service[calico-bird]

+    srange  => ($NETWORK_INFRA 10.64.155.1)
+    proto   => tcp
+    notrack => False
+    ensure  => present
+    prio    => 10
+    port    => 179
+    desc    =>

Exec[Generate cert mlserve__calico-cni refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__calico-cni refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__calico-cni refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calico-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calico-cni

Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10
+    tag    => _etc_apt_sources.list.d_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

Content differences:

--- component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.orig
+++ component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: trixie-wikimedia
+Components: component/kubernetes131
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Class[Lvs::Realserver]

Parameters differences:

--- Class[Lvs::Realserver].orig
+++ Class[Lvs::Realserver]

+    realserver_ips => ['10.2.2.63', '10.2.2.84']

File[/usr/libexec/cpupower]

Parameters differences:

--- File[/usr/libexec/cpupower].orig
+++ File[/usr/libexec/cpupower]

+    mode   => 0555
+    group  => root
+    source => puppet:///modules/cpufrequtils/cpupower.sh
+    ensure => present
+    owner  => root

File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]

Parameters differences:

--- File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem].orig
+++ File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/discovery-cert.pem
+    owner  => dragonfly

File[/etc/kubernetes]

Parameters differences:

--- File[/etc/kubernetes].orig
+++ File[/etc/kubernetes]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

K8s::Package[kubelet]

Parameters differences:

--- K8s::Package[kubelet].orig
+++ K8s::Package[kubelet]

+    priority        => 1001
+    ensure_packages => True
+    version         => 1.31
+    require         => ['Class[K8s::Base_dirs]']
+    uri             => http://apt.wikimedia.org/wikimedia
+    package         => node
+    distro          => trixie-wikimedia

Systemd::Unit[rsyslog-release-deleted-inotify-watches.timer]

Parameters differences:

--- Systemd::Unit[rsyslog-release-deleted-inotify-watches.timer].orig
+++ Systemd::Unit[rsyslog-release-deleted-inotify-watches.timer]

+    override          => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    restart           => False
+    require           => ['Class[Systemd]']
+    unit              => rsyslog-release-deleted-inotify-watches.timer

Systemd::Timer[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Systemd::Timer[rsyslog-release-deleted-inotify-watches].orig
+++ Systemd::Timer[rsyslog-release-deleted-inotify-watches]

+    splay              => 0
+    ensure             => absent
+    unit_name          => rsyslog-release-deleted-inotify-watches.service
+    accuracy           => 15sec
+    fixed_random_delay => False
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* *:7:00'}]

K8s::Kubeconfig[/etc/cni/net.d/istio-kubeconfig]

Parameters differences:

--- K8s::Kubeconfig[/etc/cni/net.d/istio-kubeconfig].orig
+++ K8s::Kubeconfig[/etc/cni/net.d/istio-kubeconfig]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__istio-cni.pem', 'key': '/etc/kubernetes/pki/mlserve__istio-cni-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__istio-cni.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__istio-cni.chained.pem'}
+    group       => root
+    mode        => 0400
+    ensure      => present
+    owner       => root
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['File[/etc/cni/net.d]', 'Class[K8s::Base_dirs]']
+    username    => istio-cni

File[/etc/update-motd.d/05-ml-k8s--insetup-gpu]

Parameters differences:

--- File[/etc/update-motd.d/05-ml-k8s--insetup-gpu].orig
+++ File[/etc/update-motd.d/05-ml-k8s--insetup-gpu]

-    owner  => root
-    group  => root
-    mode   => 0555
-    ensure => present

Content differences:

--- /etc/update-motd.d/05-ml-k8s--insetup-gpu.orig
+++ /etc/update-motd.d/05-ml-k8s--insetup-gpu
@@ -1,2 +0,0 @@
-#!/bin/sh
-printf "%s\n" "ml-serve1014 is a Machine Learning GPU host in setup. (ml_k8s::insetup_gpu)"

File[/etc/systemd/system/amd-k8s-node-labeller.service.d]

Parameters differences:

--- File[/etc/systemd/system/amd-k8s-node-labeller.service.d].orig
+++ File[/etc/systemd/system/amd-k8s-node-labeller.service.d]

+    owner  => root
+    group  => root
+    mode   => 0555
+    ensure => directory

Class[Profile::Base::Production]

Parameters differences:

--- Class[Profile::Base::Production].orig
+++ Class[Profile::Base::Production]

@@
-    role_description => Machine Learning GPU host in setup.
+    role_description => ML Kubernetes worker node

Cfssl::Cert[mlserve__istio-cni]

Parameters differences:

--- Cfssl::Cert[mlserve__istio-cni].orig
+++ Cfssl::Cert[mlserve__istio-cni]

+    outdir          => /etc/kubernetes/pki
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => root
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => istio-cni
+    hosts           => []
+    notify_services => []

Concat_file[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    tag            => _etc_apt_sources.list.d_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    owner          => root
+    backup         => puppet
+    format         => plain
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    show_diff      => True
+    force          => False

File[/etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => kube
+    show_diff => False
+    backup    => False

Class[Base::Kernel]

Parameters differences:

--- Class[Base::Kernel].orig
+++ Class[Base::Kernel]

@@
-    overlayfs => False
+    overlayfs => True

File[/lib/systemd/system/cpupower.service]

Parameters differences:

--- File[/lib/systemd/system/cpupower.service].orig
+++ File[/lib/systemd/system/cpupower.service]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for cpupower.service (cpupower)]

Content differences:

--- /lib/systemd/system/cpupower.service.orig
+++ /lib/systemd/system/cpupower.service
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: Apache-2.0
+# Adapted from:
+# https://sources.debian.org/src/linux/6.16.3-1/tools/power/cpupower/cpupower.service.in
+
+[Unit]
+Description=Apply cpupower configuration
+
+[Service]
+Type=oneshot
+EnvironmentFile=-/etc/default/cpupower
+ExecStart=/usr/libexec/cpupower
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target

Class[Containerd::Configuration]

Parameters differences:

--- Class[Containerd::Configuration].orig
+++ Class[Containerd::Configuration]

+    sandbox_image     => docker-registry.discovery.wmnet/pause:3.6-1
+    registry_username => kubernetes
+    ensure            => present
+    dragonfly_enabled => True

File[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr].orig
+++ File[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr.orig
+++ /etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr
@@ -0,0 +1,13 @@
+{
+  "CN": "amdgpu-node-labeller",
+  "hosts": [
+    "amdgpu-node-labeller"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

Cfssl::Csr[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]

+    ensure      => present
+    common_name => amdgpu-node-labeller
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => []

File[/var/lib/kubelet]

Parameters differences:

--- File[/var/lib/kubelet].orig
+++ File[/var/lib/kubelet]

+    owner  => root
+    group  => root
+    mode   => 0700
+    ensure => directory

File[/var/lib/prometheus/node.d/role_owner.prom]

Content differences:

--- /var/lib/prometheus/node.d/role_owner.prom.orig
+++ /var/lib/prometheus/node.d/role_owner.prom
@@ -1,3 +1,3 @@
 # HELP role_owner The team owner of the server role
 # TYPE role_owner gauge
-role_owner{team="machine-learning",role="ml_k8s::insetup_gpu",cluster="ml_serve"} 1.0
+role_owner{team="machine-learning",role="ml_k8s::worker",cluster="ml_serve"} 1.0

Systemd::Service[rsyslog-imfile-remedy]

Parameters differences:

--- Systemd::Service[rsyslog-imfile-remedy].orig
+++ Systemd::Service[rsyslog-imfile-remedy]

+    override                 => False
+    ensure                   => present
+    restart                  => False
+    monitoring_critical      => False
+    unit_type                => timer
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_enabled       => False
+    migration_task           => T407130
+    require                  => Systemd::Unit[rsyslog-imfile-remedy.service]

Service[containerd]

Parameters differences:

--- Service[containerd].orig
+++ Service[containerd]

+    ensure => running

Sysctl::Conffile[kube_proxy_conntrack]

Parameters differences:

--- Sysctl::Conffile[kube_proxy_conntrack].orig
+++ Sysctl::Conffile[kube_proxy_conntrack]

+    priority => 75
+    ensure   => present

Exec[/sbin/modprobe overlay]

Parameters differences:

--- Exec[/sbin/modprobe overlay].orig
+++ Exec[/sbin/modprobe overlay]

+    unless      => /bin/lsmod | /bin/grep -q '^overlay '
+    refreshonly => True

File[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__system_kube-proxy.csr].orig
+++ File[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__system_kube-proxy.csr.orig
+++ /etc/cfssl/csr/mlserve__system_kube-proxy.csr
@@ -0,0 +1,19 @@
+{
+  "CN": "system:kube-proxy",
+  "hosts": [
+    "system:kube-proxy"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+    {
+      "C": null,
+      "L": null,
+      "O": "system:node-proxier",
+      "OU": null,
+      "S": null
+    }
+  ]
+}

File[/etc/cfssl/csr/mlserve__kubelet_server.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__kubelet_server.csr].orig
+++ File[/etc/cfssl/csr/mlserve__kubelet_server.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__kubelet_server.csr.orig
+++ /etc/cfssl/csr/mlserve__kubelet_server.csr
@@ -0,0 +1,17 @@
+{
+  "CN": "kubelet",
+  "hosts": [
+    "ml-serve1014",
+    "ml-serve1014.eqiad.wmnet",
+    "10.64.155.8",
+    "2620:0:861:123:10:64:155:8",
+    "kubelet"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

Cfssl::Csr[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]

+    ensure      => present
+    common_name => ml-serve1014.eqiad.wmnet
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => ['ml-serve1014', 'ml-serve1014.eqiad.wmnet', 'docker-registry.discovery.wmnet', '127.0.0.1', '::1', 'localhost']
+    names       => []

Systemd::Timer::Job[rsyslog-imfile-remedy]

Parameters differences:

--- Systemd::Timer::Job[rsyslog-imfile-remedy].orig
+++ Systemd::Timer::Job[rsyslog-imfile-remedy]

+    user                      => root
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    send_mail_only_on_error   => True
+    environment               => {}
+    ignore_errors             => False
+    send_mail_to              => root@ml-serve1014.eqiad.wmnet
+    syslog_match_startswith   => True
+    command                   => /usr/bin/systemctl try-restart rsyslog
+    logfile_basedir           => /var/log
+    logging_enabled           => False
+    logfile_group             => root
+    logfile_perms             => all
+    syslog_force_stop         => True
+    success_exit_status       => []
+    ensure                    => present
+    fixed_random_delay        => False
+    private_tmp               => False
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* 00/3:8:00'}
+    splay                     => 30
+    send_mail                 => False
+    monitoring_enabled        => False
+    description               => Restart rsyslog T357616
+    logfile_name              => syslog.log
+    monitoring_contact_groups => admins

File[/etc/kubernetes/proxy.conf]

Parameters differences:

--- File[/etc/kubernetes/proxy.conf].orig
+++ File[/etc/kubernetes/proxy.conf]

+    owner  => kube
+    group  => kube
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/kubernetes/proxy.conf.orig
+++ /etc/kubernetes/proxy.conf
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: default-proxy
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: default-proxy
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__system_kube-proxy.pem
+    client-key: /etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem

Package[amd-k8s-node-labeller]

Parameters differences:

--- Package[amd-k8s-node-labeller].orig
+++ Package[amd-k8s-node-labeller]

+    require  => K8s::Kubeconfig[/etc/amd/node-labeller-kubeconfig]
+    ensure   => present
+    provider => apt

File[/etc/systemd/system/ferm.service.d/ferm-service-auto-restart.conf]

Parameters differences:

--- File[/etc/systemd/system/ferm.service.d/ferm-service-auto-restart.conf].orig
+++ File[/etc/systemd/system/ferm.service.d/ferm-service-auto-restart.conf]

+    mode   => 0444
+    group  => root
+    source => puppet:///modules/profile/kubernetes/node/ferm_systemd_override
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for ferm.service (ferm-ferm-service-auto-restart)]

Class[K8s::Kubelet]

Parameters differences:

--- Class[K8s::Kubelet].orig
+++ Class[K8s::Kubelet]

+    v_log_level                     => 0
+    cluster_domain                  => cluster.local
+    cni_bin_dir                     => /opt/cni/bin
+    kubelet_cert                    => {'cert': '/etc/kubernetes/pki/mlserve__kubelet_server.pem', 'key': '/etc/kubernetes/pki/mlserve__kubelet_server-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem'}
+    cni_conf_dir                    => /etc/cni/net.d
+    cluster_dns                     => ['10.67.0.3']
+    node_taints                     => []
+    ipv6dualstack                   => False
+    version                         => 1.31
+    system_reserved                 => {'cpu': '45.3', 'memory': '50.83Gi'}
+    pod_infra_container_image       => docker-registry.discovery.wmnet/pause:3.6-1
+    node_labels                     => ['topology.kubernetes.io/region=eqiad', 'topology.kubernetes.io/zone=row-e11', 'node.kubernetes.io/disk-type=ssd']
+    docker_kubernetes_user_password => somepassword2
+    kubeconfig                      => /etc/kubernetes/kubelet.conf

Exec[create chained cert /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]

+    subscribe => ['Exec[renew certificate - mlserve__system_node_ml-serve1014_eqiad_wmnet]', 'File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem > /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem
+    require   => Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh on intermediate ca change]
+    notify    => ['Service[kubelet]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem | sha512sum)"

Apt::Package_from_component[calico329]

Parameters differences:

--- Apt::Package_from_component[calico329].orig
+++ Apt::Package_from_component[calico329]

+    component       => component/calico329
+    priority        => 1001
+    ensure_packages => True
+    ensure          => present
+    packages        => {'calicoctl': '>=3.29 <3.30', 'calico-cni': '>=3.29 <3.30'}
+    uri             => http://apt.wikimedia.org/wikimedia
+    distro          => trixie-wikimedia

Exec[apt_repository_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Exec[apt_repository_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Exec[apt_repository_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Kmod::Blacklist[wmf_overlay]

Parameters differences:

--- Kmod::Blacklist[wmf_overlay].orig
+++ Kmod::Blacklist[wmf_overlay]

@@
-    modules => ['overlayfs', 'overlay']
+    modules => []
@@
-    ensure  => present
+    ensure  => absent

Class[Base::Sysctl::Inotify]

Parameters differences:

--- Class[Base::Sysctl::Inotify].orig
+++ Class[Base::Sysctl::Inotify]

+    max_user_watches   => 32768
+    max_user_instances => 512

Sysctl::Conffile[ipv6-fowarding-accept-ra]

Parameters differences:

--- Sysctl::Conffile[ipv6-fowarding-accept-ra].orig
+++ Sysctl::Conffile[ipv6-fowarding-accept-ra]

+    priority => 70
+    ensure   => present

Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet

+    notify      => ['Service[kubelet]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Class[Dragonfly::Dfdaemon]

Parameters differences:

--- Class[Dragonfly::Dfdaemon].orig
+++ Class[Dragonfly::Dfdaemon]

+    ratelimit            => 100M
+    docker_registry_fqdn => docker-registry.discovery.wmnet
+    ensure               => present
+    supernodes           => ['dragonfly-supernode1001.eqiad.wmnet:8002=1']
+    proxy_urls_regex     => ['wikimedia/machinelearning-liftwing.*/blobs/sha256.*', 'amd-pytorch.*/blobs/sha256.*']
+    dfdaemon_ssl_cert    => /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem
+    dfdaemon_ssl_key     => /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem

File[/etc/cni/net.d]

Parameters differences:

--- File[/etc/cni/net.d].orig
+++ File[/etc/cni/net.d]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem]

Parameters differences:

--- File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem].orig
+++ File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem]

+    owner   => dragonfly
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]

File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer]

Parameters differences:

--- File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer].orig
+++ File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer]

+    group  => root
+    mode   => 0444
+    ensure => absent
+    owner  => root
+    notify => Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)]

Content differences:

--- /lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer.orig
+++ /lib/systemd/system/rsyslog-release-deleted-inotify-watches.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of rsyslog-release-deleted-inotify-watches.service
+
+[Timer]
+Unit=rsyslog-release-deleted-inotify-watches.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* *:7:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Unit[rsyslog-imfile-remedy.service]

Parameters differences:

--- Systemd::Unit[rsyslog-imfile-remedy.service].orig
+++ Systemd::Unit[rsyslog-imfile-remedy.service]

+    override          => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    restart           => False
+    require           => ['Class[Systemd]']
+    unit              => rsyslog-imfile-remedy.service

Package[firmware-amd-graphics]

Parameters differences:

--- Package[firmware-amd-graphics].orig
+++ Package[firmware-amd-graphics]

@@
-    ensure => installed
+    ensure => 20251021-1~bpo13+1

Exec[create chained cert /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem].orig
+++ Exec[create chained cert /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]

+    subscribe => ['Exec[renew certificate - mlserve__rsyslog]', 'File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]', 'File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem]']
+    command   => /bin/cat /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem > /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem
+    require   => Exec[Generate cert mlserve__rsyslog refresh on intermediate ca change]
+    notify    => ['Service[rsyslog]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem | sha512sum)" == "$(/bin/cat /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem | sha512sum)"

Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh]

Parameters differences:

--- Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh].orig
+++ Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh]

+    subscribe   => File[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label discovery  /etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet

+    notify      => ['Service[dragonfly-dfdaemon]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Exec[renew certificate - mlserve__rsyslog]

Parameters differences:

--- Exec[renew certificate - mlserve__rsyslog].orig
+++ Exec[renew certificate - mlserve__rsyslog]

+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog

+    require     => Exec[Generate cert mlserve__rsyslog]
+    notify      => ['Service[rsyslog]']
+    unless      => /usr/bin/openssl x509 -in /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']

Service[rsyslog-release-deleted-inotify-watches.timer]

Parameters differences:

--- Service[rsyslog-release-deleted-inotify-watches.timer].orig
+++ Service[rsyslog-release-deleted-inotify-watches.timer]

+    enable   => False
+    ensure   => stopped
+    provider => systemd
+    before   => ['Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.timer (rsyslog-release-deleted-inotify-watches.timer)]']

Nrpe::Monitor_service[disk_space]

Parameters differences:

--- Nrpe::Monitor_service[disk_space].orig
+++ Nrpe::Monitor_service[disk_space]

@@
-    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(containerd|kubelet)/*' --exclude-type=tracefs

File[/etc/kubernetes/pki/mlserve__kubelet_server-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__kubelet_server-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__kubelet_server-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => kube
+    show_diff => False
+    backup    => False

Concat::Fragment[main contacts]

Docker::Credentials[/var/lib/kubelet/config.json]

Parameters differences:

--- Docker::Credentials[/var/lib/kubelet/config.json].orig
+++ Docker::Credentials[/var/lib/kubelet/config.json]

+    registry_password => somepassword2
+    registry_username => kubernetes
+    group             => root
+    registry          => docker-registry.discovery.wmnet
+    owner             => root
+    allow_group       => True

Package[nerdctl]

Parameters differences:

--- Package[nerdctl].orig
+++ Package[nerdctl]

+    ensure   => installed
+    provider => apt

Apt::Package_from_bpo[linux-6.16-trixie]

Parameters differences:

--- Apt::Package_from_bpo[linux-6.16-trixie].orig
+++ Apt::Package_from_bpo[linux-6.16-trixie]

+    priority        => 1001
+    ensure_packages => True
+    distro          => trixie
+    packages        => {'linux-image-6.16.3+deb13-amd64': 'present'}

Exec[Generate cert mlserve__system_kube-proxy refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__system_kube-proxy refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__system_kube-proxy refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_kube-proxy.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_kube-proxy

+    notify      => ['Service[kube-proxy]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Exec[create chained cert /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]

+    require   => Exec[Generate cert mlserve__amdgpu-node-labeller refresh on intermediate ca change]
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem | sha512sum)"

+    subscribe => ['Exec[renew certificate - mlserve__amdgpu-node-labeller]', 'File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem > /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem

File[/etc/apparmor.d/abstractions]

Parameters differences:

--- File[/etc/apparmor.d/abstractions].orig
+++ File[/etc/apparmor.d/abstractions]

+    group   => root
+    mode    => 0755
+    ensure  => directory
+    require => Package[apparmor]
+    owner   => root

Motd::Message[ml_k8s::insetup_gpu]

Parameters differences:

--- Motd::Message[ml_k8s::insetup_gpu].orig
+++ Motd::Message[ml_k8s::insetup_gpu]

-    message  => ml-serve1014 is a Machine Learning GPU host in setup. (ml_k8s::insetup_gpu)
-    priority => 5
-    ensure   => present

Cfssl::Csr[/etc/cfssl/csr/mlserve__calico-cni.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__calico-cni.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__calico-cni.csr]

+    ensure      => present
+    common_name => calico-cni
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => []

File[/etc/kubernetes/pki/mlserve__calico-cni.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calico-cni.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calico-cni.pem]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

File[/etc/kubernetes/pki/mlserve__istio-cni.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__istio-cni.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__istio-cni.pem]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh]

Parameters differences:

--- Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh].orig
+++ Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet

+    notify      => ['Service[kubelet]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Systemd::Unit[cpupower]

Parameters differences:

--- Systemd::Unit[cpupower].orig
+++ Systemd::Unit[cpupower]

+    override          => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    restart           => True
+    require           => ['Class[Systemd]']
+    unit              => cpupower

File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chained.pem]

+    owner   => root
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]

File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

Exec[Generate cert mlserve__istio-cni]

Parameters differences:

--- Exec[Generate cert mlserve__istio-cni].orig
+++ Exec[Generate cert mlserve__istio-cni]

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__istio-cni.csr]
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__istio-cni.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__istio-cni-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__istio-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__istio-cni

Exec[apt_package_from_component_kubernetes131]

Parameters differences:

--- Exec[apt_package_from_component_kubernetes131].orig
+++ Exec[apt_package_from_component_kubernetes131]

+    subscribe   => Apt::Repository[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    before      => []
+    refreshonly => True
+    command     => /usr/bin/apt-get update

Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat_fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10
+    tag    => _etc_apt_sources.list.d_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

Content differences:

--- component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.orig
+++ component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: trixie-wikimedia
+Components: component/istio115
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Cfssl::Csr[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__system_kube-proxy.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]

+    ensure      => present
+    common_name => system:kube-proxy
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => [{'organisation': 'system:node-proxier'}]

Systemd::Unit[rsyslog-imfile-remedy.timer]

Parameters differences:

--- Systemd::Unit[rsyslog-imfile-remedy.timer].orig
+++ Systemd::Unit[rsyslog-imfile-remedy.timer]

+    override          => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    restart           => False
+    require           => ['Class[Systemd]']
+    unit              => rsyslog-imfile-remedy.timer

Class[Profile::Apt]

Parameters differences:

--- Class[Profile::Apt].orig
+++ Class[Profile::Apt]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[starship]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[linux-sysctl-defaults]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[atop]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[rocm-smi]', 'Package[python3-requests]', 'Package[firmware-amd-graphics]', 'Package[ruby-concurrent]', 'Package[ruby]', 'Package[libruby]', 'Package[puppet-agent]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[starship]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[linux-sysctl-defaults]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[atop]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[dragonfly-dfdaemon]', 'Package[dragonfly-dfget]', 'Package[crictl]', 'Package[containerd]', 'Package[nerdctl]', 'Package[rsyslog-kubernetes]', 'Package[linux-cpupower]', 'Package[apparmor]', 'Package[socat]', 'Package[amd-k8s-device-plugin]', 'Package[amd-k8s-node-labeller]', 'Package[rocm-smi]', 'Package[python3-requests]', 'Package[wikimedia-lvs-realserver]', 'Package[ruby-concurrent]', 'Package[ruby]', 'Package[libruby]', 'Package[puppet-agent]', 'Package[linux-image-6.16.3+deb13-amd64]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[kubernetes-node]', 'Package[calicoctl]', 'Package[calico-cni]', 'Package[istio-cni]', 'Package[firmware-amd-graphics]']

Apt::Package_from_component[kubernetes131]

Parameters differences:

--- Apt::Package_from_component[kubernetes131].orig
+++ Apt::Package_from_component[kubernetes131]

+    component       => component/kubernetes131
+    priority        => 1001
+    ensure_packages => True
+    ensure          => present
+    packages        => []
+    uri             => http://apt.wikimedia.org/wikimedia
+    distro          => trixie-wikimedia

File[/etc/udev/rules.d/75-kube_proxy_conntrack.rules]

Parameters differences:

--- File[/etc/udev/rules.d/75-kube_proxy_conntrack.rules].orig
+++ File[/etc/udev/rules.d/75-kube_proxy_conntrack.rules]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[udev_reload]

Content differences:

--- /etc/udev/rules.d/75-kube_proxy_conntrack.rules.orig
+++ /etc/udev/rules.d/75-kube_proxy_conntrack.rules
@@ -0,0 +1,2 @@
+ACTION=="add", SUBSYSTEM=="module", KERNEL=="nf_conntrack", \
+    RUN+="/usr/lib/systemd/systemd-sysctl --prefix net.netfilter.nf_conntrack_max"

Exec[systemd daemon-reload for amd-k8s-node-labeller.service (amd-k8s-node-labeller-amd-devplugin-after-labeller)]

Parameters differences:

--- Exec[systemd daemon-reload for amd-k8s-node-labeller.service (amd-k8s-node-labeller-amd-devplugin-after-labeller)].orig
+++ Exec[systemd daemon-reload for amd-k8s-node-labeller.service (amd-k8s-node-labeller-amd-devplugin-after-labeller)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Concat_file[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    tag            => _etc_apt_sources.list.d_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    owner          => root
+    backup         => puppet
+    format         => plain
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    show_diff      => True
+    force          => False

Kmod::Module[overlay]

Parameters differences:

--- Kmod::Module[overlay].orig
+++ Kmod::Module[overlay]

+    ensure => present

Apt::Repository[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Apt::Repository[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Apt::Repository[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    allow_releaseinfo_change => False
+    source                   => True
+    ensure                   => present
+    trust_repo               => False
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    uri                      => http://apt.wikimedia.org/wikimedia
+    dist                     => trixie-wikimedia
+    components               => component/calico329
+    bin                      => True

Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01

File[/etc/amd]

Parameters differences:

--- File[/etc/amd].orig
+++ File[/etc/amd]

+    owner  => root
+    group  => root
+    ensure => directory

Apt::Pin[apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]

Parameters differences:

--- Apt::Pin[apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo].orig
+++ Apt::Pin[apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]

+    priority => 1001
+    ensure   => present
+    before   => ['Package[firmware-amd-graphics]']
+    notify   => Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo]
+    package  => firmware-amd-graphics
+    pin      => release a=trixie-backports

Exec[Generate cert mlserve__kubelet_server]

Parameters differences:

--- Exec[Generate cert mlserve__kubelet_server].orig
+++ Exec[Generate cert mlserve__kubelet_server]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve -profile server /etc/cfssl/csr/mlserve__kubelet_server.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__kubelet_server

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__kubelet_server.csr]
+    notify      => ['Service[kubelet]']
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__kubelet_server.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__kubelet_server-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']

Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet].orig
+++ Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label discovery  /etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet

+    require     => Cfssl::Csr[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]
+    notify      => ['Service[dragonfly-dfdaemon]']
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']

Sysctl::Conffile[kube_proxy_icmp]

Parameters differences:

--- Sysctl::Conffile[kube_proxy_icmp].orig
+++ Sysctl::Conffile[kube_proxy_icmp]

+    priority => 75
+    ensure   => present

Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo]

Parameters differences:

--- Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo].orig
+++ Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Exec[Generate cert mlserve__kubelet_server refresh]

Parameters differences:

--- Exec[Generate cert mlserve__kubelet_server refresh].orig
+++ Exec[Generate cert mlserve__kubelet_server refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__kubelet_server.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve -profile server /etc/cfssl/csr/mlserve__kubelet_server.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__kubelet_server

+    notify      => ['Service[kubelet]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Exec[systemd daemon-reload for kubelet.service (kubelet-container-runtime)]

Parameters differences:

--- Exec[systemd daemon-reload for kubelet.service (kubelet-container-runtime)].orig
+++ Exec[systemd daemon-reload for kubelet.service (kubelet-container-runtime)]

+    notify      => ['Service[kubelet]']
+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Class[Profile::Kubernetes::Node]

Parameters differences:

--- Class[Profile::Kubernetes::Node].orig
+++ Class[Profile::Kubernetes::Node]

+    kubernetes_cluster_name         => ml-serve-eqiad
+    feature_flags                   => {}
+    require                         => ['Class[Profile::Rsyslog::Kubernetes]', 'Class[Profile::Netbox::Host]']
+    kubelet_node_taints             => []
+    docker_kubernetes_user_password => somepassword2
+    kubelet_node_labels             => []

Class[Cpufrequtils]

Parameters differences:

--- Class[Cpufrequtils].orig
+++ Class[Cpufrequtils]

+    ensure   => present
+    governor => performance

File[/etc/kubernetes/kube-proxy-config.yaml]

Parameters differences:

--- File[/etc/kubernetes/kube-proxy-config.yaml].orig
+++ File[/etc/kubernetes/kube-proxy-config.yaml]

+    group   => kube
+    mode    => 0400
+    ensure  => file
+    require => K8s::Package[proxy]
+    owner   => kube
+    notify  => Service[kube-proxy]

Content differences:

--- /etc/kubernetes/kube-proxy-config.yaml.orig
+++ /etc/kubernetes/kube-proxy-config.yaml
@@ -0,0 +1,12 @@
+---
+apiVersion: kubeproxy.config.k8s.io/v1alpha1
+kind: KubeProxyConfiguration
+hostnameOverride: ml-serve1014.eqiad.wmnet
+clientConnection:
+  kubeconfig: "/etc/kubernetes/proxy.conf"
+clusterCIDR: 10.67.16.0/21
+mode: iptables
+metricsBindAddress: 0.0.0.0
+nodePortAddresses:
+- 0.0.0.0/0
+- "::/0"

Rsyslog::Conf[kubernetes-node-filters]

Parameters differences:

--- Rsyslog::Conf[kubernetes-node-filters].orig
+++ Rsyslog::Conf[kubernetes-node-filters]

+    mode     => 0444
+    priority => 10
+    source   => puppet:///modules/profile/kubernetes/node/kubernetes-node-filters.rsyslog.conf
+    ensure   => present

Apt::Repository[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Apt::Repository[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Apt::Repository[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    allow_releaseinfo_change => False
+    source                   => True
+    ensure                   => present
+    trust_repo               => False
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    uri                      => http://apt.wikimedia.org/wikimedia
+    dist                     => trixie-wikimedia
+    components               => component/kubernetes131
+    bin                      => True

File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem]

+    owner   => amd-nodelabeller
+    group   => amd-nodelabeller
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]

Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat::Fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10

Sysctl::Parameters[ipv6-fowarding-accept-ra]

Parameters differences:

--- Sysctl::Parameters[ipv6-fowarding-accept-ra].orig
+++ Sysctl::Parameters[ipv6-fowarding-accept-ra]

+    priority => 70
+    values   => {'net.ipv6.conf.all.forwarding': 1, 'net.ipv6.conf.ens51f1np1.accept_ra': 2}
+    ensure   => present

File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem]

+    owner  => amd-nodelabeller
+    group  => amd-nodelabeller
+    mode   => 0440
+    ensure => file

Exec[Generate cert mlserve__calicoctl refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__calicoctl refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__calicoctl refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calicoctl.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calicoctl

Motd::Script[ml_k8s::worker]

Parameters differences:

--- Motd::Script[ml_k8s::worker].orig
+++ Motd::Script[ml_k8s::worker]

+    priority => 5
+    ensure   => present

Cfssl::Csr[/etc/cfssl/csr/mlserve__istio-cni.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__istio-cni.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__istio-cni.csr]

+    ensure      => present
+    common_name => istio-cni
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => []

File[/etc/containerd/config.toml]

Parameters differences:

--- File[/etc/containerd/config.toml].orig
+++ File[/etc/containerd/config.toml]

+    group  => root
+    mode   => 0440
+    ensure => file
+    owner  => root
+    notify => Service[containerd]

Content differences:

--- /etc/containerd/config.toml.orig
+++ /etc/containerd/config.toml
@@ -0,0 +1,47 @@
+# SPDX-License-Identifier: Apache-2.0
+# This is based on the config shipped with the containerd package in Debian (1.6.20~ds1-1+b1)
+#
+# All possible config values including their defaults can be found by running:
+# containerd config default
+version = 2
+
+[plugins]
+  [plugins."io.containerd.grpc.v1.cri"]
+    # Define our sandbox image
+    sandbox_image = "docker-registry.discovery.wmnet/pause:3.6-1"
+    # max_container_log_line_size is the maximum log line size in bytes for a container.
+    # Log line longer than the limit will be split into multiple lines. -1 means no
+    # limit.
+    max_container_log_line_size = -1
+    # By default docker does set net.ipv4.ip_unprivileged_port_start=0 allowing containers to bind to ports
+    # below 1024 without explicit NET_BIND_SERVICE capability.
+    # It also sets net.ipv4.ping_group_range="0 2147483647", allowing ICMP sockets without CAP_NET_RAW.
+    # The following two options ensure compatibility with current workloads.
+    #
+    # enable_unprivileged_ports configures net.ipv4.ip_unprivileged_port_start=0
+    # for all containers which are not using host network and if it is not overwritten by PodSandboxConfig
+    # Note that currently default is set to disabled but target change it in future, see:
+    # https://github.com/kubernetes/kubernetes/issues/102612
+    enable_unprivileged_ports = true
+    # enable_unprivileged_icmp configures net.ipv4.ping_group_range="0 2147483647"
+    # for all containers which are not using host network, are not running in user namespace and if it is not
+    # overwritten by PodSandboxConfig.
+    # Note that currently default is set to disabled but target change it in future together with enable_unprivileged_ports
+    enable_unprivileged_icmp = true
+    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
+      # Re-define the runtime type as defining runc.options would shadow the default setting.
+      # Without this kubelet will fail to run containers with the following error:
+      # failed to create containerd container: create container failed validation: container.Runtime.Name must be set: invalid argument
+      runtime_type = "io.containerd.runc.v2"
+      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
+        # With cgroup v2 we need to use the systemd cgroup driver
+        SystemdCgroup = true
+    # If dragonfly is enabled, configure the local dfget as registry mirror
+    # https://d7y.io/docs/v2.0.2/setup/runtime/containerd/mirror
+    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker-registry.discovery.wmnet"]
+      endpoint = ["https://127.0.0.1:65001","https://docker-registry.discovery.wmnet"]
+    [plugins."io.containerd.grpc.v1.cri".cni]
+      bin_dir = "/opt/cni/bin"
+  [plugins."io.containerd.internal.v1.opt"]
+    # Debian overrides path from /opt/containerd
+    path = "/var/lib/containerd/opt"

Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat::Fragment[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10

Exec[renew certificate - mlserve__calico-cni]

Parameters differences:

--- Exec[renew certificate - mlserve__calico-cni].orig
+++ Exec[renew certificate - mlserve__calico-cni]

+    require     => Exec[Generate cert mlserve__calico-cni]
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__calico-cni.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__calico-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calico-cni

Cfssl::Csr[/etc/cfssl/csr/mlserve__kubelet_server.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__kubelet_server.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__kubelet_server.csr]

+    ensure      => present
+    common_name => kubelet
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => ['ml-serve1014', 'ml-serve1014.eqiad.wmnet', '10.64.155.8', '2620:0:861:123:10:64:155:8']
+    names       => []

File[/lib/systemd/system/rsyslog-imfile-remedy.timer]

Parameters differences:

--- File[/lib/systemd/system/rsyslog-imfile-remedy.timer].orig
+++ File[/lib/systemd/system/rsyslog-imfile-remedy.timer]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for rsyslog-imfile-remedy.timer (rsyslog-imfile-remedy.timer)]

Content differences:

--- /lib/systemd/system/rsyslog-imfile-remedy.timer.orig
+++ /lib/systemd/system/rsyslog-imfile-remedy.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of rsyslog-imfile-remedy.service
+
+[Timer]
+Unit=rsyslog-imfile-remedy.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* 00/3:8:00
+RandomizedDelaySec=30
+
+[Install]
+WantedBy=multi-user.target

File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => kube

Rsyslog::Conf[imfile]

Parameters differences:

--- Rsyslog::Conf[imfile].orig
+++ Rsyslog::Conf[imfile]

+    priority => 0
+    mode     => 0444
+    ensure   => present

File[/etc/update-motd.d/05-ml-k8s--worker]

Parameters differences:

--- File[/etc/update-motd.d/05-ml-k8s--worker].orig
+++ File[/etc/update-motd.d/05-ml-k8s--worker]

+    owner  => root
+    group  => root
+    mode   => 0555
+    ensure => present

Content differences:

--- /etc/update-motd.d/05-ml-k8s--worker.orig
+++ /etc/update-motd.d/05-ml-k8s--worker
@@ -0,0 +1,2 @@
+#!/bin/sh
+printf "%s\n" "ml-serve1014 is a ML Kubernetes worker node (ml_k8s::worker)"

Apt::Pin[apt_pin_linux-6.16-trixie_trixie-bpo]

Parameters differences:

--- Apt::Pin[apt_pin_linux-6.16-trixie_trixie-bpo].orig
+++ Apt::Pin[apt_pin_linux-6.16-trixie_trixie-bpo]

+    priority => 1001
+    ensure   => present
+    before   => ['Package[linux-image-6.16.3+deb13-amd64]']
+    notify   => Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo]
+    package  => linux-image-6.16.3+deb13-amd64
+    pin      => release a=trixie-backports

Class[Profile::Containerd]

Parameters differences:

--- Class[Profile::Containerd].orig
+++ Class[Profile::Containerd]

+    kubernetes_cluster_name => ml-serve-eqiad
+    registry_username       => kubernetes
+    ensure                  => present

Cfssl::Csr[/etc/cfssl/csr/mlserve__calicoctl.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/mlserve__calicoctl.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/mlserve__calicoctl.csr]

+    ensure      => present
+    common_name => calicoctl
+    key         => {'algo': 'ecdsa', 'size': 256}
+    hosts       => []
+    names       => []

File[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.list].orig
+++ File[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

+    owner  => root
+    group  => root
+    ensure => absent

Concat[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    ensure         => present
+    owner          => root
+    backup         => puppet
+    format         => plain
+    path           => /etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    notify         => Exec[apt_repository_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    show_diff      => True
+    warn           => False
+    force          => False

File[/etc/modules-load.d/overlay.conf]

Parameters differences:

--- File[/etc/modules-load.d/overlay.conf].orig
+++ File[/etc/modules-load.d/overlay.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[/sbin/modprobe overlay]

Content differences:

--- /etc/modules-load.d/overlay.conf.orig
+++ /etc/modules-load.d/overlay.conf
@@ -0,0 +1 @@
+overlay

File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.service]

Parameters differences:

--- File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.service].orig
+++ File[/lib/systemd/system/rsyslog-release-deleted-inotify-watches.service]

+    group  => root
+    mode   => 0444
+    ensure => absent
+    owner  => root
+    notify => Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.service (rsyslog-release-deleted-inotify-watches.service)]

Content differences:

--- /lib/systemd/system/rsyslog-release-deleted-inotify-watches.service.orig
+++ /lib/systemd/system/rsyslog-release-deleted-inotify-watches.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Restart rsyslog to release inotify watches of deleted container logs
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/rsyslog-release-deleted-inotify-watches

File[/etc/cni/net.d/istio-kubeconfig]

Parameters differences:

--- File[/etc/cni/net.d/istio-kubeconfig].orig
+++ File[/etc/cni/net.d/istio-kubeconfig]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/cni/net.d/istio-kubeconfig.orig
+++ /etc/cni/net.d/istio-kubeconfig
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: istio-cni
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: istio-cni
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__istio-cni.pem
+    client-key: /etc/kubernetes/pki/mlserve__istio-cni-key.pem

Class[Profile::Dragonfly::Dfdaemon]

Parameters differences:

--- Class[Profile::Dragonfly::Dfdaemon].orig
+++ Class[Profile::Dragonfly::Dfdaemon]

+    ratelimit            => 100M
+    docker_registry_fqdn => docker-registry.discovery.wmnet
+    ensure               => present
+    supernodes           => ['dragonfly-supernode1001.eqiad.wmnet:8002=1']
+    proxy_urls_regex     => ['wikimedia/machinelearning-liftwing.*/blobs/sha256.*', 'amd-pytorch.*/blobs/sha256.*']

Cfssl::Cert[mlserve__rsyslog]

Parameters differences:

--- Cfssl::Cert[mlserve__rsyslog].orig
+++ Cfssl::Cert[mlserve__rsyslog]

+    hosts           => []
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    owner           => root
+    before_services => []
+    names           => [{'organisation': 'view'}]
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    renew_seconds   => 952200
+    common_name     => rsyslog
+    auto_renew      => True
+    label           => mlserve
+    notify_services => ['rsyslog']

Exec[systemd daemon-reload for rsyslog-imfile-remedy.timer (rsyslog-imfile-remedy.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for rsyslog-imfile-remedy.timer (rsyslog-imfile-remedy.timer)].orig
+++ Exec[systemd daemon-reload for rsyslog-imfile-remedy.timer (rsyslog-imfile-remedy.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True
+    before      => ['Service[rsyslog-imfile-remedy.timer]']

File[/etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref]

Parameters differences:

--- File[/etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref].orig
+++ File[/etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[exec-apt-get-update-firmware-amd-graphics-trixie-bpo_trixie-bpo]

Content differences:

--- /etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref.orig
+++ /etc/apt/preferences.d/apt_pin_firmware_amd_graphics_trixie_bpo_trixie_bpo.pref
@@ -0,0 +1,3 @@
+Package: firmware-amd-graphics
+Pin: release a=trixie-backports
+Pin-Priority: 1001

Exec[apt_repository_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Exec[apt_repository_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Exec[apt_repository_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

File[/etc/default/cpupower]

Parameters differences:

--- File[/etc/default/cpupower].orig
+++ File[/etc/default/cpupower]

+    require => Package[linux-cpupower]
+    owner   => root
+    group   => root

Content differences:

--- /etc/default/cpupower.orig
+++ /etc/default/cpupower
@@ -0,0 +1 @@
+GOVERNOR=performance

Concat[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat[/etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    ensure         => present
+    owner          => root
+    backup         => puppet
+    format         => plain
+    path           => /etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    notify         => Exec[apt_repository_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    show_diff      => True
+    warn           => False
+    force          => False

File[/etc/default/prometheus-node-exporter]

Content differences:

--- /etc/default/prometheus-node-exporter.orig
+++ /etc/default/prometheus-node-exporter
@@ -15,6 +15,7 @@
  --collector.netdev \
  --collector.netstat \
  --collector.netstat.fields=^(.*) \
+ --collector.processes \
  --collector.sockstat \
  --collector.stat \
  --collector.systemd.enable-restarts-metrics \

File[/etc/containerd]

Parameters differences:

--- File[/etc/containerd].orig
+++ File[/etc/containerd]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

Exec[systemd daemon-reload for rsyslog-imfile-remedy.service (rsyslog-imfile-remedy.service)]

Parameters differences:

--- Exec[systemd daemon-reload for rsyslog-imfile-remedy.service (rsyslog-imfile-remedy.service)].orig
+++ Exec[systemd daemon-reload for rsyslog-imfile-remedy.service (rsyslog-imfile-remedy.service)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem]

+    group     => amd-nodelabeller
+    mode      => 0440
+    ensure    => file
+    owner     => amd-nodelabeller
+    show_diff => False
+    backup    => False

Nrpe::Check[check_disk_space]

Parameters differences:

--- Nrpe::Check[check_disk_space].orig
+++ Nrpe::Check[check_disk_space]

@@
-    command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    command => /usr/lib/nagios/plugins/check_disk -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(containerd|kubelet)/*' --exclude-type=tracefs

Ferm::Service[kubelet-http]

Parameters differences:

--- Ferm::Service[kubelet-http].orig
+++ Ferm::Service[kubelet-http]

+    srange  => (@resolve((ml-serve-ctrl1001.eqiad.wmnet ml-serve-ctrl1002.eqiad.wmnet)) @resolve((ml-serve-ctrl1001.eqiad.wmnet ml-serve-ctrl1002.eqiad.wmnet), AAAA))
+    proto   => tcp
+    notrack => False
+    ensure  => present
+    prio    => 10
+    port    => 10250
+    desc    =>

Ferm::Service[calico_typha]

Parameters differences:

--- Ferm::Service[calico_typha].orig
+++ Ferm::Service[calico_typha]

+    proto    => tcp
+    notrack  => False
+    ensure   => present
+    prio     => 10
+    port     => 5473
+    src_sets => ['DOMAIN_NETWORKS']
+    desc     =>

Systemd::Service[cpupower]

Parameters differences:

--- Systemd::Service[cpupower].orig
+++ Systemd::Service[cpupower]

+    override                 => False
+    ensure                   => present
+    restart                  => True
+    monitoring_critical      => False
+    unit_type                => service
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_enabled       => False
+    migration_task           => T407130

Cfssl::Cert[mlserve__calico-cni]

Parameters differences:

--- Cfssl::Cert[mlserve__calico-cni].orig
+++ Cfssl::Cert[mlserve__calico-cni]

+    outdir          => /etc/kubernetes/pki
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => root
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => calico-cni
+    hosts           => []
+    notify_services => []

File[/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__kubelet_server.chained.pem]

+    owner   => kube
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]

File[/etc/kubernetes/pki/mlserve__calicoctl.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calicoctl.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calicoctl.pem]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Package[calicoctl]

Parameters differences:

--- Package[calicoctl].orig
+++ Package[calicoctl]

+    ensure   => >=3.29 <3.30
+    provider => apt

File[/etc/systemd/system/kube-proxy.service.d]

Parameters differences:

--- File[/etc/systemd/system/kube-proxy.service.d].orig
+++ File[/etc/systemd/system/kube-proxy.service.d]

+    owner  => root
+    group  => root
+    mode   => 0555
+    ensure => directory

Package[wikimedia-lvs-realserver]

Parameters differences:

--- Package[wikimedia-lvs-realserver].orig
+++ Package[wikimedia-lvs-realserver]

+    require  => File[/etc/default/wikimedia-lvs-realserver]
+    ensure   => present
+    provider => apt

K8s::Kubeconfig[/etc/cni/net.d/calico-kubeconfig]

Parameters differences:

--- K8s::Kubeconfig[/etc/cni/net.d/calico-kubeconfig].orig
+++ K8s::Kubeconfig[/etc/cni/net.d/calico-kubeconfig]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__calico-cni.pem', 'key': '/etc/kubernetes/pki/mlserve__calico-cni-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__calico-cni.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__calico-cni.chained.pem'}
+    group       => root
+    mode        => 0400
+    ensure      => present
+    owner       => root
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['File[/etc/cni/net.d]', 'Class[K8s::Base_dirs]']
+    username    => calico-cni

File[/etc/nerdctl]

Parameters differences:

--- File[/etc/nerdctl].orig
+++ File[/etc/nerdctl]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

Firewall::Service[calico-typha]

Parameters differences:

--- Firewall::Service[calico-typha].orig
+++ Firewall::Service[calico-typha]

+    proto    => tcp
+    notrack  => False
+    ensure   => present
+    prio     => 10
+    port     => 5473
+    src_sets => ['DOMAIN_NETWORKS']
+    desc     =>

Class[K8s::Proxy]

Parameters differences:

--- Class[K8s::Proxy].orig
+++ Class[K8s::Proxy]

+    cluster_cidr  => {'v4': '10.67.16.0/21', 'v6': '2620:0:861:300::/64'}
+    proxy_mode    => iptables
+    ipv6dualstack => False
+    version       => 1.31
+    v_log_level   => 0
+    kubeconfig    => /etc/kubernetes/proxy.conf

Cfssl::Cert[discovery__ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Cfssl::Cert[discovery__ml-serve1014_eqiad_wmnet].orig
+++ Cfssl::Cert[discovery__ml-serve1014_eqiad_wmnet]

+    hosts           => ['ml-serve1014', 'ml-serve1014.eqiad.wmnet', 'docker-registry.discovery.wmnet', '127.0.0.1', '::1', 'localhost']
+    outdir          => /etc/dragonfly
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    owner           => dragonfly
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => ml-serve1014.eqiad.wmnet
+    label           => discovery
+    notify_services => ['dragonfly-dfdaemon']

File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem]

Parameters differences:

--- File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem].orig
+++ File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => dragonfly
+    show_diff => False
+    backup    => False

File[/etc/cni]

Parameters differences:

--- File[/etc/cni].orig
+++ File[/etc/cni]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

Motd::Script[ml_k8s::insetup_gpu]

Parameters differences:

--- Motd::Script[ml_k8s::insetup_gpu].orig
+++ Motd::Script[ml_k8s::insetup_gpu]

-    priority => 5
-    ensure   => present

Systemd::Unit[rsyslog-release-deleted-inotify-watches.service]

Parameters differences:

--- Systemd::Unit[rsyslog-release-deleted-inotify-watches.service].orig
+++ Systemd::Unit[rsyslog-release-deleted-inotify-watches.service]

+    override          => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    restart           => False
+    require           => ['Class[Systemd]']
+    unit              => rsyslog-release-deleted-inotify-watches.service

Cfssl::Cert[mlserve__kubelet_server]

Parameters differences:

--- Cfssl::Cert[mlserve__kubelet_server].orig
+++ Cfssl::Cert[mlserve__kubelet_server]

+    hosts           => ['ml-serve1014', 'ml-serve1014.eqiad.wmnet', '10.64.155.8', '2620:0:861:123:10:64:155:8']
+    outdir          => /etc/kubernetes/pki
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    owner           => kube
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => kubelet
+    profile         => server
+    label           => mlserve
+    notify_services => ['kubelet']

File[/etc/rsyslog.d/10-kubernetes-node-filters.conf]

Parameters differences:

--- File[/etc/rsyslog.d/10-kubernetes-node-filters.conf].orig
+++ File[/etc/rsyslog.d/10-kubernetes-node-filters.conf]

+    group  => root
+    source => puppet:///modules/profile/kubernetes/node/kubernetes-node-filters.rsyslog.conf
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Exec[Generate cert mlserve__system_kube-proxy]

Parameters differences:

--- Exec[Generate cert mlserve__system_kube-proxy].orig
+++ Exec[Generate cert mlserve__system_kube-proxy]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_kube-proxy.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_kube-proxy

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]
+    notify      => ['Service[kube-proxy]']
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__system_kube-proxy.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chained.pem]

+    owner   => kube
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]

Exec[Generate cert mlserve__kubelet_server refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__kubelet_server refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__kubelet_server refresh on intermediate ca change]

+    subscribe   => File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve -profile server /etc/cfssl/csr/mlserve__kubelet_server.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__kubelet_server

+    notify      => ['Service[kubelet]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Exec[systemd daemon-reload for ferm.service (ferm-ferm-service-auto-restart)]

Parameters differences:

--- Exec[systemd daemon-reload for ferm.service (ferm-ferm-service-auto-restart)].orig
+++ Exec[systemd daemon-reload for ferm.service (ferm-ferm-service-auto-restart)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True
+    before      => ['Service[ferm]']

Package[dragonfly-dfdaemon]

Parameters differences:

--- Package[dragonfly-dfdaemon].orig
+++ Package[dragonfly-dfdaemon]

+    ensure   => installed
+    provider => apt

File[/etc/sysctl.d/75-kube_proxy_icmp.conf]

Parameters differences:

--- File[/etc/sysctl.d/75-kube_proxy_icmp.conf].orig
+++ File[/etc/sysctl.d/75-kube_proxy_icmp.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/75-kube_proxy_icmp.conf.orig
+++ /etc/sysctl.d/75-kube_proxy_icmp.conf
@@ -0,0 +1,4 @@
+# sysctl parameters managed by Puppet.
+net.ipv4.conf.all.send_redirects = 0
+net.ipv4.conf.default.send_redirects = 0
+net.ipv4.conf.ens51f1np1.send_redirects = 0

Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo]

Parameters differences:

--- Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo].orig
+++ Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Udev::Rule[kube_proxy_conntrack]

Parameters differences:

--- Udev::Rule[kube_proxy_conntrack].orig
+++ Udev::Rule[kube_proxy_conntrack]

+    priority => 75
+    ensure   => present

Exec[Generate cert mlserve__amdgpu-node-labeller refresh]

Parameters differences:

--- Exec[Generate cert mlserve__amdgpu-node-labeller refresh].orig
+++ Exec[Generate cert mlserve__amdgpu-node-labeller refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__amdgpu-node-labeller

File[/etc/kubernetes/pki/mlserve__calicoctl-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calicoctl-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calicoctl-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => root
+    show_diff => False
+    backup    => False

File[/etc/calico/calicoctl-kubeconfig]

Parameters differences:

--- File[/etc/calico/calicoctl-kubeconfig].orig
+++ File[/etc/calico/calicoctl-kubeconfig]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/calico/calicoctl-kubeconfig.orig
+++ /etc/calico/calicoctl-kubeconfig
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: calicoctl
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: calicoctl
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__calicoctl.pem
+    client-key: /etc/kubernetes/pki/mlserve__calicoctl-key.pem

Cfssl::Cert[mlserve__calicoctl]

Parameters differences:

--- Cfssl::Cert[mlserve__calicoctl].orig
+++ Cfssl::Cert[mlserve__calicoctl]

+    outdir          => /etc/kubernetes/pki
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => root
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => calicoctl
+    hosts           => []
+    notify_services => []

Cfssl::Cert[mlserve__amdgpu-node-labeller]

Parameters differences:

--- Cfssl::Cert[mlserve__amdgpu-node-labeller].orig
+++ Cfssl::Cert[mlserve__amdgpu-node-labeller]

+    outdir          => /etc/kubernetes/pki
+    names           => []
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    group           => amd-nodelabeller
+    provide_chain   => True
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => amd-nodelabeller
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => amdgpu-node-labeller
+    hosts           => []
+    notify_services => []

File[/etc/systemd/system/kube-proxy.service.d/puppet-override.conf]

Parameters differences:

--- File[/etc/systemd/system/kube-proxy.service.d/puppet-override.conf].orig
+++ File[/etc/systemd/system/kube-proxy.service.d/puppet-override.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for kube-proxy.service (kube-proxy)]

Content differences:

--- /etc/systemd/system/kube-proxy.service.d/puppet-override.conf.orig
+++ /etc/systemd/system/kube-proxy.service.d/puppet-override.conf
@@ -0,0 +1,2 @@
+[Unit]
+After = ferm.service

File[/etc/cfssl/csr/mlserve__rsyslog.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__rsyslog.csr].orig
+++ File[/etc/cfssl/csr/mlserve__rsyslog.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__rsyslog.csr.orig
+++ /etc/cfssl/csr/mlserve__rsyslog.csr
@@ -0,0 +1,19 @@
+{
+  "CN": "rsyslog",
+  "hosts": [
+    "rsyslog"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+    {
+      "C": null,
+      "L": null,
+      "O": "view",
+      "OU": null,
+      "S": null
+    }
+  ]
+}

File[/etc/udev/rules.d/70-kfd.rules]

Parameters differences:

--- File[/etc/udev/rules.d/70-kfd.rules].orig
+++ File[/etc/udev/rules.d/70-kfd.rules]

+    owner => root
+    group => root
+    mode  => 0544

Content differences:

--- /etc/udev/rules.d/70-kfd.rules.orig
+++ /etc/udev/rules.d/70-kfd.rules
@@ -0,0 +1 @@
+SUBSYSTEM=="kfd", KERNEL=="kfd", MODE="0666"

Systemd::Unit[amd-k8s-node-labeller-amd-devplugin-after-labeller]

Parameters differences:

--- Systemd::Unit[amd-k8s-node-labeller-amd-devplugin-after-labeller].orig
+++ Systemd::Unit[amd-k8s-node-labeller-amd-devplugin-after-labeller]

+    override          => True
+    override_filename => amd-devplugin-after-labeller
+    restart           => True
+    ensure            => present
+    require           => ['Class[Systemd]']
+    unit              => amd-k8s-node-labeller

File[/etc/kubernetes/pki/mlserve__kubelet_server.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__kubelet_server.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__kubelet_server.csr]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

File[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr.orig
+++ /etc/cfssl/csr/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr
@@ -0,0 +1,19 @@
+{
+  "CN": "system:node:ml-serve1014.eqiad.wmnet",
+  "hosts": [
+    "system:node:ml-serve1014.eqiad.wmnet"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+    {
+      "C": null,
+      "L": null,
+      "O": "system:nodes",
+      "OU": null,
+      "S": null
+    }
+  ]
+}

Exec[create chained cert /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem].orig
+++ Exec[create chained cert /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]

+    subscribe => ['Exec[renew certificate - discovery__ml-serve1014_eqiad_wmnet]', 'File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]', 'File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem]']
+    command   => /bin/cat /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem > /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem
+    require   => Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh on intermediate ca change]
+    notify    => ['Service[dragonfly-dfdaemon]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem | sha512sum)" == "$(/bin/cat /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem | sha512sum)"

Exec[Generate cert mlserve__rsyslog refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert mlserve__rsyslog refresh on intermediate ca change].orig
+++ Exec[Generate cert mlserve__rsyslog refresh on intermediate ca change]

+    subscribe   => File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__rsyslog.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog

+    notify      => ['Service[rsyslog]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/kubernetes/pki/mlserve__calico-cni-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calico-cni-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calico-cni-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => root
+    show_diff => False
+    backup    => False

File[/etc/rsyslog.d/00-imfile.conf]

Parameters differences:

--- File[/etc/rsyslog.d/00-imfile.conf].orig
+++ File[/etc/rsyslog.d/00-imfile.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/00-imfile.conf.orig
+++ /etc/rsyslog.d/00-imfile.conf
@@ -0,0 +1 @@
+module(load="imfile")

Exec[/usr/sbin/dpkg-reconfigure -p critical -f noninteractive wikimedia-lvs-realserver]

Parameters differences:

--- Exec[/usr/sbin/dpkg-reconfigure -p critical -f noninteractive wikimedia-lvs-realserver].orig
+++ Exec[/usr/sbin/dpkg-reconfigure -p critical -f noninteractive wikimedia-lvs-realserver]

+    require     => Package[wikimedia-lvs-realserver]
+    subscribe   => File[/etc/default/wikimedia-lvs-realserver]
+    path        => /bin:/sbin:/usr/bin:/usr/sbin
+    refreshonly => True

File[/etc/calico/calicoctl.cfg]

Parameters differences:

--- File[/etc/calico/calicoctl.cfg].orig
+++ File[/etc/calico/calicoctl.cfg]

+    owner  => root
+    group  => root
+    mode   => 0444
+    ensure => file

Content differences:

--- /etc/calico/calicoctl.cfg.orig
+++ /etc/calico/calicoctl.cfg
@@ -0,0 +1,10 @@
+# This configures calicoctl to use the kubernetes datastore.
+# The user referenced in the kubeconfig file probably needs broad permissions, see:
+# https://docs.projectcalico.org/getting-started/clis/calicoctl/configure/overview
+# https://docs.projectcalico.org/getting-started/kubernetes/hardway/end-user-rbac
+apiVersion: projectcalico.org/v3
+kind: CalicoAPIConfig
+metadata:
+spec:
+  datastoreType: "kubernetes"
+  kubeconfig: "/etc/calico/calicoctl-kubeconfig"

Exec[Generate cert mlserve__rsyslog refresh]

Parameters differences:

--- Exec[Generate cert mlserve__rsyslog refresh].orig
+++ Exec[Generate cert mlserve__rsyslog refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__rsyslog.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__rsyslog.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog

+    notify      => ['Service[rsyslog]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/ferm/conf.d/10_calico_typha]

Parameters differences:

--- File[/etc/ferm/conf.d/10_calico_typha].orig
+++ File[/etc/ferm/conf.d/10_calico_typha]

+    group   => root
+    mode    => 0400
+    ensure  => present
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    owner   => root
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_calico_typha.orig
+++ /etc/ferm/conf.d/10_calico_typha
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 5473, $DOMAIN_NETWORKS);
+
+

File[/etc/udev/rules.d/70-render.rules]

Parameters differences:

--- File[/etc/udev/rules.d/70-render.rules].orig
+++ File[/etc/udev/rules.d/70-render.rules]

+    owner => root
+    group => root
+    mode  => 0544

Content differences:

--- /etc/udev/rules.d/70-render.rules.orig
+++ /etc/udev/rules.d/70-render.rules
@@ -0,0 +1 @@
+SUBSYSTEM=="drm", KERNEL=="renderD*", MODE="0666"

File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Concat::Fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    target => /etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 10

Concat_file[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    tag            => _etc_apt_sources.list.d_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    owner          => root
+    backup         => puppet
+    format         => plain
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    show_diff      => True
+    force          => False

File[/etc/cfssl/csr/mlserve__calicoctl.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__calicoctl.csr].orig
+++ File[/etc/cfssl/csr/mlserve__calicoctl.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__calicoctl.csr.orig
+++ /etc/cfssl/csr/mlserve__calicoctl.csr
@@ -0,0 +1,13 @@
+{
+  "CN": "calicoctl",
+  "hosts": [
+    "calicoctl"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

K8s::Kubeconfig[/etc/calico/calicoctl-kubeconfig]

Parameters differences:

--- K8s::Kubeconfig[/etc/calico/calicoctl-kubeconfig].orig
+++ K8s::Kubeconfig[/etc/calico/calicoctl-kubeconfig]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__calicoctl.pem', 'key': '/etc/kubernetes/pki/mlserve__calicoctl-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__calicoctl.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__calicoctl.chained.pem'}
+    group       => root
+    mode        => 0400
+    ensure      => present
+    owner       => root
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['Class[K8s::Base_dirs]']
+    username    => calicoctl

Firewall::Service[dragonfly_dfget]

Parameters differences:

--- Firewall::Service[dragonfly_dfget].orig
+++ Firewall::Service[dragonfly_dfget]

+    proto    => tcp
+    notrack  => False
+    ensure   => present
+    prio     => 10
+    port     => 15001
+    src_sets => ['DOMAIN_NETWORKS']
+    desc     =>

Exec[cpupower_reload]

Parameters differences:

--- Exec[cpupower_reload].orig
+++ Exec[cpupower_reload]

+    require => File[/etc/default/cpupower]
+    unless  => /usr/bin/cpupower frequency-info -p | /bin/grep -wq performance
+    command => /usr/bin/systemctl restart cpupower

Exec[Generate cert mlserve__calico-cni]

Parameters differences:

--- Exec[Generate cert mlserve__calico-cni].orig
+++ Exec[Generate cert mlserve__calico-cni]

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__calico-cni.csr]
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__calico-cni.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__calico-cni-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calico-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calico-cni

Class[Profile::Base]

Parameters differences:

--- Class[Profile::Base].orig
+++ Class[Profile::Base]

@@
-    use_linux_from_bpo_on_trixie => False
+    use_linux_from_bpo_on_trixie => True
@@
-    overlayfs                    => False
+    overlayfs                    => True

Service[cpupower]

Parameters differences:

--- Service[cpupower].orig
+++ Service[cpupower]

+    enable => True
+    ensure => running

File[/etc/calico/pki]

Parameters differences:

--- File[/etc/calico/pki].orig
+++ File[/etc/calico/pki]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => absent

Rsyslog::Conf[output_kafka_k8s]

Parameters differences:

--- Rsyslog::Conf[output_kafka_k8s].orig
+++ Rsyslog::Conf[output_kafka_k8s]

+    priority => 35
+    mode     => 0444
+    ensure   => present

K8s::Kubeconfig[/etc/kubernetes/proxy.conf]

Parameters differences:

--- K8s::Kubeconfig[/etc/kubernetes/proxy.conf].orig
+++ K8s::Kubeconfig[/etc/kubernetes/proxy.conf]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__system_kube-proxy.pem', 'key': '/etc/kubernetes/pki/mlserve__system_kube-proxy-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem'}
+    group       => kube
+    mode        => 0400
+    ensure      => present
+    owner       => kube
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['Class[K8s::Base_dirs]']
+    username    => default-proxy

File[/etc/kubernetes/pki/mlserve__istio-cni.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__istio-cni.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__istio-cni.csr]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

File[/etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf].orig
+++ File[/etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf]

+    group  => root
+    mode   => 0444
+    ensure => absent
+    owner  => root
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf.orig
+++ /etc/rsyslog.d/40-rsyslog-release-deleted-inotify-watches.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "rsyslog-release-deleted-inotify-watches" then {
+    action(
+        type="omfile" file="/var/log/rsyslog-release-deleted-inotify-watches/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/amd/node-labeller-kubeconfig]

Parameters differences:

--- File[/etc/amd/node-labeller-kubeconfig].orig
+++ File[/etc/amd/node-labeller-kubeconfig]

+    owner  => amd-nodelabeller
+    group  => amd-nodelabeller
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/amd/node-labeller-kubeconfig.orig
+++ /etc/amd/node-labeller-kubeconfig
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: amdgpu-node-labeller
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: amdgpu-node-labeller
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem
+    client-key: /etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem

File[/etc/kubernetes/pki/mlserve__istio-cni-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__istio-cni-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__istio-cni-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => root
+    show_diff => False
+    backup    => False

File[/etc/kubernetes/pki/mlserve__system_kube-proxy.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_kube-proxy.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_kube-proxy.pem]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

Exec[apt_repository_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

Parameters differences:

--- Exec[apt_repository_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia].orig
+++ Exec[apt_repository_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Exec[create chained cert /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]

+    subscribe => ['Exec[renew certificate - mlserve__kubelet_server]', 'File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__kubelet_server.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__kubelet_server.pem /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem > /etc/kubernetes/pki/mlserve__kubelet_server.chained.pem
+    require   => Exec[Generate cert mlserve__kubelet_server refresh on intermediate ca change]
+    notify    => ['Service[kubelet]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__kubelet_server.pem /etc/kubernetes/pki/mlserve__kubelet_server.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__kubelet_server.chained.pem | sha512sum)"

Class[Containerd]

Parameters differences:

--- Class[Containerd].orig
+++ Class[Containerd]

+    require => ['Class[Containerd::Configuration]']
+    ensure  => present

Package[rsyslog-kubernetes]

Parameters differences:

--- Package[rsyslog-kubernetes].orig
+++ Package[rsyslog-kubernetes]

+    ensure   => installed
+    provider => apt

Exec[apt_package_from_component_istio115]

Parameters differences:

--- Exec[apt_package_from_component_istio115].orig
+++ Exec[apt_package_from_component_istio115]

+    subscribe   => Apt::Repository[component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    before      => ['Package[istio-cni]']
+    refreshonly => True
+    command     => /usr/bin/apt-get update

File[/var/log/rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- File[/var/log/rsyslog-release-deleted-inotify-watches].orig
+++ File[/var/log/rsyslog-release-deleted-inotify-watches]

+    group  => root
+    mode   => 0755
+    ensure => absent
+    owner  => root
+    backup => False
+    force  => True

File[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr.orig
+++ /etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr
@@ -0,0 +1,18 @@
+{
+  "CN": "ml-serve1014.eqiad.wmnet",
+  "hosts": [
+    "ml-serve1014",
+    "ml-serve1014.eqiad.wmnet",
+    "docker-registry.discovery.wmnet",
+    "127.0.0.1",
+    "::1",
+    "localhost"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

Exec[Generate cert mlserve__rsyslog]

Parameters differences:

--- Exec[Generate cert mlserve__rsyslog].orig
+++ Exec[Generate cert mlserve__rsyslog]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__rsyslog.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__rsyslog.csr]
+    notify      => ['Service[rsyslog]']
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']

Class[Profile::Monitoring]

Parameters differences:

--- Class[Profile::Monitoring].orig
+++ Class[Profile::Monitoring]

@@
-    nrpe_check_disk_options => -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_check_disk_options => -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(containerd|kubelet)/*' --exclude-type=tracefs

Systemd::Timer[rsyslog-imfile-remedy]

Parameters differences:

--- Systemd::Timer[rsyslog-imfile-remedy].orig
+++ Systemd::Timer[rsyslog-imfile-remedy]

+    splay              => 30
+    ensure             => present
+    unit_name          => rsyslog-imfile-remedy.service
+    accuracy           => 15sec
+    fixed_random_delay => False
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* 00/3:8:00'}]

Package[linux-image-6.16.3+deb13-amd64]

Parameters differences:

--- Package[linux-image-6.16.3+deb13-amd64].orig
+++ Package[linux-image-6.16.3+deb13-amd64]

+    ensure   => installed
+    provider => apt

Exec[renew certificate - mlserve__system_kube-proxy]

Parameters differences:

--- Exec[renew certificate - mlserve__system_kube-proxy].orig
+++ Exec[renew certificate - mlserve__system_kube-proxy]

+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__system_kube-proxy.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_kube-proxy

+    require     => Exec[Generate cert mlserve__system_kube-proxy]
+    notify      => ['Service[kube-proxy]']
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__system_kube-proxy.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']

Class[Profile::Lvs::Realserver]

Parameters differences:

--- Class[Profile::Lvs::Realserver].orig
+++ Class[Profile::Lvs::Realserver]

+    pools                => {'inference': {}, 'k8s-ingress-ml-serve': {}}
+    use_conftool         => False
+    poolcounter_backends => [{'label': 'pc1', 'fqdn': 'poolcounter1006.eqiad.wmnet'}, {'label': 'pc2', 'fqdn': 'poolcounter1007.eqiad.wmnet'}]

Package[apparmor]

Parameters differences:

--- Package[apparmor].orig
+++ Package[apparmor]

+    ensure   => installed
+    provider => apt

Systemd::Service[kube-proxy]

Parameters differences:

--- Systemd::Service[kube-proxy].orig
+++ Systemd::Service[kube-proxy]

+    override                 => True
+    ensure                   => present
+    restart                  => True
+    monitoring_critical      => False
+    unit_type                => service
+    monitoring_contact_group => admins
+    service_params           => {}
+    subscribe                => File[/etc/kubernetes/proxy.conf]
+    monitoring_enabled       => False
+    migration_task           => T407130

Sysctl::Parameters[kube_proxy_icmp]

Parameters differences:

--- Sysctl::Parameters[kube_proxy_icmp].orig
+++ Sysctl::Parameters[kube_proxy_icmp]

+    priority => 75
+    values   => {'net.ipv4.conf.all.send_redirects': 0, 'net.ipv4.conf.default.send_redirects': 0, 'net.ipv4.conf.ens51f1np1.send_redirects': 0}
+    ensure   => present

File[/etc/calico]

Parameters differences:

--- File[/etc/calico].orig
+++ File[/etc/calico]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

File[/etc/logrotate.d/rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- File[/etc/logrotate.d/rsyslog-release-deleted-inotify-watches].orig
+++ File[/etc/logrotate.d/rsyslog-release-deleted-inotify-watches]

+    owner  => root
+    group  => root
+    mode   => 0444
+    ensure => absent

Content differences:

--- /etc/logrotate.d/rsyslog-release-deleted-inotify-watches.orig
+++ /etc/logrotate.d/rsyslog-release-deleted-inotify-watches
@@ -0,0 +1,12 @@
+# logrotate(8) config for rsyslog-release-deleted-inotify-watches
+
+/var/log/rsyslog-release-deleted-inotify-watches/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/etc/cfssl/csr/mlserve__istio-cni.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__istio-cni.csr].orig
+++ File[/etc/cfssl/csr/mlserve__istio-cni.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__istio-cni.csr.orig
+++ /etc/cfssl/csr/mlserve__istio-cni.csr
@@ -0,0 +1,13 @@
+{
+  "CN": "istio-cni",
+  "hosts": [
+    "istio-cni"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

Package[istio-cni]

Parameters differences:

--- Package[istio-cni].orig
+++ Package[istio-cni]

+    ensure   => installed
+    provider => apt

File[/etc/kubernetes/pki/mlserve__istio-cni.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__istio-cni.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__istio-cni.chained.pem]

+    owner   => root
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__istio-cni.chain.pem]

Exec[create chained cert /etc/kubernetes/pki/mlserve__istio-cni.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__istio-cni.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__istio-cni.chain.pem]

+    require   => Exec[Generate cert mlserve__istio-cni refresh on intermediate ca change]
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__istio-cni.pem /etc/kubernetes/pki/mlserve__istio-cni.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__istio-cni.chained.pem | sha512sum)"

+    subscribe => ['Exec[renew certificate - mlserve__istio-cni]', 'File[/etc/kubernetes/pki/mlserve__istio-cni.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__istio-cni.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__istio-cni.pem /etc/kubernetes/pki/mlserve__istio-cni.chain.pem > /etc/kubernetes/pki/mlserve__istio-cni.chained.pem

File[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.list].orig
+++ File[/etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

+    owner  => root
+    group  => root
+    ensure => absent

File[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.list].orig
+++ File[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.list]

+    owner  => root
+    group  => root
+    ensure => absent

File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__kubelet_server.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => kube

File[/etc/cni/net.d/calico-kubeconfig]

Parameters differences:

--- File[/etc/cni/net.d/calico-kubeconfig].orig
+++ File[/etc/cni/net.d/calico-kubeconfig]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/cni/net.d/calico-kubeconfig.orig
+++ /etc/cni/net.d/calico-kubeconfig
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: calico-cni
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: calico-cni
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__calico-cni.pem
+    client-key: /etc/kubernetes/pki/mlserve__calico-cni-key.pem

Class[Prometheus::Node_exporter]

Parameters differences:

--- Class[Prometheus::Node_exporter].orig
+++ Class[Prometheus::Node_exporter]

@@
-    collectors_extra => []
+    collectors_extra => ['processes']

Systemd::Syslog[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Systemd::Syslog[rsyslog-release-deleted-inotify-watches].orig
+++ Systemd::Syslog[rsyslog-release-deleted-inotify-watches]

+    log_filename           => syslog.log
+    readable_by            => all
+    group                  => root
+    ensure                 => absent
+    owner                  => root
+    force_stop             => True
+    programname_comparison => startswith
+    base_dir               => /var/log

K8s::Package[proxy]

Parameters differences:

--- K8s::Package[proxy].orig
+++ K8s::Package[proxy]

+    priority        => 1001
+    ensure_packages => True
+    version         => 1.31
+    require         => ['Class[K8s::Base_dirs]']
+    uri             => http://apt.wikimedia.org/wikimedia
+    package         => node
+    distro          => trixie-wikimedia

File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.csr].orig
+++ File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.csr]

+    owner  => dragonfly
+    group  => root
+    mode   => 0440
+    ensure => file

Package[kubernetes-node]

Parameters differences:

--- Package[kubernetes-node].orig
+++ Package[kubernetes-node]

+    require  => Apt::Package_from_component[kubernetes131]
+    ensure   => >=1.31 <1.32
+    provider => apt

File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

Exec[create chained cert /etc/kubernetes/pki/mlserve__calicoctl.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__calicoctl.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__calicoctl.chain.pem]

+    require   => Exec[Generate cert mlserve__calicoctl refresh on intermediate ca change]
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__calicoctl.pem /etc/kubernetes/pki/mlserve__calicoctl.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__calicoctl.chained.pem | sha512sum)"

+    subscribe => ['Exec[renew certificate - mlserve__calicoctl]', 'File[/etc/kubernetes/pki/mlserve__calicoctl.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__calicoctl.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__calicoctl.pem /etc/kubernetes/pki/mlserve__calicoctl.chain.pem > /etc/kubernetes/pki/mlserve__calicoctl.chained.pem

Rsyslog::Conf[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Rsyslog::Conf[rsyslog-release-deleted-inotify-watches].orig
+++ Rsyslog::Conf[rsyslog-release-deleted-inotify-watches]

+    require  => File[/var/log/rsyslog-release-deleted-inotify-watches]
+    priority => 40
+    mode     => 0444
+    ensure   => absent

File[/usr/local/sbin/rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- File[/usr/local/sbin/rsyslog-release-deleted-inotify-watches].orig
+++ File[/usr/local/sbin/rsyslog-release-deleted-inotify-watches]

+    owner  => root
+    group  => root
+    mode   => 0544
+    ensure => absent

Package[amd-k8s-device-plugin]

Parameters differences:

--- Package[amd-k8s-device-plugin].orig
+++ Package[amd-k8s-device-plugin]

+    ensure   => present
+    provider => apt

Class[Profile::Calico::Kubernetes]

Parameters differences:

--- Class[Profile::Calico::Kubernetes].orig
+++ Class[Profile::Calico::Kubernetes]

+    kubernetes_cluster_name => ml-serve-eqiad

Package[dragonfly-dfget]

Parameters differences:

--- Package[dragonfly-dfget].orig
+++ Package[dragonfly-dfget]

+    ensure   => installed
+    provider => apt

File[/etc/default/wikimedia-lvs-realserver]

Parameters differences:

--- File[/etc/default/wikimedia-lvs-realserver].orig
+++ File[/etc/default/wikimedia-lvs-realserver]

+    owner  => root
+    group  => root
+    mode   => 0444
+    ensure => present

Content differences:

--- /etc/default/wikimedia-lvs-realserver.orig
+++ /etc/default/wikimedia-lvs-realserver
@@ -0,0 +1,10 @@
+# This file is managed by puppet!
+
+
+
+# Location of the sysctl file containing LVS ARP settings
+SYSCTLFILE=/usr/share/wikimedia-lvs-realserver/sysctl.conf
+
+# LVS service IPs to be bound to the loopback interface,
+# separate using spaces
+LVS_SERVICE_IPS="10.2.2.63 10.2.2.84"

File[/etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref]

Parameters differences:

--- File[/etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref].orig
+++ File[/etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[exec-apt-get-update-linux-6.16-trixie_trixie-bpo]

Content differences:

--- /etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref.orig
+++ /etc/apt/preferences.d/apt_pin_linux_6_16_trixie_trixie_bpo.pref
@@ -0,0 +1,3 @@
+Package: linux-image-6.16.3+deb13-amd64
+Pin: release a=trixie-backports
+Pin-Priority: 1001

File[/etc/kubernetes/pki/mlserve__calico-cni.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calico-cni.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__calico-cni.csr]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Class[Profile::Rsyslog::Kubernetes]

Parameters differences:

--- Class[Profile::Rsyslog::Kubernetes].orig
+++ Class[Profile::Rsyslog::Kubernetes]

+    kubernetes_cluster_name => ml-serve-eqiad
+    kafka_brokers           => ['kafka-logging1001.eqiad.wmnet:9093', 'kafka-logging1002.eqiad.wmnet:9093', 'kafka-logging1003.eqiad.wmnet:9093', 'kafka-logging1004.eqiad.wmnet:9093', 'kafka-logging1005.eqiad.wmnet:9093']
+    enable                  => True

File[/etc/nerdctl/nerdctl.toml]

Parameters differences:

--- File[/etc/nerdctl/nerdctl.toml].orig
+++ File[/etc/nerdctl/nerdctl.toml]

+    owner  => root
+    group  => root
+    mode   => 0644
+    ensure => file

Content differences:

--- /etc/nerdctl/nerdctl.toml.orig
+++ /etc/nerdctl/nerdctl.toml
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: Apache-2.0
+#
+# For documentation of the available options, see:
+# https://github.com/containerd/nerdctl/blob/main/docs/config.md
+namespace = "k8s.io"

File[/etc/dragonfly]

Parameters differences:

--- File[/etc/dragonfly].orig
+++ File[/etc/dragonfly]

+    group   => root
+    recurse => True
+    ensure  => directory
+    mode    => 0740
+    owner   => dragonfly

File[/etc/default/kubelet]

Parameters differences:

--- File[/etc/default/kubelet].orig
+++ File[/etc/default/kubelet]

+    group  => root
+    mode   => 0644
+    ensure => file
+    owner  => root
+    notify => Service[kubelet]

Content differences:

--- /etc/default/kubelet.orig
+++ /etc/default/kubelet
@@ -0,0 +1,11 @@
+###
+# kubernetes kubelet (minion) config
+
+DAEMON_ARGS="--config=/etc/kubernetes/kubelet-config.yaml \
+ --hostname-override=ml-serve1014.eqiad.wmnet \
+ --kubeconfig=/etc/kubernetes/kubelet.conf \
+ --node-ip=10.64.155.8 \
+ --node-labels=node.kubernetes.io/disk-type=ssd,topology.kubernetes.io/region=eqiad,topology.kubernetes.io/zone=row-e11 \
+ --register-schedulable=false \
+ --system-reserved=cpu=45.3,memory=50.83Gi \
+ --v=0"

File[/etc/sysctl.d/70-increase_inotify_limits.conf]

Parameters differences:

--- File[/etc/sysctl.d/70-increase_inotify_limits.conf].orig
+++ File[/etc/sysctl.d/70-increase_inotify_limits.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/70-increase_inotify_limits.conf.orig
+++ /etc/sysctl.d/70-increase_inotify_limits.conf
@@ -0,0 +1,3 @@
+# sysctl parameters managed by Puppet.
+fs.inotify.max_user_instances = 512
+fs.inotify.max_user_watches = 32768

Sysctl::Conffile[increase_inotify_limits]

Parameters differences:

--- Sysctl::Conffile[increase_inotify_limits].orig
+++ Sysctl::Conffile[increase_inotify_limits]

+    priority => 70
+    ensure   => present

File[/etc/cfssl/ssl/mlserve__rsyslog]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog]

+    group   => root
+    recurse => True
+    ensure  => directory
+    mode    => 0740
+    owner   => root

File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.csr]

Parameters differences:

--- File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.csr].orig
+++ File[/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.csr]

+    owner  => root
+    group  => root
+    mode   => 0440
+    ensure => file

Systemd::Override[amd-devplugin-after-labeller]

Parameters differences:

--- Systemd::Override[amd-devplugin-after-labeller].orig
+++ Systemd::Override[amd-devplugin-after-labeller]

+    unit    => amd-k8s-node-labeller
+    ensure  => present
+    restart => True

Exec[Generate cert mlserve__calicoctl refresh]

Parameters differences:

--- Exec[Generate cert mlserve__calicoctl refresh].orig
+++ Exec[Generate cert mlserve__calicoctl refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__calicoctl.csr]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calicoctl.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calicoctl

File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem]

Parameters differences:

--- File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem].orig
+++ File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.pem]

+    owner  => dragonfly
+    group  => root
+    mode   => 0440
+    ensure => file

File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.csr]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.csr].orig
+++ File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.csr]

+    owner  => amd-nodelabeller
+    group  => amd-nodelabeller
+    mode   => 0440
+    ensure => file

File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem]

+    owner   => kube
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

File[/etc/kubernetes/kubelet.conf]

Parameters differences:

--- File[/etc/kubernetes/kubelet.conf].orig
+++ File[/etc/kubernetes/kubelet.conf]

+    owner  => kube
+    group  => kube
+    mode   => 0400
+    ensure => present

Content differences:

--- /etc/kubernetes/kubelet.conf.orig
+++ /etc/kubernetes/kubelet.conf
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Config
+preferences: {}
+current-context: default-system
+contexts:
+- name: default-system
+  context:
+    cluster: default-cluster
+    user: default-auth
+clusters:
+- name: default-cluster
+  cluster:
+    server: https://ml-ctrl.svc.eqiad.wmnet:6443
+users:
+- name: default-auth
+  user:
+    client-certificate: /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem
+    client-key: /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem

Exec[apt_package_from_component_calico329]

Parameters differences:

--- Exec[apt_package_from_component_calico329].orig
+++ Exec[apt_package_from_component_calico329]

+    subscribe   => Apt::Repository[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    before      => ['Package[calicoctl]', 'Package[calico-cni]']
+    refreshonly => True
+    command     => /usr/bin/apt-get update

File[/etc/dragonfly/dfdaemon.yml]

Parameters differences:

--- File[/etc/dragonfly/dfdaemon.yml].orig
+++ File[/etc/dragonfly/dfdaemon.yml]

+    group  => root
+    mode   => 0644
+    ensure => file
+    owner  => root
+    notify => Service[dragonfly-dfdaemon]

Content differences:

--- /etc/dragonfly/dfdaemon.yml.orig
+++ /etc/dragonfly/dfdaemon.yml
@@ -0,0 +1,49 @@
+# This is only used if dfdaemon is configured in registry-mirrors of
+# /etc/docker/daemon.json.
+# It is not used in our configuration, but I'll keep it here as the default is
+# the upstream docker hub and that looks very phishy in logs.
+registry_mirror:
+  remote: "https://docker-registry.discovery.wmnet"
+
+# dfdaemon will listen on TCP 65001 by default for connections to be proxied to the registry.
+# If certpem and keypem is set, dfdaemon will only accept HTTPS connections.
+# Prometheus metrics are served as well using this port, so we need to bind to 0.0.0.0.
+hostIp: 0.0.0.0
+port: 65001
+# We can't use HTTP even though we're localhost only as docker will refuse to send credentials
+# for a registry via HTTPS. So this needs to be a certificate for: 127.0.0.1, ::1, localhost and $(hostname -f) (for scraping metrics).
+certpem: "/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem"
+keypem: "/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem"
+
+# Requests will be handled by the first matching rule
+# "use_https: true" can be used to upgrade incoming HTTP requests to HTTPS (this means connections
+# from dfdaemon to the source registry will always use HTTPS).
+proxies:
+  - regx: "wikimedia/machinelearning-liftwing.*/blobs/sha256.*"
+    use_https: true
+  - regx: "amd-pytorch.*/blobs/sha256.*"
+    use_https: true
+
+# If an https request's host matches any of the hijacking rules, dfdaemon will
+# decrypt the request with given key pair and proxy it with the proxy rules.
+hijack_https:
+  # Cert and key of docker-registry.discovery.wmnet
+  cert: "/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chained.pem"
+  key: "/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet-key.pem"
+  hosts:
+    - regx: "docker-registry.discovery.wmnet"
+      # Puppet-ca here, for validation of the cert from source
+      certs: ["/etc/ssl/certs/wmf-ca-certificates.crt"]
+
+# Configure dfget to use a specific home directory (will be created if it does not exist).
+#
+# The IP used here is not only a listen IP but the IP annonced to the supernode. Using 0.0.0.0
+# leads to the network still functioning but every client will ask it's own dfget server for parts
+# which will make them fetch the parts from the docker-registry istead of peers.
+#
+# Port needs to be specified here as dfget will choose a random one (per invocation) if not.
+dfget_flags: ["--home", "/var/lib/dragonfly-dfdaemon/dfget", "--ip", "10.64.155.8", "--port", "15001"]
+
+# Network bandwith rate limit, will actually be used as "--locallimit" and "--totallimit" for dfget.
+# In format of G(B)/g/M(B)/m/K(B)/k/B, pure number will also be parsed as Byte.
+ratelimit: 100M

File[/etc/kubernetes/pki/mlserve__kubelet_server.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__kubelet_server.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__kubelet_server.pem]

+    owner  => kube
+    group  => root
+    mode   => 0440
+    ensure => file

File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem]

+    group  => amd-nodelabeller
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => amd-nodelabeller

Rsyslog::Conf[kubernetes]

Parameters differences:

--- Rsyslog::Conf[kubernetes].orig
+++ Rsyslog::Conf[kubernetes]

+    priority => 9
+    mode     => 0444
+    ensure   => present

File[/etc/rsyslog.d/08-input-file-kubernetes-json.conf]

Parameters differences:

--- File[/etc/rsyslog.d/08-input-file-kubernetes-json.conf].orig
+++ File[/etc/rsyslog.d/08-input-file-kubernetes-json.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/08-input-file-kubernetes-json.conf.orig
+++ /etc/rsyslog.d/08-input-file-kubernetes-json.conf
@@ -0,0 +1,8 @@
+# This file managed by puppet rsyslog::input::file
+
+input(type="imfile"
+      File="/var/log/containers/*.log"
+      reopenOnTruncate="on"
+      addMetadata="on"
+      addCeeTag="on"
+      Tag="input-file-kubernetes")

File[/lib/systemd/system/rsyslog-imfile-remedy.service]

Parameters differences:

--- File[/lib/systemd/system/rsyslog-imfile-remedy.service].orig
+++ File[/lib/systemd/system/rsyslog-imfile-remedy.service]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Exec[systemd daemon-reload for rsyslog-imfile-remedy.service (rsyslog-imfile-remedy.service)]

Content differences:

--- /lib/systemd/system/rsyslog-imfile-remedy.service.orig
+++ /lib/systemd/system/rsyslog-imfile-remedy.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Restart rsyslog T357616
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/systemctl try-restart rsyslog

K8s::Kubeconfig[/etc/amd/node-labeller-kubeconfig]

Parameters differences:

--- K8s::Kubeconfig[/etc/amd/node-labeller-kubeconfig].orig
+++ K8s::Kubeconfig[/etc/amd/node-labeller-kubeconfig]

+    auth_cert   => {'cert': '/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem', 'key': '/etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem', 'chain': '/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chain.pem', 'chained': '/etc/kubernetes/pki/mlserve__amdgpu-node-labeller.chained.pem'}
+    group       => amd-nodelabeller
+    mode        => 0400
+    ensure      => present
+    owner       => amd-nodelabeller
+    master_host => ml-ctrl.svc.eqiad.wmnet
+    require     => ['File[/etc/amd]', 'Class[K8s::Base_dirs]']
+    username    => amdgpu-node-labeller

Exec[renew certificate - mlserve__system_node_ml-serve1014_eqiad_wmnet]

Parameters differences:

--- Exec[renew certificate - mlserve__system_node_ml-serve1014_eqiad_wmnet].orig
+++ Exec[renew certificate - mlserve__system_node_ml-serve1014_eqiad_wmnet]

+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet

+    require     => Exec[Generate cert mlserve__system_node_ml-serve1014_eqiad_wmnet]
+    notify      => ['Service[kubelet]']
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']

File[/etc/sysctl.d/75-kube_proxy_conntrack.conf]

Parameters differences:

--- File[/etc/sysctl.d/75-kube_proxy_conntrack.conf].orig
+++ File[/etc/sysctl.d/75-kube_proxy_conntrack.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/75-kube_proxy_conntrack.conf.orig
+++ /etc/sysctl.d/75-kube_proxy_conntrack.conf
@@ -0,0 +1,2 @@
+# sysctl parameters managed by Puppet.
+net.netfilter.nf_conntrack_max = 1048576

Exec[apt_pin_apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]

Parameters differences:

--- Exec[apt_pin_apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo].orig
+++ Exec[apt_pin_apt_pin_firmware-amd-graphics-trixie-bpo_trixie-bpo]

+    refreshonly => True
+    command     => /usr/bin/apt-get update

Motd::Message[ml_k8s::worker]

Parameters differences:

--- Motd::Message[ml_k8s::worker].orig
+++ Motd::Message[ml_k8s::worker]

+    message  => ml-serve1014 is a ML Kubernetes worker node (ml_k8s::worker)
+    priority => 5
+    ensure   => present

Exec[create chained cert /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem].orig
+++ Exec[create chained cert /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]

+    subscribe => ['Exec[renew certificate - mlserve__system_kube-proxy]', 'File[/etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem]', 'File[/etc/kubernetes/pki/mlserve__system_kube-proxy.pem]']
+    command   => /bin/cat /etc/kubernetes/pki/mlserve__system_kube-proxy.pem /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem > /etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem
+    require   => Exec[Generate cert mlserve__system_kube-proxy refresh on intermediate ca change]
+    notify    => ['Service[kube-proxy]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/kubernetes/pki/mlserve__system_kube-proxy.pem /etc/kubernetes/pki/mlserve__system_kube-proxy.chain.pem | sha512sum)" == "$(/bin/cat /etc/kubernetes/pki/mlserve__system_kube-proxy.chained.pem | sha512sum)"

Exec[renew certificate - mlserve__istio-cni]

Parameters differences:

--- Exec[renew certificate - mlserve__istio-cni].orig
+++ Exec[renew certificate - mlserve__istio-cni]

+    require     => Exec[Generate cert mlserve__istio-cni]
+    unless      => /usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__istio-cni.pem -checkend 952200
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/kubernetes/pki/mlserve__istio-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__istio-cni

Service[kubelet]

Parameters differences:

--- Service[kubelet].orig
+++ Service[kubelet]

+    subscribe => ['File[/etc/kubernetes/kubelet.conf]']
+    enable    => True
+    ensure    => running

Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh on intermediate ca change].orig
+++ Exec[Generate cert discovery__ml-serve1014_eqiad_wmnet refresh on intermediate ca change]

+    subscribe   => File[/etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label discovery  /etc/cfssl/csr/discovery__ml-serve1014_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/dragonfly/discovery__ml-serve1014_eqiad_wmnet

+    notify      => ['Service[dragonfly-dfdaemon]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Class[Containerd::Nerdctl]

Parameters differences:

--- Class[Containerd::Nerdctl].orig
+++ Class[Containerd::Nerdctl]

+    ensure    => present
+    namespace => k8s.io

File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calico-cni.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => root

Systemd::Override[ferm-service-auto-restart]

Parameters differences:

--- Systemd::Override[ferm-service-auto-restart].orig
+++ Systemd::Override[ferm-service-auto-restart]

+    source  => puppet:///modules/profile/kubernetes/node/ferm_systemd_override
+    unit    => ferm
+    ensure  => present
+    restart => False

File[/var/lib/kubelet/config.json]

Parameters differences:

--- File[/var/lib/kubelet/config.json].orig
+++ File[/var/lib/kubelet/config.json]

+    group     => root
+    mode      => 0440
+    ensure    => present
+    owner     => root
+    show_diff => False

Content differences:

--- /var/lib/kubelet/config.json.orig
+++ /var/lib/kubelet/config.json
@@ -0,0 +1,7 @@
+{
+  "auths": {
+    "https://docker-registry.discovery.wmnet": {
+      "auth": "a3ViZXJuZXRlczpzb21lcGFzc3dvcmQy"
+    }
+  }
+}

File[/etc/default/kube-proxy]

Parameters differences:

--- File[/etc/default/kube-proxy].orig
+++ File[/etc/default/kube-proxy]

+    group  => root
+    mode   => 0644
+    ensure => file
+    owner  => root
+    notify => Service[kube-proxy]

Content differences:

--- /etc/default/kube-proxy.orig
+++ /etc/default/kube-proxy
@@ -0,0 +1,7 @@
+###
+# Kubernetes proxy config.
+
+# default config should be adequate
+
+DAEMON_ARGS="--config=/etc/kubernetes/kube-proxy-config.yaml \
+ --v=0"

File[/etc/kubernetes/pki/mlserve__calicoctl.chained.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__calicoctl.chained.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__calicoctl.chained.pem]

+    owner   => root
+    group   => root
+    ensure  => file
+    require => Exec[create chained cert /etc/kubernetes/pki/mlserve__calicoctl.chain.pem]

File[/etc/kubernetes/pki]

Parameters differences:

--- File[/etc/kubernetes/pki].orig
+++ File[/etc/kubernetes/pki]

+    owner  => root
+    group  => root
+    mode   => 0755
+    ensure => directory

Cfssl::Cert[mlserve__system_kube-proxy]

Parameters differences:

--- Cfssl::Cert[mlserve__system_kube-proxy].orig
+++ Cfssl::Cert[mlserve__system_kube-proxy]

+    outdir          => /etc/kubernetes/pki
+    names           => [{'organisation': 'system:node-proxier'}]
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    auto_renew      => True
+    provide_chain   => True
+    group           => root
+    ensure          => present
+    mode            => 0740
+    key             => {'algo': 'ecdsa', 'size': 256}
+    label           => mlserve
+    owner           => kube
+    before_services => []
+    renew_seconds   => 952200
+    common_name     => system:kube-proxy
+    hosts           => []
+    notify_services => ['kube-proxy']

Exec[Generate cert mlserve__calico-cni refresh]

Parameters differences:

--- Exec[Generate cert mlserve__calico-cni refresh].orig
+++ Exec[Generate cert mlserve__calico-cni refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__calico-cni.csr]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__calico-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__calico-cni

File[/etc/ferm/conf.d/10_dragonfly_dfget]

Parameters differences:

--- File[/etc/ferm/conf.d/10_dragonfly_dfget].orig
+++ File[/etc/ferm/conf.d/10_dragonfly_dfget]

+    group   => root
+    mode    => 0400
+    ensure  => present
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    owner   => root
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_dragonfly_dfget.orig
+++ /etc/ferm/conf.d/10_dragonfly_dfget
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 15001, $DOMAIN_NETWORKS);
+
+

Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.service (rsyslog-release-deleted-inotify-watches.service)]

Parameters differences:

--- Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.service (rsyslog-release-deleted-inotify-watches.service)].orig
+++ Exec[systemd daemon-reload for rsyslog-release-deleted-inotify-watches.service (rsyslog-release-deleted-inotify-watches.service)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Exec[Generate cert mlserve__system_kube-proxy refresh]

Parameters differences:

--- Exec[Generate cert mlserve__system_kube-proxy refresh].orig
+++ Exec[Generate cert mlserve__system_kube-proxy refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__system_kube-proxy.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__system_kube-proxy.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__system_kube-proxy

+    notify      => ['Service[kube-proxy]']
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']

Sysctl::Parameters[kube_proxy_conntrack]

Parameters differences:

--- Sysctl::Parameters[kube_proxy_conntrack].orig
+++ Sysctl::Parameters[kube_proxy_conntrack]

+    priority => 75
+    module   => nf_conntrack
+    values   => {'net.netfilter.nf_conntrack_max': 1048576}
+    ensure   => present

Class[Toil::Rsyslog_imfile_remedy]

Parameters differences:

--- Class[Toil::Rsyslog_imfile_remedy].orig
+++ Class[Toil::Rsyslog_imfile_remedy]

+    period_hours => 3
+    ensure       => present

Exec[Generate cert mlserve__amdgpu-node-labeller]

Parameters differences:

--- Exec[Generate cert mlserve__amdgpu-node-labeller].orig
+++ Exec[Generate cert mlserve__amdgpu-node-labeller]

+    require     => Cfssl::Csr[/etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr]
+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/kubernetes/pki/mlserve__amdgpu-node-labeller.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/kubernetes/pki/mlserve__amdgpu-node-labeller-key.pem 2>&1)"

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__amdgpu-node-labeller.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__amdgpu-node-labeller

File[/etc/cfssl/csr/mlserve__calico-cni.csr]

Parameters differences:

--- File[/etc/cfssl/csr/mlserve__calico-cni.csr].orig
+++ File[/etc/cfssl/csr/mlserve__calico-cni.csr]

+    owner  => root
+    group  => root
+    mode   => 0400
+    ensure => file

Content differences:

--- /etc/cfssl/csr/mlserve__calico-cni.csr.orig
+++ /etc/cfssl/csr/mlserve__calico-cni.csr
@@ -0,0 +1,13 @@
+{
+  "CN": "calico-cni",
+  "hosts": [
+    "calico-cni"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

File[/etc/nagios/nrpe.d/check_disk_space.cfg]

Content differences:

--- /etc/nagios/nrpe.d/check_disk_space.cfg.orig
+++ /etc/nagios/nrpe.d/check_disk_space.cfg
@@ -1,2 +1,2 @@
 # File generated by puppet. DO NOT edit by hand
-command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 10% -c 5% -W 6% -K 3% -l -e -A -i '/(var/lib|run)/(containerd|kubelet)/*' --exclude-type=tracefs

File[/var/run/kubernetes]

Parameters differences:

--- File[/var/run/kubernetes].orig
+++ File[/var/run/kubernetes]

+    owner  => root
+    group  => root
+    mode   => 0700
+    ensure => directory

Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat_fragment[component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01
+    tag    => _etc_apt_sources.list.d_component-kubernetes131-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

File[/etc/rsyslog.d/35-output-kafka-k8s.conf]

Parameters differences:

--- File[/etc/rsyslog.d/35-output-kafka-k8s.conf].orig
+++ File[/etc/rsyslog.d/35-output-kafka-k8s.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/35-output-kafka-k8s.conf.orig
+++ /etc/rsyslog.d/35-output-kafka-k8s.conf
@@ -0,0 +1,24 @@
+
+
+if ( $.log_outputs contains "k8s" ) then {
+    action(type="mmjsonparse" name="mmjsonparse_kafka_k8s")
+
+    action(type="omkafka"
+           name="omkafka_k8s"
+           broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1003.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093"]
+           topic="k8s-ml-serve-eqiad"
+           partitions.auto="on"
+           template="syslog_cee"
+           queue.type="LinkedList" queue.size="10000" queue.filename="output_kafka_k8s"
+           queue.highWatermark="7000" queue.lowWatermark="6000"
+           queue.checkpointInterval="5"
+           queue.maxDiskSpace="40960000"
+           confParam=[ "security.protocol=ssl",
+                       "ssl.ca.location=/etc/ssl/certs/wmf-ca-certificates.crt",
+                       "compression.codec=snappy",
+                       "socket.timeout.ms=10000",
+                       "socket.keepalive.enable=true",
+                       "queue.buffering.max.ms=50",
+                       "batch.num.messages=1000" ]
+    )
+}

Systemd::Unit[kubelet-container-runtime]

Parameters differences:

--- Systemd::Unit[kubelet-container-runtime].orig
+++ Systemd::Unit[kubelet-container-runtime]

+    override          => True
+    override_filename => container-runtime
+    restart           => True
+    ensure            => present
+    require           => ['Class[Systemd]']
+    unit              => kubelet

File[/etc/rsyslog.d/09-kubernetes.conf]

Parameters differences:

--- File[/etc/rsyslog.d/09-kubernetes.conf].orig
+++ File[/etc/rsyslog.d/09-kubernetes.conf]

+    group  => root
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/09-kubernetes.conf.orig
+++ /etc/rsyslog.d/09-kubernetes.conf
@@ -0,0 +1,9 @@
+module(load="mmkubernetes"
+        KubernetesURL="https://ml-ctrl.svc.eqiad.wmnet:6443"
+        tls.mycert="/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog.pem"
+        tls.myprivkey="/etc/cfssl/ssl/mlserve__rsyslog/mlserve__rsyslog-key.pem")
+action(type="mmkubernetes"
+       name="mmkubernetes"
+       action.resumeRetryCount="-1"
+       action.resumeIntervalMax="300"
+       action.reportSuspensionContinuation="on")

Package[crictl]

Parameters differences:

--- Package[crictl].orig
+++ Package[crictl]

+    ensure   => installed
+    provider => apt

Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

Parameters differences:

--- Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header].orig
+++ Concat_fragment[component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order  => 01
+    tag    => _etc_apt_sources.list.d_component-calico329-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources

File[/etc/rsyslog.d/20-shellbox.conf]

Parameters differences:

--- File[/etc/rsyslog.d/20-shellbox.conf].orig
+++ File[/etc/rsyslog.d/20-shellbox.conf]

+    group  => root
+    source => puppet:///modules/profile/rsyslog/shellbox.rsyslog.conf
+    mode   => 0444
+    ensure => present
+    owner  => root
+    notify => Service[rsyslog]

Logrotate::Conf[rsyslog-release-deleted-inotify-watches]

Parameters differences:

--- Logrotate::Conf[rsyslog-release-deleted-inotify-watches].orig
+++ Logrotate::Conf[rsyslog-release-deleted-inotify-watches]

+    ensure => absent

Exec[Generate cert mlserve__istio-cni refresh]

Parameters differences:

--- Exec[Generate cert mlserve__istio-cni refresh].orig
+++ Exec[Generate cert mlserve__istio-cni refresh]

+    subscribe   => File[/etc/cfssl/csr/mlserve__istio-cni.csr]
+    refreshonly => True
+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/ml-serve1014.eqiad.wmnet.pem -label mlserve  /etc/cfssl/csr/mlserve__istio-cni.csr | /usr/bin/cfssljson -bare /etc/kubernetes/pki/mlserve__istio-cni

Class[Profile::Amd_gpu]

Parameters differences:

--- Class[Profile::Amd_gpu].orig
+++ Class[Profile::Amd_gpu]

+    kubernetes_cluster_name => ml-serve-eqiad
@@
-    is_kubernetes_node      => False
+    is_kubernetes_node      => True
@@
-    firmwares_from_bpo      => False
+    firmwares_from_bpo      => True
@@
-    is_basic_gpu_node       => True
+    is_basic_gpu_node       => False

File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet.chain.pem]

+    group  => root
+    mode   => 0440
+    ensure => file
+    source => puppet:///modules/profile/pki/intermediates/mlserve-cert.pem
+    owner  => kube

Class[Adduser]

Parameters differences:

--- Class[Adduser].orig
+++ Class[Adduser]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[starship]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[linux-sysctl-defaults]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[atop]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[rocm-smi]', 'Package[python3-requests]', 'Package[firmware-amd-graphics]', 'Package[ruby-concurrent]', 'Package[ruby]', 'Package[libruby]', 'Package[puppet-agent]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[starship]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[linux-sysctl-defaults]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[atop]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[dragonfly-dfdaemon]', 'Package[dragonfly-dfget]', 'Package[crictl]', 'Package[containerd]', 'Package[nerdctl]', 'Package[rsyslog-kubernetes]', 'Package[linux-cpupower]', 'Package[apparmor]', 'Package[socat]', 'Package[amd-k8s-device-plugin]', 'Package[amd-k8s-node-labeller]', 'Package[rocm-smi]', 'Package[python3-requests]', 'Package[wikimedia-lvs-realserver]', 'Package[ruby-concurrent]', 'Package[ruby]', 'Package[libruby]', 'Package[puppet-agent]', 'Package[linux-image-6.16.3+deb13-amd64]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[kubernetes-node]', 'Package[calicoctl]', 'Package[calico-cni]', 'Package[istio-cni]', 'Package[firmware-amd-graphics]']

File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem]

Parameters differences:

--- File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem].orig
+++ File[/etc/kubernetes/pki/mlserve__system_node_ml-serve1014_eqiad_wmnet-key.pem]

+    group     => root
+    mode      => 0440
+    ensure    => file
+    owner     => kube
+    show_diff => False
+    backup    => False

Service[rsyslog-imfile-remedy.timer]

Parameters differences:

--- Service[rsyslog-imfile-remedy.timer].orig
+++ Service[rsyslog-imfile-remedy.timer]

+    enable   => True
+    provider => systemd
+    ensure   => running

File[/etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf]

Parameters differences:

--- File[/etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf].orig
+++ File[/etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf.orig
+++ /etc/sysctl.d/70-ipv6-fowarding-accept-ra.conf
@@ -0,0 +1,3 @@
+# sysctl parameters managed by Puppet.
+net.ipv6.conf.all.forwarding = 1
+net.ipv6.conf.ens51f1np1.accept_ra = 2

File[/etc/cni/net.d/10-calico.conflist]

Parameters differences:

--- File[/etc/cni/net.d/10-calico.conflist].orig
+++ File[/etc/cni/net.d/10-calico.conflist]

+    owner => root
+    group => root
+    mode  => 0755

Content differences:

--- /etc/cni/net.d/10-calico.conflist.orig
+++ /etc/cni/net.d/10-calico.conflist
@@ -0,0 +1,39 @@
+{
+  "name": "k8s-pod-network",
+  "cniVersion": "0.3.1",
+  "plugins": [
+    {
+      "type": "calico",
+      "log_level": "info",
+      "datastore_type": "kubernetes",
+      "mtu": 1460,
+      "ipam": {
+        "type": "calico-ipam",
+        "assign_ipv4": "true",
+        "assign_ipv6": "true"
+      },
+      "policy": {
+        "type": "k8s"
+      },
+      "kubernetes": {
+        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
+      }
+    },
+    {
+      "name": "istio-cni",
+      "type": "istio-cni",
+      "log_level": "info",
+      "kubernetes": {
+        "kubeconfig": "/etc/cni/net.d/istio-kubeconfig",
+        "cni_bin_dir": "/opt/cni/bin",
+        "exclude_namespaces": [
+          "istio-system",
+          "kube-system",
+          "knative-serving",
+          "cert-manager",
+          "kserve"
+        ]
+      }
+    }
+  ]
+}

Concat[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources].orig
+++ Concat[/etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources]

+    group          => root
+    mode           => 0444
+    ensure         => present
+    owner          => root
+    backup         => puppet
+    format         => plain
+    path           => /etc/apt/sources.list.d/component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia.sources
+    order          => alpha
+    replace        => True
+    ensure_newline => False
+    notify         => Exec[apt_repository_component-istio115-apt.wikimedia.org-wikimedia-trixie-wikimedia]
+    show_diff      => True
+    warn           => False
+    force          => False

Concat_fragment[main contacts]

Content differences:

--- main contacts.orig
+++ main contacts
@@ -1,3 +1,3 @@
 ---
-role::ml_k8s::insetup_gpu:
+role::ml_k8s::worker:
 - Machine Learning

Compilation results for ml-serve1014.eqiad.wmnet: System changes detected

Catalog differences

Summary

Resources only in the new catalog

Resources only in the old catalog

Resources modified

Relevant files