Compilation results for relforge1008.eqiad.wmnet: System changes detected

You can retrieve this result from host.json.

Catalog differences

Summary

Total Resources:	2900
Resources added:	634
Resources removed:	8
Resources modified:	646
Change percentage:	44.41%

Resources only in the new catalog

Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Class[Profile::Java]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]
Monitoring::Service[check_push_cross_cluster_settings_9200_status]
Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]
Rsyslog::Conf[opensearch-disable-readahead]
Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]
File[/etc/apt/sources.list.d/wikimedia-opensearch.list]
Rsyslog::Conf[push_cross_cluster_settings_9200]
Logrotate::Rule[opensearch]
File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]
Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Service[opensearch-disable-readahead-relforge-eqiad.timer]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]
Ferm::Service[opensearch-https-9443]
Nrpe::Check[check_opensearch_shard_size_check_9400]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]
Class[Profile::Opensearch::Server]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]
Package[prometheus-elasticsearch-exporter]
Systemd::Unit[opensearch-hot-threads-log.service]
Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]
Monitoring::Service[opensearch_unassigned_shard_check_9400]
Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
File[/etc/opensearch]
Systemd::Unit[push_cross_cluster_settings_9200.timer]
Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]
File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]
Exec[mask_default_prometheus_elasticsearch_exporter]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]
Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]
Concat::Fragment[wikimedia-opensearch-plugins]
File[/etc/opensearch/relforge-eqiad/opensearch.keystore]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]
File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]
Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]
File[/var/log/push_cross_cluster_settings_9200]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]
Systemd::Syslog[clean_up_dlq]
Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]
Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
File[/etc/logstash/logstash.yml]
Systemd::Service[push_cross_cluster_settings_9400]
Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]
Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]
File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]
Systemd::Unit[prometheus-elasticsearch-exporter-9200]
Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]
Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
File[/etc/update-motd.d/96-cluster-memberships]
Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Nginx::Site[relforge-eqiad-small-alpha]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]
Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
File[/usr/local/bin/jheapdump]
Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]
File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]
Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
Systemd::Timer[push_cross_cluster_settings_9200]
File[/var/log/opensearch-disable-readahead]
Tlsproxy::Localssl[relforge-eqiad-small-alpha]
File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]
File[/var/log/opensearch]
Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]
File[/etc/sysctl.d/70-TCP-Fast-Open.conf]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]
Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]
Sysctl::Parameters[opensearch]
Java::Package[openjdk-jdk-17]
Exec[apt_repository_wikimedia-opensearch-plugins]
Prometheus::Elasticsearch_exporter[localhost:9200]
File[/etc/nginx/sites-available/relforge-eqiad]
Java::Cacert[wmf:puppetca.pem]
File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]
Opensearch::Instance[relforge-eqiad-small-alpha]
Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]
File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]
File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]
Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
Monitoring::Service[opensearch_unassigned_shard_check_9200]
Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]
Package[openjdk-17-jdk]
File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]
Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]
File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]
Service[nrpe2nodexp-opensearch_shards_9400.timer]
Sysctl::Conffile[TCP Fast Open]
File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]
File[/etc/ferm/conf.d/10_opensearch-https-9443]
Nrpe::Check[check_opensearch_shards_9400]
File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]
File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]
Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
Opensearch::Log::Hot_threads_cluster[relforge-eqiad]
Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Service[prometheus-wmf-elasticsearch-exporter-9200]
File[/etc/logrotate.d/nginx]
Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]
Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]
Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Service[nginx]
Nrpe::Monitor_service[opensearch_shards_9400]
File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]
Systemd::Unit[clean_up_dlq.service]
Exec[tlsproxy localssl default_server on ports [9243]]
Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]
File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]
Rsyslog::Conf[clean_up_dlq]
Service[prometheus-elasticsearch-exporter-9200]
Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]
Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]
File[/etc/opensearch/relforge-eqiad-small-alpha]
Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
File[/usr/local/bin/logstash-config-test]
Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]
Systemd::Timer::Job[opensearch-disable-readahead]
Service[logstash]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]
File[/etc/logrotate.d/push_cross_cluster_settings_9200]
File[/lib/systemd/system/opensearch_1@.service]
File[/etc/opensearch/relforge-eqiad/scripts]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]
Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]
File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]
Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]
File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]
Sudo::User[nrpe-check_opensearch_shard_size_check_9200]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]
File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]
Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]
File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]
File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]
Service[prometheus-wmf-elasticsearch-exporter-9400]
File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]
File[/etc/ferm/conf.d/10_opensearch-https-9243]
Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]
File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]
Motd::Script[cirrus::relforge]
Systemd::Timer::Job[clean_up_dlq]
Systemd::Unit[opensearch-hot-threads-log.timer]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Class[Profile::Opensearch::Cirrus::Server]
Monitoring::Service[opensearch_shards_9400]
Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]
File[/etc/default/opensearch]
Package[opensearch]
Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]
Systemd::Unit[push_cross_cluster_settings_9400.timer]
File[/etc/logrotate.d/opensearch]
Monitoring::Service[opensearch_shard_size_check_9400]
File[/lib/systemd/system/opensearch-disable-readahead.timer]
Service[prometheus-elasticsearch-exporter-9400]
Logstash::Output::Udp[gelf_relay]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
Sysctl::Conffile[opensearch]
Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]
Elasticsearch::Tlsproxy[relforge-eqiad]
Service[prometheus-elasticsearch-exporter]
Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]
Systemd::Service[opensearch-disable-readahead-relforge-eqiad]
Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Exec[java__cacert_wmf:puppetca.pem]
File[/etc/rsyslog.d/40-clean-up-dlq.conf]
File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]
Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]
Exec[unmask_nginx.service]
Monitoring::Service[opensearch_shards_9200]
File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]
Class[Profile::Tlsproxy::Instance]
Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]
Nrpe::Monitor_service[opensearch_shards_9200]
File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]
Service[opensearch-hot-threads-log.timer]
File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]
Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]
File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]
Service[push_cross_cluster_settings_9400.timer]
File[/lib/systemd/system/clean_up_dlq.service]
Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]
File[/etc/ferm/conf.d/10_elastic-main-https-9243]
File[/lib/systemd/system/opensearch-disable-readahead.service]
Package[nginx-light]
File[/etc/logrotate.d/push_cross_cluster_settings_9400]
Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]
File[/etc/systemd/system/nginx.service.d/security.conf]
File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]
File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Rsyslog::Conf[udp_json_logback_compat]
Systemd::Monitor[push_cross_cluster_settings_9200]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
File[/etc/systemd/system/nginx.service.d/numa.conf]
Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
File[/usr/local/bin/opensearch_hot_threads_logger.py]
Mount[/var/lib/nginx]
Systemd::Unit[opensearch-disable-readahead.service]
Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]
File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]
Package[edac-utils]
Concat::Fragment[wikimedia-opensearch-plugins-header]
File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]
File[/etc/logrotate.d/clean_up_dlq]
Apt::Repository[wikimedia-opensearch]
Systemd::Syslog[opensearch-disable-readahead]
Systemd::Timer[opensearch-disable-readahead]
Class[Nginx]
File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]
Systemd::Timer[opensearch-hot-threads-log]
File[/srv/opensearch]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
Logstash::Conf[input-gelf-gelf_relay]
File[/etc/sysctl.d/opensearch.conf]
File[/usr/local/bin/cleanup-dlq]
File[/etc/opensearch/instances]
File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]
Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]
File[/etc/opensearch/relforge-eqiad/logging.yml]
Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]
File[/etc/opensearch/relforge-eqiad/log4j2.properties]
File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
File[/srv/opensearch/relforge-eqiad-small-alpha]
Class[Profile::Opensearch::Monitoring::Base_checks]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]
Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]
File[/etc/opensearch_hot_threads.d]
File[/etc/nginx/sites-available]
File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]
File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]
Nrpe::Monitor_service[opensearch_shard_size_check_9200]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]
Package[opensearch-madvise]
Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]
Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]
Systemd::Unit[prometheus-elasticsearch-exporter-9400]
Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]
Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]
Exec[opensearch-create-keystore-relforge-eqiad]
Logrotate::Conf[opensearch]
Service[opensearch]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]
File[/etc/opensearch/relforge-eqiad/java-security.policy]
Logrotate::Conf[push_cross_cluster_settings_9400]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
File[/etc/elasticsearch]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]
Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]
Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]
Class[Profile::Rsyslog::Udp_json_logback_compat]
Logrotate::Conf[clean_up_dlq]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]
Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
File[/var/log/clean_up_dlq]
File[/usr/lib/nagios/plugins/check_elasticsearch]
Sysctl::Parameters[TCP Fast Open]
File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]
Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]
Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
File[/var/log/push_cross_cluster_settings_9400]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]
Apt::Repository[wikimedia-opensearch-plugins]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]
Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]
Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]
File[/var/log/opensearch/opensearch_index_search_slowlog.log]
File[/lib/systemd/system/opensearch-hot-threads-log.service]
File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]
Package[nginx-common]
File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]
Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Ferm::Service[opensearch-https-9243]
Systemd::Unit[clean_up_dlq.timer]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]
Class[Profile::Tcp_fast_open]
Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]
Systemd::Mask[nginx.service]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]
File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]
Nrpe::Monitor_service[opensearch_shard_size_check_9400]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]
Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]
Concat_fragment[wikimedia-opensearch-header]
Motd::Script[cluster_memberships]
Systemd::Service[nrpe2nodexp-opensearch_shards_9200]
Class[Opensearch]
Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Systemd::Service[prometheus-elasticsearch-exporter-9200]
Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]
File[/etc/opensearch/relforge-eqiad/opensearch.yml]
File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]
Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]
File[/etc/ssl/dhparam.pem]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]
Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]
File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]
File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]
Logrotate::Conf[opensearch-disable-readahead]
Concat::Fragment[wikimedia-opensearch-header]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]
File[/etc/opensearch/relforge-eqiad/jvm.options]
File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]
Class[Java::Tools]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]
Nrpe::Check[check_opensearch_unassigned_shard_check_9200]
Logstash::Input::Gelf[gelf_relay]
Nrpe::Plugin[check_systemd_unit_status]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
Package[logstash-oss]
Class[Java]
File[/lib/systemd/system/clean_up_dlq.timer]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
Class[Profile::Logstash::Gelf_relay]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]
Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]
Nrpe::Check[check_opensearch_shard_size_check_9200]
Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]
Systemd::Syslog[push_cross_cluster_settings_9200]
File[/etc/logstash/jvm.options]
File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]
File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]
Class[Prometheus::Elasticsearch_exporter::Common]
Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
File[/etc/opensearch/relforge-eqiad]
File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]
Class[Prometheus::Wmf_elasticsearch_exporter::Common]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
Monitoring::Service[elasticsearch-https-relforge-eqiad]
Service[nrpe2nodexp-opensearch_shards_9200.timer]
Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]
Systemd::Timer[push_cross_cluster_settings_9400]
Class[Opensearch::Packages]
Class[Icinga::Elasticsearch::Base_plugin]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]
Exec[s3-credentials-user-relforge-eqiad]
Exec[Refresh tmpfile opensearch-relforge-eqiad]
Class[Logstash]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]
Opensearch::Cross_cluster_settings[relforge-eqiad]
File[/srv/opensearch/relforge-eqiad]
Systemd::Unit[push_cross_cluster_settings_9200.service]
Exec[stop-default-nginx]
Systemd::Service[nrpe2nodexp-opensearch_shards_9400]
File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
File[/usr/local/bin/opensearch-disable-readahead.sh]
Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]
File[/etc/logrotate.d/opensearch-disable-readahead]
Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]
Service[clean_up_dlq.timer]
Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]
Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]
File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]
Systemd::Service[clean_up_dlq]
Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]
File[/etc/udev/rules.d/40-opensearch-readahead.rules]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]
Concat::Fragment[wikimedia-opensearch]
File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]
File[/etc/nginx/nginx.conf]
Service[opensearch_1@relforge-eqiad]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]
Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]
File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]
Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]
Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]
Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
Systemd::Unit[opensearch-disable-readahead.timer]
Concat_fragment[wikimedia-opensearch-plugins-header]
Package[wmf-opensearch-search-plugins]
Nrpe::Check[check_opensearch_shards_9200]
Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]
Nginx::Site[relforge-eqiad]
Prometheus::Elasticsearch_exporter[localhost:9400]
File[/var/log/opensearch/opensearch.log]
Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]
Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]
File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]
Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]
Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]
Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]
Package[logstash-old-name]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]
Systemd::Timer::Job[opensearch-hot-threads-log]
Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Service[push_cross_cluster_settings_9200.timer]
Concat_fragment[wikimedia-opensearch]
Ferm::Service[elastic-small-alpha-https-9443]
Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]
Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]
Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]
Systemd::Monitor[push_cross_cluster_settings_9400]
File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]
Systemd::Service[opensearch-disable-readahead]
File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]
Systemd::Timer[clean_up_dlq]
File[/etc/nginx]
Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]
Ferm::Service[opensearch_inter_node_9500]
Exec[update_java_alternatives_17]
Systemd::Unmask[nginx.service]
Exec[nginx-reload]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
File[/etc/default/logstash]
Concat_fragment[wikimedia-opensearch-plugins]
File[/var/lib/nginx]
Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]
Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]
File[/etc/logstash/log4j2.properties]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
Sudo::User[nrpe-check_opensearch_shard_size_check_9400]
Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]
Nrpe::Check[check_opensearch_unassigned_shard_check_9400]
Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]
Opensearch::Keystore[s3-credentials-user-relforge-eqiad]
Logrotate::Conf[nginx]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]
Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]
Tlsproxy::Localssl[relforge-eqiad]
Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]
Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]
Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
Exec[s3-credentials-user-relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]
File[/etc/update-motd.d/05-cirrus--relforge]
File[/etc/default/prometheus-elasticsearch-exporter]
File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]
Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]
Exec[systemd reload for nginx systemd fragments]
Sudo::User[nrpe-check_opensearch_shards_9400]
File[/lib/systemd/system/push_cross_cluster_settings_9200.service]
Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]
File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]
Ferm::Service[elastic-main-https-9243]
File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]
File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]
Motd::Message[cirrus::relforge]
Systemd::Timer::Job[push_cross_cluster_settings_9200]
File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]
File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]
Systemd::Syslog[push_cross_cluster_settings_9400]
File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Timer::Job[push_cross_cluster_settings_9400]
Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]
File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]
Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]
Service[opensearch-disable-readahead.timer]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]
File[/lib/systemd/system/push_cross_cluster_settings_9400.service]
File[/usr/share/opensearch/plugins]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]
Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
File[/etc/nginx/sites-enabled/relforge-eqiad]
File[/etc/nginx/conf.d]
Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]
Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]
Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]
Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]
Class[Profile::Opensearch::Cirrus::Relforge]
Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Alternatives::Java[17]
Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]
File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]
File[/var/log/opensearch-disable-readahead-relforge-eqiad]
Logstash::Conf[output-udp-gelf_relay]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
Class[Opensearch::Log::Hot_threads]
File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]
Monitoring::Service[check_push_cross_cluster_settings_9400_status]
Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]
Opensearch::Instance[relforge-eqiad]
File[/etc/nginx/sites-enabled]
Package[python3-requests]
Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]
File[/etc/logstash/conf.d/README]
Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]
File_line[opensearch_bin_bashrc]
Systemd::Service[prometheus-elasticsearch-exporter-9400]
Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
Service[opensearch_1@relforge-eqiad-small-alpha]
Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]
Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]
Firewall::Service[opensearch-inter-node-9500]
Class[Role::Cirrus::Relforge]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
File[/usr/lib/nagios/plugins/check_elasticsearch.py]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
Monitoring::Service[opensearch_shard_size_check_9200]
Systemd::Tmpfile[opensearch-relforge-eqiad]
Exec[mask_nginx.service]
File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]
Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]
File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
Java::Cacert[wmf:Wikimedia_Internal_Root_CA]
Systemd::Unit[opensearch_1@.service]
File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]
Exec[apt_repository_wikimedia-opensearch]
Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]
Systemd::Unit[push_cross_cluster_settings_9400.service]
Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Sudo::User[nrpe-check_opensearch_shards_9200]
Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]
Rsyslog::Conf[push_cross_cluster_settings_9400]
Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
File[/etc/systemd/system/nginx.service.d]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]
Systemd::Service[push_cross_cluster_settings_9200]
Exec[tlsproxy localssl default_server on ports [9443]]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]
Logrotate::Conf[push_cross_cluster_settings_9200]
File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]
File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]
File[/lib/systemd/system/opensearch-hot-threads-log.timer]
Systemd::Service[opensearch-hot-threads-log]
Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]
Class[Sslcert::Dhparam]
Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]
File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]
Udev::Rule[opensearch-readahead]
Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]
Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Resources only in the old catalog

Class[Role::Cirrus::Test]
Ferm::Service[opensearch_http_9200]
Motd::Message[cirrus::test]
Firewall::Service[opensearch-http-9200]
File[/etc/update-motd.d/05-cirrus--test]
Class[Profile::Opensearch::Cirrus::Test]
File[/etc/ferm/conf.d/10_opensearch_http_9200]
Motd::Script[cirrus::test]

Resources modified

Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Systemd::Unit[opensearch-hot-threads-log.service]

Parameters differences:

--- Systemd::Unit[opensearch-hot-threads-log.service].orig
+++ Systemd::Unit[opensearch-hot-threads-log.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-hot-threads-log.service

Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad-small-alpha /srv/opensearch

Class[Profile::Apt]

Parameters differences:

--- Class[Profile::Apt].orig
+++ Class[Profile::Apt]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[edac-utils]', 'Package[opensearch]', 'Package[python3-requests]', 'Package[wmf-opensearch-search-plugins]', 'Package[opensearch-madvise]', 'Package[logstash-oss]', 'Package[logstash-old-name]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[openjdk-17-jdk]', 'Package[prometheus-elasticsearch-exporter]', 'Package[nginx-light]', 'Package[nginx-common]']

File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]

Parameters differences:

--- File[/usr/local/lib/nagios/plugins/check_systemd_unit_status].orig
+++ File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]

+    owner   => root
+    source  => puppet:///modules/systemd/check_systemd_unit_status
+    require => File[/usr/local/lib/nagios/plugins/]
+    group   => root
+    tag     => nrpe::plugin
+    ensure  => file
+    mode    => 0555

Nginx::Site[relforge-eqiad-small-alpha]

Parameters differences:

--- Nginx::Site[relforge-eqiad-small-alpha].orig
+++ Nginx::Site[relforge-eqiad-small-alpha]

+    require => Exec[tlsproxy localssl default_server on ports [9443]]
+    ensure  => present

Sysctl::Parameters[opensearch]

Parameters differences:

--- Sysctl::Parameters[opensearch].orig
+++ Sysctl::Parameters[opensearch]

+    ensure             => present
+    no_priority_prefix => True
+    priority           => 70
+    values             => {'vm.max_map_count': 1048576}

Java::Package[openjdk-jdk-17]

Parameters differences:

--- Java::Package[openjdk-jdk-17].orig
+++ Java::Package[openjdk-jdk-17]

+    package_info => {'version': '17', 'variant': 'jdk'}
+    egd_source   => /dev/random
+    hardened_tls => False

Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400].orig
+++ Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

+    prometheus_port    => 9121
+    indices_to_monitor => []
+    elasticsearch_port => 9400

Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Service[prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => running
+    enable => True

Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status].orig
+++ Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]

+    check_command          => nrpe_check!check_check_push_cross_cluster_settings_9200_status!10
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => admins
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 10
+    notifications_enabled  => 0
+    max_check_attempts     => 2
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => Check unit status of push_cross_cluster_settings_9200
+    notes_url              => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Systemd::Unit[clean_up_dlq.service]

Parameters differences:

--- Systemd::Unit[clean_up_dlq.service].orig
+++ Systemd::Unit[clean_up_dlq.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => clean_up_dlq.service

Class[Profile::Base::Production]

Parameters differences:

--- Class[Profile::Base::Production].orig
+++ Class[Profile::Base::Production]

@@
-    role_description => Unpuppetised system for testing
+    role_description => Elasticsearch relforge

Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    send_mail                 => False
+    user                      => root
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9400
+    private_tmp               => False
+    description               => Auto restart job: prometheus-wmf-elasticsearch-exporter-9400
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- Service[opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

File[/etc/ferm/conf.d/10_opensearch-https-9243]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch-https-9243].orig
+++ File[/etc/ferm/conf.d/10_opensearch-https-9243]

+    owner   => root
+    group   => root
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    mode    => 0400
+    ensure  => present
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_opensearch-https-9243.orig
+++ /etc/ferm/conf.d/10_opensearch-https-9243
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9243, (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS)));
+
+

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    ensure => present

Service[prometheus-elasticsearch-exporter]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter].orig
+++ Service[prometheus-elasticsearch-exporter]

+    ensure  => stopped
+    require => Package[prometheus-elasticsearch-exporter]

File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch_inter_node_9500].orig
+++ File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]

+    owner   => root
+    group   => root
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    mode    => 0400
+    ensure  => present
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_opensearch_inter_node_9500.orig
+++ /etc/ferm/conf.d/10_opensearch_inter_node_9500
@@ -0,0 +1,8 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9500, (10.64.164.14 10.64.32.100 10.64.48.120 2620:0:861:103:10:64:32:100 2620:0:861:107:10:64:48:120 2620:0:861:12c:10:64:164:14));
+
+
+
+&NO_TRACK(tcp, 9500);

File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]

+    owner  => root
+    group  => root
+    ensure => absent

File[/usr/local/bin/opensearch_hot_threads_logger.py]

Parameters differences:

--- File[/usr/local/bin/opensearch_hot_threads_logger.py].orig
+++ File[/usr/local/bin/opensearch_hot_threads_logger.py]

+    mode   => 0555
+    source => puppet:///modules/opensearch/opensearch_hot_threads_logger.py
+    owner  => root
+    group  => root

Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "2a63eb7dcbe368b7f538118765d629d6" --timeout 10 --check-command "check_opensearch_shards_9400"
+    private_tmp               => False
+    splay                     => 60
+    description               => execution of nrpe2nodexp for the check_opensearch_shards_9400 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_shards_9400
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Package[edac-utils]

Parameters differences:

--- Package[edac-utils].orig
+++ Package[edac-utils]

+    provider => apt
+    ensure   => latest

Systemd::Timer[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Timer[opensearch-hot-threads-log].orig
+++ Systemd::Timer[opensearch-hot-threads-log]

+    unit_name          => opensearch-hot-threads-log.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* *:00/5:00'}]
+    accuracy           => 15sec

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service

File[/etc/opensearch/instances]

Parameters differences:

--- File[/etc/opensearch/instances].orig
+++ File[/etc/opensearch/instances]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/instances.orig
+++ /etc/opensearch/instances
@@ -0,0 +1,2 @@
+opensearch_1@relforge-eqiad
+opensearch_1@relforge-eqiad-small-alpha

File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shard_size_check_9400]=/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4

Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/bin/find /var/log/opensearch -name 'relforge-eqiad_jvm_gc.*.log*' -mtime +30 -delete
+    private_tmp               => False
+    description               => Cleanup GC logs
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_unassigned_shard_check_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "a0a4892478c289e50484ffd2fb4c3202" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9400"

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3].orig
+++ Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]

+    for                => 720m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_unassigned_shard_check___9200
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch unassigned shard check - 9200
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="54ca99505516d5452912b573aa7d35d3",check_name="check_opensearch_unassigned_shard_check_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_unassigned_shard_check_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch unassigned shard check - 9200

Class[Profile::Rsyslog::Udp_json_logback_compat]

Parameters differences:

--- Class[Profile::Rsyslog::Udp_json_logback_compat].orig
+++ Class[Profile::Rsyslog::Udp_json_logback_compat]

+    logging_kafka_brokers => ['kafka-logging1001.eqiad.wmnet:9093', 'kafka-logging1002.eqiad.wmnet:9093', 'kafka-logging1003.eqiad.wmnet:9093', 'kafka-logging1004.eqiad.wmnet:9093', 'kafka-logging1005.eqiad.wmnet:9093']
+    queue_enabled_sites   => ['ulsfo', 'esams', 'eqsin', 'eqiad', 'codfw', 'drmrs', 'magru']
+    port                  => 11514

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/usr/lib/nagios/plugins/check_elasticsearch]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch]

+    owner  => root
+    mode   => 0755
+    group  => root
+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shard_size_check_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-check_push_cross_cluster_settings_9200_status" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Ferm::Service[opensearch-https-9243]

Parameters differences:

--- Ferm::Service[opensearch-https-9243].orig
+++ Ferm::Service[opensearch-https-9243]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => False
+    srange              => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    prio                => 10
+    desc                => 
+    port                => 9243
+    proto               => tcp

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/scripts].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => directory

File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]

+    owner  => opensearch
+    source => puppet:///modules/profile/opensearch/java-security.policy
+    group  => opensearch
+    ensure => file
+    mode   => 0440

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem 2>&1)"

+    require     => Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/nginx/sites-available/relforge-eqiad-small-alpha].orig
+++ File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]

+    before => ['Service[nginx]']
+    owner  => root
+    group  => root
+    ensure => present
+    notify => Exec[nginx-reload]

Content differences:

--- /etc/nginx/sites-available/relforge-eqiad-small-alpha.orig
+++ /etc/nginx/sites-available/relforge-eqiad-small-alpha
@@ -0,0 +1,59 @@
+# Proxy site configuration file for relforge-eqiad-small-alpha
+# This file is managed by Puppet!
+upstream local_fe_relforge-eqiad-small-alpha {
+    server 10.64.32.100:9400 max_fails=0;
+}
+
+# SSL proxying
+server {
+    # Enabling TCP Fast Open is safe for HTTP over TLS. There is no idempotency
+    # concern replaying TLS Client Hello.
+    # https://tools.ietf.org/html/rfc7413#section-6.3.2
+    listen [::]:9443 default_server deferred backlog=16384 reuseport ipv6only=on fastopen=150 ssl ;
+    listen 9443 default_server deferred backlog=16384 reuseport fastopen=150 ssl ;
+    server_name relforge1008.eqiad.wmnet relforge.svc.eqiad.wmnet;
+
+    error_log   /var/log/nginx/relforge-eqiad-small-alpha.error.log;
+    access_log   off;
+
+    ssl_certificate /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem;
+    ssl_certificate_key /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem;
+
+    keepalive_timeout 60;
+    keepalive_requests 100;
+
+    location / {
+        proxy_pass http://local_fe_relforge-eqiad-small-alpha;
+        proxy_http_version 1.1;
+        proxy_set_header Connection close; # should be default
+
+        # this should be in sync with Varnish's first_byte_timeout
+        # and PHP's max_execution_time
+        proxy_read_timeout 180s;
+
+        proxy_set_header Host $host;
+        proxy_set_header X-Client-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Connection-Properties "H2=$h2; SSR=$session_reused; SSL=$ssl_protocol; C=$ssl_cipher;";
+
+        # Don't clobber the Server header from the backend.
+        proxy_pass_header Server;
+
+        proxy_redirect off;
+        proxy_buffering off;
+        proxy_request_buffering on;
+
+        # With proxy buffering off, proxy_buffer_size is still used to
+        # define the singular chunk of buffer the content passes
+        # through.  Apparently if nginx can't read all the response
+        # headers in the first such chunk, it fails with 502.  Upping
+        # this to 8k (from def 4k) solves the one reported case in
+        # https://phabricator.wikimedia.org/T161819
+        # --
+        # Raising this further from 8K to 32K, at least temporarily, due to T226840
+        # Note 32KB is also the similar default limit in our varnishes (http_resp_size)
+        proxy_buffer_size       32k;
+        proxy_buffers         4 32k;
+    }
+}

Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    send_mail                 => False
+    user                      => root
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9400
+    private_tmp               => False
+    description               => Auto restart job: prometheus-elasticsearch-exporter-9400
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Concat::Fragment[main contacts]

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer

Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    send_mail                 => False
+    user                      => root
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9200
+    private_tmp               => False
+    description               => Auto restart job: prometheus-elasticsearch-exporter-9200
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet].orig
+++ Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]

+    notify_services => []
+    ensure          => present
+    common_name     => relforge1008.eqiad.wmnet
+    hosts           => ['relforge.svc.eqiad.wmnet']
+    renew_seconds   => 952200
+    key             => {'algo': 'ecdsa', 'size': 256}
+    provide_chain   => True
+    owner           => root
+    group           => root
+    auto_renew      => True
+    label           => discovery2026
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    before_services => []
+    mode            => 0740
+    names           => []

Systemd::Timer::Job[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Timer::Job[opensearch-hot-threads-log].orig
+++ Systemd::Timer::Job[opensearch-hot-threads-log]

+    send_mail                 => False
+    user                      => opensearch
+    require                   => ['Package[opensearch]', 'File[/usr/local/bin/opensearch_hot_threads_logger.py]']
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/opensearch_hot_threads_logger.py
+    private_tmp               => False
+    description               => Archive exception logs of hot opensearch threads
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* *:00/5:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/etc/nginx]

Parameters differences:

--- File[/etc/nginx].orig
+++ File[/etc/nginx]

+    before  => ['Service[nginx]']
+    owner   => root
+    group   => root
+    require => Package[nginx-light]
+    ensure  => directory

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]

+    owner   => root
+    group   => root
+    require => Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
+    ensure  => file
+    notify  => ['Exec[nginx-reload]', 'Exec[nginx-reload]']

Logrotate::Conf[nginx]

Parameters differences:

--- Logrotate::Conf[nginx].orig
+++ Logrotate::Conf[nginx]

+    source => puppet:///modules/profile/tlsproxy/logrotate
+    tag    => nginx
+    ensure => present

Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad].orig
+++ Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]

+    ensure => present

File[/etc/update-motd.d/05-cirrus--relforge]

Parameters differences:

--- File[/etc/update-motd.d/05-cirrus--relforge].orig
+++ File[/etc/update-motd.d/05-cirrus--relforge]

+    mode   => 0555
+    owner  => root
+    group  => root
+    ensure => present

Content differences:

--- /etc/update-motd.d/05-cirrus--relforge.orig
+++ /etc/update-motd.d/05-cirrus--relforge
@@ -0,0 +1,2 @@
+#!/bin/sh
+printf "%s\n" "relforge1008 is a Elasticsearch relforge (cirrus::relforge)"

File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-relforge-eqiad-gc-log-cleanup.service
+
+[Timer]
+Unit=opensearch-relforge-eqiad-gc-log-cleanup.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* 02:12:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]

+    owner  => root
+    group  => root
+    ensure => absent

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead-relforge-eqiad.service
+
+[Timer]
+Unit=opensearch-disable-readahead-relforge-eqiad.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha].orig
+++ File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]

+    before => ['Service[nginx]']
+    owner  => root
+    target => /etc/nginx/sites-available/relforge-eqiad-small-alpha
+    group  => root
+    ensure => link

File[/etc/nginx/conf.d]

Parameters differences:

--- File[/etc/nginx/conf.d].orig
+++ File[/etc/nginx/conf.d]

+    before  => ['Service[nginx]']
+    owner   => root
+    group   => root
+    recurse => True
+    force   => True
+    ensure  => directory
+    purge   => True

Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml].orig
+++ File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]

+    owner  => root
+    mode   => 0755
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch_hot_threads.d/relforge-eqiad.yml.orig
+++ /etc/opensearch_hot_threads.d/relforge-eqiad.yml
@@ -0,0 +1,3 @@
+---
+port: 9200
+log_file: "/var/log/opensearch/opensearch_hot_threads-relforge-eqiad.log"

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-wmf-elasticsearch-exporter-9400
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9400

File[/etc/nginx/sites-enabled]

Parameters differences:

--- File[/etc/nginx/sites-enabled].orig
+++ File[/etc/nginx/sites-enabled]

+    before  => ['Service[nginx]']
+    owner   => root
+    group   => root
+    recurse => True
+    force   => True
+    ensure  => directory
+    purge   => True

File[/etc/systemd/system/nginx.service.d]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d].orig
+++ File[/etc/systemd/system/nginx.service.d]

+    mode   => 0555
+    owner  => root
+    group  => root
+    ensure => directory

Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200].orig
+++ Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]

+    extra_config       => 
+    prometheus_port    => 9108
+    elasticsearch_port => 9200

File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]

+    path    => /etc/debdeploy-client/autorestarts.conf
+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    line    => prometheus-wmf-elasticsearch-exporter-9400
+    ensure  => present

Udev::Rule[opensearch-readahead]

Parameters differences:

--- Udev::Rule[opensearch-readahead].orig
+++ Udev::Rule[opensearch-readahead]

+    priority => 40
+    ensure   => present

Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

+    ensure => present

Nrpe::Monitor_service[disk_space]

Parameters differences:

--- Nrpe::Monitor_service[disk_space].orig
+++ Nrpe::Monitor_service[disk_space]

@@
-    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

Logrotate::Rule[opensearch]

Parameters differences:

--- Logrotate::Rule[opensearch].orig
+++ Logrotate::Rule[opensearch]

+    post_rotate    => []
+    frequency      => daily
+    ensure         => present
+    copy_truncate  => True
+    compress       => True
+    date_yesterday => False
+    no_create      => False
+    date_ext       => False
+    missing_ok     => True
+    file_glob      => /var/log/opensearch/*.log
+    not_if_empty   => True
+    rotate         => 7

Class[Profile::Opensearch::Server]

Parameters differences:

--- Class[Profile::Opensearch::Server].orig
+++ Class[Profile::Opensearch::Server]

+    common_settings       => {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True}
+    enable_curator        => False
+    exporter_extra_config => 
+    instances             => {'relforge-eqiad': {'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    require               => ['Class[Profile::Java]', 'Class[Profile::Netbox::Host]']
+    native_lib_path       => /usr/share/opensearch/plugins/opensearch-knn/lib
+    s3_username           => search:platform
+    base_data_dir         => /srv/opensearch
+    logstash_logback_port => 11514
+    version               => 1.3.20
+    logstash_host         => localhost
+    java_home             => /usr/lib/jvm/java-11-openjdk-amd64
+    dc_settings           => {'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2}

Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties
@@ -0,0 +1,201 @@
+status = error
+
+# log action execution errors for easier debugging
+logger.action.name = org.opensearch.action
+logger.action.level = debug
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+######## Server JSON ############################
+appender.rolling.type = RollingFile
+appender.rolling.name = rolling
+appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.json
+appender.rolling.layout.type = OpenSearchJsonLayout
+appender.rolling.layout.type_name = server
+
+appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128MB
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.fileIndex = nomax
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+######## Server -  old style pattern ###########
+appender.rolling_old.type = RollingFile
+appender.rolling_old.name = rolling_old
+appender.rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}.log
+appender.rolling_old.layout.type = PatternLayout
+appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling_old.policies.type = Policies
+appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling_old.policies.time.interval = 1
+appender.rolling_old.policies.time.modulate = true
+appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling_old.policies.size.size = 128MB
+appender.rolling_old.strategy.type = DefaultRolloverStrategy
+appender.rolling_old.strategy.fileIndex = nomax
+appender.rolling_old.strategy.action.type = Delete
+appender.rolling_old.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling_old.strategy.action.condition.type = IfFileName
+appender.rolling_old.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = rolling
+rootLogger.appenderRef.rolling_old.ref = rolling_old
+
+######## Deprecation JSON #######################
+appender.deprecation_rolling.type = RollingFile
+appender.deprecation_rolling.name = deprecation_rolling
+appender.deprecation_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.json
+appender.deprecation_rolling.layout.type = OpenSearchJsonLayout
+appender.deprecation_rolling.layout.type_name = deprecation
+appender.deprecation_rolling.layout.opensearchmessagefields=x-opaque-id
+
+appender.deprecation_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation-%i.json.gz
+appender.deprecation_rolling.policies.type = Policies
+appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling.policies.size.size = 1GB
+appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling.strategy.max = 4
+#################################################
+######## Deprecation -  old style pattern #######
+appender.deprecation_rolling_old.type = RollingFile
+appender.deprecation_rolling_old.name = deprecation_rolling_old
+appender.deprecation_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.log
+appender.deprecation_rolling_old.layout.type = PatternLayout
+appender.deprecation_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.deprecation_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _deprecation-%i.log.gz
+appender.deprecation_rolling_old.policies.type = Policies
+appender.deprecation_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling_old.policies.size.size = 1GB
+appender.deprecation_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling_old.strategy.max = 4
+#################################################
+logger.deprecation.name = org.opensearch.deprecation
+logger.deprecation.level = warn
+logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
+logger.deprecation.appenderRef.deprecation_rolling_old.ref = deprecation_rolling_old
+logger.deprecation.additivity = false
+
+######## Search slowlog JSON ####################
+appender.index_search_slowlog_rolling.type = RollingFile
+appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
+appender.index_search_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog.json
+appender.index_search_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog
+appender.index_search_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,total_hits,types,stats,search_type,total_shards,source,id
+
+appender.index_search_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog-%i.json.gz
+appender.index_search_slowlog_rolling.policies.type = Policies
+appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling.policies.size.size = 1GB
+appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling.strategy.max = 4
+#################################################
+######## Search slowlog -  old style pattern ####
+appender.index_search_slowlog_rolling_old.type = RollingFile
+appender.index_search_slowlog_rolling_old.name = index_search_slowlog_rolling_old
+appender.index_search_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog.log
+appender.index_search_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_search_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_search_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog-%i.log.gz
+appender.index_search_slowlog_rolling_old.policies.type = Policies
+appender.index_search_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_search_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling_old.strategy.max = 4
+#################################################
+logger.index_search_slowlog_rolling.name = index.search.slowlog
+logger.index_search_slowlog_rolling.level = trace
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling_old.ref = index_search_slowlog_rolling_old
+logger.index_search_slowlog_rolling.additivity = false
+
+######## Indexing slowlog JSON ##################
+appender.index_indexing_slowlog_rolling.type = RollingFile
+appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
+appender.index_indexing_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.json
+appender.index_indexing_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog
+appender.index_indexing_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,doc_type,id,routing,source
+
+appender.index_indexing_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.json.gz
+appender.index_indexing_slowlog_rolling.policies.type = Policies
+appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling.strategy.max = 4
+#################################################
+######## Indexing slowlog -  old style pattern ##
+appender.index_indexing_slowlog_rolling_old.type = RollingFile
+appender.index_indexing_slowlog_rolling_old.name = index_indexing_slowlog_rolling_old
+appender.index_indexing_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.log
+appender.index_indexing_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_indexing_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_indexing_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.log.gz
+appender.index_indexing_slowlog_rolling_old.policies.type = Policies
+appender.index_indexing_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling_old.strategy.max = 4
+#################################################
+
+logger.index_indexing_slowlog.name = index.indexing.slowlog.index
+logger.index_indexing_slowlog.level = trace
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling_old.ref = index_indexing_slowlog_rolling_old
+logger.index_indexing_slowlog.additivity = false
+
+
+### Custom settings
+monitorInterval=600
+
+# Add references to Logstash appender for interesting loggers
+
+rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+
+# Define Logstash appender
+appender.ship_to_logstash.name=ship_to_logstash
+appender.ship_to_logstash.type=syslog
+appender.ship_to_logstash.protocol=udp
+appender.ship_to_logstash.facility=LOCAL0
+appender.ship_to_logstash.layout.type=OpenSearchJsonLayout
+appender.ship_to_logstash.layout.compact=true
+appender.ship_to_logstash.layout.complete=true
+appender.ship_to_logstash.host=localhost
+appender.ship_to_logstash.port=11514
+appender.ship_to_logstash.layout.type_name=opensearch
+

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service

Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400].orig
+++ Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

+    prometheus_port    => 9121
+    indices_to_monitor => []
+    elasticsearch_port => 9400

Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]

+    before      => ['Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

Opensearch::Instance[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Instance[relforge-eqiad-small-alpha].orig
+++ Opensearch::Instance[relforge-eqiad-small-alpha]

+    configure_curator                  => False
+    compatibility_mode                 => False
+    short_cluster_name                 => omega
+    base_data_dir                      => /srv/opensearch
+    use_cms_gc                         => False
+    recovery_max_bytes_per_sec         => 40mb
+    certificate_name                   => relforge.svc.eqiad.wmnet
+    script_max_compilations_per_minute => 10000
+    publish_host                       => 10.64.32.100
+    node_name                          => relforge1008-relforge-eqiad-small-alpha
+    plugins_mandatory                  => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    security_plugin_certificates       => {}
+    reindex_remote_whitelist           => search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243
+    auto_create_index                  => True
+    row                                => eqiad-row-c
+    watermarks                         => {}
+    recover_after_nodes                => 1
+    bind_networks                      => ['_local_', '_site_']
+    unicast_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    send_logs_to_logstash              => True
+    gc_log                             => True
+    require                            => Package[opensearch]
+    bulk_thread_pool_executors         => 32
+    ltr_cache_size                     => 100mb
+    cluster_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    bulk_thread_pool_capacity          => 1000
+    recover_after_time                 => 1m
+    holds_data                         => True
+    logstash_logback_port              => 11514
+    filter_cache_size                  => 20%
+    logstash_host                      => localhost
+    version                            => 1.3.20
+    search_shard_count_limit           => 1000
+    disable_security_plugin            => True
+    expected_nodes                     => 2
+    http_port                          => 9400
+    load_fixed_bitset_filters_eagerly  => True
+    tls_port                           => 9443
+    transport_tcp_port                 => 9500
+    heap_memory                        => 4G
+    cluster_name                       => relforge-eqiad-small-alpha
+    cms_gc_init_occupancy_fraction     => 75
+    rack                               => C4
+    curator_uses_unicast_hosts         => True
+    awareness_attributes               => row

Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    unit_name          => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00'}]
+    accuracy           => 15sec

Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400].orig
+++ Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]

+    ensure => present

Nrpe::Check[check_opensearch_shards_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_shards_9400].orig
+++ Nrpe::Check[check_opensearch_shards_9400]

+    before  => Monitoring::Service[opensearch_shards_9400]
+    command => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4
+    ensure  => present

Opensearch::Log::Hot_threads_cluster[relforge-eqiad]

Parameters differences:

--- Opensearch::Log::Hot_threads_cluster[relforge-eqiad].orig
+++ Opensearch::Log::Hot_threads_cluster[relforge-eqiad]

+    http_port    => 9200
+    cluster_name => relforge-eqiad

Exec[tlsproxy localssl default_server on ports [9243]]

Parameters differences:

--- Exec[tlsproxy localssl default_server on ports [9243]].orig
+++ Exec[tlsproxy localssl default_server on ports [9243]]

+    command     => /bin/true
+    onlyif      => /bin/false
+    refreshonly => True

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad /srv/opensearch

File[/lib/systemd/system/opensearch_1@.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch_1@.service].orig
+++ File[/lib/systemd/system/opensearch_1@.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]

Content differences:

--- /lib/systemd/system/opensearch_1@.service.orig
+++ /lib/systemd/system/opensearch_1@.service
@@ -0,0 +1,73 @@
+[Unit]
+Description=OpenSearch (cluster %i)
+PartOf=opensearch.service
+Documentation=https://opensearch.org/docs
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=notify
+RuntimeDirectory=opensearch
+PrivateTmp=true
+Environment=OPENSEARCH_HOME=/usr/share/opensearch
+Environment=OPENSEARCH_PATH_CONF=/etc/opensearch/%i
+Environment=OPENSEARCH_JVM_OPTIONS=/etc/opensearch/%i/jvm.options
+Environment=PID_DIR=/run/opensearch-%i
+Environment=OPENSEARCH_SD_NOTIFY=true
+
+Environment=LD_LIBRARY_PATH=/usr/share/opensearch/plugins/opensearch-knn/lib
+Environment=JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
+
+WorkingDirectory=/usr/share/opensearch
+
+User=opensearch
+Group=opensearch
+
+# Force creation of /run/opensearch (see T276198)
+ExecStartPre=/bin/systemd-tmpfiles --create /etc/tmpfiles.d/opensearch-%i.conf
+ExecStart=/usr/share/opensearch/bin/opensearch \
+        -p ${PID_DIR}/%i.pid \
+        --quiet \
+        -Epath.logs=/var/log/opensearch \
+        -Epath.data=/srv/opensearch/%i
+
+# StandardOutput is configured to redirect to journalctl since
+# some error messages may be logged in standard output before
+# opensearch logging system is initialized. OpenSearch
+# stores its logs in /var/log/opensearch and does not use
+# journalctl by default. If you also want to enable journalctl
+# logging, you can simply remove the "quiet" option from ExecStart.
+StandardOutput=journal
+StandardError=inherit
+
+# Specifies the maximum file descriptor number that can be opened by this process
+LimitNOFILE=65535
+
+# Specifies the maximum number of processes
+LimitNPROC=4096
+
+# Specifies the maximum size of virtual memory
+LimitAS=infinity
+
+# Specifies the maximum file size
+LimitFSIZE=infinity
+
+# Disable timeout logic and wait until process is stopped
+TimeoutStopSec=0
+
+# SIGTERM signal is used to stop the Java process
+KillSignal=SIGTERM
+
+# Send the signal only to the JVM rather than its control group
+KillMode=process
+
+# Java process is never killed
+SendSIGKILL=no
+
+# When a JVM receives a SIGTERM signal it exits with code 143
+SuccessExitStatus=143
+
+[Install]
+WantedBy=multi-user.target
+
+# Built for packages-7.4.1 (packages)

Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-elasticsearch-exporter-9200
+
+/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Service[prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => running
+    enable => True

File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Systemd::Unit[push_cross_cluster_settings_9400.timer]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9400.timer].orig
+++ Systemd::Unit[push_cross_cluster_settings_9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => push_cross_cluster_settings_9400.timer

Logstash::Output::Udp[gelf_relay]

Parameters differences:

--- Logstash::Output::Udp[gelf_relay].orig
+++ Logstash::Output::Udp[gelf_relay]

+    priority  => 90
+    port      => 11514
+    ensure    => present
+    codec     => json
+    host      => localhost
+    plugin_id => output/udp/gelf_relay

File[/lib/systemd/system/opensearch-disable-readahead.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh

File[/etc/systemd/system/nginx.service.d/security.conf]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d/security.conf].orig
+++ File[/etc/systemd/system/nginx.service.d/security.conf]

+    before  => Class[Nginx]
+    owner   => root
+    require => File[/etc/systemd/system/nginx.service.d]
+    source  => puppet:///modules/profile/tlsproxy/nginx-security.conf
+    group   => root
+    ensure  => present
+    mode    => 0444

File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

File[/etc/systemd/system/nginx.service.d/numa.conf]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d/numa.conf].orig
+++ File[/etc/systemd/system/nginx.service.d/numa.conf]

+    before  => Class[Nginx]
+    owner   => root
+    require => File[/etc/systemd/system/nginx.service.d]
+    group   => root
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/systemd/system/nginx.service.d/numa.conf.orig
+++ /etc/systemd/system/nginx.service.d/numa.conf
@@ -0,0 +1,3 @@
+[Service]
+ExecStart=
+ExecStart=/usr/bin/numactl -a -N 0,1 /usr/sbin/nginx -g 'daemon on; master_process on;'

Mount[/var/lib/nginx]

Parameters differences:

--- Mount[/var/lib/nginx].orig
+++ Mount[/var/lib/nginx]

+    before  => Service[nginx]
+    options => defaults,noatime,uid=0,gid=0,mode=755,size=1g
+    pass    => 0
+    device  => tmpfs
+    fstype  => tmpfs
+    ensure  => mounted
+    require => File[/var/lib/nginx]
+    dump    => 0

File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]

+    owner  => root
+    group  => root
+    ensure => absent

File[/usr/local/bin/cleanup-dlq]

Parameters differences:

--- File[/usr/local/bin/cleanup-dlq].orig
+++ File[/usr/local/bin/cleanup-dlq]

+    owner  => root
+    group  => root
+    ensure => absent

File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]

Parameters differences:

--- File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter].orig
+++ File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]

+    owner  => root
+    source => puppet:///modules/prometheus/usr/local/bin/prometheus-wmf-elasticsearch-exporter.py
+    group  => root
+    ensure => present
+    mode   => 0555

File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad.orig
+++ /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead-relforge-eqiad
+
+/var/log/opensearch-disable-readahead-relforge-eqiad/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/srv/opensearch/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/srv/opensearch/relforge-eqiad-small-alpha].orig
+++ File[/srv/opensearch/relforge-eqiad-small-alpha]

+    owner  => opensearch
+    mode   => 0755
+    group  => opensearch
+    ensure => directory

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-elasticsearch-exporter-9200" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]

Parameters differences:

--- Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi].orig
+++ Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]

+    creates => /etc/opensearch/relforge-eqiad-small-alpha/sudachi
+    command => /usr/bin/ln -s '/usr/share/opensearch/config/sudachi' '/etc/opensearch/relforge-eqiad-small-alpha/sudachi'
+    onlyif  => /usr/bin/test -d '/usr/share/opensearch/config/sudachi'

File[/etc/opensearch_hot_threads.d]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d].orig
+++ File[/etc/opensearch_hot_threads.d]

+    owner  => root
+    mode   => 0755
+    group  => root
+    ensure => directory

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Nrpe::Monitor_service[opensearch_shard_size_check_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shard_size_check_9200].orig
+++ Nrpe::Monitor_service[opensearch_shard_size_check_9200]

+    retry_interval              => 180
+    check_interval              => 1440
+    ensure                      => present
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 3
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch shard size check - 9200
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

Package[opensearch-madvise]

Parameters differences:

--- Package[opensearch-madvise].orig
+++ Package[opensearch-madvise]

+    provider => apt
+    ensure   => present

File[/etc/opensearch/relforge-eqiad/java-security.policy]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/java-security.policy].orig
+++ File[/etc/opensearch/relforge-eqiad/java-security.policy]

+    owner  => opensearch
+    source => puppet:///modules/profile/opensearch/java-security.policy
+    group  => opensearch
+    ensure => file
+    mode   => 0440

Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]

+    check_command          => nrpe_check!check_opensearch_shards_9400!10
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch health check for shards on 9400
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration

File[/var/log/opensearch/opensearch_index_search_slowlog.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch_index_search_slowlog.log].orig
+++ File[/var/log/opensearch/opensearch_index_search_slowlog.log]

+    owner  => root
+    group  => root
+    ensure => absent

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9400.timer

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-elasticsearch-exporter-9200
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9200

File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]

Content differences:

--- /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Prometheus WMF Elasticsearch Exporter
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/local/bin/prometheus-wmf-elasticsearch-exporter \
+    --listen :9121 \
+    --port 9400 \
+    --indices 
+
+[Install]
+WantedBy=multi-user.target

Nrpe::Check[check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_unassigned_shard_check_9200].orig
+++ Nrpe::Check[check_opensearch_unassigned_shard_check_9200]

+    before  => Monitoring::Service[opensearch_unassigned_shard_check_9200]
+    command => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4
+    ensure  => present

Logstash::Input::Gelf[gelf_relay]

Parameters differences:

--- Logstash::Input::Gelf[gelf_relay].orig
+++ Logstash::Input::Gelf[gelf_relay]

+    priority  => 10
+    port      => 12201
+    ensure    => present
+    host      => localhost
+    tags      => ['input-gelf-relay']
+    plugin_id => input/gelf/12201

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=720min
+OnActiveSec=1s
+RandomizedDelaySec=43200
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]

+    path    => /etc/debdeploy-client/autorestarts.conf
+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    line    => prometheus-wmf-elasticsearch-exporter-9200
+    ensure  => present

File[/var/lib/prometheus/node.d/role_owner.prom]

Content differences:

--- /var/lib/prometheus/node.d/role_owner.prom.orig
+++ /var/lib/prometheus/node.d/role_owner.prom
@@ -1,3 +1,3 @@
 # HELP role_owner The team owner of the server role
 # TYPE role_owner gauge
-role_owner{team="search-platform",role="cirrus::test",cluster="relforge"} 1.0
+role_owner{team="search-platform",role="cirrus::relforge",cluster="relforge"} 1.0

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead-relforge-eqiad.timer

Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)].orig
+++ Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9400.timer].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9400.timer.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9400.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of push_cross_cluster_settings_9400.service
+
+[Timer]
+Unit=push_cross_cluster_settings_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=15min
+OnActiveSec=1s
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha].orig
+++ Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]

+    command     => /bin/systemd-tmpfiles --create --remove '/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf'
+    subscribe   => File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]
+    user        => root
+    refreshonly => True

Systemd::Monitor[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Monitor[push_cross_cluster_settings_9400].orig
+++ Systemd::Monitor[push_cross_cluster_settings_9400]

+    retries        => 2
+    check_interval => 10
+    ensure         => present
+    migration_task => T407130
+    contact_group  => admins
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    critical       => False

File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]

+    unit_name          => nrpe2nodexp-opensearch_shards_9200.service
+    fixed_random_delay => True
+    splay              => 60
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Exec[nginx-reload]

Parameters differences:

--- Exec[nginx-reload].orig
+++ Exec[nginx-reload]

+    command     => /usr/sbin/service nginx reload
+    refreshonly => True

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-wmf-elasticsearch-exporter-9200
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9200

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shards_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shards_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=1min
+OnActiveSec=1s
+RandomizedDelaySec=60
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad-small-alpha /srv/opensearch
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/etc/logstash/log4j2.properties]

Parameters differences:

--- File[/etc/logstash/log4j2.properties].orig
+++ File[/etc/logstash/log4j2.properties]

+    owner   => root
+    group   => root
+    require => Package[logstash-oss]
+    mode    => 0444
+    ensure  => file

Content differences:

--- /etc/logstash/log4j2.properties.orig
+++ /etc/logstash/log4j2.properties
@@ -0,0 +1,51 @@
+status = error
+name = LogstashPropertiesConfig
+
+appender.rolling.type = RollingFile
+appender.rolling.name = plain_rolling
+appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
+appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.layout.type = PatternLayout
+appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128M
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.max = 4
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:ls.logs}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
+appender.rolling.strategy.action.ifAny.type = IfAny
+appender.rolling.strategy.action.ifAny.ifLastModified.type = IfLastModified
+appender.rolling.strategy.action.ifAny.ifLastModified.age = 30D
+appender.rolling.strategy.action.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.ifAny.ifAccumulatedFileSize.exceeds = 15GB
+
+appender.json_rolling.type = RollingFile
+appender.json_rolling.name = json_rolling
+appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
+appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log
+appender.json_rolling.policies.type = Policies
+appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.json_rolling.policies.time.interval = 1
+appender.json_rolling.policies.time.modulate = true
+appender.json_rolling.layout.type = JSONLayout
+appender.json_rolling.layout.compact = true
+appender.json_rolling.layout.eventEol = true
+appender.json_rolling.strategy.type = DefaultRolloverStrategy
+appender.json_rolling.strategy.action.type = Delete
+appender.json_rolling.strategy.action.basepath = ${sys:ls.logs}
+appender.json_rolling.strategy.action.condition.type = IfFileName
+appender.json_rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
+appender.json_rolling.strategy.action.ifAny.type = IfAny
+appender.json_rolling.strategy.action.ifAny.ifLastModified.type = IfLastModified
+appender.json_rolling.strategy.action.ifAny.ifLastModified.age = 30D
+appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
+appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.exceeds = 10GB
+
+rootLogger.level = ${sys:ls.log.level}
+rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling

Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    send_mail                 => False
+    user                      => root
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9200
+    private_tmp               => False
+    description               => Auto restart job: prometheus-wmf-elasticsearch-exporter-9200
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/bin/find /var/log/opensearch -name 'relforge-eqiad-small-alpha_jvm_gc.*.log*' -mtime +30 -delete
+    private_tmp               => False
+    description               => Cleanup GC logs
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    subscribe   => File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    require     => Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
+    refreshonly => True

File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/usr/share/opensearch/plugins]

Parameters differences:

--- File[/usr/share/opensearch/plugins].orig
+++ File[/usr/share/opensearch/plugins]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755

Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "de4a199c43946c09f819ca4ffea40ac0" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9400_status"
+    private_tmp               => False
+    splay                     => 300
+    description               => execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9400_status command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-check_push_cross_cluster_settings_9400_status
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => True
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => prometheus-wmf-elasticsearch-exporter-9200

Systemd::Tmpfile[opensearch-relforge-eqiad]

Parameters differences:

--- Systemd::Tmpfile[opensearch-relforge-eqiad].orig
+++ Systemd::Tmpfile[opensearch-relforge-eqiad]

+    owner  => root
+    group  => root
+    ensure => present

File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]

Parameters differences:

--- File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf].orig
+++ File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]

+    owner        => logstash
+    group        => logstash
+    validate_cmd => /usr/local/bin/logstash-config-test %
+    ensure       => present
+    mode         => 0440
+    notify       => Service[logstash]

Content differences:

--- /etc/logstash/conf.d/90-output-udp-gelf_relay.conf.orig
+++ /etc/logstash/conf.d/90-output-udp-gelf_relay.conf
@@ -0,0 +1,9 @@
+output {
+
+      udp {
+          host => 'localhost'
+          port => '11514'
+          codec =>'json'
+      }
+
+}

File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+
+[Timer]
+Unit=opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* 02:12:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => present

Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
+    ensure   => present

Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]

+    check_command          => nrpe_check!check_opensearch_shard_size_check_9400!10
+    retry_interval         => 180
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1440
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch shard size check - 9400
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

+    tag            => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    format         => plain
+    force          => False
+    show_diff      => True
+    ensure_newline => False
+    order          => alpha
+    owner          => root
+    group          => root
+    mode           => 0444
+    backup         => puppet
+    replace        => True

File[/var/log/push_cross_cluster_settings_9200]

Parameters differences:

--- File[/var/log/push_cross_cluster_settings_9200].orig
+++ File[/var/log/push_cross_cluster_settings_9200]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]

+    before      => ['Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)].orig
+++ Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]

+    command     => /bin/systemctl daemon-reload
+    notify      => ['Service[prometheus-wmf-elasticsearch-exporter-9400]']
+    refreshonly => True

File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cleanup GC logs
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/find /var/log/opensearch -name 'relforge-eqiad-small-alpha_jvm_gc.*.log*' -mtime +30 -delete

Systemd::Timer[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Timer[push_cross_cluster_settings_9200].orig
+++ Systemd::Timer[push_cross_cluster_settings_9200]

+    unit_name          => push_cross_cluster_settings_9200.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '15min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

File[/var/log/opensearch]

Parameters differences:

--- File[/var/log/opensearch].orig
+++ File[/var/log/opensearch]

+    owner   => opensearch
+    group   => opensearch
+    require => Package[opensearch]
+    mode    => 0755
+    ensure  => directory

Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => True
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => prometheus-wmf-elasticsearch-exporter-9400

Java::Cacert[wmf:puppetca.pem]

Parameters differences:

--- Java::Cacert[wmf:puppetca.pem].orig
+++ Java::Cacert[wmf:puppetca.pem]

+    owner     => root
+    require   => Alternatives::Java[17]
+    group     => root
+    ensure    => present
+    path      => /etc/ssl/certs/Puppet_Internal_CA.pem
+    storepass => changeit

Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    unit_name          => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+    fixed_random_delay => True
+    splay              => 300
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml].orig
+++ File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]

+    owner  => root
+    mode   => 0755
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml.orig
+++ /etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml
@@ -0,0 +1,3 @@
+---
+port: 9400
+log_file: "/var/log/opensearch/opensearch_hot_threads-relforge-eqiad-small-alpha.log"

File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg.orig
+++ /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_check_push_cross_cluster_settings_9200_status]=/usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200

File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]

Content differences:

--- /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Prometheus WMF Elasticsearch Exporter
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/local/bin/prometheus-wmf-elasticsearch-exporter \
+    --listen :9120 \
+    --port 9200 \
+    --indices 
+
+[Install]
+WantedBy=multi-user.target

Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem].orig
+++ Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

+    subscribe => ['Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]', 'File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]', 'File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]']
+    command   => /bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem > /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem
+    require   => Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]
+    notify    => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem | sha512sum)" == "$(/bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem | sha512sum)"

File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shard_size_check_9200]=/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4

Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter-9200].orig
+++ Service[prometheus-elasticsearch-exporter-9200]

+    ensure => running
+    enable => True

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shard_size_check_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shard_size_check_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=600min
+OnActiveSec=1s
+RandomizedDelaySec=36000
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list].orig
+++ File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]

+    owner  => root
+    group  => root
+    ensure => absent

File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf.orig
+++ /etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-relforge-eqiad-small-alpha-gc-log-cleanup" then {
+    action(
+        type="omfile" file="/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer

Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha].orig
+++ Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]

+    check_command          => check_ssl_on_port!relforge1008.eqiad.wmnet!9443
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => team-data-platform,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => Elasticsearch HTTPS for relforge-eqiad-small-alpha
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/default/opensearch]

Parameters differences:

--- File[/etc/default/opensearch].orig
+++ File[/etc/default/opensearch]

+    owner   => root
+    group   => root
+    require => Package[opensearch]
+    mode    => 0444
+    ensure  => file

Content differences:

--- /etc/default/opensearch.orig
+++ /etc/default/opensearch
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# Default env file provided by upstream always reset OPENSEARCH_PATH_CONF
+# when called from /usr/share/opensearch/bin/opensearch-env
+# only set it when not defined.
+# This file is still required to set OPENSEARCH_PATH_CONF by postinst scripts
+OPENSEARCH_PATH_CONF=${OPENSEARCH_PATH_CONF-/etc/opensearch}

Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400].orig
+++ Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]

+    extra_config       => 
+    prometheus_port    => 9109
+    elasticsearch_port => 9400

Class[Profile::Tlsproxy::Instance]

Parameters differences:

--- Class[Profile::Tlsproxy::Instance].orig
+++ Class[Profile::Tlsproxy::Instance]

+    ssl_compatibility_mode     => compat
+    nginx_client_max_body_size => 100m
+    bootstrap_protection       => False
+    require                    => ['Class[Profile::Tcp_fast_open]']
+    numa_iface                 => lo
+    nginx_variant              => light

Service[opensearch-hot-threads-log.timer]

Parameters differences:

--- Service[opensearch-hot-threads-log.timer].orig
+++ Service[opensearch-hot-threads-log.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha].orig
+++ Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]

+    owner  => root
+    group  => root
+    ensure => present

Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    unit_name          => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}]
+    accuracy           => 15sec

File[/etc/sysctl.d/opensearch.conf]

Parameters differences:

--- File[/etc/sysctl.d/opensearch.conf].orig
+++ File[/etc/sysctl.d/opensearch.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/opensearch.conf.orig
+++ /etc/sysctl.d/opensearch.conf
@@ -0,0 +1,2 @@
+# sysctl parameters managed by Puppet.
+vm.max_map_count = 1048576

File[/etc/opensearch/relforge-eqiad/log4j2.properties]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/log4j2.properties].orig
+++ File[/etc/opensearch/relforge-eqiad/log4j2.properties]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad/log4j2.properties.orig
+++ /etc/opensearch/relforge-eqiad/log4j2.properties
@@ -0,0 +1,201 @@
+status = error
+
+# log action execution errors for easier debugging
+logger.action.name = org.opensearch.action
+logger.action.level = debug
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+######## Server JSON ############################
+appender.rolling.type = RollingFile
+appender.rolling.name = rolling
+appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.json
+appender.rolling.layout.type = OpenSearchJsonLayout
+appender.rolling.layout.type_name = server
+
+appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128MB
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.fileIndex = nomax
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+######## Server -  old style pattern ###########
+appender.rolling_old.type = RollingFile
+appender.rolling_old.name = rolling_old
+appender.rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}.log
+appender.rolling_old.layout.type = PatternLayout
+appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling_old.policies.type = Policies
+appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling_old.policies.time.interval = 1
+appender.rolling_old.policies.time.modulate = true
+appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling_old.policies.size.size = 128MB
+appender.rolling_old.strategy.type = DefaultRolloverStrategy
+appender.rolling_old.strategy.fileIndex = nomax
+appender.rolling_old.strategy.action.type = Delete
+appender.rolling_old.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling_old.strategy.action.condition.type = IfFileName
+appender.rolling_old.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = rolling
+rootLogger.appenderRef.rolling_old.ref = rolling_old
+
+######## Deprecation JSON #######################
+appender.deprecation_rolling.type = RollingFile
+appender.deprecation_rolling.name = deprecation_rolling
+appender.deprecation_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.json
+appender.deprecation_rolling.layout.type = OpenSearchJsonLayout
+appender.deprecation_rolling.layout.type_name = deprecation
+appender.deprecation_rolling.layout.opensearchmessagefields=x-opaque-id
+
+appender.deprecation_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation-%i.json.gz
+appender.deprecation_rolling.policies.type = Policies
+appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling.policies.size.size = 1GB
+appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling.strategy.max = 4
+#################################################
+######## Deprecation -  old style pattern #######
+appender.deprecation_rolling_old.type = RollingFile
+appender.deprecation_rolling_old.name = deprecation_rolling_old
+appender.deprecation_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.log
+appender.deprecation_rolling_old.layout.type = PatternLayout
+appender.deprecation_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.deprecation_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _deprecation-%i.log.gz
+appender.deprecation_rolling_old.policies.type = Policies
+appender.deprecation_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling_old.policies.size.size = 1GB
+appender.deprecation_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling_old.strategy.max = 4
+#################################################
+logger.deprecation.name = org.opensearch.deprecation
+logger.deprecation.level = warn
+logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
+logger.deprecation.appenderRef.deprecation_rolling_old.ref = deprecation_rolling_old
+logger.deprecation.additivity = false
+
+######## Search slowlog JSON ####################
+appender.index_search_slowlog_rolling.type = RollingFile
+appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
+appender.index_search_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog.json
+appender.index_search_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog
+appender.index_search_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,total_hits,types,stats,search_type,total_shards,source,id
+
+appender.index_search_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog-%i.json.gz
+appender.index_search_slowlog_rolling.policies.type = Policies
+appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling.policies.size.size = 1GB
+appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling.strategy.max = 4
+#################################################
+######## Search slowlog -  old style pattern ####
+appender.index_search_slowlog_rolling_old.type = RollingFile
+appender.index_search_slowlog_rolling_old.name = index_search_slowlog_rolling_old
+appender.index_search_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog.log
+appender.index_search_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_search_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_search_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog-%i.log.gz
+appender.index_search_slowlog_rolling_old.policies.type = Policies
+appender.index_search_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_search_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling_old.strategy.max = 4
+#################################################
+logger.index_search_slowlog_rolling.name = index.search.slowlog
+logger.index_search_slowlog_rolling.level = trace
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling_old.ref = index_search_slowlog_rolling_old
+logger.index_search_slowlog_rolling.additivity = false
+
+######## Indexing slowlog JSON ##################
+appender.index_indexing_slowlog_rolling.type = RollingFile
+appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
+appender.index_indexing_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.json
+appender.index_indexing_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog
+appender.index_indexing_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,doc_type,id,routing,source
+
+appender.index_indexing_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.json.gz
+appender.index_indexing_slowlog_rolling.policies.type = Policies
+appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling.strategy.max = 4
+#################################################
+######## Indexing slowlog -  old style pattern ##
+appender.index_indexing_slowlog_rolling_old.type = RollingFile
+appender.index_indexing_slowlog_rolling_old.name = index_indexing_slowlog_rolling_old
+appender.index_indexing_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.log
+appender.index_indexing_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_indexing_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_indexing_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.log.gz
+appender.index_indexing_slowlog_rolling_old.policies.type = Policies
+appender.index_indexing_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling_old.strategy.max = 4
+#################################################
+
+logger.index_indexing_slowlog.name = index.indexing.slowlog.index
+logger.index_indexing_slowlog.level = trace
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling_old.ref = index_indexing_slowlog_rolling_old
+logger.index_indexing_slowlog.additivity = false
+
+
+### Custom settings
+monitorInterval=600
+
+# Add references to Logstash appender for interesting loggers
+
+rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+
+# Define Logstash appender
+appender.ship_to_logstash.name=ship_to_logstash
+appender.ship_to_logstash.type=syslog
+appender.ship_to_logstash.protocol=udp
+appender.ship_to_logstash.facility=LOCAL0
+appender.ship_to_logstash.layout.type=OpenSearchJsonLayout
+appender.ship_to_logstash.layout.compact=true
+appender.ship_to_logstash.layout.complete=true
+appender.ship_to_logstash.host=localhost
+appender.ship_to_logstash.port=11514
+appender.ship_to_logstash.layout.type_name=opensearch
+

File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.orig
+++ /etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup
+
+/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]

+    owner     => root
+    backup    => False
+    group     => root
+    ensure    => file
+    mode      => 0440
+    show_diff => False
+    notify    => ['Exec[nginx-reload]', 'Exec[nginx-reload]']

Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

+    key         => {'algo': 'ecdsa', 'size': 256}
+    ensure      => present
+    names       => []
+    common_name => relforge1008.eqiad.wmnet
+    hosts       => ['relforge.svc.eqiad.wmnet']

Service[opensearch]

Parameters differences:

--- Service[opensearch].orig
+++ Service[opensearch]

+    ensure  => stopped
+    require => Package[opensearch]
+    enable  => False

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shards_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shards_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "2a63eb7dcbe368b7f538118765d629d6" --timeout 10 --check-command "check_opensearch_shards_9400"

Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status].orig
+++ Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

File[/lib/systemd/system/opensearch-hot-threads-log.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-hot-threads-log.service].orig
+++ File[/lib/systemd/system/opensearch-hot-threads-log.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]

Content differences:

--- /lib/systemd/system/opensearch-hot-threads-log.service.orig
+++ /lib/systemd/system/opensearch-hot-threads-log.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Archive exception logs of hot opensearch threads
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=opensearch
+ExecStart=/usr/local/bin/opensearch_hot_threads_logger.py

Systemd::Unit[clean_up_dlq.timer]

Parameters differences:

--- Systemd::Unit[clean_up_dlq.timer].orig
+++ Systemd::Unit[clean_up_dlq.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => clean_up_dlq.timer

Nrpe::Monitor_service[opensearch_shard_size_check_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shard_size_check_9400].orig
+++ Nrpe::Monitor_service[opensearch_shard_size_check_9400]

+    retry_interval              => 180
+    check_interval              => 1440
+    ensure                      => present
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 3
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch shard size check - 9400
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    unit_name          => opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    accuracy           => 15sec

File[/etc/ssl/dhparam.pem]

Parameters differences:

--- File[/etc/ssl/dhparam.pem].orig
+++ File[/etc/ssl/dhparam.pem]

+    owner  => root
+    source => puppet:///modules/sslcert/dhparam.pem
+    group  => root
+    ensure => present
+    mode   => 0444

Concat::Fragment[wikimedia-opensearch-header]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-header].orig
+++ Concat::Fragment[wikimedia-opensearch-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    order  => 01

Nrpe::Plugin[check_systemd_unit_status]

Parameters differences:

--- Nrpe::Plugin[check_systemd_unit_status].orig
+++ Nrpe::Plugin[check_systemd_unit_status]

+    source => puppet:///modules/systemd/check_systemd_unit_status
+    ensure => present

Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead-relforge-eqiad-small-alpha.timer

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+
+[Timer]
+Unit=opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Timer[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Timer[push_cross_cluster_settings_9400].orig
+++ Systemd::Timer[push_cross_cluster_settings_9400]

+    unit_name          => push_cross_cluster_settings_9400.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '15min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Opensearch::Cross_cluster_settings[relforge-eqiad]

Parameters differences:

--- Opensearch::Cross_cluster_settings[relforge-eqiad].orig
+++ Opensearch::Cross_cluster_settings[relforge-eqiad]

+    enable_remote_search => False
+    instance_name        => relforge-eqiad
+    settings             => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}

Systemd::Service[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shards_9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Nrpe::Check[check_disk_space]

Parameters differences:

--- Nrpe::Check[check_disk_space].orig
+++ Nrpe::Check[check_disk_space]

@@
-    command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    command => /usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

Concat::Fragment[wikimedia-opensearch]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch].orig
+++ Concat::Fragment[wikimedia-opensearch]

+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    order  => 10

Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200].orig
+++ Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]

+    retry_interval              => 120
+    check_interval              => 720
+    ensure                      => present
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 1
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch unassigned shard check - 9200
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration

Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
+    ensure   => present

Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Ferm::Service[elastic-small-alpha-https-9443]

Parameters differences:

--- Ferm::Service[elastic-small-alpha-https-9443].orig
+++ Ferm::Service[elastic-small-alpha-https-9443]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => False
+    srange              => (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49)
+    prio                => 10
+    desc                => 
+    port                => 9443
+    proto               => tcp

Systemd::Timer[clean_up_dlq]

Parameters differences:

--- Systemd::Timer[clean_up_dlq].orig
+++ Systemd::Timer[clean_up_dlq]

+    unit_name          => clean_up_dlq.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '1h'}]
+    accuracy           => 15sec

Concat_fragment[wikimedia-opensearch-plugins]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-plugins].orig
+++ Concat_fragment[wikimedia-opensearch-plugins]

+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    order  => 10

Content differences:

--- wikimedia-opensearch-plugins.orig
+++ wikimedia-opensearch-plugins
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: bookworm-wikimedia
+Components: component/opensearch13
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]
+    ensure   => present

File[/etc/default/prometheus-elasticsearch-exporter]

Parameters differences:

--- File[/etc/default/prometheus-elasticsearch-exporter].orig
+++ File[/etc/default/prometheus-elasticsearch-exporter]

+    owner  => root
+    group  => root
+    ensure => absent

Sudo::User[nrpe-check_opensearch_shards_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shards_9400].orig
+++ Sudo::User[nrpe-check_opensearch_shards_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-check_push_cross_cluster_settings_9400_status" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

File[/var/log/opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead-relforge-eqiad].orig
+++ File[/var/log/opensearch-disable-readahead-relforge-eqiad]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

File[/etc/nagios/nrpe.d/check_disk_space.cfg]

Content differences:

--- /etc/nagios/nrpe.d/check_disk_space.cfg.orig
+++ /etc/nagios/nrpe.d/check_disk_space.cfg
@@ -1,2 +1,2 @@
 # File generated by puppet. DO NOT edit by hand
-command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

Exec[mask_nginx.service]

Parameters differences:

--- Exec[mask_nginx.service].orig
+++ Exec[mask_nginx.service]

+    command => /bin/systemctl mask nginx.service
+    unless  => /usr/bin/dpkg -s nginx-light | /bin/grep -q '^Status: install ok installed$'
+    creates => /etc/systemd/system/nginx.service

File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]

+    owner  => root
+    mode   => 0755
+    group  => root
+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch_shard_size.py

Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Monitoring::Service[check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Monitoring::Service[check_push_cross_cluster_settings_9200_status].orig
+++ Monitoring::Service[check_push_cross_cluster_settings_9200_status]

+    check_command  => nrpe_check!check_check_push_cross_cluster_settings_9200_status!10
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 10
+    config_dir     => /etc/nagios
+    migration_task => T407130
+    contact_group  => admins
+    host           => relforge1008
+    critical       => False
+    retries        => 2
+    passive        => False
+    description    => Check unit status of push_cross_cluster_settings_9200
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

File[/etc/apt/sources.list.d/wikimedia-opensearch.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/wikimedia-opensearch.list].orig
+++ File[/etc/apt/sources.list.d/wikimedia-opensearch.list]

+    owner  => root
+    group  => root
+    ensure => absent

Ferm::Service[opensearch-https-9443]

Parameters differences:

--- Ferm::Service[opensearch-https-9443].orig
+++ Ferm::Service[opensearch-https-9443]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => False
+    srange              => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    prio                => 10
+    desc                => 
+    port                => 9443
+    proto               => tcp

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-elasticsearch-exporter-9400" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Unit[push_cross_cluster_settings_9200.timer]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9200.timer].orig
+++ Systemd::Unit[push_cross_cluster_settings_9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => push_cross_cluster_settings_9200.timer

Exec[mask_default_prometheus_elasticsearch_exporter]

Parameters differences:

--- Exec[mask_default_prometheus_elasticsearch_exporter].orig
+++ Exec[mask_default_prometheus_elasticsearch_exporter]

+    command => /bin/systemctl mask prometheus-elasticsearch-exporter.service
+    creates => /etc/systemd/system/prometheus-elasticsearch-exporter.service

Systemd::Unit[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Unit[prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Unit[prometheus-elasticsearch-exporter-9200]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => prometheus-elasticsearch-exporter-9200

Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-relforge-eqiad-gc-log-cleanup.service

Exec[apt_repository_wikimedia-opensearch-plugins]

Parameters differences:

--- Exec[apt_repository_wikimedia-opensearch-plugins].orig
+++ Exec[apt_repository_wikimedia-opensearch-plugins]

+    command     => /usr/bin/apt-get update 
+    refreshonly => True

File[/etc/nginx/sites-available/relforge-eqiad]

Parameters differences:

--- File[/etc/nginx/sites-available/relforge-eqiad].orig
+++ File[/etc/nginx/sites-available/relforge-eqiad]

+    before => ['Service[nginx]']
+    owner  => root
+    group  => root
+    ensure => present
+    notify => Exec[nginx-reload]

Content differences:

--- /etc/nginx/sites-available/relforge-eqiad.orig
+++ /etc/nginx/sites-available/relforge-eqiad
@@ -0,0 +1,59 @@
+# Proxy site configuration file for relforge-eqiad
+# This file is managed by Puppet!
+upstream local_fe_relforge-eqiad {
+    server 10.64.32.100:9200 max_fails=0;
+}
+
+# SSL proxying
+server {
+    # Enabling TCP Fast Open is safe for HTTP over TLS. There is no idempotency
+    # concern replaying TLS Client Hello.
+    # https://tools.ietf.org/html/rfc7413#section-6.3.2
+    listen [::]:9243 default_server deferred backlog=16384 reuseport ipv6only=on fastopen=150 ssl ;
+    listen 9243 default_server deferred backlog=16384 reuseport fastopen=150 ssl ;
+    server_name relforge1008.eqiad.wmnet relforge.svc.eqiad.wmnet;
+
+    error_log   /var/log/nginx/relforge-eqiad.error.log;
+    access_log   off;
+
+    ssl_certificate /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem;
+    ssl_certificate_key /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem;
+
+    keepalive_timeout 60;
+    keepalive_requests 100;
+
+    location / {
+        proxy_pass http://local_fe_relforge-eqiad;
+        proxy_http_version 1.1;
+        proxy_set_header Connection close; # should be default
+
+        # this should be in sync with Varnish's first_byte_timeout
+        # and PHP's max_execution_time
+        proxy_read_timeout 180s;
+
+        proxy_set_header Host $host;
+        proxy_set_header X-Client-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Connection-Properties "H2=$h2; SSR=$session_reused; SSL=$ssl_protocol; C=$ssl_cipher;";
+
+        # Don't clobber the Server header from the backend.
+        proxy_pass_header Server;
+
+        proxy_redirect off;
+        proxy_buffering off;
+        proxy_request_buffering on;
+
+        # With proxy buffering off, proxy_buffer_size is still used to
+        # define the singular chunk of buffer the content passes
+        # through.  Apparently if nginx can't read all the response
+        # headers in the first such chunk, it fails with 502.  Upping
+        # this to 8k (from def 4k) solves the one reported case in
+        # https://phabricator.wikimedia.org/T161819
+        # --
+        # Raising this further from 8K to 32K, at least temporarily, due to T226840
+        # Note 32KB is also the similar default limit in our varnishes (http_resp_size)
+        proxy_buffer_size       32k;
+        proxy_buffers         4 32k;
+    }
+}

File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json
@@ -0,0 +1,14 @@
+{
+  "persistent": {
+    "cluster.remote.chi.seeds": [
+      "relforge1008.eqiad.wmnet:9300",
+      "relforge1009.eqiad.wmnet:9300",
+      "relforge1010.eqiad.wmnet:9300"
+    ],
+    "cluster.remote.omega.seeds": [
+      "relforge1008.eqiad.wmnet:9500",
+      "relforge1009.eqiad.wmnet:9500",
+      "relforge1010.eqiad.wmnet:9500"
+    ]
+  }
+}

File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom].orig
+++ File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]

+    owner  => root
+    group  => root
+    ensure => absent

Service[nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_shards_9400.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shards_9400]=/usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4

Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => present

Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha.orig
+++ /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead-relforge-eqiad-small-alpha
+
+/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6].orig
+++ Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]

+    for                => 3m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_health_check_for_shards_on_9400
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch health check for shards on 9400
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="2a63eb7dcbe368b7f538118765d629d6",check_name="check_opensearch_shards_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shards_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch health check for shards on 9400

Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]

Parameters differences:

--- Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA].orig
+++ Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]

+    command => /usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-certificates/wikimedia/Wikimedia_Internal_Root_CA.crt -storepass changeit -alias wmf:Wikimedia_Internal_Root_CA

+    group   => root
+    user    => root
+    unless  => /usr/bin/keytool -list -cacerts -noprompt -storepass changeit -alias wmf:Wikimedia_Internal_Root_CA

File[/etc/ferm/conf.d/10_opensearch_inter_node_9300]

Content differences:

--- /etc/ferm/conf.d/10_opensearch_inter_node_9300.orig
+++ /etc/ferm/conf.d/10_opensearch_inter_node_9300
@@ -4,3 +4,5 @@
 &R_SERVICE(tcp, 9300, (10.64.164.14 10.64.32.100 10.64.48.120 2620:0:861:103:10:64:32:100 2620:0:861:107:10:64:48:120 2620:0:861:12c:10:64:164:14));
 
 
+
+&NO_TRACK(tcp, 9300);

File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]

Parameters differences:

--- File[/usr/local/bin/set-cross-cluster-seeds_9200.sh].orig
+++ File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]

+    owner  => root
+    mode   => 0700
+    group  => root
+    ensure => file

Content differences:

--- /usr/local/bin/set-cross-cluster-seeds_9200.sh.orig
+++ /usr/local/bin/set-cross-cluster-seeds_9200.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# set-cross-cluster-seeds.sh. Given a json file with opensearch remote seeds,
+# update OpenSearch cluster with correct settings.
+# Author: Brian King
+
+set -eo pipefail
+
+data="/etc/opensearch/relforge-eqiad/cirrus_check_settings.json"
+
+logger "updating cross cluster seed settings via ${data}"
+
+curl -XPUT \
+  -H 'content-type: application/json' \
+  --data "@${data}" \
+  http://localhost:9200/_cluster/settings

Motd::Script[cirrus::relforge]

Parameters differences:

--- Motd::Script[cirrus::relforge].orig
+++ Motd::Script[cirrus::relforge]

+    priority => 5
+    ensure   => present

Class[Profile::Opensearch::Cirrus::Server]

Parameters differences:

--- Class[Profile::Opensearch::Cirrus::Server].orig
+++ Class[Profile::Opensearch::Cirrus::Server]

+    ferm_srange                     => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    certificate_domains             => ['relforge.svc.eqiad.wmnet']
+    enable_performance_cpu_governor => False
+    enable_remote_search            => False
+    base_data_dir                   => /srv/opensearch
+    ferm_ro_srange                  => 
+    storage_device                  => md1
+    version                         => 1.3.20
+    expose_http                     => False
+    cluster                         => relforge
+    ssl_provider                    => cfssl

Package[opensearch]

Parameters differences:

--- Package[opensearch].orig
+++ Package[opensearch]

+    provider => apt
+    ensure   => 1.3.20

Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter-9400].orig
+++ Service[prometheus-elasticsearch-exporter-9400]

+    ensure => running
+    enable => True

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9200.timer

Sysctl::Conffile[opensearch]

Parameters differences:

--- Sysctl::Conffile[opensearch].orig
+++ Sysctl::Conffile[opensearch]

+    ensure             => present
+    priority           => 70
+    no_priority_prefix => True

File[/lib/systemd/system/clean_up_dlq.service]

Parameters differences:

--- File[/lib/systemd/system/clean_up_dlq.service].orig
+++ File[/lib/systemd/system/clean_up_dlq.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]

Content differences:

--- /lib/systemd/system/clean_up_dlq.service.orig
+++ /lib/systemd/system/clean_up_dlq.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Clean up dead letter queue and restart logstash
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/cleanup-dlq

File[/etc/logrotate.d/push_cross_cluster_settings_9400]

Parameters differences:

--- File[/etc/logrotate.d/push_cross_cluster_settings_9400].orig
+++ File[/etc/logrotate.d/push_cross_cluster_settings_9400]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/push_cross_cluster_settings_9400.orig
+++ /etc/logrotate.d/push_cross_cluster_settings_9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for push_cross_cluster_settings_9400
+
+/var/log/push_cross_cluster_settings_9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Monitor[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Monitor[push_cross_cluster_settings_9200].orig
+++ Systemd::Monitor[push_cross_cluster_settings_9200]

+    retries        => 2
+    check_interval => 10
+    ensure         => present
+    migration_task => T407130
+    contact_group  => admins
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    critical       => False

Apt::Repository[wikimedia-opensearch]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch].orig
+++ Apt::Repository[wikimedia-opensearch]

+    before                   => Class[Opensearch]
+    bin                      => True
+    components               => thirdparty/opensearch1
+    ensure                   => present
+    dist                     => bookworm-wikimedia
+    source                   => True
+    trust_repo               => False
+    uri                      => http://apt.wikimedia.org/wikimedia
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    allow_releaseinfo_change => False

Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Logrotate::Conf[opensearch]

Parameters differences:

--- Logrotate::Conf[opensearch].orig
+++ Logrotate::Conf[opensearch]

+    ensure => present

File[/etc/elasticsearch]

Parameters differences:

--- File[/etc/elasticsearch].orig
+++ File[/etc/elasticsearch]

+    owner   => root
+    target  => /etc/opensearch
+    group   => root
+    require => File[/etc/opensearch/instances]
+    ensure  => link

File[/var/log/clean_up_dlq]

Parameters differences:

--- File[/var/log/clean_up_dlq].orig
+++ File[/var/log/clean_up_dlq]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => absent
+    mode   => 0755
+    backup => False

Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shards_9400.timer

File[/var/log/push_cross_cluster_settings_9400]

Parameters differences:

--- File[/var/log/push_cross_cluster_settings_9400].orig
+++ File[/var/log/push_cross_cluster_settings_9400]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Apt::Repository[wikimedia-opensearch-plugins]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch-plugins].orig
+++ Apt::Repository[wikimedia-opensearch-plugins]

+    bin                      => True
+    components               => component/opensearch13
+    ensure                   => present
+    dist                     => bookworm-wikimedia
+    source                   => True
+    trust_repo               => False
+    uri                      => http://apt.wikimedia.org/wikimedia
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    allow_releaseinfo_change => False

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service

File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)].orig
+++ Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]

+    command     => /bin/systemctl daemon-reload
+    notify      => ['Service[prometheus-wmf-elasticsearch-exporter-9200]']
+    refreshonly => True

Concat_fragment[wikimedia-opensearch-header]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-header].orig
+++ Concat_fragment[wikimedia-opensearch-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    order  => 01

Motd::Script[cluster_memberships]

Parameters differences:

--- Motd::Script[cluster_memberships].orig
+++ Motd::Script[cluster_memberships]

+    source   => puppet:///modules/opensearch/opensearch.motd
+    priority => 96
+    ensure   => present

Systemd::Service[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shards_9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Class[Opensearch]

Parameters differences:

--- Class[Opensearch].orig
+++ Class[Opensearch]

+    before                  => ['File[/usr/share/opensearch/plugins]']
+    enable_curator          => False
+    default_instance_params => {}
+    plugins_mandatory       => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    instances               => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    rack                    => C4
+    row                     => eqiad-row-c
+    native_lib_path         => /usr/share/opensearch/plugins/opensearch-knn/lib
+    logstash_logback_port   => 11514
+    base_data_dir           => /srv/opensearch
+    logstash_host           => localhost
+    version                 => 1.3.20
+    java_home               => /usr/lib/jvm/java-11-openjdk-amd64

Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status].orig
+++ Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]

+    check_command          => nrpe_check!check_check_push_cross_cluster_settings_9400_status!10
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => admins
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 10
+    notifications_enabled  => 0
+    max_check_attempts     => 2
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => Check unit status of push_cross_cluster_settings_9400
+    notes_url              => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer

File[/etc/opensearch/relforge-eqiad/opensearch.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/opensearch.yml].orig
+++ File[/etc/opensearch/relforge-eqiad/opensearch.yml]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad/opensearch.yml.orig
+++ /etc/opensearch/relforge-eqiad/opensearch.yml
@@ -0,0 +1,199 @@
+# ======================== OpenSearch Configuration =========================
+#
+# NOTE: OpenSearch comes with reasonable defaults for most settings.
+#       Before you set out to tweak and tune the configuration, make sure you
+#       understand what are you trying to accomplish and the consequences.
+#
+# The primary way of configuring a node is via this file. This template lists
+# the most important settings you may want to configure for a production cluster.
+#
+# Please consult the documentation for further information on configuration options:
+# https://opensearch.org/docs/opensearch/configuration/
+#
+# ---------------------------------- Cluster -----------------------------------
+#
+# Use a descriptive name for your cluster:
+#
+cluster.name: relforge-eqiad
+#
+# ------------------------------------ Node ------------------------------------
+#
+# Use a descriptive name for the node:
+#
+node.name: relforge1008-relforge-eqiad
+#
+# Add custom attributes to the node:
+#
+#node.attr.rack: r1
+#
+# Row/rack awareness attributes
+#
+cluster.routing.allocation.awareness.attributes: row
+node.attr.row: eqiad-row-c
+node.attr.rack: C4
+#
+# Other attributes
+#
+node.attr.hostname: relforge1008
+node.attr.fqdn: relforge1008.eqiad.wmnet
+#
+# Allow this node to be eligible as a master node (enabled by default):
+node.master: true
+# Allow this node to store data (enabled by default):
+node.data: true
+#
+# ----------------------------------- Paths ------------------------------------
+#
+# Path to directory where to store the data (separate multiple locations by comma):
+#
+path.data: /srv/opensearch/relforge-eqiad
+#
+# Path to log files:
+#
+path.logs: /var/log/opensearch
+#
+# ----------------------------------- Memory -----------------------------------
+#
+# Lock the memory on startup:
+#
+bootstrap.memory_lock: false # we don't have swap anyway
+#
+# Make sure that the heap size is set to about half the memory available
+# on the system and that the owner of the process is allowed to use this
+# limit.
+#
+# OpenSearch performs poorly when the system is swapping the memory.
+#
+# ---------------------------------- Network -----------------------------------
+#
+# Set the bind address to a specific IP (IPv4 or IPv6):
+#
+#network.host: 192.168.0.1
+#
+# Set a custom port for HTTP:
+#
+http.port: 9200
+#
+# For more information, consult the network module documentation.
+#
+# Set the address other nodes will use to communicate with this node. If not
+# set, it is automatically derived. It must point to an actual IP address.
+#
+network.publish_host: 10.64.32.100
+
+# Set both 'bind_host' and 'publish_host':
+#
+network.host: [ _local_,_site_ ]
+
+# Set a custom port for the node to node communication (9300 by default):
+#
+transport.tcp.port: 9300
+#
+# --------------------------------- Discovery ----------------------------------
+#
+# Pass an initial list of hosts to perform discovery when this node is started:
+# The default list of hosts is ["127.0.0.1", "[::1]"]
+#
+#discovery.seed_hosts: ["host1", "host2"]
+#
+# Bootstrap the cluster using an initial set of master-eligible nodes:
+#
+#cluster.initial_master_nodes: ["node-1", "node-2"]
+#
+discovery.zen.ping.unicast.hosts: ["relforge1008.eqiad.wmnet", "relforge1009.eqiad.wmnet", "relforge1010.eqiad.wmnet"]
+#
+# For more information, consult the discovery and cluster formation module documentation.
+#
+# ---------------------------------- Gateway -----------------------------------
+#
+# Block initial recovery after a full cluster restart until N nodes are started:
+#
+gateway.recover_after_nodes: 1
+#
+# For more information, consult the gateway module documentation.
+#
+#
+gateway:
+    recover_after_time: 1m
+    expected_nodes: 2
+#
+# ---------------------------------- Various -----------------------------------
+#
+# Require explicit names when deleting indices:
+#
+#action.destructive_requires_name: true
+#
+# ---------------------------------- Custom settings ---------------------------
+#
+# If a plugin listed here is not installed for current node, the node will not start.
+#
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+
+# Specify the pattern of index names allowed to be created automatically
+action.auto_create_index: true
+
+# Protect against accidental close/delete operations on all indices. You can
+# still close/delete individual indices.
+action.destructive_requires_name: true
+
+# Allow up to 1000 shards to be queried at a time. The default
+# 1k is too low to allow mwgrep to operate.
+action.search.shard_count.limit: 1000
+
+# Enable the disk space aware shard allocator
+cluster.routing.allocation.disk.threshold_enabled: true
+# Stop allocating new shards to nodes when this threshold is reached
+cluster.routing.allocation.disk.watermark.low: 0.75
+# Move shards off of nodes when this threshold is reached
+cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node when this threshold is reached
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+# Check disk utilization every 60 seconds
+cluster.info.update.interval: 60s
+
+# Set the number of concurrent recoveries happening on a node:
+#
+# 1. During the initial recovery
+cluster.routing.allocation.node_initial_primaries_recoveries: 3
+
+# 2. During adding/removing nodes, rebalancing, etc
+cluster.routing.allocation.node_concurrent_recoveries: 2
+
+# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
+indices.recovery.max_bytes_per_sec: 40mb
+
+##
+# OpenSearch cluster balance configuration
+##
+# This puts much more weight on distributing the shards per index then the defaults.  This will make sure that,
+# for example, each live node gets a copy of enwiki.
+cluster.routing.allocation.balance:
+    shard: 0.195
+    index: 0.8
+
+##
+# Filter cache size
+##
+indices.queries.cache.size: 20%
+
+##
+# Thread pool settings
+##
+thread_pool:
+    write:
+        size: 32
+        queue_size: 1000
+
+
+
+
+##
+# Hosts allowed as data sources for reindexing
+reindex.remote.whitelist: "search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243"
+
+
+##
+# Size the LTR cache to prevent churn under normal and AB testing scenarios
+ltr.caches.max_mem: 100mb
+
+

Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => present

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]

+    before      => ['Service[opensearch-disable-readahead-relforge-eqiad.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9400.service

Class[Opensearch::Packages]

Parameters differences:

--- Class[Opensearch::Packages].orig
+++ Class[Opensearch::Packages]

+    send_logs_to_logstash => True
+    version               => 1.3.20

Exec[s3-credentials-user-relforge-eqiad]

Parameters differences:

--- Exec[s3-credentials-user-relforge-eqiad].orig
+++ Exec[s3-credentials-user-relforge-eqiad]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad']
+    command     => echo 'search:platform' | /usr/share/opensearch/bin/opensearch-keystore add s3.client.default.access_key
+    group       => opensearch
+    require     => File[/etc/opensearch/relforge-eqiad/opensearch.keystore]
+    path        => /bin:/usr/bin
+    unless      => /usr/share/opensearch/bin/opensearch-keystore list | grep s3.client.default.access_key

Class[Logstash]

Parameters differences:

--- Class[Logstash].orig
+++ Class[Logstash]

+    pipeline_batch_delay => 50
+    dlq_hosts            => []
+    gc_log               => True
+    log_format           => json
+    pipeline_workers     => 2
+    heap_memory          => 192m
+    pipeline_batch_size  => 125
+    enable_dlq           => False
+    dlq_max_bytes        => 1024mb
+    java_package         => openjdk-17-jdk
+    version              => 7.16.3-1
+    manage_service       => True

Systemd::Unit[push_cross_cluster_settings_9200.service]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9200.service].orig
+++ Systemd::Unit[push_cross_cluster_settings_9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => push_cross_cluster_settings_9200.service

Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Service[clean_up_dlq.timer]

Parameters differences:

--- Service[clean_up_dlq.timer].orig
+++ Service[clean_up_dlq.timer]

+    before   => ['Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead-relforge-eqiad-small-alpha" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup.orig
+++ /etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-relforge-eqiad-gc-log-cleanup
+
+/var/log/opensearch-relforge-eqiad-gc-log-cleanup/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Service[opensearch_1@relforge-eqiad]

Parameters differences:

--- Service[opensearch_1@relforge-eqiad].orig
+++ Service[opensearch_1@relforge-eqiad]

+    provider => systemd
+    tag      => opensearch_services
+    require  => ['Systemd::Unit[opensearch_1@.service]', 'File[/etc/opensearch/relforge-eqiad/opensearch.yml]', 'File[/etc/opensearch/relforge-eqiad/logging.yml]', 'File[/etc/opensearch/relforge-eqiad/log4j2.properties]', 'File[/etc/opensearch/relforge-eqiad/jvm.options]', 'File[/srv/opensearch/relforge-eqiad]', 'Systemd::Tmpfile[opensearch-relforge-eqiad]']
+    ensure   => running
+    enable   => True

Systemd::Unit[opensearch-disable-readahead.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead.timer

Concat_fragment[wikimedia-opensearch-plugins-header]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-plugins-header].orig
+++ Concat_fragment[wikimedia-opensearch-plugins-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    order  => 01

Prometheus::Elasticsearch_exporter[localhost:9400]

Parameters differences:

--- Prometheus::Elasticsearch_exporter[localhost:9400].orig
+++ Prometheus::Elasticsearch_exporter[localhost:9400]

+    extra_config       => 
+    prometheus_port    => 9109
+    elasticsearch_port => 9400

File[/var/log/opensearch/opensearch.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch.log].orig
+++ File[/var/log/opensearch/opensearch.log]

+    owner  => root
+    group  => root
+    ensure => absent

Package[logstash-old-name]

Parameters differences:

--- Package[logstash-old-name].orig
+++ Package[logstash-old-name]

+    name     => logstash
+    provider => apt
+    before   => Package[logstash-oss]
+    ensure   => absent

Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad /srv/opensearch
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]

+    unit_name          => opensearch-relforge-eqiad-gc-log-cleanup.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}]
+    accuracy           => 15sec

File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shards_9200]=/usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+
+[Timer]
+Unit=nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=5min
+OnActiveSec=1s
+RandomizedDelaySec=300
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shards_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shards_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "fd6eadf747c645a72e11bd74109a8156" --timeout 10 --check-command "check_opensearch_shards_9200"

Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    ensure => present

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+
+[Timer]
+Unit=nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=5min
+OnActiveSec=1s
+RandomizedDelaySec=300
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd reload for nginx systemd fragments]

Parameters differences:

--- Exec[systemd reload for nginx systemd fragments].orig
+++ Exec[systemd reload for nginx systemd fragments]

+    command     => /bin/systemctl daemon-reload
+    subscribe   => ['File[/etc/systemd/system/nginx.service.d/numa.conf]', 'File[/etc/systemd/system/nginx.service.d/security.conf]']
+    refreshonly => True

File[/lib/systemd/system/push_cross_cluster_settings_9200.service]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9200.service].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9200.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9200.service.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto set remote cluster seeds for relforge-eqiad
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/bin/bash /usr/local/bin/set-cross-cluster-seeds_9200.sh

Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Ferm::Service[elastic-main-https-9243]

Parameters differences:

--- Ferm::Service[elastic-main-https-9243].orig
+++ Ferm::Service[elastic-main-https-9243]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => False
+    srange              => (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49)
+    prio                => 10
+    desc                => 
+    port                => 9243
+    proto               => tcp

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "a0a4892478c289e50484ffd2fb4c3202" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9400"
+    private_tmp               => False
+    splay                     => 43200
+    description               => execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9400 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_unassigned_shard_check_9400
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
+    ensure   => present

Logstash::Conf[output-udp-gelf_relay]

Parameters differences:

--- Logstash::Conf[output-udp-gelf_relay].orig
+++ Logstash::Conf[output-udp-gelf_relay]

+    priority => 90
+    ensure   => present

Monitoring::Service[check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Monitoring::Service[check_push_cross_cluster_settings_9400_status].orig
+++ Monitoring::Service[check_push_cross_cluster_settings_9400_status]

+    check_command  => nrpe_check!check_check_push_cross_cluster_settings_9400_status!10
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 10
+    config_dir     => /etc/nagios
+    migration_task => T407130
+    contact_group  => admins
+    host           => relforge1008
+    critical       => False
+    retries        => 2
+    passive        => False
+    description    => Check unit status of push_cross_cluster_settings_9400
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]
+    ensure   => present

File[/usr/lib/nagios/plugins/check_elasticsearch.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch.py]

+    owner  => root
+    mode   => 0755
+    group  => root
+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch.py

Exec[apt_repository_wikimedia-opensearch]

Parameters differences:

--- Exec[apt_repository_wikimedia-opensearch].orig
+++ Exec[apt_repository_wikimedia-opensearch]

+    command     => /usr/bin/apt-get update 
+    refreshonly => True

Logrotate::Conf[push_cross_cluster_settings_9200]

Parameters differences:

--- Logrotate::Conf[push_cross_cluster_settings_9200].orig
+++ Logrotate::Conf[push_cross_cluster_settings_9200]

+    ensure => present

File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml
@@ -0,0 +1,199 @@
+# ======================== OpenSearch Configuration =========================
+#
+# NOTE: OpenSearch comes with reasonable defaults for most settings.
+#       Before you set out to tweak and tune the configuration, make sure you
+#       understand what are you trying to accomplish and the consequences.
+#
+# The primary way of configuring a node is via this file. This template lists
+# the most important settings you may want to configure for a production cluster.
+#
+# Please consult the documentation for further information on configuration options:
+# https://opensearch.org/docs/opensearch/configuration/
+#
+# ---------------------------------- Cluster -----------------------------------
+#
+# Use a descriptive name for your cluster:
+#
+cluster.name: relforge-eqiad-small-alpha
+#
+# ------------------------------------ Node ------------------------------------
+#
+# Use a descriptive name for the node:
+#
+node.name: relforge1008-relforge-eqiad-small-alpha
+#
+# Add custom attributes to the node:
+#
+#node.attr.rack: r1
+#
+# Row/rack awareness attributes
+#
+cluster.routing.allocation.awareness.attributes: row
+node.attr.row: eqiad-row-c
+node.attr.rack: C4
+#
+# Other attributes
+#
+node.attr.hostname: relforge1008
+node.attr.fqdn: relforge1008.eqiad.wmnet
+#
+# Allow this node to be eligible as a master node (enabled by default):
+node.master: true
+# Allow this node to store data (enabled by default):
+node.data: true
+#
+# ----------------------------------- Paths ------------------------------------
+#
+# Path to directory where to store the data (separate multiple locations by comma):
+#
+path.data: /srv/opensearch/relforge-eqiad-small-alpha
+#
+# Path to log files:
+#
+path.logs: /var/log/opensearch
+#
+# ----------------------------------- Memory -----------------------------------
+#
+# Lock the memory on startup:
+#
+bootstrap.memory_lock: false # we don't have swap anyway
+#
+# Make sure that the heap size is set to about half the memory available
+# on the system and that the owner of the process is allowed to use this
+# limit.
+#
+# OpenSearch performs poorly when the system is swapping the memory.
+#
+# ---------------------------------- Network -----------------------------------
+#
+# Set the bind address to a specific IP (IPv4 or IPv6):
+#
+#network.host: 192.168.0.1
+#
+# Set a custom port for HTTP:
+#
+http.port: 9400
+#
+# For more information, consult the network module documentation.
+#
+# Set the address other nodes will use to communicate with this node. If not
+# set, it is automatically derived. It must point to an actual IP address.
+#
+network.publish_host: 10.64.32.100
+
+# Set both 'bind_host' and 'publish_host':
+#
+network.host: [ _local_,_site_ ]
+
+# Set a custom port for the node to node communication (9300 by default):
+#
+transport.tcp.port: 9500
+#
+# --------------------------------- Discovery ----------------------------------
+#
+# Pass an initial list of hosts to perform discovery when this node is started:
+# The default list of hosts is ["127.0.0.1", "[::1]"]
+#
+#discovery.seed_hosts: ["host1", "host2"]
+#
+# Bootstrap the cluster using an initial set of master-eligible nodes:
+#
+#cluster.initial_master_nodes: ["node-1", "node-2"]
+#
+discovery.zen.ping.unicast.hosts: ["relforge1008.eqiad.wmnet", "relforge1009.eqiad.wmnet", "relforge1010.eqiad.wmnet"]
+#
+# For more information, consult the discovery and cluster formation module documentation.
+#
+# ---------------------------------- Gateway -----------------------------------
+#
+# Block initial recovery after a full cluster restart until N nodes are started:
+#
+gateway.recover_after_nodes: 1
+#
+# For more information, consult the gateway module documentation.
+#
+#
+gateway:
+    recover_after_time: 1m
+    expected_nodes: 2
+#
+# ---------------------------------- Various -----------------------------------
+#
+# Require explicit names when deleting indices:
+#
+#action.destructive_requires_name: true
+#
+# ---------------------------------- Custom settings ---------------------------
+#
+# If a plugin listed here is not installed for current node, the node will not start.
+#
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+
+# Specify the pattern of index names allowed to be created automatically
+action.auto_create_index: true
+
+# Protect against accidental close/delete operations on all indices. You can
+# still close/delete individual indices.
+action.destructive_requires_name: true
+
+# Allow up to 1000 shards to be queried at a time. The default
+# 1k is too low to allow mwgrep to operate.
+action.search.shard_count.limit: 1000
+
+# Enable the disk space aware shard allocator
+cluster.routing.allocation.disk.threshold_enabled: true
+# Stop allocating new shards to nodes when this threshold is reached
+cluster.routing.allocation.disk.watermark.low: 0.75
+# Move shards off of nodes when this threshold is reached
+cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node when this threshold is reached
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+# Check disk utilization every 60 seconds
+cluster.info.update.interval: 60s
+
+# Set the number of concurrent recoveries happening on a node:
+#
+# 1. During the initial recovery
+cluster.routing.allocation.node_initial_primaries_recoveries: 3
+
+# 2. During adding/removing nodes, rebalancing, etc
+cluster.routing.allocation.node_concurrent_recoveries: 2
+
+# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
+indices.recovery.max_bytes_per_sec: 40mb
+
+##
+# OpenSearch cluster balance configuration
+##
+# This puts much more weight on distributing the shards per index then the defaults.  This will make sure that,
+# for example, each live node gets a copy of enwiki.
+cluster.routing.allocation.balance:
+    shard: 0.195
+    index: 0.8
+
+##
+# Filter cache size
+##
+indices.queries.cache.size: 20%
+
+##
+# Thread pool settings
+##
+thread_pool:
+    write:
+        size: 32
+        queue_size: 1000
+
+
+
+
+##
+# Hosts allowed as data sources for reindexing
+reindex.remote.whitelist: "search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243"
+
+
+##
+# Size the LTR cache to prevent churn under normal and AB testing scenarios
+ltr.caches.max_mem: 100mb
+
+

File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf].orig
+++ File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf.orig
+++ /etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "push_cross_cluster_settings_9400" then {
+    action(
+        type="omfile" file="/var/log/push_cross_cluster_settings_9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Unit[opensearch-disable-readahead.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead.service].orig
+++ Systemd::Unit[opensearch-disable-readahead.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead.service

Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Class[Profile::Java]

Parameters differences:

--- Class[Profile::Java].orig
+++ Class[Profile::Java]

+    trust_puppet_ca => True
+    enable_dbg      => False
+    hardened_tls    => False
+    extra_args      => {}
+    egd_source      => /dev/random
+    java_packages   => []

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service

Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]

Parameters differences:

--- Exec[symlink /etc/opensearch/relforge-eqiad/sudachi].orig
+++ Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]

+    creates => /etc/opensearch/relforge-eqiad/sudachi
+    command => /usr/bin/ln -s '/usr/share/opensearch/config/sudachi' '/etc/opensearch/relforge-eqiad/sudachi'
+    onlyif  => /usr/bin/test -d '/usr/share/opensearch/config/sudachi'

Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]

+    subscribe                => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    override                 => False
+    restart                  => True
+    require                  => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    ensure                   => present
+    unit_type                => service
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/opensearch]

Parameters differences:

--- File[/etc/opensearch].orig
+++ File[/etc/opensearch]

+    owner   => root
+    group   => root
+    recurse => True
+    force   => True
+    ensure  => directory
+    mode    => 0755
+    purge   => True

File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/opensearch.keystore].orig
+++ File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

+    owner => root
+    mode  => 0640
+    group => opensearch

File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9200.timer].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9200.timer.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9200.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of push_cross_cluster_settings_9200.service
+
+[Timer]
+Unit=push_cross_cluster_settings_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=15min
+OnActiveSec=1s
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/etc/update-motd.d/96-cluster-memberships]

Parameters differences:

--- File[/etc/update-motd.d/96-cluster-memberships].orig
+++ File[/etc/update-motd.d/96-cluster-memberships]

+    owner  => root
+    source => puppet:///modules/opensearch/opensearch.motd
+    group  => root
+    ensure => present
+    mode   => 0555

Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/usr/local/bin/jheapdump]

Parameters differences:

--- File[/usr/local/bin/jheapdump].orig
+++ File[/usr/local/bin/jheapdump]

+    owner  => root
+    source => puppet:///modules/java/jheapdump
+    group  => root
+    ensure => file
+    mode   => 0555

File[/etc/sysctl.d/70-TCP-Fast-Open.conf]

Parameters differences:

--- File[/etc/sysctl.d/70-TCP-Fast-Open.conf].orig
+++ File[/etc/sysctl.d/70-TCP-Fast-Open.conf]

+    owner  => root
+    notify => Exec[update_sysctl]
+    group  => root
+    ensure => present

Content differences:

--- /etc/sysctl.d/70-TCP-Fast-Open.conf.orig
+++ /etc/sysctl.d/70-TCP-Fast-Open.conf
@@ -0,0 +1,2 @@
+# sysctl parameters managed by Puppet.
+net.ipv4.tcp_fastopen = 3

Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha].orig
+++ Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]

+    key          => s3.client.default.access_key
+    value        => search:platform
+    cluster_name => relforge-eqiad-small-alpha

File[/etc/logrotate.d/nginx]

Parameters differences:

--- File[/etc/logrotate.d/nginx].orig
+++ File[/etc/logrotate.d/nginx]

+    before => ['Service[nginx]']
+    owner  => root
+    source => puppet:///modules/profile/tlsproxy/logrotate
+    group  => root
+    ensure => present
+    mode   => 0444

Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha].orig
+++ Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]

+    enable_remote_search => False
+    instance_name        => relforge-eqiad-small-alpha
+    settings             => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shard_size_check_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Rsyslog::Conf[clean_up_dlq]

Parameters differences:

--- Rsyslog::Conf[clean_up_dlq].orig
+++ Rsyslog::Conf[clean_up_dlq]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/clean_up_dlq]
+    ensure   => absent

File[/usr/local/bin/logstash-config-test]

Parameters differences:

--- File[/usr/local/bin/logstash-config-test].orig
+++ File[/usr/local/bin/logstash-config-test]

+    owner   => root
+    source  => puppet:///modules/logstash/logstash-config-test
+    group   => root
+    require => Package[logstash-oss]
+    mode    => 0555

Systemd::Timer::Job[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Service[logstash]

Parameters differences:

--- Service[logstash].orig
+++ Service[logstash]

+    provider   => systemd
+    hasrestart => True
+    ensure     => running
+    hasstatus  => True
+    enable     => True

Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]

+    before      => ['Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]

+    owner  => root
+    group  => root
+    ensure => absent

Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)].orig
+++ Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet].orig
+++ Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    unless      => /usr/bin/openssl x509 -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem -checkend 952200
+    require     => Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]

Monitoring::Service[opensearch_shards_9400]

Parameters differences:

--- Monitoring::Service[opensearch_shards_9400].orig
+++ Monitoring::Service[opensearch_shards_9400]

+    check_command  => nrpe_check!check_opensearch_shards_9400!10
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => OpenSearch health check for shards on 9400
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration

File[/etc/logrotate.d/opensearch]

Parameters differences:

--- File[/etc/logrotate.d/opensearch].orig
+++ File[/etc/logrotate.d/opensearch]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/opensearch.orig
+++ /etc/logrotate.d/opensearch
@@ -0,0 +1,12 @@
+# This file is managed by Puppet.
+# puppet:///logrotate/logrotate.erb
+
+/var/log/opensearch/*.log {
+    notifempty
+    daily
+    copytruncate
+    rotate 7
+    compress
+    delaycompress
+    missingok
+}

Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800].orig
+++ Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]

+    for                => 1800m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_shard_size_check___9200
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch shard size check - 9200
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="c62725841c61e20ac3f6819b4c0ad800",check_name="check_opensearch_shard_size_check_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shard_size_check_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch shard size check - 9200

Systemd::Service[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Service[opensearch-disable-readahead-relforge-eqiad]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    ensure => present

Exec[unmask_nginx.service]

Parameters differences:

--- Exec[unmask_nginx.service].orig
+++ Exec[unmask_nginx.service]

+    command     => /bin/systemctl unmask nginx.service
+    onlyif      => /bin/readlink -f /etc/systemd/system/nginx.service | grep -q /dev/null
+    refreshonly => True

Monitoring::Service[opensearch_shards_9200]

Parameters differences:

--- Monitoring::Service[opensearch_shards_9200].orig
+++ Monitoring::Service[opensearch_shards_9200]

+    check_command  => nrpe_check!check_opensearch_shards_9200!10
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => OpenSearch health check for shards on 9200
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration

File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json].orig
+++ File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad/cirrus_check_settings.json.orig
+++ /etc/opensearch/relforge-eqiad/cirrus_check_settings.json
@@ -0,0 +1,14 @@
+{
+  "persistent": {
+    "cluster.remote.chi.seeds": [
+      "relforge1008.eqiad.wmnet:9300",
+      "relforge1009.eqiad.wmnet:9300",
+      "relforge1010.eqiad.wmnet:9300"
+    ],
+    "cluster.remote.omega.seeds": [
+      "relforge1008.eqiad.wmnet:9500",
+      "relforge1009.eqiad.wmnet:9500",
+      "relforge1010.eqiad.wmnet:9500"
+    ]
+  }
+}

Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    unit_name          => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+    fixed_random_delay => True
+    splay              => 300
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]

Parameters differences:

--- File[/usr/local/bin/set-cross-cluster-seeds_9400.sh].orig
+++ File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]

+    owner  => root
+    mode   => 0700
+    group  => root
+    ensure => file

Content differences:

--- /usr/local/bin/set-cross-cluster-seeds_9400.sh.orig
+++ /usr/local/bin/set-cross-cluster-seeds_9400.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# set-cross-cluster-seeds.sh. Given a json file with opensearch remote seeds,
+# update OpenSearch cluster with correct settings.
+# Author: Brian King
+
+set -eo pipefail
+
+data="/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json"
+
+logger "updating cross cluster seed settings via ${data}"
+
+curl -XPUT \
+  -H 'content-type: application/json' \
+  --data "@${data}" \
+  http://localhost:9400/_cluster/settings

Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)].orig
+++ Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources].orig
+++ Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

+    warn           => False
+    format         => plain
+    ensure         => present
+    force          => False
+    show_diff      => True
+    ensure_newline => False
+    order          => alpha
+    owner          => root
+    group          => root
+    path           => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    mode           => 0444
+    backup         => puppet
+    notify         => Exec[apt_repository_wikimedia-opensearch]
+    replace        => True

Class[Profile::Opensearch::Monitoring::Base_checks]

Parameters differences:

--- Class[Profile::Opensearch::Monitoring::Base_checks].orig
+++ Class[Profile::Opensearch::Monitoring::Base_checks]

+    threshold                     => >=0.15
+    shard_size_warning            => 110
+    require                       => ['Class[Profile::Opensearch::Server]']
+    enable_shard_size_check       => True
+    shard_size_critical           => 140
+    enable_unassigned_shard_check => True
+    timeout                       => 4
+    enable_health_check           => True

Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    ensure => present

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Ferm::Service[opensearch_inter_node_9300]

Parameters differences:

--- Ferm::Service[opensearch_inter_node_9300].orig
+++ Ferm::Service[opensearch_inter_node_9300]

@@
-    notrack => False
+    notrack => True
@@
-    port    => [9300]
+    port    => 9300

Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]

+    unit_name          => nrpe2nodexp-opensearch_shard_size_check_9200.service
+    fixed_random_delay => True
+    splay              => 36000
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400].orig
+++ File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]

+    path    => /etc/debdeploy-client/autorestarts.conf
+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    line    => prometheus-elasticsearch-exporter-9400
+    ensure  => present

Firewall::Service[opensearch-inter-node-9300]

Parameters differences:

--- Firewall::Service[opensearch-inter-node-9300].orig
+++ Firewall::Service[opensearch-inter-node-9300]

@@
-    notrack => False
+    notrack => True
@@
-    port    => [9300]
+    port    => 9300

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_unassigned_shard_check_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    unit_name          => nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+    fixed_random_delay => True
+    splay              => 43200
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Package[nginx-common]

Parameters differences:

--- Package[nginx-common].orig
+++ Package[nginx-common]

+    provider => apt
+    ensure   => installed

Logrotate::Conf[opensearch-disable-readahead]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead].orig
+++ Logrotate::Conf[opensearch-disable-readahead]

+    ensure => absent

Package[logstash-oss]

Parameters differences:

--- Package[logstash-oss].orig
+++ Package[logstash-oss]

+    ensure   => 1:7.16.3-1
+    provider => apt
+    require  => Package[openjdk-17-jdk]

File[/lib/systemd/system/clean_up_dlq.timer]

Parameters differences:

--- File[/lib/systemd/system/clean_up_dlq.timer].orig
+++ File[/lib/systemd/system/clean_up_dlq.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]

Content differences:

--- /lib/systemd/system/clean_up_dlq.timer.orig
+++ /lib/systemd/system/clean_up_dlq.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of clean_up_dlq.service
+
+[Timer]
+Unit=clean_up_dlq.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=1h
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Class[Profile::Logstash::Gelf_relay]

Parameters differences:

--- Class[Profile::Logstash::Gelf_relay].orig
+++ Class[Profile::Logstash::Gelf_relay]

+    version => 7.16.3-1

Exec[Refresh tmpfile opensearch-relforge-eqiad]

Parameters differences:

--- Exec[Refresh tmpfile opensearch-relforge-eqiad].orig
+++ Exec[Refresh tmpfile opensearch-relforge-eqiad]

+    command     => /bin/systemd-tmpfiles --create --remove '/etc/tmpfiles.d/opensearch-relforge-eqiad.conf'
+    subscribe   => File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]
+    user        => root
+    refreshonly => True

File[/srv/opensearch/relforge-eqiad]

Parameters differences:

--- File[/srv/opensearch/relforge-eqiad].orig
+++ File[/srv/opensearch/relforge-eqiad]

+    owner  => opensearch
+    mode   => 0755
+    group  => opensearch
+    ensure => directory

Exec[stop-default-nginx]

Parameters differences:

--- Exec[stop-default-nginx].orig
+++ Exec[stop-default-nginx]

+    before      => Service[nginx]
+    command     => /usr/sbin/service nginx stop
+    subscribe   => Package[nginx-light]
+    refreshonly => True

Ferm::Service[opensearch_http_9200]

Parameters differences:

--- Ferm::Service[opensearch_http_9200].orig
+++ Ferm::Service[opensearch_http_9200]

-    unrestricted_access => False
-    ensure              => present
-    notrack             => False
-    prio                => 10
-    src_sets            => ['ANALYTICS_NETWORKS']
-    desc                => 
-    port                => [9200]
-    proto               => tcp

Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "54ca99505516d5452912b573aa7d35d3" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9200"
+    private_tmp               => False
+    splay                     => 43200
+    description               => execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9200 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_unassigned_shard_check_9200
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]

+    unit_name          => nrpe2nodexp-opensearch_shards_9400.service
+    fixed_random_delay => True
+    splay              => 60
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008].orig
+++ Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]

+    for                => 1800m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_shard_size_check___9400
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch shard size check - 9400
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="aefe2d8b10db3bc84662052af1e20008",check_name="check_opensearch_shard_size_check_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shard_size_check_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch shard size check - 9400

Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shards_9200.timer

Nrpe::Check[check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_unassigned_shard_check_9400].orig
+++ Nrpe::Check[check_opensearch_unassigned_shard_check_9400]

+    before  => Monitoring::Service[opensearch_unassigned_shard_check_9400]
+    command => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4
+    ensure  => present

Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "aefe2d8b10db3bc84662052af1e20008" --timeout 10 --check-command "check_opensearch_shard_size_check_9400"
+    private_tmp               => False
+    splay                     => 36000
+    description               => execution of nrpe2nodexp for the check_opensearch_shard_size_check_9400 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_shard_size_check_9400
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Tlsproxy::Localssl[relforge-eqiad]

Parameters differences:

--- Tlsproxy::Localssl[relforge-eqiad].orig
+++ Tlsproxy::Localssl[relforge-eqiad]

+    keepalive_timeout  => 60
+    default_server     => True
+    require            => ['Class[Profile::Tlsproxy::Instance]']
+    keepalive_requests => 100
+    cfssl_paths        => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    skip_private       => False
+    access_log         => False
+    read_timeout       => 180
+    only_get_requests  => False
+    acme_chief         => False
+    server_name        => relforge1008.eqiad.wmnet
+    acme_certname      => relforge-eqiad
+    enable_http2       => False
+    server_aliases     => ['relforge.svc.eqiad.wmnet']
+    tls_ports          => [9243]
+    upstream_ports     => [9200]
+    upstream_ip        => 10.64.32.100

Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources].orig
+++ Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

+    warn           => False
+    format         => plain
+    ensure         => present
+    force          => False
+    show_diff      => True
+    ensure_newline => False
+    order          => alpha
+    owner          => root
+    group          => root
+    path           => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    mode           => 0444
+    backup         => puppet
+    notify         => Exec[apt_repository_wikimedia-opensearch-plugins]
+    replace        => True

Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400].orig
+++ Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Firewall::Service[opensearch-http-9200]

Parameters differences:

--- Firewall::Service[opensearch-http-9200].orig
+++ Firewall::Service[opensearch-http-9200]

-    unrestricted_access => False
-    ensure              => present
-    notrack             => False
-    prio                => 10
-    src_sets            => ['ANALYTICS_NETWORKS']
-    desc                => 
-    port                => [9200]
-    proto               => tcp

File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]

Content differences:

--- /lib/systemd/system/prometheus-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/prometheus-elasticsearch-exporter-9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus exporter for Elasticsearch
+Documentation=https://prometheus.io/docs/introduction/overview/
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/bin/prometheus-elasticsearch-exporter --es.uri=http://localhost:9200 --web.listen-address=:9108 
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]

+    before      => ['Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Alternatives::Java[17]

Parameters differences:

--- Alternatives::Java[17].orig
+++ Alternatives::Java[17]

+    require => Java::Package[openjdk-jdk-17]

Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Opensearch::Instance[relforge-eqiad]

Parameters differences:

--- Opensearch::Instance[relforge-eqiad].orig
+++ Opensearch::Instance[relforge-eqiad]

+    configure_curator                  => False
+    compatibility_mode                 => False
+    short_cluster_name                 => chi
+    base_data_dir                      => /srv/opensearch
+    use_cms_gc                         => False
+    recovery_max_bytes_per_sec         => 40mb
+    certificate_name                   => relforge.svc.eqiad.wmnet
+    script_max_compilations_per_minute => 10000
+    publish_host                       => 10.64.32.100
+    node_name                          => relforge1008-relforge-eqiad
+    plugins_mandatory                  => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    security_plugin_certificates       => {}
+    reindex_remote_whitelist           => search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243
+    auto_create_index                  => True
+    row                                => eqiad-row-c
+    watermarks                         => {}
+    recover_after_nodes                => 1
+    bind_networks                      => ['_local_', '_site_']
+    unicast_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    send_logs_to_logstash              => True
+    gc_log                             => True
+    require                            => Package[opensearch]
+    bulk_thread_pool_executors         => 32
+    ltr_cache_size                     => 100mb
+    cluster_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    bulk_thread_pool_capacity          => 1000
+    recover_after_time                 => 1m
+    holds_data                         => True
+    logstash_logback_port              => 11514
+    filter_cache_size                  => 20%
+    logstash_host                      => localhost
+    version                            => 1.3.20
+    search_shard_count_limit           => 1000
+    disable_security_plugin            => True
+    expected_nodes                     => 2
+    http_port                          => 9200
+    load_fixed_bitset_filters_eagerly  => True
+    tls_port                           => 9243
+    transport_tcp_port                 => 9300
+    heap_memory                        => 30G
+    cluster_name                       => relforge-eqiad
+    cms_gc_init_occupancy_fraction     => 75
+    rack                               => C4
+    curator_uses_unicast_hosts         => True
+    awareness_attributes               => row

File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400
+
+/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Java::Cacert[wmf:Wikimedia_Internal_Root_CA]

Parameters differences:

--- Java::Cacert[wmf:Wikimedia_Internal_Root_CA].orig
+++ Java::Cacert[wmf:Wikimedia_Internal_Root_CA]

+    owner     => root
+    require   => Alternatives::Java[17]
+    group     => root
+    ensure    => present
+    path      => /usr/share/ca-certificates/wikimedia/Wikimedia_Internal_Root_CA.crt
+    storepass => changeit

Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    unit_name          => nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+    fixed_random_delay => True
+    splay              => 43200
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[opensearch-disable-readahead]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead].orig
+++ Rsyslog::Conf[opensearch-disable-readahead]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/opensearch-disable-readahead]
+    ensure   => absent

Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    unit_name          => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00'}]
+    accuracy           => 15sec

Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

Package[prometheus-elasticsearch-exporter]

Parameters differences:

--- Package[prometheus-elasticsearch-exporter].orig
+++ Package[prometheus-elasticsearch-exporter]

+    provider => apt
+    ensure   => installed

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)].orig
+++ Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]

+    before      => ['Service[prometheus-elasticsearch-exporter-9400]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "379fbc6cbce5b297397c866a15729f1a" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9200_status"
+    private_tmp               => False
+    splay                     => 300
+    description               => execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9200_status command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-check_push_cross_cluster_settings_9200_status
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

+    owner  => root
+    mode   => 0400
+    group  => root
+    ensure => file

Content differences:

--- /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr.orig
+++ /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr
@@ -0,0 +1,14 @@
+{
+  "CN": "relforge1008.eqiad.wmnet",
+  "hosts": [
+    "relforge.svc.eqiad.wmnet",
+    "relforge1008.eqiad.wmnet"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[opensearch-create-keystore-relforge-eqiad-small-alpha].orig
+++ Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad-small-alpha']
+    command     => /usr/share/opensearch/bin/opensearch-keystore create
+    creates     => /etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore
+    before      => File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]

+    check_command          => nrpe_check!check_opensearch_shards_9200!10
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch health check for shards on 9200
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration

Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status].orig
+++ Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service

Service[nginx]

Parameters differences:

--- Service[nginx].orig
+++ Service[nginx]

+    hasrestart => True
+    enable     => True
+    require    => Package[nginx-light]
+    ensure     => running

Class[Profile::Monitoring]

Parameters differences:

--- Class[Profile::Monitoring].orig
+++ Class[Profile::Monitoring]

@@
-    nrpe_check_disk_options => -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_check_disk_options => -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

File[/etc/update-motd.d/05-cirrus--test]

Parameters differences:

--- File[/etc/update-motd.d/05-cirrus--test].orig
+++ File[/etc/update-motd.d/05-cirrus--test]

-    mode   => 0555
-    owner  => root
-    group  => root
-    ensure => present

Content differences:

--- /etc/update-motd.d/05-cirrus--test.orig
+++ /etc/update-motd.d/05-cirrus--test
@@ -1,2 +0,0 @@
-#!/bin/sh
-printf "%s\n" "relforge1008 is a Unpuppetised system for testing (cirrus::test)"

Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
+    ensure   => present

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shard_size_check_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shard_size_check_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=600min
+OnActiveSec=1s
+RandomizedDelaySec=36000
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/etc/opensearch/relforge-eqiad/scripts]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/scripts].orig
+++ File[/etc/opensearch/relforge-eqiad/scripts]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => directory

Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status].orig
+++ Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]

+    retry_interval              => 1
+    check_interval              => 10
+    ensure                      => present
+    migration_task              => T407130
+    nrpe_command                => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200
+    contact_group               => admins
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 2
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => Check unit status of push_cross_cluster_settings_9200
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Elasticsearch::Tlsproxy[relforge-eqiad]

Parameters differences:

--- Elasticsearch::Tlsproxy[relforge-eqiad].orig
+++ Elasticsearch::Tlsproxy[relforge-eqiad]

+    tls_port          => 9243
+    read_only         => False
+    acme_chief        => False
+    server_name       => relforge1008.eqiad.wmnet
+    enable_http2      => False
+    upstream_port     => 9200
+    server_aliases    => ['relforge.svc.eqiad.wmnet']
+    cfssl_paths       => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    certificate_names => []

Exec[java__cacert_wmf:puppetca.pem]

Parameters differences:

--- Exec[java__cacert_wmf:puppetca.pem].orig
+++ Exec[java__cacert_wmf:puppetca.pem]

+    command => /usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /etc/ssl/certs/Puppet_Internal_CA.pem -storepass changeit -alias wmf:puppetca.pem

+    group   => root
+    user    => root
+    unless  => /usr/bin/keytool -list -cacerts -noprompt -storepass changeit -alias wmf:puppetca.pem

File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]

Parameters differences:

--- Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha].orig
+++ Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]

+    tls_port          => 9443
+    read_only         => False
+    acme_chief        => False
+    server_name       => relforge1008.eqiad.wmnet
+    enable_http2      => False
+    upstream_port     => 9400
+    server_aliases    => ['relforge.svc.eqiad.wmnet']
+    cfssl_paths       => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    certificate_names => []

Nrpe::Monitor_service[opensearch_shards_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shards_9200].orig
+++ Nrpe::Monitor_service[opensearch_shards_9200]

+    retry_interval              => 1
+    ensure                      => present
+    check_interval              => 1
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 3
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch health check for shards on 9200
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration

File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]

Parameters differences:

--- File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf].orig
+++ File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]

+    owner        => logstash
+    group        => logstash
+    validate_cmd => /usr/local/bin/logstash-config-test %
+    ensure       => present
+    mode         => 0440
+    notify       => Service[logstash]

Content differences:

--- /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf.orig
+++ /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf
@@ -0,0 +1,11 @@
+input {
+  # Graylog extended logging format
+  gelf {
+    type  => "gelf"
+    port  => 12201
+    remap => false
+    host  => "localhost"
+    id   => "input/gelf/12201"
+    tags => ["input-gelf-relay"]
+  }
+}

File[/etc/ferm/conf.d/10_elastic-main-https-9243]

Parameters differences:

--- File[/etc/ferm/conf.d/10_elastic-main-https-9243].orig
+++ File[/etc/ferm/conf.d/10_elastic-main-https-9243]

+    owner   => root
+    group   => root
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    mode    => 0400
+    ensure  => present
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_elastic-main-https-9243.orig
+++ /etc/ferm/conf.d/10_elastic-main-https-9243
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9243, (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49));
+
+

Class[Adduser]

Parameters differences:

--- Class[Adduser].orig
+++ Class[Adduser]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[edac-utils]', 'Package[opensearch]', 'Package[python3-requests]', 'Package[wmf-opensearch-search-plugins]', 'Package[opensearch-madvise]', 'Package[logstash-oss]', 'Package[logstash-old-name]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[openjdk-17-jdk]', 'Package[prometheus-elasticsearch-exporter]', 'Package[nginx-light]', 'Package[nginx-common]']

Concat::Fragment[wikimedia-opensearch-plugins-header]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-plugins-header].orig
+++ Concat::Fragment[wikimedia-opensearch-plugins-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    order  => 01

File[/etc/logrotate.d/clean_up_dlq]

Parameters differences:

--- File[/etc/logrotate.d/clean_up_dlq].orig
+++ File[/etc/logrotate.d/clean_up_dlq]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => absent

Content differences:

--- /etc/logrotate.d/clean_up_dlq.orig
+++ /etc/logrotate.d/clean_up_dlq
@@ -0,0 +1,12 @@
+# logrotate(8) config for clean_up_dlq
+
+/var/log/clean_up_dlq/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Syslog[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead].orig
+++ Systemd::Syslog[opensearch-disable-readahead]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => absent
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]

+    owner  => root
+    group  => root
+    ensure => absent

File[/srv/opensearch]

Parameters differences:

--- File[/srv/opensearch].orig
+++ File[/srv/opensearch]

+    owner   => opensearch
+    group   => opensearch
+    require => Package[opensearch]
+    mode    => 0755
+    ensure  => directory

Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9200.service

Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-relforge-eqiad-gc-log-cleanup.timer

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9200_status command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-check_push_cross_cluster_settings_9200_status
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "379fbc6cbce5b297397c866a15729f1a" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9200_status"

Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200].orig
+++ Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Logrotate::Conf[clean_up_dlq]

Parameters differences:

--- Logrotate::Conf[clean_up_dlq].orig
+++ Logrotate::Conf[clean_up_dlq]

+    ensure => absent

Sysctl::Parameters[TCP Fast Open]

Parameters differences:

--- Sysctl::Parameters[TCP Fast Open].orig
+++ Sysctl::Parameters[TCP Fast Open]

+    ensure             => present
+    no_priority_prefix => False
+    priority           => 70
+    values             => {'net.ipv4.tcp_fastopen': 3}

File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg.orig
+++ /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_check_push_cross_cluster_settings_9400_status]=/usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400

Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    mode     => 0444
+    priority => 25
+    ensure   => absent

File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Systemd::Mask[nginx.service]

Parameters differences:

--- Systemd::Mask[nginx.service].orig
+++ Systemd::Mask[nginx.service]

+    before => ['Package[nginx-light]']
+    unless => /usr/bin/dpkg -s nginx-light | /bin/grep -q '^Status: install ok installed$'
+    unit   => nginx.service

Systemd::Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Service[prometheus-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    require                  => Package[prometheus-elasticsearch-exporter]
+    ensure                   => present
+    unit_type                => service
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {'ensure': 'running'}
+    monitoring_critical      => False

File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/jvm.options.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/jvm.options
@@ -0,0 +1,109 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://opensearch.org/docs/opensearch/install/important-settings/
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms4G
+-Xmx4G
+
+################################################################
+## Numa Awareness
+################################################################
+-XX:+UseNUMA
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# log4j 2
+-Dlog4j.skipJansi=true
+-Dlog4j2.formatMsgNoLookups=true
+
+# security policy, needed for OpenSearch to access non-default files/sockets/ports T395571
+
+-Djava.security.policy=/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+-XX:HeapDumpPath=/srv/opensearch/relforge-eqiad-small-alpha
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+## Custom Wikimedia configuration
+
+# Defines a limit of proportion of time spent in GC before
+# an OutOfMemory error is thrown.  The goal is to bail and
+# restart the server if throughput is suffering due to
+# GC stalls.
+# TODO: Why 70? why not 60 or 80?
+-XX:GCTimeLimit=70
+
+# Defines minimum percentage of free space after a full GC
+# before an OutOfMemoryError is thrown. Typically a full GC
+# should leave > 50% of heap free. If we are down to 10%
+# something is wrong and bailing is a reasonable solution.
+-XX:GCHeapFreeLimit=10
+
+# Exit on OutOfMemoryError, since in most cases the JVM does not recover
+-XX:+ExitOnOutOfMemoryError
+
+# Custom WMF GC flags
+-Xlog:gc*:file=/var/log/opensearch/relforge-eqiad-small-alpha_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace
+-Xlog:safepoint

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9200.service

File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch_index_indexing_slowlog.log].orig
+++ File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]

+    owner  => root
+    group  => root
+    ensure => absent

Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
+    ensure   => present

Monitoring::Service[elasticsearch-https-relforge-eqiad]

Parameters differences:

--- Monitoring::Service[elasticsearch-https-relforge-eqiad].orig
+++ Monitoring::Service[elasticsearch-https-relforge-eqiad]

+    check_command  => check_ssl_on_port!relforge1008.eqiad.wmnet!9243
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1
+    config_dir     => /etc/nagios
+    migration_task => T384998
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => Elasticsearch HTTPS for relforge-eqiad
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search

Service[nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_shards_9200.timer]

+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]

+    owner   => root
+    group   => root
+    recurse => True
+    ensure  => directory
+    mode    => 0740

File[/usr/local/bin/opensearch-disable-readahead.sh]

Parameters differences:

--- File[/usr/local/bin/opensearch-disable-readahead.sh].orig
+++ File[/usr/local/bin/opensearch-disable-readahead.sh]

+    owner  => root
+    source => puppet:///modules/profile/opensearch/cirrus/opensearch-disable-readahead.sh
+    group  => root
+    ensure => file
+    mode   => 0555

Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

File[/etc/udev/rules.d/40-opensearch-readahead.rules]

Parameters differences:

--- File[/etc/udev/rules.d/40-opensearch-readahead.rules].orig
+++ File[/etc/udev/rules.d/40-opensearch-readahead.rules]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[udev_reload]

Content differences:

--- /etc/udev/rules.d/40-opensearch-readahead.rules.orig
+++ /etc/udev/rules.d/40-opensearch-readahead.rules
@@ -0,0 +1 @@
+SUBSYSTEM=="block", KERNEL=="md1", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="16"

File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer

Nrpe::Check[check_opensearch_shards_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_shards_9200].orig
+++ Nrpe::Check[check_opensearch_shards_9200]

+    before  => Monitoring::Service[opensearch_shards_9200]
+    command => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4
+    ensure  => present

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]

+    before      => ['Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Service[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead].orig
+++ Systemd::Service[opensearch-disable-readahead]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-disable-readahead.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Motd::Script[cirrus::test]

Parameters differences:

--- Motd::Script[cirrus::test].orig
+++ Motd::Script[cirrus::test]

-    priority => 5
-    ensure   => present

File[/etc/default/logstash]

Parameters differences:

--- File[/etc/default/logstash].orig
+++ File[/etc/default/logstash]

+    owner   => root
+    group   => root
+    require => Package[logstash-oss]
+    mode    => 0444
+    notify  => Service[logstash]

Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Nrpe::Check[check_check_push_cross_cluster_settings_9400_status].orig
+++ Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]

+    before  => Monitoring::Service[check_push_cross_cluster_settings_9400_status]
+    command => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400
+    ensure  => present

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service

Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf].orig
+++ File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf.orig
+++ /etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "push_cross_cluster_settings_9200" then {
+    action(
+        type="omfile" file="/var/log/push_cross_cluster_settings_9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_unassigned_shard_check_9200]=/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4

Systemd::Syslog[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Syslog[push_cross_cluster_settings_9400].orig
+++ Systemd::Syslog[push_cross_cluster_settings_9400]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Systemd::Timer::Job[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Timer::Job[push_cross_cluster_settings_9400].orig
+++ Systemd::Timer::Job[push_cross_cluster_settings_9400]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => True
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /bin/bash /usr/local/bin/set-cross-cluster-seeds_9400.sh
+    private_tmp               => False
+    description               => Auto set remote cluster seeds for relforge-eqiad-small-alpha
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnUnitActiveSec', 'interval': '15min'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/lib/systemd/system/push_cross_cluster_settings_9400.service]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9400.service].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9400.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9400.service.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto set remote cluster seeds for relforge-eqiad-small-alpha
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/bin/bash /usr/local/bin/set-cross-cluster-seeds_9400.sh

Package[python3-requests]

Parameters differences:

--- Package[python3-requests].orig
+++ Package[python3-requests]

+    provider => apt
+    ensure   => installed

Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]

Parameters differences:

--- Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)].orig
+++ Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Firewall::Service[opensearch-inter-node-9500]

Parameters differences:

--- Firewall::Service[opensearch-inter-node-9500].orig
+++ Firewall::Service[opensearch-inter-node-9500]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => True
+    srange              => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    prio                => 10
+    desc                => 
+    port                => 9500
+    proto               => tcp

Monitoring::Service[opensearch_shard_size_check_9200]

Parameters differences:

--- Monitoring::Service[opensearch_shard_size_check_9200].orig
+++ Monitoring::Service[opensearch_shard_size_check_9200]

+    check_command  => nrpe_check!check_opensearch_shard_size_check_9200!10
+    retry_interval => 180
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1440
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => OpenSearch shard size check - 9200
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]

Parameters differences:

--- Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a].orig
+++ Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]

+    for                => 11m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_Check_unit_status_of_push_cross_cluster_settings_9200
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: Check unit status of push_cross_cluster_settings_9200
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="379fbc6cbce5b297397c866a15729f1a",check_name="check_check_push_cross_cluster_settings_9200_status", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_check_push_cross_cluster_settings_9200_status))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: Check unit status of push_cross_cluster_settings_9200

Systemd::Unit[push_cross_cluster_settings_9400.service]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9400.service].orig
+++ Systemd::Unit[push_cross_cluster_settings_9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => push_cross_cluster_settings_9400.service

Sudo::User[nrpe-check_opensearch_shards_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shards_9200].orig
+++ Sudo::User[nrpe-check_opensearch_shards_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Rsyslog::Conf[push_cross_cluster_settings_9200]

Parameters differences:

--- Rsyslog::Conf[push_cross_cluster_settings_9200].orig
+++ Rsyslog::Conf[push_cross_cluster_settings_9200]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/push_cross_cluster_settings_9200]
+    ensure   => present

Nrpe::Check[check_opensearch_shard_size_check_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_shard_size_check_9400].orig
+++ Nrpe::Check[check_opensearch_shard_size_check_9400]

+    before  => Monitoring::Service[opensearch_shard_size_check_9400]
+    command => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    ensure  => present

File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]

+    owner  => root
+    group  => root
+    ensure => absent

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    subscribe   => File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
+    refreshonly => True

Concat::Fragment[wikimedia-opensearch-plugins]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-plugins].orig
+++ Concat::Fragment[wikimedia-opensearch-plugins]

+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    order  => 10

Systemd::Service[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Service[push_cross_cluster_settings_9400].orig
+++ Systemd::Service[push_cross_cluster_settings_9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[push_cross_cluster_settings_9400.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]

Parameters differences:

--- File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf].orig
+++ File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]

+    mode   => 0444
+    owner  => root
+    group  => root
+    ensure => present

Content differences:

--- /etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf.orig
+++ /etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf
@@ -0,0 +1 @@
+d    /run/opensearch-relforge-eqiad-small-alpha  0755 opensearch opensearch - -

File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]

Content differences:

--- /lib/systemd/system/prometheus-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/prometheus-elasticsearch-exporter-9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus exporter for Elasticsearch
+Documentation=https://prometheus.io/docs/introduction/overview/
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/bin/prometheus-elasticsearch-exporter --es.uri=http://localhost:9400 --web.listen-address=:9109 
+
+[Install]
+WantedBy=multi-user.target

Sysctl::Conffile[TCP Fast Open]

Parameters differences:

--- Sysctl::Conffile[TCP Fast Open].orig
+++ Sysctl::Conffile[TCP Fast Open]

+    ensure             => present
+    priority           => 70
+    no_priority_prefix => False

File[/etc/ferm/conf.d/10_opensearch-https-9443]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch-https-9443].orig
+++ File[/etc/ferm/conf.d/10_opensearch-https-9443]

+    owner   => root
+    group   => root
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    mode    => 0400
+    ensure  => present
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_opensearch-https-9443.orig
+++ /etc/ferm/conf.d/10_opensearch-https-9443
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9443, (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS)));
+
+

File[/etc/logrotate.d/push_cross_cluster_settings_9200]

Parameters differences:

--- File[/etc/logrotate.d/push_cross_cluster_settings_9200].orig
+++ File[/etc/logrotate.d/push_cross_cluster_settings_9200]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/push_cross_cluster_settings_9200.orig
+++ /etc/logrotate.d/push_cross_cluster_settings_9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for push_cross_cluster_settings_9200
+
+/var/log/push_cross_cluster_settings_9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Sudo::User[nrpe-check_opensearch_shard_size_check_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shard_size_check_9200].orig
+++ Sudo::User[nrpe-check_opensearch_shard_size_check_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]

Parameters differences:

--- File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443].orig
+++ File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]

+    owner   => root
+    group   => root
+    tag     => ferm
+    require => File[/etc/ferm/conf.d]
+    mode    => 0400
+    ensure  => present
+    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_elastic-small-alpha-https-9443.orig
+++ /etc/ferm/conf.d/10_elastic-small-alpha-https-9443
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9443, (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49));
+
+

Systemd::Timer::Job[clean_up_dlq]

Parameters differences:

--- Systemd::Timer::Job[clean_up_dlq].orig
+++ Systemd::Timer::Job[clean_up_dlq]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /usr/local/bin/cleanup-dlq
+    private_tmp               => False
+    description               => Clean up dead letter queue and restart logstash
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnCalendar', 'interval': '1h'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=720min
+OnActiveSec=1s
+RandomizedDelaySec=43200
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha].orig
+++ Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]

+    http_port    => 9400
+    cluster_name => relforge-eqiad-small-alpha

Monitoring::Service[opensearch_shard_size_check_9400]

Parameters differences:

--- Monitoring::Service[opensearch_shard_size_check_9400].orig
+++ Monitoring::Service[opensearch_shard_size_check_9400]

+    check_command  => nrpe_check!check_opensearch_shard_size_check_9400!10
+    retry_interval => 180
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1440
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => OpenSearch shard size check - 9400
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

File[/lib/systemd/system/opensearch-disable-readahead.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead.service
+
+[Timer]
+Unit=opensearch-disable-readahead.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "c62725841c61e20ac3f6819b4c0ad800" --timeout 10 --check-command "check_opensearch_shard_size_check_9200"
+    private_tmp               => False
+    splay                     => 36000
+    description               => execution of nrpe2nodexp for the check_opensearch_shard_size_check_9200 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_shard_size_check_9200
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/etc/rsyslog.d/40-clean-up-dlq.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-clean-up-dlq.conf].orig
+++ File[/etc/rsyslog.d/40-clean-up-dlq.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-clean-up-dlq.conf.orig
+++ /etc/rsyslog.d/40-clean-up-dlq.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "clean_up_dlq" then {
+    action(
+        type="omfile" file="/var/log/clean_up_dlq/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad].orig
+++ Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/opensearch-disable-readahead-relforge-eqiad]
+    ensure   => present

File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    tag     => nrpe::check
+    ensure  => present
+    mode    => 0444
+    notify  => Service[nagios-nrpe-server]

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_unassigned_shard_check_9400]=/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4

Rsyslog::Conf[udp_json_logback_compat]

Parameters differences:

--- Rsyslog::Conf[udp_json_logback_compat].orig
+++ Rsyslog::Conf[udp_json_logback_compat]

+    mode     => 0444
+    priority => 50
+    ensure   => present

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9400_status command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-check_push_cross_cluster_settings_9400_status
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "de4a199c43946c09f819ca4ffea40ac0" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9400_status"

Class[Nginx]

Parameters differences:

--- Class[Nginx].orig
+++ Class[Nginx]

+    modules      => []
+    managed      => False
+    lib_on_tmpfs => True
+    variant      => light
+    ensure       => present
+    tmpfs_size   => 1g

Logstash::Conf[input-gelf-gelf_relay]

Parameters differences:

--- Logstash::Conf[input-gelf-gelf_relay].orig
+++ Logstash::Conf[input-gelf-gelf_relay]

+    priority => 10
+    ensure   => present

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

+    owner  => root
+    source => puppet:///modules/profile/pki/intermediates/discovery2026-cert.pem
+    group  => root
+    ensure => file
+    mode   => 0440
+    notify => ['Exec[nginx-reload]', 'Exec[nginx-reload]']

File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

+    owner => root
+    mode  => 0640
+    group => opensearch

File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]

Parameters differences:

--- File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf].orig
+++ File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]

+    mode   => 0444
+    owner  => root
+    group  => root
+    ensure => present

Content differences:

--- /etc/tmpfiles.d/opensearch-relforge-eqiad.conf.orig
+++ /etc/tmpfiles.d/opensearch-relforge-eqiad.conf
@@ -0,0 +1 @@
+d    /run/opensearch-relforge-eqiad  0755 opensearch opensearch - -

Exec[opensearch-create-keystore-relforge-eqiad]

Parameters differences:

--- Exec[opensearch-create-keystore-relforge-eqiad].orig
+++ Exec[opensearch-create-keystore-relforge-eqiad]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad']
+    command     => /usr/share/opensearch/bin/opensearch-keystore create
+    creates     => /etc/opensearch/relforge-eqiad/opensearch.keystore
+    before      => File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service

Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status].orig
+++ File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

File[/etc/opensearch/relforge-eqiad/jvm.options]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/jvm.options].orig
+++ File[/etc/opensearch/relforge-eqiad/jvm.options]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => file

Content differences:

--- /etc/opensearch/relforge-eqiad/jvm.options.orig
+++ /etc/opensearch/relforge-eqiad/jvm.options
@@ -0,0 +1,109 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://opensearch.org/docs/opensearch/install/important-settings/
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms30G
+-Xmx30G
+
+################################################################
+## Numa Awareness
+################################################################
+-XX:+UseNUMA
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# log4j 2
+-Dlog4j.skipJansi=true
+-Dlog4j2.formatMsgNoLookups=true
+
+# security policy, needed for OpenSearch to access non-default files/sockets/ports T395571
+
+-Djava.security.policy=/etc/opensearch/relforge-eqiad/java-security.policy
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+-XX:HeapDumpPath=/srv/opensearch/relforge-eqiad
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+## Custom Wikimedia configuration
+
+# Defines a limit of proportion of time spent in GC before
+# an OutOfMemory error is thrown.  The goal is to bail and
+# restart the server if throughput is suffering due to
+# GC stalls.
+# TODO: Why 70? why not 60 or 80?
+-XX:GCTimeLimit=70
+
+# Defines minimum percentage of free space after a full GC
+# before an OutOfMemoryError is thrown. Typically a full GC
+# should leave > 50% of heap free. If we are down to 10%
+# something is wrong and bailing is a reasonable solution.
+-XX:GCHeapFreeLimit=10
+
+# Exit on OutOfMemoryError, since in most cases the JVM does not recover
+-XX:+ExitOnOutOfMemoryError
+
+# Custom WMF GC flags
+-Xlog:gc*:file=/var/log/opensearch/relforge-eqiad_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace
+-Xlog:safepoint

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer

Systemd::Syslog[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Syslog[push_cross_cluster_settings_9200].orig
+++ Systemd::Syslog[push_cross_cluster_settings_9200]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => present
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

Motd::Message[cirrus::test]

Parameters differences:

--- Motd::Message[cirrus::test].orig
+++ Motd::Message[cirrus::test]

-    priority => 5
-    message  => relforge1008 is a Unpuppetised system for testing (cirrus::test)
-    ensure   => present

Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]

+    send_mail                 => False
+    user                      => nagios
+    monitoring_enabled        => False
+    logfile_name              => syslog.log
+    logging_enabled           => False
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "fd6eadf747c645a72e11bd74109a8156" --timeout 10 --check-command "check_opensearch_shards_9200"
+    private_tmp               => False
+    splay                     => 60
+    description               => execution of nrpe2nodexp for the check_opensearch_shards_9200 command.
+    syslog_force_stop         => True
+    ignore_errors             => True
+    send_mail_only_on_error   => True
+    fixed_random_delay        => True
+    logfile_perms             => all
+    syslog_identifier         => nrpe2nodexp-opensearch_shards_9200
+    ensure                    => absent
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    group                     => prometheus-node-exporter
+    success_exit_status       => []
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}]
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]

Parameters differences:

--- File[/etc/rsyslog.d/50-udp-json-logback-compat.conf].orig
+++ File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/50-udp-json-logback-compat.conf.orig
+++ /etc/rsyslog.d/50-udp-json-logback-compat.conf
@@ -0,0 +1,43 @@
+# Provide a UDP input to accept JSON payloads on localhost and forward them to logstash via Kakfa.
+
+module(load="imudp")
+module(load="mmjsonparse")
+module(load="omkafka")
+
+template(name="template_udp_json_logback_compat" type="list") {
+  property(name="$!all-json")
+}
+
+# Use the parsed json "level" field value as the kafka topic suffix
+template(name="udp_json_logback_compat_topic" type="string" string="logback-%!level:::lowercase%")
+
+# Use a separate (in memory) queue to limit message processing to this ruleset only.
+ruleset(name="ruleset_udp_json_logback_compat" queue.type="LinkedList") {
+
+  action(type="mmjsonparse" name="mmjsonparse_udp_json_logback_compat" cookie="" useRawMsg="on")
+
+  if $parsesuccess == "OK" then {
+    action(type="omkafka"
+           broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1003.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093"]
+           topic="udp_json_logback_compat_topic"
+           dynatopic="on"
+           dynatopic.cachesize="1000"
+           partitions.auto="on"
+           template="template_udp_json_logback_compat"
+           queue.type="LinkedList" queue.size="10000" queue.filename="udp_json_logback_compat"
+           queue.highWatermark="7000" queue.lowWatermark="6000"
+           queue.checkpointInterval="5"
+           queue.maxDiskSpace="40960000"
+           confParam=[ "security.protocol=ssl",
+                       "ssl.ca.location=/etc/ssl/certs/wmf-ca-certificates.crt",
+                       "compression.codec=snappy",
+                       "socket.timeout.ms=60000",
+                       "socket.keepalive.enable=true",
+                       "queue.buffering.max.ms=50",
+                       "batch.num.messages=1000" ]
+    )
+  }
+
+}
+
+input(type="imudp" port="11514" address="localhost" ruleset="ruleset_udp_json_logback_compat")

Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/nginx/nginx.conf]

Parameters differences:

--- File[/etc/nginx/nginx.conf].orig
+++ File[/etc/nginx/nginx.conf]

+    owner  => root
+    group  => root
+    tag    => nginx
+    before => ['Service[nginx]']

Content differences:

--- /etc/nginx/nginx.conf.orig
+++ /etc/nginx/nginx.conf
@@ -0,0 +1,67 @@
+# This file is managed by Puppet!
+
+user www-data www-data;
+worker_processes 48;
+worker_cpu_affinity 000000000000000000000001000000000000000000000001 000000000000000000000001000000000000000000000001 000000000000000000000010000000000000000000000010 000000000000000000000010000000000000000000000010 000000000000000000000100000000000000000000000100 000000000000000000000100000000000000000000000100 000000000000000000001000000000000000000000001000 000000000000000000001000000000000000000000001000 000000000000000000010000000000000000000000010000 000000000000000000010000000000000000000000010000 000000000000000000100000000000000000000000100000 000000000000000000100000000000000000000000100000 000000000000000001000000000000000000000001000000 000000000000000001000000000000000000000001000000 000000000000000010000000000000000000000010000000 000000000000000010000000000000000000000010000000 000000000000000100000000000000000000000100000000 000000000000000100000000000000000000000100000000 000000000000001000000000000000000000001000000000 000000000000001000000000000000000000001000000000 000000000000010000000000000000000000010000000000 000000000000010000000000000000000000010000000000 000000000000100000000000000000000000100000000000 000000000000100000000000000000000000100000000000 000000000001000000000000000000000001000000000000 000000000001000000000000000000000001000000000000 000000000010000000000000000000000010000000000000 000000000010000000000000000000000010000000000000 000000000100000000000000000000000100000000000000 000000000100000000000000000000000100000000000000 000000001000000000000000000000001000000000000000 000000001000000000000000000000001000000000000000 000000010000000000000000000000010000000000000000 000000010000000000000000000000010000000000000000 000000100000000000000000000000100000000000000000 000000100000000000000000000000100000000000000000 000001000000000000000000000001000000000000000000 000001000000000000000000000001000000000000000000 000010000000000000000000000010000000000000000000 000010000000000000000000000010000000000000000000 000100000000000000000000000100000000000000000000 000100000000000000000000000100000000000000000000 001000000000000000000000001000000000000000000000 001000000000000000000000001000000000000000000000 010000000000000000000000010000000000000000000000 010000000000000000000000010000000000000000000000 100000000000000000000000100000000000000000000000 100000000000000000000000100000000000000000000000;
+worker_rlimit_nofile 262144;
+
+error_log  /var/log/nginx/error.log;
+pid        /run/nginx.pid;
+
+events {
+    worker_connections 131072;
+    multi_accept off; # accept 1 conn per epoll hit, basically
+    accept_mutex off; # better latencies at high connection rates
+}
+
+http {
+    include       /etc/nginx/mime.types;
+
+    access_log	/var/log/nginx/access.log;
+
+    client_max_body_size 100m;
+    client_body_buffer_size 64k;
+    http2_max_field_size 8k;
+
+    sendfile        on;
+    #tcp_nopush     on;
+
+    #keepalive_timeout  0;
+    keepalive_timeout  65;
+    tcp_nodelay        on;
+
+    gzip  off;
+    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
+
+    # SSL sessionid caching.  1m is ~4000 sessions.
+    # Still experimenting!
+    ssl_session_cache shared:SSL:1024m;
+    ssl_session_timeout 25h;
+
+    # Disable RFC5077 tickets (may revisit later when client support is better)
+    ssl_session_tickets off;
+
+    # nginx's $http2 is either 'h2' or '' (empty string) for http/1
+    # Map it to $h2 as a boolean string value '1' or '0' for use in headers
+    map $http2 $h2 {
+        default '1';
+        '' '0';
+    }
+
+    # Nginx indicates session reuse with 'r' and '.', which is rather cryptic.
+    # Map this value to '1' and '0' instead.
+    map $ssl_session_reused $session_reused {
+        'r' '1';
+        '.' '0';
+    }
+
+
+    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
+    ssl_ciphers -ALL:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA;
+    ssl_ecdh_curve X25519:prime256v1;
+    ssl_prefer_server_ciphers on;
+    ssl_dhparam /etc/ssl/dhparam.pem;
+
+    include /etc/nginx/conf.d/*.conf;
+    include /etc/nginx/sites-enabled/*;
+}

File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead-relforge-eqiad" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead-relforge-eqiad/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]

+    check_command          => nrpe_check!check_opensearch_unassigned_shard_check_9400!10
+    retry_interval         => 120
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 720
+    notifications_enabled  => 0
+    max_check_attempts     => 1
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch unassigned shard check - 9400
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration

Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]

+    before      => ['Service[opensearch-hot-threads-log.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200].orig
+++ Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

+    prometheus_port    => 9120
+    indices_to_monitor => []
+    elasticsearch_port => 9200

Service[push_cross_cluster_settings_9200.timer]

Parameters differences:

--- Service[push_cross_cluster_settings_9200.timer].orig
+++ Service[push_cross_cluster_settings_9200.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Concat_fragment[wikimedia-opensearch]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch].orig
+++ Concat_fragment[wikimedia-opensearch]

+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    order  => 10

Content differences:

--- wikimedia-opensearch.orig
+++ wikimedia-opensearch
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: bookworm-wikimedia
+Components: thirdparty/opensearch1
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Ferm::Service[opensearch_inter_node_9500]

Parameters differences:

--- Ferm::Service[opensearch_inter_node_9500].orig
+++ Ferm::Service[opensearch_inter_node_9500]

+    unrestricted_access => False
+    ensure              => present
+    notrack             => True
+    srange              => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    prio                => 10
+    desc                => 
+    port                => 9500
+    proto               => tcp

Exec[update_java_alternatives_17]

Parameters differences:

--- Exec[update_java_alternatives_17].orig
+++ Exec[update_java_alternatives_17]

+    command => /usr/sbin/update-java-alternatives -s /usr/lib/jvm/java-1.17.0-openjdk-amd64
+    unless  => /usr/bin/update-alternatives --query java | /bin/grep 'Value: /usr/lib/jvm/java-17-openjdk-amd64'

Systemd::Unmask[nginx.service]

Parameters differences:

--- Systemd::Unmask[nginx.service].orig
+++ Systemd::Unmask[nginx.service]

+    unit        => nginx.service
+    refreshonly => True

Sudo::User[nrpe-check_opensearch_shard_size_check_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shard_size_check_9400].orig
+++ Sudo::User[nrpe-check_opensearch_shard_size_check_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shards_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shards_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=1min
+OnActiveSec=1s
+RandomizedDelaySec=60
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shard_size_check_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shard_size_check_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "aefe2d8b10db3bc84662052af1e20008" --timeout 10 --check-command "check_opensearch_shard_size_check_9400"

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shards_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]

+    unit_name          => opensearch-disable-readahead-relforge-eqiad.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    accuracy           => 15sec

Service[opensearch-disable-readahead.timer]

Parameters differences:

--- Service[opensearch-disable-readahead.timer].orig
+++ Service[opensearch-disable-readahead.timer]

+    before   => ['Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]']
+    provider => systemd
+    enable   => False
+    ensure   => stopped

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9400.service

Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]

+    before      => ['Service[push_cross_cluster_settings_9200.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202].orig
+++ Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]

+    for                => 720m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_unassigned_shard_check___9400
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch unassigned shard check - 9400
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="a0a4892478c289e50484ffd2fb4c3202",check_name="check_opensearch_unassigned_shard_check_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_unassigned_shard_check_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch unassigned shard check - 9400

Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]

+    check_command          => nrpe_check!check_opensearch_unassigned_shard_check_9200!10
+    retry_interval         => 120
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 720
+    notifications_enabled  => 0
+    max_check_attempts     => 1
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch unassigned shard check - 9200
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration

Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Nrpe::Check[check_check_push_cross_cluster_settings_9200_status].orig
+++ Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]

+    before  => Monitoring::Service[check_push_cross_cluster_settings_9200_status]
+    command => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200
+    ensure  => present

File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cleanup GC logs
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/find /var/log/opensearch -name 'relforge-eqiad_jvm_gc.*.log*' -mtime +30 -delete

Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]

+    before      => ['Service[push_cross_cluster_settings_9400.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/logstash/conf.d/README]

Parameters differences:

--- File[/etc/logstash/conf.d/README].orig
+++ File[/etc/logstash/conf.d/README]

+    owner   => logstash
+    source  => puppet:///modules/logstash/conf.d/README
+    group   => logstash
+    require => Package[logstash-oss]
+    ensure  => present

File_line[opensearch_bin_bashrc]

Parameters differences:

--- File_line[opensearch_bin_bashrc].orig
+++ File_line[opensearch_bin_bashrc]

+    path   => /root/.bashrc
+    line   => PATH=${PATH}:/usr/share/opensearch/bin  # Managed by puppet
+    ensure => present

Systemd::Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Service[prometheus-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    require                  => Package[prometheus-elasticsearch-exporter]
+    ensure                   => present
+    unit_type                => service
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {'ensure': 'running'}
+    monitoring_critical      => False

Systemd::Unit[opensearch_1@.service]

Parameters differences:

--- Systemd::Unit[opensearch_1@.service].orig
+++ Systemd::Unit[opensearch_1@.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch_1@.service

Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad].orig
+++ Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]

+    check_command          => check_ssl_on_port!relforge1008.eqiad.wmnet!9243
+    retry_interval         => 1
+    active_checks_enabled  => 1
+    contact_groups         => team-data-platform,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => Elasticsearch HTTPS for relforge-eqiad
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search

Rsyslog::Conf[push_cross_cluster_settings_9400]

Parameters differences:

--- Rsyslog::Conf[push_cross_cluster_settings_9400].orig
+++ Rsyslog::Conf[push_cross_cluster_settings_9400]

+    mode     => 0444
+    priority => 40
+    require  => File[/var/log/push_cross_cluster_settings_9400]
+    ensure   => present

Exec[tlsproxy localssl default_server on ports [9443]]

Parameters differences:

--- Exec[tlsproxy localssl default_server on ports [9443]].orig
+++ Exec[tlsproxy localssl default_server on ports [9443]]

+    command     => /bin/true
+    onlyif      => /bin/false
+    refreshonly => True

File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf.orig
+++ /etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-relforge-eqiad-gc-log-cleanup" then {
+    action(
+        type="omfile" file="/var/log/opensearch-relforge-eqiad-gc-log-cleanup/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead-relforge-eqiad-small-alpha.service

File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Service[opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- Service[opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ Service[opensearch-disable-readahead-relforge-eqiad.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]

+    owner  => root
+    group  => root
+    ensure => file
+    mode   => 0440
+    notify => ['Exec[nginx-reload]', 'Exec[nginx-reload]']

Monitoring::Service[opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Monitoring::Service[opensearch_unassigned_shard_check_9400].orig
+++ Monitoring::Service[opensearch_unassigned_shard_check_9400]

+    check_command  => nrpe_check!check_opensearch_unassigned_shard_check_9400!10
+    retry_interval => 120
+    freshness      => 36000
+    ensure         => present
+    check_interval => 720
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 1
+    passive        => False
+    description    => OpenSearch unassigned shard check - 9400
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration

Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => present

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_unassigned_shard_check_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Syslog[clean_up_dlq]

Parameters differences:

--- Systemd::Syslog[clean_up_dlq].orig
+++ Systemd::Syslog[clean_up_dlq]

+    readable_by            => all
+    force_stop             => True
+    ensure                 => absent
+    programname_comparison => startswith
+    log_filename           => syslog.log
+    owner                  => root
+    group                  => root
+    base_dir               => /var/log

File[/etc/logstash/logstash.yml]

Parameters differences:

--- File[/etc/logstash/logstash.yml].orig
+++ File[/etc/logstash/logstash.yml]

+    owner   => root
+    group   => root
+    require => Package[logstash-oss]
+    mode    => 0444
+    notify  => Service[logstash]

Content differences:

--- /etc/logstash/logstash.yml.orig
+++ /etc/logstash/logstash.yml
@@ -0,0 +1,11 @@
+---
+path.data: "/var/lib/logstash"
+path.config: "/etc/logstash/conf.d"
+path.logs: "/var/log/logstash"
+pipeline.workers: 2
+pipeline.batch.size: 125
+pipeline.batch.delay: 50
+log.format: json
+dead_letter_queue.enable: false
+dead_letter_queue.max_bytes: 1024mb
+http.port: '9675'

File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]

+    owner  => root
+    group  => root
+    ensure => absent

Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

+    tag            => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    format         => plain
+    force          => False
+    show_diff      => True
+    ensure_newline => False
+    order          => alpha
+    owner          => root
+    group          => root
+    mode           => 0444
+    backup         => puppet
+    replace        => True

Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_unassigned_shard_check_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "54ca99505516d5452912b573aa7d35d3" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9200"

File[/var/log/opensearch-disable-readahead]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead].orig
+++ File[/var/log/opensearch-disable-readahead]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => absent
+    mode   => 0755
+    backup => False

Tlsproxy::Localssl[relforge-eqiad-small-alpha]

Parameters differences:

--- Tlsproxy::Localssl[relforge-eqiad-small-alpha].orig
+++ Tlsproxy::Localssl[relforge-eqiad-small-alpha]

+    keepalive_timeout  => 60
+    default_server     => True
+    require            => ['Class[Profile::Tlsproxy::Instance]']
+    keepalive_requests => 100
+    cfssl_paths        => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    skip_private       => False
+    access_log         => False
+    read_timeout       => 180
+    only_get_requests  => False
+    acme_chief         => False
+    server_name        => relforge1008.eqiad.wmnet
+    acme_certname      => relforge-eqiad-small-alpha
+    enable_http2       => False
+    server_aliases     => ['relforge.svc.eqiad.wmnet']
+    tls_ports          => [9443]
+    upstream_ports     => [9400]
+    upstream_ip        => 10.64.32.100

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]

+    before      => ['Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Prometheus::Elasticsearch_exporter[localhost:9200]

Parameters differences:

--- Prometheus::Elasticsearch_exporter[localhost:9200].orig
+++ Prometheus::Elasticsearch_exporter[localhost:9200]

+    extra_config       => 
+    prometheus_port    => 9108
+    elasticsearch_port => 9200

File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200].orig
+++ File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]

+    path    => /etc/debdeploy-client/autorestarts.conf
+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    line    => prometheus-elasticsearch-exporter-9200
+    ensure  => present

Monitoring::Service[opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Monitoring::Service[opensearch_unassigned_shard_check_9200].orig
+++ Monitoring::Service[opensearch_unassigned_shard_check_9200]

+    check_command  => nrpe_check!check_opensearch_unassigned_shard_check_9200!10
+    retry_interval => 120
+    freshness      => 36000
+    ensure         => present
+    check_interval => 720
+    config_dir     => /etc/nagios
+    migration_task => T371083
+    contact_group  => admins,team-discovery
+    host           => relforge1008
+    critical       => False
+    retries        => 1
+    passive        => False
+    description    => OpenSearch unassigned shard check - 9200
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration

Package[openjdk-17-jdk]

Parameters differences:

--- Package[openjdk-17-jdk].orig
+++ Package[openjdk-17-jdk]

+    provider => apt
+    ensure   => installed

Nrpe::Monitor_service[opensearch_shards_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shards_9400].orig
+++ Nrpe::Monitor_service[opensearch_shards_9400]

+    retry_interval              => 1
+    ensure                      => present
+    check_interval              => 1
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 3
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch health check for shards on 9400
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration

Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]

Parameters differences:

--- Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha].orig
+++ Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]

+    check_command  => check_ssl_on_port!relforge1008.eqiad.wmnet!9443
+    retry_interval => 1
+    freshness      => 36000
+    ensure         => present
+    check_interval => 1
+    config_dir     => /etc/nagios
+    migration_task => T384998
+    host           => relforge1008
+    critical       => False
+    retries        => 3
+    passive        => False
+    description    => Elasticsearch HTTPS for relforge-eqiad-small-alpha
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search

Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]

+    check_command          => nrpe_check!check_opensearch_shard_size_check_9200!10
+    retry_interval         => 180
+    active_checks_enabled  => 1
+    contact_groups         => admins,team-discovery
+    host_name              => relforge1008
+    check_freshness        => 0
+    notification_period    => 24x7
+    servicegroups          => relforge_eqiad
+    notification_interval  => 0
+    ensure                 => present
+    notification_options   => c,r,f
+    check_interval         => 1440
+    notifications_enabled  => 0
+    max_check_attempts     => 3
+    is_volatile            => 0
+    passive_checks_enabled => 1
+    check_period           => 24x7
+    service_description    => OpenSearch shard size check - 9200
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed

Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/opensearch/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha]

+    owner  => root
+    mode   => 0755
+    group  => root
+    ensure => directory

Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    unit_name          => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00'}]
+    accuracy           => 15sec

Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200].orig
+++ Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

+    prometheus_port    => 9120
+    indices_to_monitor => []
+    elasticsearch_port => 9200

Concat_fragment[main contacts]

Content differences:

--- main contacts.orig
+++ main contacts
@@ -1,3 +1,3 @@
 ---
-role::cirrus::test:
+role::cirrus::relforge:
 - Search Platform

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]

+    owner  => root
+    mode   => 0440
+    group  => root
+    ensure => file

Systemd::Unit[opensearch-hot-threads-log.timer]

Parameters differences:

--- Systemd::Unit[opensearch-hot-threads-log.timer].orig
+++ Systemd::Unit[opensearch-hot-threads-log.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-hot-threads-log.timer

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-elasticsearch-exporter-9400
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9400

Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]

Parameters differences:

--- Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0].orig
+++ Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]

+    for                => 11m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_Check_unit_status_of_push_cross_cluster_settings_9400
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: Check unit status of push_cross_cluster_settings_9400
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="de4a199c43946c09f819ca4ffea40ac0",check_name="check_check_push_cross_cluster_settings_9400_status", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_check_push_cross_cluster_settings_9400_status))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: Check unit status of push_cross_cluster_settings_9400

File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status].orig
+++ File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Service[push_cross_cluster_settings_9400.timer]

Parameters differences:

--- Service[push_cross_cluster_settings_9400.timer].orig
+++ Service[push_cross_cluster_settings_9400.timer]

+    ensure   => running
+    provider => systemd
+    enable   => True

Package[nginx-light]

Parameters differences:

--- Package[nginx-light].orig
+++ Package[nginx-light]

+    provider => apt
+    notify   => ['Systemd::Unmask[nginx.service]']
+    ensure   => installed

Systemd::Timer[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead].orig
+++ Systemd::Timer[opensearch-disable-readahead]

+    unit_name          => opensearch-disable-readahead.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    accuracy           => 15sec

File[/etc/opensearch/relforge-eqiad/logging.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/logging.yml].orig
+++ File[/etc/opensearch/relforge-eqiad/logging.yml]

+    owner  => root
+    group  => root
+    ensure => absent

File[/etc/nginx/sites-available]

Parameters differences:

--- File[/etc/nginx/sites-available].orig
+++ File[/etc/nginx/sites-available]

+    before  => ['Service[nginx]']
+    owner   => root
+    group   => root
+    recurse => True
+    force   => True
+    ensure  => directory
+    purge   => True

File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]

+    owner  => root
+    mode   => 0755
+    group  => root
+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch_unassigned_shards.py

Systemd::Unit[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Unit[prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Unit[prometheus-elasticsearch-exporter-9400]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => prometheus-elasticsearch-exporter-9400

Logrotate::Conf[push_cross_cluster_settings_9400]

Parameters differences:

--- Logrotate::Conf[push_cross_cluster_settings_9400].orig
+++ Logrotate::Conf[push_cross_cluster_settings_9400]

+    ensure => present

Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)].orig
+++ Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]

+    before      => ['Service[prometheus-elasticsearch-exporter-9200]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156].orig
+++ Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]

+    for                => 3m
+    team               => observability
+    ensure             => absent
+    dashboard          => TODO
+    instance           => ops
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    severity           => info
+    def_label_whitelst => ['team', 'severity']
+    alert_name         => nrpe_OpenSearch_health_check_for_shards_on_9200
+    site               => eqiad
+    group              => nrpechecks
+    summary            => NRPE CHECK: OpenSearch health check for shards on 9200
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="fd6eadf747c645a72e11bd74109a8156",check_name="check_opensearch_shards_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shards_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    description        => NRPE CHECK: OpenSearch health check for shards on 9200

Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]

+    unit_name          => nrpe2nodexp-opensearch_shard_size_check_9400.service
+    fixed_random_delay => True
+    splay              => 36000
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    accuracy           => 15sec

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => present
+    override_filename => puppet-override.conf
+    unit              => opensearch-disable-readahead-relforge-eqiad.service

File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]

+    owner   => root
+    require => Package[nagios-nrpe-server]
+    group   => root
+    ensure  => absent

Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shards_9400.service

Class[Java]

Parameters differences:

--- Class[Java].orig
+++ Class[Java]

+    before        => ['Java::Cacert[wmf:puppetca.pem]', 'Java::Cacert[wmf:Wikimedia_Internal_Root_CA]', 'Java::Cacert[wmf:puppetca.pem]', 'Java::Cacert[wmf:Wikimedia_Internal_Root_CA]']
+    enable_dbg    => False
+    hardened_tls  => False
+    require       => Package[wmf-certificates]
+    egd_source    => /dev/random
+    java_packages => [{'version': '17', 'variant': 'jdk'}]

Nrpe::Check[check_opensearch_shard_size_check_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_shard_size_check_9200].orig
+++ Nrpe::Check[check_opensearch_shard_size_check_9200]

+    before  => Monitoring::Service[opensearch_shard_size_check_9200]
+    command => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    ensure  => present

File[/etc/logstash/jvm.options]

Parameters differences:

--- File[/etc/logstash/jvm.options].orig
+++ File[/etc/logstash/jvm.options]

+    owner   => root
+    group   => root
+    require => Package[logstash-oss]
+    mode    => 0444
+    notify  => Service[logstash]

Content differences:

--- /etc/logstash/jvm.options.orig
+++ /etc/logstash/jvm.options
@@ -0,0 +1,91 @@
+#####################################################################
+## THSI FILE IS MANAGED BY PUPPET
+## puppet://modules/logstash/jvm.options.erb
+#####################################################################
+
+## JVM configuration
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms192m
+-Xmx192m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+## locale
+# Set the locale language
+#-Duser.language=en
+
+# Set the locale country
+#-Duser.country=US
+
+# Set the locale variant, if any
+#-Duser.variant=
+
+## basic
+
+# set the I/O temp directory
+#-Djava.io.tmpdir=$HOME
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+#-Djna.nosys=true
+
+# log4j2
+-Dlog4j2.formatMsgNoLookups=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof
+
+## GC logging
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${LS_GC_LOG_FILE}
+
+# Entropy source for randomness
+-Djava.security.egd=file:/dev/urandom
+
+# Copy the logging context from parent threads to children
+-Dlog4j2.isThreadContextMapInheritable=true
+
+--add-opens java.base/sun.nio.ch=ALL-UNNAMED
+--add-opens java.base/java.io=ALL-UNNAMED
+
+# Custom flags for GC logging.
+-Xlog:gc*:file=/var/log/logstash/logstash_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace

File[/etc/opensearch/relforge-eqiad]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad].orig
+++ File[/etc/opensearch/relforge-eqiad]

+    owner  => root
+    mode   => 0755
+    group  => root
+    ensure => directory

Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-opensearch_shards_9200.service

File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    owner  => root
+    group  => root
+    force  => True
+    ensure => directory
+    mode   => 0755
+    backup => False

Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status].orig
+++ Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]

+    retry_interval              => 1
+    check_interval              => 10
+    ensure                      => present
+    migration_task              => T407130
+    nrpe_command                => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400
+    contact_group               => admins
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 2
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => Check unit status of push_cross_cluster_settings_9400
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

File[/etc/logrotate.d/opensearch-disable-readahead]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => absent

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead.orig
+++ /etc/logrotate.d/opensearch-disable-readahead
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead
+
+/var/log/opensearch-disable-readahead/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Service[clean_up_dlq]

Parameters differences:

--- Systemd::Service[clean_up_dlq].orig
+++ Systemd::Service[clean_up_dlq]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[clean_up_dlq.service]
+    ensure                   => absent
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Service[rsyslog]

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shards_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-elasticsearch-exporter-9400
+
+/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Package[wmf-opensearch-search-plugins]

Parameters differences:

--- Package[wmf-opensearch-search-plugins].orig
+++ Package[wmf-opensearch-search-plugins]

+    before   => ['Service[opensearch_1@relforge-eqiad]', 'Service[opensearch_1@relforge-eqiad-small-alpha]']
+    provider => apt
+    require  => ['Class[Java]', 'Package[opensearch]']
+    ensure   => present

Nginx::Site[relforge-eqiad]

Parameters differences:

--- Nginx::Site[relforge-eqiad].orig
+++ Nginx::Site[relforge-eqiad]

+    require => Exec[tlsproxy localssl default_server on ports [9243]]
+    ensure  => present

File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    owner  => root
+    mode   => 0444
+    group  => root
+    ensure => present

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200
+
+/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/var/lib/nginx]

Parameters differences:

--- File[/var/lib/nginx].orig
+++ File[/var/lib/nginx]

+    owner  => root
+    group  => root
+    before => ['Service[nginx]']
+    ensure => directory

Exec[apt-get update]

Parameters differences:

--- Exec[apt-get update].orig
+++ Exec[apt-get update]

+    before => ['Class[Opensearch]']

Opensearch::Keystore[s3-credentials-user-relforge-eqiad]

Parameters differences:

--- Opensearch::Keystore[s3-credentials-user-relforge-eqiad].orig
+++ Opensearch::Keystore[s3-credentials-user-relforge-eqiad]

+    key          => s3.client.default.access_key
+    value        => search:platform
+    cluster_name => relforge-eqiad

Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200].orig
+++ Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]

+    ensure => present

Exec[s3-credentials-user-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[s3-credentials-user-relforge-eqiad-small-alpha].orig
+++ Exec[s3-credentials-user-relforge-eqiad-small-alpha]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad-small-alpha']
+    command     => echo 'search:platform' | /usr/share/opensearch/bin/opensearch-keystore add s3.client.default.access_key
+    group       => opensearch
+    require     => File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]
+    path        => /bin:/usr/bin
+    unless      => /usr/share/opensearch/bin/opensearch-keystore list | grep s3.client.default.access_key

File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom].orig
+++ File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]

+    owner  => root
+    group  => root
+    ensure => absent

Motd::Message[cirrus::relforge]

Parameters differences:

--- Motd::Message[cirrus::relforge].orig
+++ Motd::Message[cirrus::relforge]

+    priority => 5
+    message  => relforge1008 is a Elasticsearch relforge (cirrus::relforge)
+    ensure   => present

Systemd::Timer::Job[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Timer::Job[push_cross_cluster_settings_9200].orig
+++ Systemd::Timer::Job[push_cross_cluster_settings_9200]

+    send_mail                 => False
+    user                      => root
+    monitoring_enabled        => True
+    logfile_name              => syslog.log
+    logging_enabled           => True
+    command                   => /bin/bash /usr/local/bin/set-cross-cluster-seeds_9200.sh
+    private_tmp               => False
+    description               => Auto set remote cluster seeds for relforge-eqiad
+    syslog_force_stop         => True
+    ignore_errors             => False
+    send_mail_only_on_error   => True
+    fixed_random_delay        => False
+    logfile_perms             => all
+    ensure                    => present
+    logfile_basedir           => /var/log
+    environment               => {}
+    logfile_group             => root
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    success_exit_status       => []
+    interval                  => {'start': 'OnUnitActiveSec', 'interval': '15min'}
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_match_startswith   => True

Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]

+    subscribe                => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    override                 => False
+    restart                  => True
+    require                  => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    ensure                   => present
+    unit_type                => service
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]

+    owner  => root
+    group  => root
+    ensure => absent
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shard_size_check_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shard_size_check_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "c62725841c61e20ac3f6819b4c0ad800" --timeout 10 --check-command "check_opensearch_shard_size_check_9200"

File[/etc/ferm/conf.d/10_opensearch_http_9200]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch_http_9200].orig
+++ File[/etc/ferm/conf.d/10_opensearch_http_9200]

-    owner   => root
-    group   => root
-    tag     => ferm
-    require => File[/etc/ferm/conf.d]
-    mode    => 0400
-    ensure  => present
-    notify  => Service[ferm]

Content differences:

--- /etc/ferm/conf.d/10_opensearch_http_9200.orig
+++ /etc/ferm/conf.d/10_opensearch_http_9200
@@ -1,6 +0,0 @@
-# Autogenerated by puppet. DO NOT EDIT BY HAND!
-#
-# 
-&R_SERVICE(tcp, 9200, $ANALYTICS_NETWORKS);
-
-

File[/etc/nginx/sites-enabled/relforge-eqiad]

Parameters differences:

--- File[/etc/nginx/sites-enabled/relforge-eqiad].orig
+++ File[/etc/nginx/sites-enabled/relforge-eqiad]

+    before => ['Service[nginx]']
+    owner  => root
+    target => /etc/nginx/sites-available/relforge-eqiad
+    group  => root
+    ensure => link

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]

+    before      => ['Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Class[Profile::Opensearch::Cirrus::Relforge]

Parameters differences:

--- Class[Profile::Opensearch::Cirrus::Relforge].orig
+++ Class[Profile::Opensearch::Cirrus::Relforge]

+    cumin_masters => ['10.64.16.154', '2620:0:861:102:10:64:16:154', '10.192.32.49', '2620:0:860:103:10:192:32:49']

Service[opensearch_1@relforge-eqiad-small-alpha]

Parameters differences:

--- Service[opensearch_1@relforge-eqiad-small-alpha].orig
+++ Service[opensearch_1@relforge-eqiad-small-alpha]

+    provider => systemd
+    tag      => opensearch_services
+    require  => ['Systemd::Unit[opensearch_1@.service]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]', 'File[/srv/opensearch/relforge-eqiad-small-alpha]', 'Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]']
+    ensure   => running
+    enable   => True

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    override          => False
+    require           => ['Class[Systemd]']
+    restart           => False
+    ensure            => absent
+    override_filename => puppet-override.conf
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer

Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400].orig
+++ Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]

+    retry_interval              => 120
+    check_interval              => 720
+    ensure                      => present
+    migration_task              => T371083
+    contact_group               => admins,team-discovery
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4
+    enable_nrpe2nodexp          => False
+    critical                    => False
+    alertmanager_team           => observability
+    retries                     => 1
+    nrpe2nodexp_parse_perf_data => False
+    enable_icinga_check         => True
+    timeout                     => 10
+    description                 => OpenSearch unassigned shard check - 9400
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration

Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    unit_name          => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+    fixed_random_delay => False
+    splay              => 0
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00'}]
+    accuracy           => 15sec

Systemd::Service[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Service[push_cross_cluster_settings_9200].orig
+++ Systemd::Service[push_cross_cluster_settings_9200]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[push_cross_cluster_settings_9200.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/lib/systemd/system/opensearch-hot-threads-log.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-hot-threads-log.timer].orig
+++ File[/lib/systemd/system/opensearch-hot-threads-log.timer]

+    owner  => root
+    group  => root
+    ensure => present
+    mode   => 0444
+    notify => Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]

Content differences:

--- /lib/systemd/system/opensearch-hot-threads-log.timer.orig
+++ /lib/systemd/system/opensearch-hot-threads-log.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-hot-threads-log.service
+
+[Timer]
+Unit=opensearch-hot-threads-log.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* *:00/5:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Service[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Service[opensearch-hot-threads-log].orig
+++ Systemd::Service[opensearch-hot-threads-log]

+    override                 => False
+    restart                  => False
+    require                  => Systemd::Unit[opensearch-hot-threads-log.service]
+    ensure                   => present
+    unit_type                => timer
+    migration_task           => T407130
+    monitoring_enabled       => False
+    monitoring_contact_group => admins
+    service_params           => {}
+    monitoring_critical      => False

Compilation results for relforge1008.eqiad.wmnet: System changes detected

Catalog differences

Summary

Resources only in the new catalog

Resources only in the old catalog

Resources modified

Relevant files