Compilation results for relforge1008.eqiad.wmnet: System changes detected

You can retrieve this result from host.json.

Catalog differences

Summary

Total Resources:	2900
Resources added:	640
Resources removed:	8
Resources modified:	652
Change percentage:	44.83%

Resources only in the new catalog

File[/etc/logrotate.d/opensearch]
Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]
File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]
Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
Service[nrpe2nodexp-opensearch_shards_9200.timer]
Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]
File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]
File[/etc/logrotate.d/opensearch-disable-readahead]
Package[edac-utils]
Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]
Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]
File[/etc/ferm/conf.d/10_opensearch-https-9443]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]
Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]
Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]
Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Concat::Fragment[wikimedia-opensearch-header]
Systemd::Syslog[push_cross_cluster_settings_9200]
File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]
Logstash::Input::Gelf[gelf_relay]
Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]
Sudo::User[nrpe-check_opensearch_shard_size_check_9200]
Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]
File[/etc/opensearch/relforge-eqiad/opensearch.yml]
File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]
Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]
File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]
Systemd::Override[opensearch_2@]
Class[Prometheus::Elasticsearch_exporter::Common]
Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]
File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]
Monitoring::Service[check_push_cross_cluster_settings_9200_status]
File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]
Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]
Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]
Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
Class[Java::Tools]
Java::Package[openjdk-jdk-17]
Systemd::Unmask[nginx.service]
Nrpe::Monitor_service[opensearch_shards_9200]
Systemd::Timer[clean_up_dlq]
File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]
File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Class[Sslcert::Dhparam]
Class[Profile::Opensearch::Monitoring::Base_checks]
Sysctl::Conffile[opensearch]
Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]
Systemd::Unit[opensearch-disable-readahead.timer]
Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]
Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]
File[/etc/opensearch/relforge-eqiad]
File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]
Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]
Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Unit[clean_up_dlq.service]
Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
Nrpe::Monitor_service[opensearch_shards_9400]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]
File[/etc/opensearch_hot_threads.d]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]
File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]
Systemd::Timer[opensearch-hot-threads-log]
Systemd::Timer[push_cross_cluster_settings_9400]
File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]
Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]
Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]
File[/var/log/opensearch-disable-readahead-relforge-eqiad]
Systemd::Unit[opensearch_2@.service]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
Package[python3-requests]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
Monitoring::Service[opensearch_unassigned_shard_check_9400]
File[/etc/systemd/system/nginx.service.d/numa.conf]
Monitoring::Service[opensearch_shard_size_check_9400]
Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]
Exec[s3-credentials-user-relforge-eqiad]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]
Service[prometheus-elasticsearch-exporter-9400]
Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]
Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
Nrpe::Check[check_opensearch_unassigned_shard_check_9400]
Rsyslog::Conf[push_cross_cluster_settings_9200]
Class[Role::Cirrus::Relforge]
File[/etc/nginx/sites-available/relforge-eqiad]
Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]
File[/etc/opensearch/relforge-eqiad/opensearch.keystore]
Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]
Systemd::Monitor[push_cross_cluster_settings_9200]
File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]
Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]
File[/etc/ssl/dhparam.pem]
File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]
Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Systemd::Timer::Job[clean_up_dlq]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
Opensearch::Keystore[s3-credentials-user-relforge-eqiad]
File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]
File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]
File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]
Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
Logstash::Output::Udp[gelf_relay]
Concat::Fragment[wikimedia-opensearch]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]
Nginx::Site[relforge-eqiad-small-alpha]
Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]
Systemd::Syslog[push_cross_cluster_settings_9400]
Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]
File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]
Systemd::Service[prometheus-elasticsearch-exporter-9400]
File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]
Package[nginx-light]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]
File[/etc/rsyslog.d/40-clean-up-dlq.conf]
Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Class[Icinga::Elasticsearch::Base_plugin]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]
File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]
Exec[nginx-reload]
Apt::Repository[wikimedia-opensearch]
Systemd::Unit[opensearch-hot-threads-log.service]
File[/etc/default/prometheus-elasticsearch-exporter]
Service[opensearch_2@relforge-eqiad-small-alpha]
Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]
File[/srv/opensearch]
File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]
File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]
Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]
File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]
File[/etc/opensearch/relforge-eqiad/java-security.policy]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
Rsyslog::Conf[opensearch-disable-readahead]
Service[push_cross_cluster_settings_9400.timer]
Alternatives::Java[17]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]
Nrpe::Check[check_opensearch_unassigned_shard_check_9200]
File[/etc/nginx/sites-enabled]
Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]
Systemd::Timer::Job[opensearch-hot-threads-log]
Package[openjdk-17-jdk]
Logrotate::Conf[push_cross_cluster_settings_9400]
Concat::Fragment[wikimedia-opensearch-plugins]
Systemd::Timer::Job[push_cross_cluster_settings_9400]
File[/etc/nginx/sites-enabled/relforge-eqiad]
Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
Monitoring::Service[opensearch_shards_9400]
File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]
Systemd::Unit[opensearch-hot-threads-log.timer]
Class[Profile::Java]
Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Exec[tlsproxy localssl default_server on ports [9443]]
Exec[opensearch-create-keystore-relforge-eqiad]
Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Systemd::Service[push_cross_cluster_settings_9400]
Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
File[/usr/local/bin/jheapdump]
Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]
File[/usr/lib/nagios/plugins/check_elasticsearch]
Systemd::Service[clean_up_dlq]
File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]
Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]
File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
Logrotate::Conf[opensearch]
File[/var/lib/nginx]
Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]
File[/var/log/clean_up_dlq]
File[/etc/nginx/conf.d]
Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]
Class[Profile::Opensearch::Cirrus::Relforge]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]
Motd::Script[cirrus::relforge]
Ferm::Service[opensearch_inter_node_9500]
Nrpe::Monitor_service[opensearch_shard_size_check_9400]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
Nginx::Site[relforge-eqiad]
Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]
Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]
Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]
Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]
Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
File[/lib/systemd/system/clean_up_dlq.service]
Exec[mask_nginx.service]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
Service[nginx]
Logrotate::Conf[opensearch-disable-readahead]
File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]
Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
Sudo::User[nrpe-check_opensearch_shards_9200]
Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]
Logrotate::Conf[clean_up_dlq]
File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]
File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]
Systemd::Syslog[clean_up_dlq]
Class[Java]
Exec[stop-default-nginx]
Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Prometheus::Elasticsearch_exporter[localhost:9200]
File[/etc/update-motd.d/96-cluster-memberships]
File[/etc/nginx]
Java::Cacert[wmf:puppetca.pem]
Logrotate::Conf[push_cross_cluster_settings_9200]
Sysctl::Parameters[opensearch]
Sysctl::Conffile[TCP Fast Open]
Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]
File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]
Rsyslog::Conf[udp_json_logback_compat]
File[/lib/systemd/system/push_cross_cluster_settings_9400.service]
Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]
Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]
File[/etc/logrotate.d/push_cross_cluster_settings_9200]
File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]
Systemd::Unit[push_cross_cluster_settings_9200.timer]
Exec[apt_repository_wikimedia-opensearch-plugins]
File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Unit[prometheus-elasticsearch-exporter-9200]
File[/etc/opensearch/relforge-eqiad/scripts]
File[/etc/sysctl.d/opensearch.conf]
Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Rsyslog::Conf[push_cross_cluster_settings_9400]
File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
File[/var/log/push_cross_cluster_settings_9400]
File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]
Service[push_cross_cluster_settings_9200.timer]
Exec[java__cacert_wmf:puppetca.pem]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]
File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]
Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
Systemd::Tmpfile[opensearch-relforge-eqiad]
Ferm::Service[opensearch-https-9243]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]
Exec[Refresh tmpfile opensearch-relforge-eqiad]
Apt::Repository[wikimedia-opensearch-plugins]
File[/etc/default/logstash]
File[/etc/update-motd.d/05-cirrus--relforge]
Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]
Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]
Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]
Class[Opensearch]
Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]
Nrpe::Plugin[check_systemd_unit_status]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]
Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]
File[/usr/share/opensearch/plugins]
Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
Class[Nginx]
File[/etc/systemd/system/nginx.service.d]
Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]
File[/etc/opensearch]
Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
File[/etc/opensearch/instances]
Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]
Concat_fragment[wikimedia-opensearch-plugins]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]
Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]
Logstash::Conf[input-gelf-gelf_relay]
File[/lib/systemd/system/opensearch_2@.service]
Nrpe::Check[check_opensearch_shard_size_check_9200]
Service[prometheus-elasticsearch-exporter]
File[/etc/logstash/logstash.yml]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]
Package[logstash-oss]
File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]
Systemd::Mask[nginx.service]
Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Opensearch::Cross_cluster_settings[relforge-eqiad]
Systemd::Monitor[push_cross_cluster_settings_9400]
Nrpe::Check[check_opensearch_shard_size_check_9400]
Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]
Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]
Package[nginx-common]
Systemd::Unit[push_cross_cluster_settings_9400.timer]
File[/usr/local/bin/opensearch_hot_threads_logger.py]
File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]
File[/etc/opensearch/relforge-eqiad/jvm.options]
File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]
Systemd::Timer[opensearch-disable-readahead]
Systemd::Service[opensearch-disable-readahead-relforge-eqiad]
Nrpe::Check[check_opensearch_shards_9200]
File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]
Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]
Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]
Class[Profile::Tcp_fast_open]
Systemd::Unit[clean_up_dlq.timer]
Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]
Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
File[/etc/logstash/log4j2.properties]
File[/etc/opensearch/relforge-eqiad-small-alpha]
Ferm::Service[elastic-main-https-9243]
File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]
File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]
Package[prometheus-elasticsearch-exporter]
Systemd::Service[nrpe2nodexp-opensearch_shards_9200]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
Logrotate::Conf[nginx]
Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]
Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
Elasticsearch::Tlsproxy[relforge-eqiad]
File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]
File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]
Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
Exec[s3-credentials-user-relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]
File[/srv/opensearch/relforge-eqiad]
Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]
File[/etc/opensearch/relforge-eqiad/log4j2.properties]
Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]
Exec[systemd reload for nginx systemd fragments]
Class[Profile::Opensearch::Server]
File[/etc/udev/rules.d/40-opensearch-readahead.rules]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]
Exec[tlsproxy localssl default_server on ports [9243]]
File[/etc/logrotate.d/nginx]
Tlsproxy::Localssl[relforge-eqiad-small-alpha]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]
Systemd::Timer[push_cross_cluster_settings_9200]
Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]
Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]
Monitoring::Service[check_push_cross_cluster_settings_9400_status]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]
File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]
Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]
Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]
Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]
Class[Opensearch::Packages]
File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]
File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]
File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]
File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]
Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]
File[/etc/sysctl.d/70-TCP-Fast-Open.conf]
Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
Mount[/var/lib/nginx]
Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]
Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]
Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Service[nrpe2nodexp-opensearch_shards_9400.timer]
File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]
Systemd::Service[opensearch-disable-readahead]
Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Service[opensearch]
Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]
Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Package[logstash-old-name]
Motd::Script[cluster_memberships]
Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]
File[/usr/local/bin/opensearch-disable-readahead.sh]
File[/etc/ferm/conf.d/10_elastic-main-https-9243]
Service[opensearch_2@relforge-eqiad]
Prometheus::Elasticsearch_exporter[localhost:9400]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]
Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
Exec[update_java_alternatives_17]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]
Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]
Sysctl::Parameters[TCP Fast Open]
File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]
Service[clean_up_dlq.timer]
Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]
Service[opensearch-disable-readahead-relforge-eqiad.timer]
Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]
File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]
Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]
Opensearch::Log::Hot_threads_cluster[relforge-eqiad]
Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]
Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]
Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]
Ferm::Service[opensearch-https-9443]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]
Monitoring::Service[opensearch_shard_size_check_9200]
Exec[install-opensearch]
Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]
File[/usr/lib/nagios/plugins/check_elasticsearch.py]
Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]
Concat_fragment[wikimedia-opensearch-header]
File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]
File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]
Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]
Exec[mask_default_prometheus_elasticsearch_exporter]
File[/usr/local/bin/logstash-config-test]
Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]
Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Exec[unmask_nginx.service]
File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]
Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]
Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]
File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]
Package[opensearch]
Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]
File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]
Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]
Systemd::Unit[push_cross_cluster_settings_9200.service]
File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]
Motd::Message[cirrus::relforge]
Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
Udev::Rule[opensearch-readahead]
Systemd::Service[prometheus-elasticsearch-exporter-9200]
Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Service[opensearch-hot-threads-log.timer]
File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]
Nrpe::Check[check_opensearch_shards_9400]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]
File[/var/log/opensearch/opensearch_index_search_slowlog.log]
Service[prometheus-wmf-elasticsearch-exporter-9400]
Rsyslog::Conf[clean_up_dlq]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
Class[Profile::Logstash::Gelf_relay]
File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]
Firewall::Service[opensearch-inter-node-9500]
Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]
Ferm::Service[elastic-small-alpha-https-9443]
Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]
File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]
File_line[opensearch_bin_bashrc]
File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]
File[/etc/logstash/conf.d/README]
Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]
File[/lib/systemd/system/clean_up_dlq.timer]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]
Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]
Opensearch::Instance[relforge-eqiad]
Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]
Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]
Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]
Systemd::Timer::Job[push_cross_cluster_settings_9200]
Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]
Tlsproxy::Localssl[relforge-eqiad]
Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]
File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]
Class[Opensearch::Log::Hot_threads]
File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]
Exec[apt_repository_wikimedia-opensearch]
Systemd::Service[push_cross_cluster_settings_9200]
File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]
File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]
File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
Systemd::Service[opensearch-hot-threads-log]
Nrpe::Monitor_service[opensearch_shard_size_check_9200]
Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]
File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]
Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
File[/lib/systemd/system/opensearch-hot-threads-log.service]
File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]
File[/var/log/opensearch-disable-readahead]
File[/var/log/push_cross_cluster_settings_9200]
File[/lib/systemd/system/push_cross_cluster_settings_9200.service]
Service[prometheus-wmf-elasticsearch-exporter-9200]
File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]
Concat::Fragment[wikimedia-opensearch-plugins-header]
File[/etc/systemd/system/opensearch_2@.service.d]
Systemd::Unit[prometheus-elasticsearch-exporter-9400]
Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]
Monitoring::Service[opensearch_unassigned_shard_check_9200]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]
File[/lib/systemd/system/opensearch-hot-threads-log.timer]
Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]
File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]
Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]
File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]
Package[opensearch-madvise]
Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]
File[/usr/local/bin/cleanup-dlq]
Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]
Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]
File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]
Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]
Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]
File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]
File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
File[/etc/ferm/conf.d/10_opensearch-https-9243]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]
File[/lib/systemd/system/opensearch-disable-readahead.timer]
Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]
Concat_fragment[wikimedia-opensearch-plugins-header]
Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]
Systemd::Unit[push_cross_cluster_settings_9400.service]
Service[logstash]
Concat_fragment[wikimedia-opensearch]
File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]
Class[Profile::Rsyslog::Udp_json_logback_compat]
Class[Profile::Tlsproxy::Instance]
Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]
Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]
Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]
File[/etc/opensearch/relforge-eqiad/logging.yml]
File[/etc/systemd/system/nginx.service.d/security.conf]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]
Sudo::User[nrpe-check_opensearch_shards_9400]
Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]
File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]
File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]
File[/etc/nginx/sites-available]
Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]
Logrotate::Rule[opensearch]
Systemd::Unit[opensearch-disable-readahead.service]
File[/var/log/opensearch]
Systemd::Timer::Job[opensearch-disable-readahead]
File[/etc/elasticsearch]
Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]
File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]
File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]
Systemd::Service[nrpe2nodexp-opensearch_shards_9400]
Opensearch::Instance[relforge-eqiad-small-alpha]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]
Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]
Logstash::Conf[output-udp-gelf_relay]
File[/var/log/opensearch/opensearch.log]
Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]
File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]
Class[Prometheus::Wmf_elasticsearch_exporter::Common]
Monitoring::Service[elasticsearch-https-relforge-eqiad]
Java::Cacert[wmf:Wikimedia_Internal_Root_CA]
File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]
File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]
File[/etc/apt/sources.list.d/wikimedia-opensearch.list]
File[/etc/logstash/jvm.options]
File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]
Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]
Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
Class[Logstash]
File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]
File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]
File[/etc/logrotate.d/clean_up_dlq]
Systemd::Unit[opensearch_2@-opensearch_2@]
File[/etc/default/opensearch]
File[/etc/logrotate.d/push_cross_cluster_settings_9400]
Systemd::Syslog[opensearch-disable-readahead]
Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]
Class[Profile::Opensearch::Cirrus::Server]
File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]
Sudo::User[nrpe-check_opensearch_shard_size_check_9400]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]
File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]
File[/srv/opensearch/relforge-eqiad-small-alpha]
Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]
Monitoring::Service[opensearch_shards_9200]
Package[wmf-opensearch-search-plugins]
Service[prometheus-elasticsearch-exporter-9200]
Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]
File[/etc/nginx/nginx.conf]
Service[opensearch-disable-readahead.timer]
File[/lib/systemd/system/opensearch-disable-readahead.service]
Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

Resources only in the old catalog

Firewall::Service[opensearch-http-9200]
Class[Profile::Opensearch::Cirrus::Test]
Class[Role::Cirrus::Test]
Ferm::Service[opensearch_http_9200]
File[/etc/ferm/conf.d/10_opensearch_http_9200]
Motd::Script[cirrus::test]
Motd::Message[cirrus::test]
File[/etc/update-motd.d/05-cirrus--test]

Resources modified

File[/etc/logrotate.d/opensearch]

Parameters differences:

--- File[/etc/logrotate.d/opensearch].orig
+++ File[/etc/logrotate.d/opensearch]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch.orig
+++ /etc/logrotate.d/opensearch
@@ -0,0 +1,12 @@
+# This file is managed by Puppet.
+# puppet:///logrotate/logrotate.erb
+
+/var/log/opensearch/*.log {
+    notifempty
+    daily
+    copytruncate
+    rotate 7
+    compress
+    delaycompress
+    missingok
+}

Package[edac-utils]

Parameters differences:

--- Package[edac-utils].orig
+++ Package[edac-utils]

+    ensure   => latest
+    provider => apt

Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202].orig
+++ Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9400_a0a4892478c289e50484ffd2fb4c3202]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch unassigned shard check - 9400
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_unassigned_shard_check___9400
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="a0a4892478c289e50484ffd2fb4c3202",check_name="check_opensearch_unassigned_shard_check_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 720m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_unassigned_shard_check_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch unassigned shard check - 9400

File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]

Parameters differences:

--- File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf].orig
+++ File[/etc/logstash/conf.d/90-output-udp-gelf_relay.conf]

+    group        => logstash
+    validate_cmd => /usr/local/bin/logstash-config-test %
+    owner        => logstash
+    notify       => Service[logstash]
+    ensure       => present
+    mode         => 0440

Content differences:

--- /etc/logstash/conf.d/90-output-udp-gelf_relay.conf.orig
+++ /etc/logstash/conf.d/90-output-udp-gelf_relay.conf
@@ -0,0 +1,9 @@
+output {
+
+      udp {
+          host => 'localhost'
+          port => '11514'
+          codec =>'json'
+      }
+
+}

File[/etc/opensearch/relforge-eqiad/opensearch.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/opensearch.yml].orig
+++ File[/etc/opensearch/relforge-eqiad/opensearch.yml]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad/opensearch.yml.orig
+++ /etc/opensearch/relforge-eqiad/opensearch.yml
@@ -0,0 +1,208 @@
+# ======================== OpenSearch Configuration =========================
+#
+# NOTE: OpenSearch comes with reasonable defaults for most settings.
+#       Before you set out to tweak and tune the configuration, make sure you
+#       understand what are you trying to accomplish and the consequences.
+#
+# The primary way of configuring a node is via this file. This template lists
+# the most important settings you may want to configure for a production cluster.
+#
+# Please consult the documentation for further information on configuration options:
+# https://opensearch.org/docs/opensearch/configuration/
+#
+# ---------------------------------- Cluster -----------------------------------
+#
+# Use a descriptive name for your cluster:
+#
+cluster.name: relforge-eqiad
+#
+# ------------------------------------ Node ------------------------------------
+#
+# Use a descriptive name for the node:
+#
+node.name: relforge1008-relforge-eqiad
+#
+# Add custom attributes to the node:
+#
+#node.attr.rack: r1
+#
+# Row/rack awareness attributes
+#
+cluster.routing.allocation.awareness.attributes: row
+node.attr.row: eqiad-row-c
+node.attr.rack: C4
+#
+# Other attributes
+#
+node.attr.hostname: relforge1008
+node.attr.fqdn: relforge1008.eqiad.wmnet
+#
+# Allow this node to be eligible as a master node (enabled by default):
+node.master: true
+# Allow this node to store data (enabled by default):
+node.data: true
+#
+# ----------------------------------- Paths ------------------------------------
+#
+# Path to directory where to store the data (separate multiple locations by comma):
+#
+path.data: /srv/opensearch/relforge-eqiad
+#
+# Path to log files:
+#
+path.logs: /var/log/opensearch
+#
+# ----------------------------------- Memory -----------------------------------
+#
+# Lock the memory on startup:
+#
+bootstrap.memory_lock: false # we don't have swap anyway
+#
+# Make sure that the heap size is set to about half the memory available
+# on the system and that the owner of the process is allowed to use this
+# limit.
+#
+# OpenSearch performs poorly when the system is swapping the memory.
+#
+# ---------------------------------- Network -----------------------------------
+#
+# Set the bind address to a specific IP (IPv4 or IPv6):
+#
+#network.host: 192.168.0.1
+#
+# Set a custom port for HTTP:
+#
+http.port: 9200
+#
+# For more information, consult the network module documentation.
+#
+# Set the address other nodes will use to communicate with this node. If not
+# set, it is automatically derived. It must point to an actual IP address.
+#
+network.publish_host: 10.64.32.100
+
+# Set both 'bind_host' and 'publish_host':
+#
+network.host: [ _local_,_site_ ]
+
+# Set a custom port for the node to node communication (9300 by default):
+#
+transport.tcp.port: 9300
+#
+# --------------------------------- Discovery ----------------------------------
+#
+# Pass an initial list of hosts to perform discovery when this node is started:
+# The default list of hosts is ["127.0.0.1", "[::1]"]
+#
+#discovery.seed_hosts: ["host1", "host2"]
+#
+# Bootstrap the cluster using an initial set of master-eligible nodes:
+#
+#cluster.initial_master_nodes: ["node-1", "node-2"]
+#
+# Set to ensure a node sees N other master eligible nodes to be considered
+# operational within the cluster. Its recommended to set it to a higher value
+# than 1 when running more than 2 nodes in the cluster.
+#
+discovery.zen.ping.unicast.hosts: ["relforge1008.eqiad.wmnet", "relforge1009.eqiad.wmnet", "relforge1010.eqiad.wmnet"]
+#
+# For more information, consult the discovery and cluster formation module documentation.
+#
+# ---------------------------------- Gateway -----------------------------------
+#
+# Block initial recovery after a full cluster restart until N nodes are started:
+#
+gateway.recover_after_nodes: 1
+#
+# For more information, consult the gateway module documentation.
+#
+#
+gateway:
+    recover_after_time: 1m
+    expected_nodes: 2
+#
+# ---------------------------------- Various -----------------------------------
+#
+# Require explicit names when deleting indices:
+#
+#action.destructive_requires_name: true
+#
+# ---------------------------------- Custom settings ---------------------------
+#
+# If a plugin listed here is not installed for current node, the node will not start.
+#
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+
+# Specify the pattern of index names allowed to be created automatically
+action.auto_create_index: true
+
+# Protect against accidental close/delete operations on all indices. You can
+# still close/delete individual indices.
+action.destructive_requires_name: true
+
+# Allow up to 1000 shards to be queried at a time. The default
+# 1k is too low to allow mwgrep to operate.
+action.search.shard_count.limit: 1000
+
+# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-cluster.html#disk-based-shard-allocation
+# Enable the disk space aware shard allocator
+cluster.routing.allocation.disk.threshold_enabled: true
+# Stop allocating shards to nodes (default 80%)
+cluster.routing.allocation.disk.watermark.low: 0.75
+# Begin evicting shards off the node (default 90%)
+cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node (default 95%)
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+
+# Check disk utilization every 60 seconds
+cluster.info.update.interval: 60s
+
+# Set the number of concurrent recoveries happening on a node:
+#
+# 1. During the initial recovery
+cluster.routing.allocation.node_initial_primaries_recoveries: 3
+
+# 2. During adding/removing nodes, rebalancing, etc
+cluster.routing.allocation.node_concurrent_recoveries: 2
+
+# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
+indices.recovery.max_bytes_per_sec: 40mb
+
+##
+# OpenSearch cluster balance configuration
+##
+# This puts much more weight on distributing the shards per index then the defaults.  This will make sure that,
+# for example, each live node gets a copy of enwiki.
+cluster.routing.allocation.balance:
+    shard: 0.195
+    index: 0.8
+
+##
+# Filter cache size
+##
+indices.queries.cache.size: 20%
+
+##
+# Thread pool settings
+##
+thread_pool:
+    write:
+        size: 32
+        queue_size: 1000
+
+
+
+
+##
+# Hosts allowed as data sources for reindexing
+reindex.remote.whitelist: "search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243"
+
+
+##
+# Size the LTR cache to prevent churn under normal and AB testing scenarios
+ltr.caches.max_mem: 100mb
+
+
+
+# Disables the security plugin and the requirement for intra-cluster TLS.
+plugins.security.disabled: true

Exec[apt-get update]

Parameters differences:

--- Exec[apt-get update].orig
+++ Exec[apt-get update]

+    before => ['Class[Opensearch]']

Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead-relforge-eqiad" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead-relforge-eqiad/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Class[Profile::Opensearch::Monitoring::Base_checks]

Parameters differences:

--- Class[Profile::Opensearch::Monitoring::Base_checks].orig
+++ Class[Profile::Opensearch::Monitoring::Base_checks]

+    enable_unassigned_shard_check => True
+    enable_shard_size_check       => True
+    shard_size_warning            => 110
+    enable_health_check           => True
+    timeout                       => 4
+    threshold                     => >=0.15
+    require                       => ['Class[Profile::Opensearch::Server]']
+    shard_size_critical           => 140

File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad.orig
+++ /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead-relforge-eqiad
+
+/var/log/opensearch-disable-readahead-relforge-eqiad/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Unit[opensearch-disable-readahead.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => opensearch-disable-readahead.timer
+    override_filename => puppet-override.conf

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]']

Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Logrotate::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    ensure => present

Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]
+    unit_type                => timer

Systemd::Unit[clean_up_dlq.service]

Parameters differences:

--- Systemd::Unit[clean_up_dlq.service].orig
+++ Systemd::Unit[clean_up_dlq.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => clean_up_dlq.service
+    override_filename => puppet-override.conf

Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9200
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00'}
+    private_tmp               => False
+    description               => Auto restart job: prometheus-wmf-elasticsearch-exporter-9200

Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9200]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 1
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch health check for shards on 9200
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration
+    retry_interval         => 1
+    check_command          => nrpe_check!check_opensearch_shards_9200!10
+    max_check_attempts     => 3
+    host_name              => relforge1008

Exec[s3-credentials-user-relforge-eqiad]

Parameters differences:

--- Exec[s3-credentials-user-relforge-eqiad].orig
+++ Exec[s3-credentials-user-relforge-eqiad]

+    command     => echo 'search:platform' | /usr/share/opensearch/bin/opensearch-keystore add s3.client.default.access_key
+    group       => opensearch
+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad']
+    require     => File[/etc/opensearch/relforge-eqiad/opensearch.keystore]
+    path        => /bin:/usr/bin
+    unless      => /usr/share/opensearch/bin/opensearch-keystore list | grep s3.client.default.access_key

Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9400]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

Systemd::Monitor[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Monitor[push_cross_cluster_settings_9200].orig
+++ Systemd::Monitor[push_cross_cluster_settings_9200]

+    migration_task => T407130
+    check_interval => 10
+    contact_group  => admins
+    retries        => 2
+    critical       => False
+    ensure         => present
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Systemd::Syslog[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Syslog[push_cross_cluster_settings_9400].orig
+++ Systemd::Syslog[push_cross_cluster_settings_9400]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shard_size_check_9200]=/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead-relforge-eqiad.service
+
+[Timer]
+Unit=opensearch-disable-readahead-relforge-eqiad.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Exec[nginx-reload]

Parameters differences:

--- Exec[nginx-reload].orig
+++ Exec[nginx-reload]

+    command     => /usr/sbin/service nginx reload
+    refreshonly => True

Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shards_9200.service
+    override_filename => puppet-override.conf

Alternatives::Java[17]

Parameters differences:

--- Alternatives::Java[17].orig
+++ Alternatives::Java[17]

+    require => Java::Package[openjdk-jdk-17]

Exec[opensearch-create-keystore-relforge-eqiad]

Parameters differences:

--- Exec[opensearch-create-keystore-relforge-eqiad].orig
+++ Exec[opensearch-create-keystore-relforge-eqiad]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad']
+    creates     => /etc/opensearch/relforge-eqiad/opensearch.keystore
+    command     => /usr/share/opensearch/bin/opensearch-keystore create
+    before      => File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)].orig
+++ Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]

+    notify      => ['Service[prometheus-wmf-elasticsearch-exporter-9200]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Nrpe::Monitor_service[disk_space]

Parameters differences:

--- Nrpe::Monitor_service[disk_space].orig
+++ Nrpe::Monitor_service[disk_space]

@@
-    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_command => /usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

Class[Profile::Opensearch::Cirrus::Relforge]

Parameters differences:

--- Class[Profile::Opensearch::Cirrus::Relforge].orig
+++ Class[Profile::Opensearch::Cirrus::Relforge]

+    cumin_masters => ['10.64.16.154', '2620:0:861:102:10:64:16:154', '10.192.32.49', '2620:0:860:103:10:192:32:49']

Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9200]

+    override          => False
+    restart           => True
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => prometheus-wmf-elasticsearch-exporter-9200
+    override_filename => puppet-override.conf

Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => present

Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)].orig
+++ Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[prometheus-elasticsearch-exporter-9400]']

Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]
+    unit_type                => timer

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+    override_filename => puppet-override.conf

Sudo::User[nrpe-check_opensearch_shards_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shards_9200].orig
+++ Sudo::User[nrpe-check_opensearch_shards_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Logrotate::Conf[clean_up_dlq]

Parameters differences:

--- Logrotate::Conf[clean_up_dlq].orig
+++ Logrotate::Conf[clean_up_dlq]

+    ensure => absent

File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]

Parameters differences:

--- File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf].orig
+++ File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/tmpfiles.d/opensearch-relforge-eqiad.conf.orig
+++ /etc/tmpfiles.d/opensearch-relforge-eqiad.conf
@@ -0,0 +1 @@
+d    /run/opensearch-relforge-eqiad  0755 opensearch opensearch - -

File[/etc/update-motd.d/96-cluster-memberships]

Parameters differences:

--- File[/etc/update-motd.d/96-cluster-memberships].orig
+++ File[/etc/update-motd.d/96-cluster-memberships]

+    group  => root
+    source => puppet:///modules/opensearch/opensearch.motd
+    owner  => root
+    ensure => present
+    mode   => 0555

Sysctl::Conffile[TCP Fast Open]

Parameters differences:

--- Sysctl::Conffile[TCP Fast Open].orig
+++ Sysctl::Conffile[TCP Fast Open]

+    ensure             => present
+    priority           => 70
+    no_priority_prefix => False

Nrpe::Plugin[check_systemd_unit_status]

Parameters differences:

--- Nrpe::Plugin[check_systemd_unit_status].orig
+++ Nrpe::Plugin[check_systemd_unit_status]

+    source => puppet:///modules/systemd/check_systemd_unit_status
+    ensure => present

File[/usr/local/bin/opensearch_hot_threads_logger.py]

Parameters differences:

--- File[/usr/local/bin/opensearch_hot_threads_logger.py].orig
+++ File[/usr/local/bin/opensearch_hot_threads_logger.py]

+    source => puppet:///modules/opensearch/opensearch_hot_threads_logger.py
+    owner  => root
+    group  => root
+    mode   => 0555

Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status].orig
+++ Nrpe::Monitor_service[check_push_cross_cluster_settings_9200_status]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200
+    migration_task              => T407130
+    check_interval              => 10
+    contact_group               => admins
+    retries                     => 2
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    ensure                      => present
+    retry_interval              => 1
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => Check unit status of push_cross_cluster_settings_9200

Systemd::Unit[clean_up_dlq.timer]

Parameters differences:

--- Systemd::Unit[clean_up_dlq.timer].orig
+++ Systemd::Unit[clean_up_dlq.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => clean_up_dlq.timer
+    override_filename => puppet-override.conf

Package[prometheus-elasticsearch-exporter]

Parameters differences:

--- Package[prometheus-elasticsearch-exporter].orig
+++ Package[prometheus-elasticsearch-exporter]

+    ensure   => installed
+    provider => apt

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shard_size_check_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Exec[tlsproxy localssl default_server on ports [9243]]

Parameters differences:

--- Exec[tlsproxy localssl default_server on ports [9243]].orig
+++ Exec[tlsproxy localssl default_server on ports [9243]]

+    command     => /bin/true
+    refreshonly => True
+    onlyif      => /bin/false

Systemd::Timer[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Timer[push_cross_cluster_settings_9200].orig
+++ Systemd::Timer[push_cross_cluster_settings_9200]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '15min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => push_cross_cluster_settings_9200.service

Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)].orig
+++ Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]

+    notify      => ['Service[prometheus-wmf-elasticsearch-exporter-9400]']
+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status].orig
+++ Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9400_status]

+    check_period           => 24x7
+    contact_groups         => admins
+    check_interval         => 10
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => Check unit status of push_cross_cluster_settings_9400
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    retry_interval         => 1
+    check_command          => nrpe_check!check_check_push_cross_cluster_settings_9400_status!10
+    max_check_attempts     => 2
+    host_name              => relforge1008

Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    ensure => present

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer
+    override_filename => puppet-override.conf

File[/etc/ferm/conf.d/10_elastic-main-https-9243]

Parameters differences:

--- File[/etc/ferm/conf.d/10_elastic-main-https-9243].orig
+++ File[/etc/ferm/conf.d/10_elastic-main-https-9243]

+    require => File[/etc/ferm/conf.d]
+    tag     => ferm
+    group   => root
+    notify  => Service[ferm]
+    owner   => root
+    ensure  => present
+    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_elastic-main-https-9243.orig
+++ /etc/ferm/conf.d/10_elastic-main-https-9243
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9243, (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49));
+
+

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shards_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shards_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=1min
+OnActiveSec=1s
+RandomizedDelaySec=60
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Monitoring::Service[opensearch_shard_size_check_9200]

Parameters differences:

--- Monitoring::Service[opensearch_shard_size_check_9200].orig
+++ Monitoring::Service[opensearch_shard_size_check_9200]

+    migration_task => T371083
+    check_interval => 1440
+    contact_group  => admins,team-discovery
+    retries        => 3
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    ensure         => present
+    passive        => False
+    retry_interval => 180
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_shard_size_check_9200!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch shard size check - 9200

Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shards_9400]

+    splay              => 60
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_shards_9400.service

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-elasticsearch-exporter-9400" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Exec[mask_default_prometheus_elasticsearch_exporter]

Parameters differences:

--- Exec[mask_default_prometheus_elasticsearch_exporter].orig
+++ Exec[mask_default_prometheus_elasticsearch_exporter]

+    creates => /etc/systemd/system/prometheus-elasticsearch-exporter.service
+    command => /bin/systemctl mask prometheus-elasticsearch-exporter.service

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shards_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shards_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "fd6eadf747c645a72e11bd74109a8156" --timeout 10 --check-command "check_opensearch_shards_9200"

Class[Adduser]

Parameters differences:

--- Class[Adduser].orig
+++ Class[Adduser]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[edac-utils]', 'Package[opensearch]', 'Package[python3-requests]', 'Package[wmf-opensearch-search-plugins]', 'Package[opensearch-madvise]', 'Package[logstash-oss]', 'Package[logstash-old-name]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[openjdk-17-jdk]', 'Package[prometheus-elasticsearch-exporter]', 'Package[nginx-light]', 'Package[nginx-common]']

Udev::Rule[opensearch-readahead]

Parameters differences:

--- Udev::Rule[opensearch-readahead].orig
+++ Udev::Rule[opensearch-readahead]

+    ensure   => present
+    priority => 40

Systemd::Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Service[prometheus-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {'ensure': 'running'}
+    monitoring_enabled       => False
+    require                  => Package[prometheus-elasticsearch-exporter]
+    unit_type                => service

Class[Profile::Logstash::Gelf_relay]

Parameters differences:

--- Class[Profile::Logstash::Gelf_relay].orig
+++ Class[Profile::Logstash::Gelf_relay]

+    version => 7.16.3-1

File[/lib/systemd/system/clean_up_dlq.timer]

Parameters differences:

--- File[/lib/systemd/system/clean_up_dlq.timer].orig
+++ File[/lib/systemd/system/clean_up_dlq.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/clean_up_dlq.timer.orig
+++ /lib/systemd/system/clean_up_dlq.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of clean_up_dlq.service
+
+[Timer]
+Unit=clean_up_dlq.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=1h
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Tlsproxy::Localssl[relforge-eqiad]

Parameters differences:

--- Tlsproxy::Localssl[relforge-eqiad].orig
+++ Tlsproxy::Localssl[relforge-eqiad]

+    skip_private       => False
+    only_get_requests  => False
+    cfssl_paths        => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    acme_certname      => relforge-eqiad
+    keepalive_timeout  => 60
+    acme_chief         => False
+    server_name        => relforge1008.eqiad.wmnet
+    enable_http2       => False
+    require            => ['Class[Profile::Tlsproxy::Instance]']
+    read_timeout       => 180
+    keepalive_requests => 100
+    server_aliases     => ['relforge.svc.eqiad.wmnet']
+    tls_ports          => [9243]
+    upstream_ports     => [9200]
+    upstream_ip        => 10.64.32.100
+    default_server     => True
+    access_log         => False

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+
+[Timer]
+Unit=nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=5min
+OnActiveSec=1s
+RandomizedDelaySec=300
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+    override_filename => puppet-override.conf

Systemd::Service[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Service[opensearch-hot-threads-log].orig
+++ Systemd::Service[opensearch-hot-threads-log]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-hot-threads-log.service]
+    unit_type                => timer

File[/var/log/push_cross_cluster_settings_9200]

Parameters differences:

--- File[/var/log/push_cross_cluster_settings_9200].orig
+++ File[/var/log/push_cross_cluster_settings_9200]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_unassigned_shard_check_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-relforge-eqiad-gc-log-cleanup.timer
+    override_filename => puppet-override.conf

File[/usr/local/bin/cleanup-dlq]

Parameters differences:

--- File[/usr/local/bin/cleanup-dlq].orig
+++ File[/usr/local/bin/cleanup-dlq]

+    owner  => root
+    ensure => absent
+    group  => root

File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf].orig
+++ File[/etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf.orig
+++ /etc/rsyslog.d/40-push-cross-cluster-settings-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "push_cross_cluster_settings_9400" then {
+    action(
+        type="omfile" file="/var/log/push_cross_cluster_settings_9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9400]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

+    backup         => puppet
+    tag            => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    owner          => root
+    replace        => True
+    ensure_newline => False
+    force          => False
+    order          => alpha
+    show_diff      => True
+    group          => root
+    format         => plain
+    mode           => 0444

File[/etc/systemd/system/nginx.service.d/security.conf]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d/security.conf].orig
+++ File[/etc/systemd/system/nginx.service.d/security.conf]

+    require => File[/etc/systemd/system/nginx.service.d]
+    group   => root
+    before  => Class[Nginx]
+    source  => puppet:///modules/profile/tlsproxy/nginx-security.conf
+    owner   => root
+    ensure  => present
+    mode    => 0444

Sudo::User[nrpe-check_opensearch_shards_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shards_9400].orig
+++ Sudo::User[nrpe-check_opensearch_shards_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf.orig
+++ /etc/rsyslog.d/40-opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-relforge-eqiad-small-alpha-gc-log-cleanup" then {
+    action(
+        type="omfile" file="/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/var/log/opensearch/opensearch.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch.log].orig
+++ File[/var/log/opensearch/opensearch.log]

+    owner  => root
+    ensure => absent
+    group  => root

Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status].orig
+++ Sudo::User[nrpe-check_check_push_cross_cluster_settings_9400_status]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]']
+    enable   => False
+    provider => systemd

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json
@@ -0,0 +1,14 @@
+{
+  "persistent": {
+    "cluster.remote.chi.seeds": [
+      "relforge1008.eqiad.wmnet:9300",
+      "relforge1009.eqiad.wmnet:9300",
+      "relforge1010.eqiad.wmnet:9300"
+    ],
+    "cluster.remote.omega.seeds": [
+      "relforge1008.eqiad.wmnet:9500",
+      "relforge1009.eqiad.wmnet:9500",
+      "relforge1010.eqiad.wmnet:9500"
+    ]
+  }
+}

Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9400]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 60
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_shards_9400
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "2a63eb7dcbe368b7f538118765d629d6" --timeout 10 --check-command "check_opensearch_shards_9400"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_shards_9400 command.

File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/scripts].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/scripts]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0444

Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    splay              => 43200
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_unassigned_shard_check_9400.service

File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9200]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

File[/etc/logrotate.d/opensearch-disable-readahead]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead]

+    owner  => root
+    ensure => absent
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead.orig
+++ /etc/logrotate.d/opensearch-disable-readahead
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead
+
+/var/log/opensearch-disable-readahead/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Sudo::User[nrpe-check_opensearch_shard_size_check_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shard_size_check_9200].orig
+++ Sudo::User[nrpe-check_opensearch_shard_size_check_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Systemd::Timer[clean_up_dlq]

Parameters differences:

--- Systemd::Timer[clean_up_dlq].orig
+++ Systemd::Timer[clean_up_dlq]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '1h'}]
+    unit_name          => clean_up_dlq.service

Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- Service[opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]

Parameters differences:

--- Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0].orig
+++ Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9400_status_de4a199c43946c09f819ca4ffea40ac0]

+    team               => observability
+    summary            => NRPE CHECK: Check unit status of push_cross_cluster_settings_9400
+    site               => eqiad
+    alert_name         => nrpe_Check_unit_status_of_push_cross_cluster_settings_9400
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="de4a199c43946c09f819ca4ffea40ac0",check_name="check_check_push_cross_cluster_settings_9400_status", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 11m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_check_push_cross_cluster_settings_9400_status))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: Check unit status of push_cross_cluster_settings_9400

File[/etc/ssl/dhparam.pem]

Parameters differences:

--- File[/etc/ssl/dhparam.pem].orig
+++ File[/etc/ssl/dhparam.pem]

+    group  => root
+    source => puppet:///modules/sslcert/dhparam.pem
+    owner  => root
+    ensure => present
+    mode   => 0444

Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha].orig
+++ Opensearch::Keystore[s3-credentials-user-relforge-eqiad-small-alpha]

+    value        => search:platform
+    cluster_name => relforge-eqiad-small-alpha
+    key          => s3.client.default.access_key

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9400.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/etc/ferm/conf.d/10_opensearch_http_9200]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch_http_9200].orig
+++ File[/etc/ferm/conf.d/10_opensearch_http_9200]

-    require => File[/etc/ferm/conf.d]
-    tag     => ferm
-    group   => root
-    notify  => Service[ferm]
-    owner   => root
-    ensure  => present
-    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_opensearch_http_9200.orig
+++ /etc/ferm/conf.d/10_opensearch_http_9200
@@ -1,6 +0,0 @@
-# Autogenerated by puppet. DO NOT EDIT BY HAND!
-#
-# 
-&R_SERVICE(tcp, 9200, $ANALYTICS_NETWORKS);
-
-

File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]

Parameters differences:

--- File[/usr/local/lib/nagios/plugins/check_systemd_unit_status].orig
+++ File[/usr/local/lib/nagios/plugins/check_systemd_unit_status]

+    require => File[/usr/local/lib/nagios/plugins/]
+    tag     => nrpe::plugin
+    group   => root
+    source  => puppet:///modules/systemd/check_systemd_unit_status
+    owner   => root
+    ensure  => file
+    mode    => 0555

Service[opensearch_2@relforge-eqiad-small-alpha]

Parameters differences:

--- Service[opensearch_2@relforge-eqiad-small-alpha].orig
+++ Service[opensearch_2@relforge-eqiad-small-alpha]

+    tag      => opensearch_services
+    provider => systemd
+    require  => ['Systemd::Unit[opensearch_2@.service]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]', 'File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]', 'File[/srv/opensearch/relforge-eqiad-small-alpha]', 'Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]']
+    ensure   => running
+    enable   => True

File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/log4j2.properties
@@ -0,0 +1,200 @@
+status = error
+
+# log action execution errors for easier debugging
+logger.action.name = org.opensearch.action
+logger.action.level = debug
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+######## Server JSON ############################
+appender.rolling.type = RollingFile
+appender.rolling.name = rolling
+appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.json
+appender.rolling.layout.type = OpenSearchJsonLayout
+appender.rolling.layout.type_name = server
+
+appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128MB
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.fileIndex = nomax
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+######## Server -  old style pattern ###########
+appender.rolling_old.type = RollingFile
+appender.rolling_old.name = rolling_old
+appender.rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}.log
+appender.rolling_old.layout.type = PatternLayout
+appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling_old.policies.type = Policies
+appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling_old.policies.time.interval = 1
+appender.rolling_old.policies.time.modulate = true
+appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling_old.policies.size.size = 128MB
+appender.rolling_old.strategy.type = DefaultRolloverStrategy
+appender.rolling_old.strategy.fileIndex = nomax
+appender.rolling_old.strategy.action.type = Delete
+appender.rolling_old.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling_old.strategy.action.condition.type = IfFileName
+appender.rolling_old.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = rolling
+rootLogger.appenderRef.rolling_old.ref = rolling_old
+
+######## Deprecation JSON #######################
+appender.deprecation_rolling.type = RollingFile
+appender.deprecation_rolling.name = deprecation_rolling
+appender.deprecation_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.json
+appender.deprecation_rolling.layout.type = OpenSearchJsonLayout
+appender.deprecation_rolling.layout.type_name = deprecation
+appender.deprecation_rolling.layout.opensearchmessagefields=x-opaque-id
+
+appender.deprecation_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation-%i.json.gz
+appender.deprecation_rolling.policies.type = Policies
+appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling.policies.size.size = 1GB
+appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling.strategy.max = 4
+#################################################
+######## Deprecation -  old style pattern #######
+appender.deprecation_rolling_old.type = RollingFile
+appender.deprecation_rolling_old.name = deprecation_rolling_old
+appender.deprecation_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.log
+appender.deprecation_rolling_old.layout.type = PatternLayout
+appender.deprecation_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.deprecation_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _deprecation-%i.log.gz
+appender.deprecation_rolling_old.policies.type = Policies
+appender.deprecation_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling_old.policies.size.size = 1GB
+appender.deprecation_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling_old.strategy.max = 4
+#################################################
+logger.deprecation.name = org.opensearch.deprecation
+logger.deprecation.level = warn
+logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
+logger.deprecation.appenderRef.deprecation_rolling_old.ref = deprecation_rolling_old
+logger.deprecation.additivity = false
+
+######## Search slowlog JSON ####################
+appender.index_search_slowlog_rolling.type = RollingFile
+appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
+appender.index_search_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog.json
+appender.index_search_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog
+appender.index_search_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,total_hits,types,stats,search_type,total_shards,source,id
+
+appender.index_search_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog-%i.json.gz
+appender.index_search_slowlog_rolling.policies.type = Policies
+appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling.policies.size.size = 1GB
+appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling.strategy.max = 4
+#################################################
+######## Search slowlog -  old style pattern ####
+appender.index_search_slowlog_rolling_old.type = RollingFile
+appender.index_search_slowlog_rolling_old.name = index_search_slowlog_rolling_old
+appender.index_search_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog.log
+appender.index_search_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_search_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_search_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog-%i.log.gz
+appender.index_search_slowlog_rolling_old.policies.type = Policies
+appender.index_search_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_search_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling_old.strategy.max = 4
+#################################################
+logger.index_search_slowlog_rolling.name = index.search.slowlog
+logger.index_search_slowlog_rolling.level = trace
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling_old.ref = index_search_slowlog_rolling_old
+logger.index_search_slowlog_rolling.additivity = false
+
+######## Indexing slowlog JSON ##################
+appender.index_indexing_slowlog_rolling.type = RollingFile
+appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
+appender.index_indexing_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.json
+appender.index_indexing_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog
+appender.index_indexing_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,doc_type,id,routing,source
+
+appender.index_indexing_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.json.gz
+appender.index_indexing_slowlog_rolling.policies.type = Policies
+appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling.strategy.max = 4
+#################################################
+######## Indexing slowlog -  old style pattern ##
+appender.index_indexing_slowlog_rolling_old.type = RollingFile
+appender.index_indexing_slowlog_rolling_old.name = index_indexing_slowlog_rolling_old
+appender.index_indexing_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.log
+appender.index_indexing_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_indexing_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_indexing_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.log.gz
+appender.index_indexing_slowlog_rolling_old.policies.type = Policies
+appender.index_indexing_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling_old.strategy.max = 4
+#################################################
+
+logger.index_indexing_slowlog.name = index.indexing.slowlog.index
+logger.index_indexing_slowlog.level = trace
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling_old.ref = index_indexing_slowlog_rolling_old
+logger.index_indexing_slowlog.additivity = false
+
+
+### Custom settings
+monitorInterval=600
+
+# Add references to Logstash appender for interesting loggers
+
+rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+
+# Define Logstash appender
+appender.ship_to_logstash.name=ship_to_logstash
+appender.ship_to_logstash.type=syslog
+appender.ship_to_logstash.protocol=udp
+appender.ship_to_logstash.facility=LOCAL0
+appender.ship_to_logstash.layout.type=OpenSearchJsonLayout
+appender.ship_to_logstash.layout.compact=true
+appender.ship_to_logstash.layout.complete=true
+appender.ship_to_logstash.host=localhost
+appender.ship_to_logstash.port=11514
+appender.ship_to_logstash.layout.type_name=opensearch
+

Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources].orig
+++ Concat[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources]

+    warn           => False
+    backup         => puppet
+    owner          => root
+    notify         => Exec[apt_repository_wikimedia-opensearch-plugins]
+    ensure         => present
+    replace        => True
+    ensure_newline => False
+    force          => False
+    order          => alpha
+    show_diff      => True
+    group          => root
+    format         => plain
+    path           => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources
+    mode           => 0444

Rsyslog::Conf[opensearch-disable-readahead]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead].orig
+++ Rsyslog::Conf[opensearch-disable-readahead]

+    mode     => 0444
+    require  => File[/var/log/opensearch-disable-readahead]
+    ensure   => absent
+    priority => 40

Service[push_cross_cluster_settings_9400.timer]

Parameters differences:

--- Service[push_cross_cluster_settings_9400.timer].orig
+++ Service[push_cross_cluster_settings_9400.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status].orig
+++ File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9400_status]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Unit[prometheus-wmf-elasticsearch-exporter-9400]

+    override          => False
+    restart           => True
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => prometheus-wmf-elasticsearch-exporter-9400
+    override_filename => puppet-override.conf

File[/var/lib/nginx]

Parameters differences:

--- File[/var/lib/nginx].orig
+++ File[/var/lib/nginx]

+    owner  => root
+    ensure => directory
+    group  => root
+    before => ['Service[nginx]']

Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400].orig
+++ Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[opensearch-relforge-eqiad-gc-log-cleanup.timer]']

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status].orig
+++ Sudo::User[nrpe-check_check_push_cross_cluster_settings_9200_status]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

File[/etc/logrotate.d/push_cross_cluster_settings_9200]

Parameters differences:

--- File[/etc/logrotate.d/push_cross_cluster_settings_9200].orig
+++ File[/etc/logrotate.d/push_cross_cluster_settings_9200]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/push_cross_cluster_settings_9200.orig
+++ /etc/logrotate.d/push_cross_cluster_settings_9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for push_cross_cluster_settings_9200
+
+/var/log/push_cross_cluster_settings_9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9200.timer].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9200.timer.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9200.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of push_cross_cluster_settings_9200.service
+
+[Timer]
+Unit=push_cross_cluster_settings_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=15min
+OnActiveSec=1s
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-elasticsearch-exporter-9400
+
+/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 43200
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_unassigned_shard_check_9400
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "a0a4892478c289e50484ffd2fb4c3202" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9400"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9400 command.

Systemd::Tmpfile[opensearch-relforge-eqiad]

Parameters differences:

--- Systemd::Tmpfile[opensearch-relforge-eqiad].orig
+++ Systemd::Tmpfile[opensearch-relforge-eqiad]

+    owner  => root
+    ensure => present
+    group  => root

Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9200]

+    splay              => 36000
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_shard_size_check_9200.service

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]
+    unit_type                => timer

Concat_fragment[wikimedia-opensearch-plugins]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-plugins].orig
+++ Concat_fragment[wikimedia-opensearch-plugins]

+    order  => 10
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources

Content differences:

--- wikimedia-opensearch-plugins.orig
+++ wikimedia-opensearch-plugins
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: bookworm-wikimedia
+Components: component/opensearch2
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Service[prometheus-elasticsearch-exporter]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter].orig
+++ Service[prometheus-elasticsearch-exporter]

+    require => Package[prometheus-elasticsearch-exporter]
+    ensure  => stopped

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Package[nginx-common]

Parameters differences:

--- Package[nginx-common].orig
+++ Package[nginx-common]

+    ensure   => installed
+    provider => apt

Nrpe::Check[check_opensearch_shards_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_shards_9200].orig
+++ Nrpe::Check[check_opensearch_shards_9200]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4
+    before  => Monitoring::Service[opensearch_shards_9200]

Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-relforge-eqiad-gc-log-cleanup.service
+    override_filename => puppet-override.conf

Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/nginx/sites-available/relforge-eqiad-small-alpha].orig
+++ File[/etc/nginx/sites-available/relforge-eqiad-small-alpha]

+    group  => root
+    before => ['Service[nginx]']
+    owner  => root
+    notify => Exec[nginx-reload]
+    ensure => present

Content differences:

--- /etc/nginx/sites-available/relforge-eqiad-small-alpha.orig
+++ /etc/nginx/sites-available/relforge-eqiad-small-alpha
@@ -0,0 +1,59 @@
+# Proxy site configuration file for relforge-eqiad-small-alpha
+# This file is managed by Puppet!
+upstream local_fe_relforge-eqiad-small-alpha {
+    server 10.64.32.100:9400 max_fails=0;
+}
+
+# SSL proxying
+server {
+    # Enabling TCP Fast Open is safe for HTTP over TLS. There is no idempotency
+    # concern replaying TLS Client Hello.
+    # https://tools.ietf.org/html/rfc7413#section-6.3.2
+    listen [::]:9443 default_server deferred backlog=16384 reuseport ipv6only=on fastopen=150 ssl ;
+    listen 9443 default_server deferred backlog=16384 reuseport fastopen=150 ssl ;
+    server_name relforge1008.eqiad.wmnet relforge.svc.eqiad.wmnet;
+
+    error_log   /var/log/nginx/relforge-eqiad-small-alpha.error.log;
+    access_log   off;
+
+    ssl_certificate /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem;
+    ssl_certificate_key /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem;
+
+    keepalive_timeout 60;
+    keepalive_requests 100;
+
+    location / {
+        proxy_pass http://local_fe_relforge-eqiad-small-alpha;
+        proxy_http_version 1.1;
+        proxy_set_header Connection close; # should be default
+
+        # this should be in sync with Varnish's first_byte_timeout
+        # and PHP's max_execution_time
+        proxy_read_timeout 180s;
+
+        proxy_set_header Host $host;
+        proxy_set_header X-Client-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Connection-Properties "H2=$h2; SSR=$session_reused; SSL=$ssl_protocol; C=$ssl_cipher;";
+
+        # Don't clobber the Server header from the backend.
+        proxy_pass_header Server;
+
+        proxy_redirect off;
+        proxy_buffering off;
+        proxy_request_buffering on;
+
+        # With proxy buffering off, proxy_buffer_size is still used to
+        # define the singular chunk of buffer the content passes
+        # through.  Apparently if nginx can't read all the response
+        # headers in the first such chunk, it fails with 502.  Upping
+        # this to 8k (from def 4k) solves the one reported case in
+        # https://phabricator.wikimedia.org/T161819
+        # --
+        # Raising this further from 8K to 32K, at least temporarily, due to T226840
+        # Note 32KB is also the similar default limit in our varnishes (http_resp_size)
+        proxy_buffer_size       32k;
+        proxy_buffers         4 32k;
+    }
+}

Logrotate::Conf[nginx]

Parameters differences:

--- Logrotate::Conf[nginx].orig
+++ Logrotate::Conf[nginx]

+    source => puppet:///modules/profile/tlsproxy/logrotate
+    ensure => present
+    tag    => nginx

Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200].orig
+++ Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

+    elasticsearch_port => 9200
+    prometheus_port    => 9120
+    indices_to_monitor => []

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer
+    override_filename => puppet-override.conf

Exec[s3-credentials-user-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[s3-credentials-user-relforge-eqiad-small-alpha].orig
+++ Exec[s3-credentials-user-relforge-eqiad-small-alpha]

+    command     => echo 'search:platform' | /usr/share/opensearch/bin/opensearch-keystore add s3.client.default.access_key
+    group       => opensearch
+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad-small-alpha']
+    require     => File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]
+    path        => /bin:/usr/bin
+    unless      => /usr/share/opensearch/bin/opensearch-keystore list | grep s3.client.default.access_key

Class[Profile::Opensearch::Server]

Parameters differences:

--- Class[Profile::Opensearch::Server].orig
+++ Class[Profile::Opensearch::Server]

+    exporter_extra_config => 
+    version               => 2.19.5
+    base_data_dir         => /srv/opensearch
+    enable_curator        => False
+    native_lib_path       => /usr/share/opensearch/plugins/opensearch-knn/lib
+    instances             => {'relforge-eqiad': {'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    s3_username           => search:platform
+    dc_settings           => {'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2}
+    java_home             => /usr/lib/jvm/java-21-openjdk-amd64
+    require               => ['Class[Profile::Java]', 'Class[Profile::Netbox::Host]']
+    logstash_host         => localhost
+    common_settings       => {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True}
+    logstash_logback_port => 11514

File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg.orig
+++ /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9200_status.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_check_push_cross_cluster_settings_9200_status]=/usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200

Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Timer::Job[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/bin/find /var/log/opensearch -name 'relforge-eqiad-small-alpha_jvm_gc.*.log*' -mtime +30 -delete
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}
+    private_tmp               => False
+    description               => Cleanup GC logs

Package[logstash-old-name]

Parameters differences:

--- Package[logstash-old-name].orig
+++ Package[logstash-old-name]

+    ensure   => absent
+    provider => apt
+    before   => Package[logstash-oss]
+    name     => logstash

Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 300
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-check_push_cross_cluster_settings_9400_status
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "de4a199c43946c09f819ca4ffea40ac0" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9400_status"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9400_status command.

File[/usr/local/bin/opensearch-disable-readahead.sh]

Parameters differences:

--- File[/usr/local/bin/opensearch-disable-readahead.sh].orig
+++ File[/usr/local/bin/opensearch-disable-readahead.sh]

+    group  => root
+    source => puppet:///modules/profile/opensearch/cirrus/opensearch-disable-readahead.sh
+    owner  => root
+    ensure => file
+    mode   => 0555

File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.timer (opensearch-relforge-eqiad-gc-log-cleanup.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-relforge-eqiad-gc-log-cleanup.service
+
+[Timer]
+Unit=opensearch-relforge-eqiad-gc-log-cleanup.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* 02:12:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Service[clean_up_dlq.timer]

Parameters differences:

--- Service[clean_up_dlq.timer].orig
+++ Service[clean_up_dlq.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]']
+    enable   => False
+    provider => systemd

File[/usr/lib/nagios/plugins/check_elasticsearch.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch.py]

+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch.py
+    owner  => root
+    group  => root
+    mode   => 0755

Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shard_size_check_9400]

+    splay              => 36000
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_shard_size_check_9400.service

Package[opensearch]

Parameters differences:

--- Package[opensearch].orig
+++ Package[opensearch]

+    ensure   => 2.19.5
+    provider => apt

Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]

Parameters differences:

--- Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha].orig
+++ Monitoring::Service[elasticsearch-https-relforge-eqiad-small-alpha]

+    migration_task => T384998
+    check_interval => 1
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search
+    retries        => 3
+    config_dir     => /etc/nagios
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    check_command  => check_ssl_on_port!relforge1008.eqiad.wmnet!9443
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => Elasticsearch HTTPS for relforge-eqiad-small-alpha

Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]
+    unit_type                => timer

File_line[opensearch_bin_bashrc]

Parameters differences:

--- File_line[opensearch_bin_bashrc].orig
+++ File_line[opensearch_bin_bashrc]

+    ensure => present
+    line   => PATH=${PATH}:/usr/share/opensearch/bin  # Managed by puppet
+    path   => /root/.bashrc

Class[Profile::Base::Production]

Parameters differences:

--- Class[Profile::Base::Production].orig
+++ Class[Profile::Base::Production]

@@
-    role_description => Unpuppetised system for testing
+    role_description => Elasticsearch relforge

Systemd::Service[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Service[push_cross_cluster_settings_9200].orig
+++ Systemd::Service[push_cross_cluster_settings_9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[push_cross_cluster_settings_9200.service]
+    unit_type                => timer

File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]

Parameters differences:

--- File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf].orig
+++ File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf.orig
+++ /etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf
@@ -0,0 +1 @@
+d    /run/opensearch-relforge-eqiad-small-alpha  0755 opensearch opensearch - -

Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha].orig
+++ Exec[Refresh tmpfile opensearch-relforge-eqiad-small-alpha]

+    command     => /bin/systemd-tmpfiles --create --remove '/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf'
+    user        => root
+    refreshonly => True
+    subscribe   => File[/etc/tmpfiles.d/opensearch-relforge-eqiad-small-alpha.conf]

Firewall::Service[opensearch-inter-node-9300]

Parameters differences:

--- Firewall::Service[opensearch-inter-node-9300].orig
+++ Firewall::Service[opensearch-inter-node-9300]

@@
-    notrack => False
+    notrack => True
@@
-    port    => [9300]
+    port    => 9300

Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Nrpe::Check[check_check_push_cross_cluster_settings_9400_status].orig
+++ Nrpe::Check[check_check_push_cross_cluster_settings_9400_status]

+    ensure  => present
+    command => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400
+    before  => Monitoring::Service[check_push_cross_cluster_settings_9400_status]

Systemd::Unit[push_cross_cluster_settings_9400.service]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9400.service].orig
+++ Systemd::Unit[push_cross_cluster_settings_9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => push_cross_cluster_settings_9400.service
+    override_filename => puppet-override.conf

File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200
+
+/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00'}]
+    unit_name          => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service

File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shards_9400]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)].orig
+++ Exec[systemd daemon-reload for clean_up_dlq.timer (clean_up_dlq.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Sudo::User[nrpe-check_opensearch_shard_size_check_9400]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_shard_size_check_9400].orig
+++ Sudo::User[nrpe-check_opensearch_shard_size_check_9400]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

Service[nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_shards_9200.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]']
+    enable   => False
+    provider => systemd

Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400].orig
+++ Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

+    elasticsearch_port => 9400
+    prometheus_port    => 9121
+    indices_to_monitor => []

File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf].orig
+++ File[/etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf.orig
+++ /etc/rsyslog.d/40-push-cross-cluster-settings-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "push_cross_cluster_settings_9200" then {
+    action(
+        type="omfile" file="/var/log/push_cross_cluster_settings_9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Nrpe::Check[check_check_push_cross_cluster_settings_9200_status].orig
+++ Nrpe::Check[check_check_push_cross_cluster_settings_9200_status]

+    ensure  => present
+    command => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9200
+    before  => Monitoring::Service[check_push_cross_cluster_settings_9200_status]

File[/var/lib/prometheus/node.d/role_owner.prom]

Content differences:

--- /var/lib/prometheus/node.d/role_owner.prom.orig
+++ /var/lib/prometheus/node.d/role_owner.prom
@@ -1,3 +1,3 @@
 # HELP role_owner The team owner of the server role
 # TYPE role_owner gauge
-role_owner{team="search-platform",role="cirrus::test",cluster="relforge"} 1.0
+role_owner{team="search-platform",role="cirrus::relforge",cluster="relforge"} 1.0

Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Service[opensearch-relforge-eqiad-gc-log-cleanup]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-relforge-eqiad-gc-log-cleanup.service]
+    unit_type                => timer

Systemd::Unmask[nginx.service]

Parameters differences:

--- Systemd::Unmask[nginx.service].orig
+++ Systemd::Unmask[nginx.service]

+    unit        => nginx.service
+    refreshonly => True

File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer
+    override_filename => puppet-override.conf

File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200].orig
+++ Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9200]

+    elasticsearch_port => 9200
+    prometheus_port    => 9120
+    indices_to_monitor => []

File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/logging.yml]

+    owner  => root
+    ensure => absent
+    group  => root

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9200_status command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-check_push_cross_cluster_settings_9200_status
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "379fbc6cbce5b297397c866a15729f1a" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9200_status"

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.service (nrpe2nodexp-opensearch_shards_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Nginx::Site[relforge-eqiad-small-alpha]

Parameters differences:

--- Nginx::Site[relforge-eqiad-small-alpha].orig
+++ Nginx::Site[relforge-eqiad-small-alpha]

+    require => Exec[tlsproxy localssl default_server on ports [9443]]
+    ensure  => present

Systemd::Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Service[prometheus-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {'ensure': 'running'}
+    monitoring_enabled       => False
+    require                  => Package[prometheus-elasticsearch-exporter]
+    unit_type                => service

File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json].orig
+++ File[/etc/opensearch/relforge-eqiad/cirrus_check_settings.json]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad/cirrus_check_settings.json.orig
+++ /etc/opensearch/relforge-eqiad/cirrus_check_settings.json
@@ -0,0 +1,14 @@
+{
+  "persistent": {
+    "cluster.remote.chi.seeds": [
+      "relforge1008.eqiad.wmnet:9300",
+      "relforge1009.eqiad.wmnet:9300",
+      "relforge1010.eqiad.wmnet:9300"
+    ],
+    "cluster.remote.omega.seeds": [
+      "relforge1008.eqiad.wmnet:9500",
+      "relforge1009.eqiad.wmnet:9500",
+      "relforge1010.eqiad.wmnet:9500"
+    ]
+  }
+}

Nrpe::Check[check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_unassigned_shard_check_9200].orig
+++ Nrpe::Check[check_opensearch_unassigned_shard_check_9200]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4
+    before  => Monitoring::Service[opensearch_unassigned_shard_check_9200]

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-disable-readahead-relforge-eqiad.service
+    override_filename => puppet-override.conf

Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Timer[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    unit_name          => opensearch-disable-readahead-relforge-eqiad-small-alpha.service

Logrotate::Conf[opensearch]

Parameters differences:

--- Logrotate::Conf[opensearch].orig
+++ Logrotate::Conf[opensearch]

+    ensure => present

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]']

Nrpe::Monitor_service[opensearch_shard_size_check_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shard_size_check_9400].orig
+++ Nrpe::Monitor_service[opensearch_shard_size_check_9400]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    migration_task              => T371083
+    check_interval              => 1440
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    contact_group               => admins,team-discovery
+    retries                     => 3
+    ensure                      => present
+    retry_interval              => 180
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch shard size check - 9400

File[/etc/ferm/conf.d/10_opensearch_inter_node_9300]

Content differences:

--- /etc/ferm/conf.d/10_opensearch_inter_node_9300.orig
+++ /etc/ferm/conf.d/10_opensearch_inter_node_9300
@@ -4,3 +4,5 @@
 &R_SERVICE(tcp, 9300, (10.64.164.14 10.64.32.100 10.64.48.120 2620:0:861:103:10:64:32:100 2620:0:861:107:10:64:48:120 2620:0:861:12c:10:64:164:14));
 
 
+
+&NO_TRACK(tcp, 9300);

Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9400]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 720
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch unassigned shard check - 9400
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration
+    retry_interval         => 120
+    check_command          => nrpe_check!check_opensearch_unassigned_shard_check_9400!10
+    max_check_attempts     => 1
+    host_name              => relforge1008

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=720min
+OnActiveSec=1s
+RandomizedDelaySec=43200
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Exec[mask_nginx.service]

Parameters differences:

--- Exec[mask_nginx.service].orig
+++ Exec[mask_nginx.service]

+    unless  => /usr/bin/dpkg -s nginx-light | /bin/grep -q '^Status: install ok installed$'
+    creates => /etc/systemd/system/nginx.service
+    command => /bin/systemctl mask nginx.service

Exec[stop-default-nginx]

Parameters differences:

--- Exec[stop-default-nginx].orig
+++ Exec[stop-default-nginx]

+    before      => Service[nginx]
+    command     => /usr/sbin/service nginx stop
+    refreshonly => True
+    subscribe   => Package[nginx-light]

Sysctl::Parameters[opensearch]

Parameters differences:

--- Sysctl::Parameters[opensearch].orig
+++ Sysctl::Parameters[opensearch]

+    ensure             => present
+    values             => {'vm.max_map_count': 1048576}
+    priority           => 70
+    no_priority_prefix => True

File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-elasticsearch-exporter-9200
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-elasticsearch-exporter-9200
+
+/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Service[push_cross_cluster_settings_9200.timer]

Parameters differences:

--- Service[push_cross_cluster_settings_9200.timer].orig
+++ Service[push_cross_cluster_settings_9200.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]

+    group  => root
+    owner  => root
+    notify => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    ensure => file
+    mode   => 0440

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]

+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    refreshonly => True
+    subscribe   => File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
+    environment => ['GODEBUG=x509ignoreCN=0']
+    require     => Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

Class[Nginx]

Parameters differences:

--- Class[Nginx].orig
+++ Class[Nginx]

+    modules      => []
+    lib_on_tmpfs => True
+    tmpfs_size   => 1g
+    variant      => light
+    ensure       => present
+    managed      => False

Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]

Parameters differences:

--- Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200].orig
+++ Profile::Prometheus::Elasticsearch_exporter[relforge1008:9200]

+    extra_config       => 
+    elasticsearch_port => 9200
+    prometheus_port    => 9108

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-elasticsearch-exporter-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-elasticsearch-exporter-9200" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Logstash::Conf[input-gelf-gelf_relay]

Parameters differences:

--- Logstash::Conf[input-gelf-gelf_relay].orig
+++ Logstash::Conf[input-gelf-gelf_relay]

+    ensure   => present
+    priority => 10

File[/lib/systemd/system/opensearch_2@.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch_2@.service].orig
+++ File[/lib/systemd/system/opensearch_2@.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch_2@.service.orig
+++ /lib/systemd/system/opensearch_2@.service
@@ -0,0 +1,73 @@
+[Unit]
+Description=OpenSearch (cluster %i)
+PartOf=opensearch.service
+Documentation=https://opensearch.org/docs
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=notify
+RuntimeDirectory=opensearch
+PrivateTmp=true
+Environment=OPENSEARCH_HOME=/usr/share/opensearch
+Environment=OPENSEARCH_PATH_CONF=/etc/opensearch/%i
+Environment=OPENSEARCH_JVM_OPTIONS=/etc/opensearch/%i/jvm.options
+Environment=PID_DIR=/run/opensearch-%i
+Environment=OPENSEARCH_SD_NOTIFY=true
+
+Environment=LD_LIBRARY_PATH=/usr/share/opensearch/plugins/opensearch-knn/lib
+Environment=JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
+
+WorkingDirectory=/usr/share/opensearch
+
+User=opensearch
+Group=opensearch
+
+# Force creation of /run/opensearch (see T276198)
+ExecStartPre=/bin/systemd-tmpfiles --create /etc/tmpfiles.d/opensearch-%i.conf
+ExecStart=/usr/share/opensearch/bin/opensearch \
+        -p ${PID_DIR}/%i.pid \
+        --quiet \
+        -Epath.logs=/var/log/opensearch \
+        -Epath.data=/srv/opensearch/%i
+
+# StandardOutput is configured to redirect to journalctl since
+# some error messages may be logged in standard output before
+# opensearch logging system is initialized. OpenSearch
+# stores its logs in /var/log/opensearch and does not use
+# journalctl by default. If you also want to enable journalctl
+# logging, you can simply remove the "quiet" option from ExecStart.
+StandardOutput=journal
+StandardError=inherit
+
+# Specifies the maximum file descriptor number that can be opened by this process
+LimitNOFILE=65535
+
+# Specifies the maximum number of processes
+LimitNPROC=4096
+
+# Specifies the maximum size of virtual memory
+LimitAS=infinity
+
+# Specifies the maximum file size
+LimitFSIZE=infinity
+
+# Disable timeout logic and wait until process is stopped
+TimeoutStopSec=0
+
+# SIGTERM signal is used to stop the Java process
+KillSignal=SIGTERM
+
+# Send the signal only to the JVM rather than its control group
+KillMode=process
+
+# Java process is never killed
+SendSIGKILL=no
+
+# When a JVM receives a SIGTERM signal it exits with code 143
+SuccessExitStatus=143
+
+[Install]
+WantedBy=multi-user.target
+
+# Built for packages-7.4.1 (packages)

Opensearch::Cross_cluster_settings[relforge-eqiad]

Parameters differences:

--- Opensearch::Cross_cluster_settings[relforge-eqiad].orig
+++ Opensearch::Cross_cluster_settings[relforge-eqiad]

+    enable_remote_search => False
+    settings             => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    instance_name        => relforge-eqiad

Systemd::Service[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Service[opensearch-disable-readahead-relforge-eqiad]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.service]
+    unit_type                => timer

Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400].orig
+++ Nrpe::Monitor_service[opensearch_unassigned_shard_check_9400]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4
+    migration_task              => T371083
+    check_interval              => 720
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration
+    contact_group               => admins,team-discovery
+    retries                     => 1
+    ensure                      => present
+    retry_interval              => 120
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch unassigned shard check - 9400

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shard_size_check_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shard_size_check_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "aefe2d8b10db3bc84662052af1e20008" --timeout 10 --check-command "check_opensearch_shard_size_check_9400"

File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9200.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Systemd::Service[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead].orig
+++ Systemd::Service[opensearch-disable-readahead]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-disable-readahead.service]
+    unit_type                => timer

Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Timer::Job[opensearch-relforge-eqiad-gc-log-cleanup]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/bin/find /var/log/opensearch -name 'relforge-eqiad_jvm_gc.*.log*' -mtime +30 -delete
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}
+    private_tmp               => False
+    description               => Cleanup GC logs

Class[Profile::Monitoring]

Parameters differences:

--- Class[Profile::Monitoring].orig
+++ Class[Profile::Monitoring]

@@
-    nrpe_check_disk_options => -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    nrpe_check_disk_options => -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9200.timer (push_cross_cluster_settings_9200.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[push_cross_cluster_settings_9200.timer]']

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9200.timer (nrpe2nodexp-opensearch_shards_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Exec[install-opensearch]

Parameters differences:

--- Exec[install-opensearch].orig
+++ Exec[install-opensearch]

+    logoutput   => False
+    command     => /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install opensearch=2.19.5
+    timeout     => 300
+    before      => Package[opensearch]
+    environment => ['OPENSEARCH_INITIAL_ADMIN_PASSWORD=OpensearchTemp1!', 'DEBIAN_FRONTEND=noninteractive']
+    unless      => /usr/bin/dpkg-query -W -f='${Status} ${Version}\n' opensearch 2>/dev/null | /bin/grep -Fxq 'install ok installed 2.19.5'

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet]

+    group   => root
+    recurse => True
+    owner   => root
+    ensure  => directory
+    mode    => 0740

Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]

Parameters differences:

--- Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA].orig
+++ Exec[java__cacert_wmf:Wikimedia_Internal_Root_CA]

+    command => /usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /usr/share/ca-certificates/wikimedia/Wikimedia_Internal_Root_CA.crt -storepass changeit -alias wmf:Wikimedia_Internal_Root_CA

+    user    => root
+    group   => root
+    unless  => /usr/bin/keytool -list -cacerts -noprompt -storepass changeit -alias wmf:Wikimedia_Internal_Root_CA

Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400].orig
+++ Profile::Prometheus::Wmf_elasticsearch_exporter[relforge1008:9400]

+    elasticsearch_port => 9400
+    prometheus_port    => 9121
+    indices_to_monitor => []

File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shard_size_check_9400]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9400_status command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-check_push_cross_cluster_settings_9400_status
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "de4a199c43946c09f819ca4ffea40ac0" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9400_status"

Service[opensearch-hot-threads-log.timer]

Parameters differences:

--- Service[opensearch-hot-threads-log.timer].orig
+++ Service[opensearch-hot-threads-log.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

File[/var/log/opensearch/opensearch_index_search_slowlog.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch_index_search_slowlog.log].orig
+++ File[/var/log/opensearch/opensearch_index_search_slowlog.log]

+    owner  => root
+    ensure => absent
+    group  => root

Rsyslog::Conf[clean_up_dlq]

Parameters differences:

--- Rsyslog::Conf[clean_up_dlq].orig
+++ Rsyslog::Conf[clean_up_dlq]

+    mode     => 0444
+    require  => File[/var/log/clean_up_dlq]
+    ensure   => absent
+    priority => 40

File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9400.service (prometheus-wmf-elasticsearch-exporter-9400)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9400.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Prometheus WMF Elasticsearch Exporter
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/local/bin/prometheus-wmf-elasticsearch-exporter \
+    --listen :9121 \
+    --port 9400 \
+    --indices 
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom].orig
+++ File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9400_status.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Exec[apt_repository_wikimedia-opensearch]

Parameters differences:

--- Exec[apt_repository_wikimedia-opensearch].orig
+++ Exec[apt_repository_wikimedia-opensearch]

+    command     => /usr/bin/apt-get update 
+    refreshonly => True

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/jvm.options]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/jvm.options.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/jvm.options
@@ -0,0 +1,109 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://opensearch.org/docs/opensearch/install/important-settings/
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms4G
+-Xmx4G
+
+################################################################
+## Numa Awareness
+################################################################
+-XX:+UseNUMA
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# log4j 2
+-Dlog4j.skipJansi=true
+-Dlog4j2.formatMsgNoLookups=true
+
+# security policy, needed for OpenSearch to access non-default files/sockets/ports T395571
+
+-Djava.security.policy=/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+-XX:HeapDumpPath=/srv/opensearch/relforge-eqiad-small-alpha
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+## Custom Wikimedia configuration
+
+# Defines a limit of proportion of time spent in GC before
+# an OutOfMemory error is thrown.  The goal is to bail and
+# restart the server if throughput is suffering due to
+# GC stalls.
+# TODO: Why 70? why not 60 or 80?
+-XX:GCTimeLimit=70
+
+# Defines minimum percentage of free space after a full GC
+# before an OutOfMemoryError is thrown. Typically a full GC
+# should leave > 50% of heap free. If we are down to 10%
+# something is wrong and bailing is a reasonable solution.
+-XX:GCHeapFreeLimit=10
+
+# Exit on OutOfMemoryError, since in most cases the JVM does not recover
+-XX:+ExitOnOutOfMemoryError
+
+# Custom WMF GC flags
+-Xlog:gc*:file=/var/log/opensearch/relforge-eqiad-small-alpha_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace
+-Xlog:safepoint

Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Concat_fragment[wikimedia-opensearch]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch].orig
+++ Concat_fragment[wikimedia-opensearch]

+    order  => 10
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources

Content differences:

--- wikimedia-opensearch.orig
+++ wikimedia-opensearch
@@ -0,0 +1,5 @@
+Types: deb deb-src
+URIs: http://apt.wikimedia.org/wikimedia
+Suites: bookworm-wikimedia
+Components: thirdparty/opensearch2
+Signed-By: /etc/apt/keyrings/wikimedia-archive-keyring.gpg

Class[Profile::Tlsproxy::Instance]

Parameters differences:

--- Class[Profile::Tlsproxy::Instance].orig
+++ Class[Profile::Tlsproxy::Instance]

+    bootstrap_protection       => False
+    nginx_variant              => light
+    nginx_client_max_body_size => 100m
+    numa_iface                 => lo
+    require                    => ['Class[Profile::Tcp_fast_open]']
+    ssl_compatibility_mode     => compat

Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Opensearch::Instance[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Instance[relforge-eqiad-small-alpha].orig
+++ Opensearch::Instance[relforge-eqiad-small-alpha]

+    recover_after_time                 => 1m
+    disable_security_plugin            => True
+    plugins_mandatory                  => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    reindex_remote_whitelist           => search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243
+    logstash_host                      => localhost
+    load_fixed_bitset_filters_eagerly  => True
+    unicast_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    version                            => 2.19.5
+    curator_uses_unicast_hosts         => True
+    bulk_thread_pool_executors         => 32
+    auto_create_index                  => True
+    transport_tcp_port                 => 9500
+    compatibility_mode                 => False
+    recover_after_nodes                => 1
+    script_max_compilations_per_minute => 10000
+    cms_gc_init_occupancy_fraction     => 75
+    base_data_dir                      => /srv/opensearch
+    use_cms_gc                         => False
+    security_plugin_certificates       => {}
+    search_shard_count_limit           => 1000
+    send_logs_to_logstash              => True
+    row                                => eqiad-row-c
+    http_port                          => 9400
+    bulk_thread_pool_capacity          => 1000
+    holds_data                         => True
+    configure_curator                  => False
+    require                            => Package[opensearch]
+    cluster_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    heap_memory                        => 4G
+    bind_networks                      => ['_local_', '_site_']
+    watermarks                         => {}
+    publish_host                       => 10.64.32.100
+    gc_log                             => True
+    certificate_name                   => relforge.svc.eqiad.wmnet
+    awareness_attributes               => row
+    filter_cache_size                  => 20%
+    expected_nodes                     => 2
+    recovery_max_bytes_per_sec         => 40mb
+    cluster_name                       => relforge-eqiad-small-alpha
+    node_name                          => relforge1008-relforge-eqiad-small-alpha
+    short_cluster_name                 => omega
+    rack                               => C4
+    tls_port                           => 9443
+    ltr_cache_size                     => 100mb
+    logstash_logback_port              => 11514

File[/etc/logstash/jvm.options]

Parameters differences:

--- File[/etc/logstash/jvm.options].orig
+++ File[/etc/logstash/jvm.options]

+    require => Package[logstash-oss]
+    group   => root
+    owner   => root
+    notify  => Service[logstash]
+    mode    => 0444

Content differences:

--- /etc/logstash/jvm.options.orig
+++ /etc/logstash/jvm.options
@@ -0,0 +1,91 @@
+#####################################################################
+## THSI FILE IS MANAGED BY PUPPET
+## puppet://modules/logstash/jvm.options.erb
+#####################################################################
+
+## JVM configuration
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms192m
+-Xmx192m
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+## GC configuration
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+## locale
+# Set the locale language
+#-Duser.language=en
+
+# Set the locale country
+#-Duser.country=US
+
+# Set the locale variant, if any
+#-Duser.variant=
+
+## basic
+
+# set the I/O temp directory
+#-Djava.io.tmpdir=$HOME
+
+# set to headless, just in case
+-Djava.awt.headless=true
+
+# ensure UTF-8 encoding by default (e.g. filenames)
+-Dfile.encoding=UTF-8
+
+# use our provided JNA always versus the system one
+#-Djna.nosys=true
+
+# log4j2
+-Dlog4j2.formatMsgNoLookups=true
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+#-XX:HeapDumpPath=${LOGSTASH_HOME}/heapdump.hprof
+
+## GC logging
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${LS_GC_LOG_FILE}
+
+# Entropy source for randomness
+-Djava.security.egd=file:/dev/urandom
+
+# Copy the logging context from parent threads to children
+-Dlog4j2.isThreadContextMapInheritable=true
+
+--add-opens java.base/sun.nio.ch=ALL-UNNAMED
+--add-opens java.base/java.io=ALL-UNNAMED
+
+# Custom flags for GC logging.
+-Xlog:gc*:file=/var/log/logstash/logstash_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace

Systemd::Unit[opensearch_2@-opensearch_2@]

Parameters differences:

--- Systemd::Unit[opensearch_2@-opensearch_2@].orig
+++ Systemd::Unit[opensearch_2@-opensearch_2@]

+    override          => True
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch_2@
+    override_filename => opensearch_2@

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service
+    override_filename => puppet-override.conf

File[/etc/nginx/nginx.conf]

Parameters differences:

--- File[/etc/nginx/nginx.conf].orig
+++ File[/etc/nginx/nginx.conf]

+    owner  => root
+    before => ['Service[nginx]']
+    tag    => nginx
+    group  => root

Content differences:

--- /etc/nginx/nginx.conf.orig
+++ /etc/nginx/nginx.conf
@@ -0,0 +1,67 @@
+# This file is managed by Puppet!
+
+user www-data www-data;
+worker_processes 48;
+worker_cpu_affinity 000000000000000000000001000000000000000000000001 000000000000000000000001000000000000000000000001 000000000000000000000010000000000000000000000010 000000000000000000000010000000000000000000000010 000000000000000000000100000000000000000000000100 000000000000000000000100000000000000000000000100 000000000000000000001000000000000000000000001000 000000000000000000001000000000000000000000001000 000000000000000000010000000000000000000000010000 000000000000000000010000000000000000000000010000 000000000000000000100000000000000000000000100000 000000000000000000100000000000000000000000100000 000000000000000001000000000000000000000001000000 000000000000000001000000000000000000000001000000 000000000000000010000000000000000000000010000000 000000000000000010000000000000000000000010000000 000000000000000100000000000000000000000100000000 000000000000000100000000000000000000000100000000 000000000000001000000000000000000000001000000000 000000000000001000000000000000000000001000000000 000000000000010000000000000000000000010000000000 000000000000010000000000000000000000010000000000 000000000000100000000000000000000000100000000000 000000000000100000000000000000000000100000000000 000000000001000000000000000000000001000000000000 000000000001000000000000000000000001000000000000 000000000010000000000000000000000010000000000000 000000000010000000000000000000000010000000000000 000000000100000000000000000000000100000000000000 000000000100000000000000000000000100000000000000 000000001000000000000000000000001000000000000000 000000001000000000000000000000001000000000000000 000000010000000000000000000000010000000000000000 000000010000000000000000000000010000000000000000 000000100000000000000000000000100000000000000000 000000100000000000000000000000100000000000000000 000001000000000000000000000001000000000000000000 000001000000000000000000000001000000000000000000 000010000000000000000000000010000000000000000000 000010000000000000000000000010000000000000000000 000100000000000000000000000100000000000000000000 000100000000000000000000000100000000000000000000 001000000000000000000000001000000000000000000000 001000000000000000000000001000000000000000000000 010000000000000000000000010000000000000000000000 010000000000000000000000010000000000000000000000 100000000000000000000000100000000000000000000000 100000000000000000000000100000000000000000000000;
+worker_rlimit_nofile 262144;
+
+error_log  /var/log/nginx/error.log;
+pid        /run/nginx.pid;
+
+events {
+    worker_connections 131072;
+    multi_accept off; # accept 1 conn per epoll hit, basically
+    accept_mutex off; # better latencies at high connection rates
+}
+
+http {
+    include       /etc/nginx/mime.types;
+
+    access_log	/var/log/nginx/access.log;
+
+    client_max_body_size 100m;
+    client_body_buffer_size 64k;
+    http2_max_field_size 8k;
+
+    sendfile        on;
+    #tcp_nopush     on;
+
+    #keepalive_timeout  0;
+    keepalive_timeout  65;
+    tcp_nodelay        on;
+
+    gzip  off;
+    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
+
+    # SSL sessionid caching.  1m is ~4000 sessions.
+    # Still experimenting!
+    ssl_session_cache shared:SSL:1024m;
+    ssl_session_timeout 25h;
+
+    # Disable RFC5077 tickets (may revisit later when client support is better)
+    ssl_session_tickets off;
+
+    # nginx's $http2 is either 'h2' or '' (empty string) for http/1
+    # Map it to $h2 as a boolean string value '1' or '0' for use in headers
+    map $http2 $h2 {
+        default '1';
+        '' '0';
+    }
+
+    # Nginx indicates session reuse with 'r' and '.', which is rather cryptic.
+    # Map this value to '1' and '0' instead.
+    map $ssl_session_reused $session_reused {
+        'r' '1';
+        '.' '0';
+    }
+
+
+    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
+    ssl_ciphers -ALL:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA;
+    ssl_ecdh_curve X25519:prime256v1;
+    ssl_prefer_server_ciphers on;
+    ssl_dhparam /etc/ssl/dhparam.pem;
+
+    include /etc/nginx/conf.d/*.conf;
+    include /etc/nginx/sites-enabled/*;
+}

Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 43200
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_unassigned_shard_check_9200
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "54ca99505516d5452912b573aa7d35d3" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9200"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9200 command.

Concat::Fragment[wikimedia-opensearch-header]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-header].orig
+++ Concat::Fragment[wikimedia-opensearch-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    order  => 01
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources

Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156].orig
+++ Prometheus::Alert::Rule[check_opensearch_shards_9200_fd6eadf747c645a72e11bd74109a8156]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch health check for shards on 9200
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_health_check_for_shards_on_9200
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="fd6eadf747c645a72e11bd74109a8156",check_name="check_opensearch_shards_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 3m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shards_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch health check for shards on 9200

File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status].orig
+++ File[/etc/sudoers.d/nrpe-check_check_push_cross_cluster_settings_9200_status]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml].orig
+++ File[/etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0755

Content differences:

--- /etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml.orig
+++ /etc/opensearch_hot_threads.d/relforge-eqiad-small-alpha.yml
@@ -0,0 +1,3 @@
+---
+port: 9400
+log_file: "/var/log/opensearch/opensearch_hot_threads-relforge-eqiad-small-alpha.log"

Firewall::Service[opensearch-http-9200]

Parameters differences:

--- Firewall::Service[opensearch-http-9200].orig
+++ Firewall::Service[opensearch-http-9200]

-    unrestricted_access => False
-    prio                => 10
-    ensure              => present
-    notrack             => False
-    port                => [9200]
-    src_sets            => ['ANALYTICS_NETWORKS']
-    desc                => 
-    proto               => tcp

File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9200-status.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-check_push_cross_cluster_settings_9200_status" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha].orig
+++ Opensearch::Cross_cluster_settings[relforge-eqiad-small-alpha]

+    enable_remote_search => False
+    settings             => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    instance_name        => relforge-eqiad-small-alpha

Systemd::Timer[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Timer[push_cross_cluster_settings_9400].orig
+++ Systemd::Timer[push_cross_cluster_settings_9400]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '15min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => push_cross_cluster_settings_9400.service

File[/etc/systemd/system/nginx.service.d/numa.conf]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d/numa.conf].orig
+++ File[/etc/systemd/system/nginx.service.d/numa.conf]

+    require => File[/etc/systemd/system/nginx.service.d]
+    group   => root
+    before  => Class[Nginx]
+    owner   => root
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/systemd/system/nginx.service.d/numa.conf.orig
+++ /etc/systemd/system/nginx.service.d/numa.conf
@@ -0,0 +1,3 @@
+[Service]
+ExecStart=
+ExecStart=/usr/bin/numactl -a -N 0,1 /usr/sbin/nginx -g 'daemon on; master_process on;'

Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad].orig
+++ Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad]

+    mode     => 0444
+    require  => File[/var/log/opensearch-disable-readahead-relforge-eqiad]
+    ensure   => present
+    priority => 40

Nrpe::Check[check_opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_unassigned_shard_check_9400].orig
+++ Nrpe::Check[check_opensearch_unassigned_shard_check_9400]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4
+    before  => Monitoring::Service[opensearch_unassigned_shard_check_9400]

Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    mode     => 0444
+    require  => File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]
+    ensure   => present
+    priority => 40

Logstash::Output::Udp[gelf_relay]

Parameters differences:

--- Logstash::Output::Udp[gelf_relay].orig
+++ Logstash::Output::Udp[gelf_relay]

+    codec     => json
+    priority  => 90
+    host      => localhost
+    plugin_id => output/udp/gelf_relay
+    ensure    => present
+    port      => 11514

Concat::Fragment[wikimedia-opensearch]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch].orig
+++ Concat::Fragment[wikimedia-opensearch]

+    order  => 10
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources

Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]
+    unit_type                => timer

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shard_size_check_9200.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shard_size_check_9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=600min
+OnActiveSec=1s
+RandomizedDelaySec=36000
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/etc/rsyslog.d/40-clean-up-dlq.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-clean-up-dlq.conf].orig
+++ File[/etc/rsyslog.d/40-clean-up-dlq.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-clean-up-dlq.conf.orig
+++ /etc/rsyslog.d/40-clean-up-dlq.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "clean_up_dlq" then {
+    action(
+        type="omfile" file="/var/log/clean_up_dlq/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-wmf-elasticsearch-exporter-9400
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9400

Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Service[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service]
+    unit_type                => timer

Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]

Parameters differences:

--- Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a].orig
+++ Prometheus::Alert::Rule[check_check_push_cross_cluster_settings_9200_status_379fbc6cbce5b297397c866a15729f1a]

+    team               => observability
+    summary            => NRPE CHECK: Check unit status of push_cross_cluster_settings_9200
+    site               => eqiad
+    alert_name         => nrpe_Check_unit_status_of_push_cross_cluster_settings_9200
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="379fbc6cbce5b297397c866a15729f1a",check_name="check_check_push_cross_cluster_settings_9200_status", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 11m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_check_push_cross_cluster_settings_9200_status))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: Check unit status of push_cross_cluster_settings_9200

Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Timer::Job[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 300
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-check_push_cross_cluster_settings_9200_status
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "379fbc6cbce5b297397c866a15729f1a" --timeout 10 --check-command "check_check_push_cross_cluster_settings_9200_status"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_check_push_cross_cluster_settings_9200_status command.

File[/usr/local/bin/jheapdump]

Parameters differences:

--- File[/usr/local/bin/jheapdump].orig
+++ File[/usr/local/bin/jheapdump]

+    group  => root
+    source => puppet:///modules/java/jheapdump
+    owner  => root
+    ensure => file
+    mode   => 0555

Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9400]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

Ferm::Service[opensearch_inter_node_9500]

Parameters differences:

--- Ferm::Service[opensearch_inter_node_9500].orig
+++ Ferm::Service[opensearch_inter_node_9500]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => True
+    port                => 9500
+    desc                => 
+    srange              => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    proto               => tcp

Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_shards_9200]

+    splay              => 60
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_shards_9200.service

Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00'}]
+    unit_name          => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service

Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shards_9400]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 1
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch health check for shards on 9400
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration
+    retry_interval         => 1
+    check_command          => nrpe_check!check_opensearch_shards_9400!10
+    max_check_attempts     => 3
+    host_name              => relforge1008

Class[Java]

Parameters differences:

--- Class[Java].orig
+++ Class[Java]

+    java_packages => [{'version': '17', 'variant': 'jdk'}]
+    before        => ['Java::Cacert[wmf:puppetca.pem]', 'Java::Cacert[wmf:Wikimedia_Internal_Root_CA]', 'Java::Cacert[wmf:puppetca.pem]', 'Java::Cacert[wmf:Wikimedia_Internal_Root_CA]']
+    enable_dbg    => False
+    require       => Package[wmf-certificates]
+    egd_source    => /dev/random
+    hardened_tls  => False

Rsyslog::Conf[udp_json_logback_compat]

Parameters differences:

--- Rsyslog::Conf[udp_json_logback_compat].orig
+++ Rsyslog::Conf[udp_json_logback_compat]

+    ensure   => present
+    mode     => 0444
+    priority => 50

File[/lib/systemd/system/push_cross_cluster_settings_9400.service]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9400.service].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9400.service (push_cross_cluster_settings_9400.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9400.service.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto set remote cluster seeds for relforge-eqiad-small-alpha
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/bin/bash /usr/local/bin/set-cross-cluster-seeds_9400.sh

Systemd::Unit[push_cross_cluster_settings_9200.timer]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9200.timer].orig
+++ Systemd::Unit[push_cross_cluster_settings_9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => push_cross_cluster_settings_9200.timer
+    override_filename => puppet-override.conf

File[/var/log/push_cross_cluster_settings_9400]

Parameters differences:

--- File[/var/log/push_cross_cluster_settings_9400].orig
+++ File[/var/log/push_cross_cluster_settings_9400]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    require     => Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    unless      => /usr/bin/test "$(/usr/bin/openssl x509 -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem -noout -pubkey 2>&1)" == "$(/usr/bin/openssl pkey -pubout -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem 2>&1)"

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+    override_filename => puppet-override.conf

Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha].orig
+++ Systemd::Tmpfile[opensearch-relforge-eqiad-small-alpha]

+    owner  => root
+    ensure => present
+    group  => root

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer
+    override_filename => puppet-override.conf

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem]

+    require => Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]
+    group   => root
+    notify  => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    owner   => root
+    ensure  => file

Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Timer[opensearch-disable-readahead-relforge-eqiad]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    unit_name          => opensearch-disable-readahead-relforge-eqiad.service

File[/usr/share/opensearch/plugins]

Parameters differences:

--- File[/usr/share/opensearch/plugins].orig
+++ File[/usr/share/opensearch/plugins]

+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]']
+    enable   => False
+    provider => systemd

Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

Parameters differences:

--- Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem].orig
+++ Exec[create chained cert /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

+    command   => /bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem > /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem
+    subscribe => ['Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]', 'File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]', 'File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem]']
+    require   => Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh on intermediate ca change]
+    notify    => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    unless    => /usr/bin/test "$(/bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem | sha512sum)" == "$(/bin/cat /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem | sha512sum)"

Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200].orig
+++ Sudo::User[nrpe-check_opensearch_unassigned_shard_check_9200]

+    privileges => []
+    tag        => nrpe::check
+    user       => nagios
+    require    => ['Class[Sudo]']
+    ensure     => absent

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad /srv/opensearch

Nrpe::Check[check_opensearch_shard_size_check_9200]

Parameters differences:

--- Nrpe::Check[check_opensearch_shard_size_check_9200].orig
+++ Nrpe::Check[check_opensearch_shard_size_check_9200]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    before  => Monitoring::Service[opensearch_shard_size_check_9200]

File[/etc/logstash/logstash.yml]

Parameters differences:

--- File[/etc/logstash/logstash.yml].orig
+++ File[/etc/logstash/logstash.yml]

+    require => Package[logstash-oss]
+    group   => root
+    owner   => root
+    notify  => Service[logstash]
+    mode    => 0444

Content differences:

--- /etc/logstash/logstash.yml.orig
+++ /etc/logstash/logstash.yml
@@ -0,0 +1,11 @@
+---
+path.data: "/var/lib/logstash"
+path.config: "/etc/logstash/conf.d"
+path.logs: "/var/log/logstash"
+pipeline.workers: 2
+pipeline.batch.size: 125
+pipeline.batch.delay: 50
+log.format: json
+dead_letter_queue.enable: false
+dead_letter_queue.max_bytes: 1024mb
+http.port: '9675'

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/logstash/log4j2.properties]

Parameters differences:

--- File[/etc/logstash/log4j2.properties].orig
+++ File[/etc/logstash/log4j2.properties]

+    require => Package[logstash-oss]
+    group   => root
+    owner   => root
+    ensure  => file
+    mode    => 0444

Content differences:

--- /etc/logstash/log4j2.properties.orig
+++ /etc/logstash/log4j2.properties
@@ -0,0 +1,51 @@
+status = error
+name = LogstashPropertiesConfig
+
+appender.rolling.type = RollingFile
+appender.rolling.name = plain_rolling
+appender.rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
+appender.rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.layout.type = PatternLayout
+appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %-.10000m%n
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128M
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.max = 4
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:ls.logs}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
+appender.rolling.strategy.action.ifAny.type = IfAny
+appender.rolling.strategy.action.ifAny.ifLastModified.type = IfLastModified
+appender.rolling.strategy.action.ifAny.ifLastModified.age = 30D
+appender.rolling.strategy.action.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.ifAny.ifAccumulatedFileSize.exceeds = 15GB
+
+appender.json_rolling.type = RollingFile
+appender.json_rolling.name = json_rolling
+appender.json_rolling.fileName = ${sys:ls.logs}/logstash-${sys:ls.log.format}.log
+appender.json_rolling.filePattern = ${sys:ls.logs}/logstash-${sys:ls.log.format}-%d{yyyy-MM-dd}.log
+appender.json_rolling.policies.type = Policies
+appender.json_rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.json_rolling.policies.time.interval = 1
+appender.json_rolling.policies.time.modulate = true
+appender.json_rolling.layout.type = JSONLayout
+appender.json_rolling.layout.compact = true
+appender.json_rolling.layout.eventEol = true
+appender.json_rolling.strategy.type = DefaultRolloverStrategy
+appender.json_rolling.strategy.action.type = Delete
+appender.json_rolling.strategy.action.basepath = ${sys:ls.logs}
+appender.json_rolling.strategy.action.condition.type = IfFileName
+appender.json_rolling.strategy.action.condition.glob = logstash-${sys:ls.log.format}-*
+appender.json_rolling.strategy.action.ifAny.type = IfAny
+appender.json_rolling.strategy.action.ifAny.ifLastModified.type = IfLastModified
+appender.json_rolling.strategy.action.ifAny.ifLastModified.age = 30D
+appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.type = IfAccumulatedFileSize
+appender.json_rolling.strategy.action.ifAny.ifAccumulatedFileSize.exceeds = 10GB
+
+rootLogger.level = ${sys:ls.log.level}
+rootLogger.appenderRef.rolling.ref = ${sys:ls.log.format}_rolling

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem]

+    notify => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    group  => root
+    source => puppet:///modules/profile/pki/intermediates/discovery2026-cert.pem
+    owner  => root
+    ensure => file
+    mode   => 0440

File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_unassigned_shard_check_9200]=/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4

Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet].orig
+++ Exec[renew certificate - discovery2026__relforge1008_eqiad_wmnet]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    require     => Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet]
+    command     => /usr/bin/cfssl sign -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    unless      => /usr/bin/openssl x509 -in /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem -checkend 952200

File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+
+[Timer]
+Unit=opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* 02:12:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]

Parameters differences:

--- File[/var/log/opensearch/opensearch_index_indexing_slowlog.log].orig
+++ File[/var/log/opensearch/opensearch_index_indexing_slowlog.log]

+    owner  => root
+    ensure => absent
+    group  => root

File[/etc/sysctl.d/70-TCP-Fast-Open.conf]

Parameters differences:

--- File[/etc/sysctl.d/70-TCP-Fast-Open.conf].orig
+++ File[/etc/sysctl.d/70-TCP-Fast-Open.conf]

+    notify => Exec[update_sysctl]
+    owner  => root
+    ensure => present
+    group  => root

Content differences:

--- /etc/sysctl.d/70-TCP-Fast-Open.conf.orig
+++ /etc/sysctl.d/70-TCP-Fast-Open.conf
@@ -0,0 +1,2 @@
+# sysctl parameters managed by Puppet.
+net.ipv4.tcp_fastopen = 3

Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Rsyslog::Conf[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    mode     => 0444
+    require  => File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]
+    ensure   => present
+    priority => 40

Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]
+    unit_type                => timer

Service[nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_shards_9400.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]']
+    enable   => False
+    provider => systemd

Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]

Parameters differences:

--- Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)].orig
+++ Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[prometheus-elasticsearch-exporter-9200]']

Prometheus::Elasticsearch_exporter[localhost:9400]

Parameters differences:

--- Prometheus::Elasticsearch_exporter[localhost:9400].orig
+++ Prometheus::Elasticsearch_exporter[localhost:9400]

+    extra_config       => 
+    elasticsearch_port => 9400
+    prometheus_port    => 9109

Service[opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- Service[opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ Service[opensearch-disable-readahead-relforge-eqiad.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)].orig
+++ Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha].orig
+++ Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad-small-alpha]

+    check_period           => 24x7
+    contact_groups         => team-data-platform,team-discovery
+    check_interval         => 1
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => Elasticsearch HTTPS for relforge-eqiad-small-alpha
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search
+    retry_interval         => 1
+    check_command          => check_ssl_on_port!relforge1008.eqiad.wmnet!9443
+    max_check_attempts     => 3
+    host_name              => relforge1008

Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Timer[opensearch-relforge-eqiad-gc-log-cleanup]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}]
+    unit_name          => opensearch-relforge-eqiad-gc-log-cleanup.service

Opensearch::Log::Hot_threads_cluster[relforge-eqiad]

Parameters differences:

--- Opensearch::Log::Hot_threads_cluster[relforge-eqiad].orig
+++ Opensearch::Log::Hot_threads_cluster[relforge-eqiad]

+    cluster_name => relforge-eqiad
+    http_port    => 9200

File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_shards_9200]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

File[/etc/nagios/nrpe.d/check_disk_space.cfg]

Content differences:

--- /etc/nagios/nrpe.d/check_disk_space.cfg.orig
+++ /etc/nagios/nrpe.d/check_disk_space.cfg
@@ -1,2 +1,2 @@
 # File generated by puppet. DO NOT edit by hand
-command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead-relforge-eqiad-small-alpha.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead-relforge-eqiad-small-alpha" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Unit[push_cross_cluster_settings_9200.service]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9200.service].orig
+++ Systemd::Unit[push_cross_cluster_settings_9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => push_cross_cluster_settings_9200.service
+    override_filename => puppet-override.conf

File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File[/etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.orig
+++ /etc/logrotate.d/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400
+
+/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Firewall::Service[opensearch-inter-node-9500]

Parameters differences:

--- Firewall::Service[opensearch-inter-node-9500].orig
+++ Firewall::Service[opensearch-inter-node-9500]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => True
+    port                => 9500
+    desc                => 
+    srange              => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    proto               => tcp

Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[opensearch-hot-threads-log.timer]']

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Opensearch::Instance[relforge-eqiad]

Parameters differences:

--- Opensearch::Instance[relforge-eqiad].orig
+++ Opensearch::Instance[relforge-eqiad]

+    recover_after_time                 => 1m
+    disable_security_plugin            => True
+    plugins_mandatory                  => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    reindex_remote_whitelist           => search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243
+    logstash_host                      => localhost
+    load_fixed_bitset_filters_eagerly  => True
+    unicast_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    version                            => 2.19.5
+    curator_uses_unicast_hosts         => True
+    bulk_thread_pool_executors         => 32
+    auto_create_index                  => True
+    transport_tcp_port                 => 9300
+    compatibility_mode                 => False
+    recover_after_nodes                => 1
+    script_max_compilations_per_minute => 10000
+    cms_gc_init_occupancy_fraction     => 75
+    base_data_dir                      => /srv/opensearch
+    use_cms_gc                         => False
+    security_plugin_certificates       => {}
+    search_shard_count_limit           => 1000
+    send_logs_to_logstash              => True
+    row                                => eqiad-row-c
+    http_port                          => 9200
+    bulk_thread_pool_capacity          => 1000
+    holds_data                         => True
+    configure_curator                  => False
+    require                            => Package[opensearch]
+    cluster_hosts                      => ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet']
+    heap_memory                        => 30G
+    bind_networks                      => ['_local_', '_site_']
+    watermarks                         => {}
+    publish_host                       => 10.64.32.100
+    gc_log                             => True
+    certificate_name                   => relforge.svc.eqiad.wmnet
+    awareness_attributes               => row
+    filter_cache_size                  => 20%
+    expected_nodes                     => 2
+    recovery_max_bytes_per_sec         => 40mb
+    cluster_name                       => relforge-eqiad
+    node_name                          => relforge1008-relforge-eqiad
+    short_cluster_name                 => chi
+    rack                               => C4
+    tls_port                           => 9243
+    ltr_cache_size                     => 100mb
+    logstash_logback_port              => 11514

Systemd::Timer::Job[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Timer::Job[push_cross_cluster_settings_9200].orig
+++ Systemd::Timer::Job[push_cross_cluster_settings_9200]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /bin/bash /usr/local/bin/set-cross-cluster-seeds_9200.sh
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => True
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnUnitActiveSec', 'interval': '15min'}
+    private_tmp               => False
+    description               => Auto set remote cluster seeds for relforge-eqiad

File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml.orig
+++ /etc/opensearch/relforge-eqiad-small-alpha/opensearch.yml
@@ -0,0 +1,208 @@
+# ======================== OpenSearch Configuration =========================
+#
+# NOTE: OpenSearch comes with reasonable defaults for most settings.
+#       Before you set out to tweak and tune the configuration, make sure you
+#       understand what are you trying to accomplish and the consequences.
+#
+# The primary way of configuring a node is via this file. This template lists
+# the most important settings you may want to configure for a production cluster.
+#
+# Please consult the documentation for further information on configuration options:
+# https://opensearch.org/docs/opensearch/configuration/
+#
+# ---------------------------------- Cluster -----------------------------------
+#
+# Use a descriptive name for your cluster:
+#
+cluster.name: relforge-eqiad-small-alpha
+#
+# ------------------------------------ Node ------------------------------------
+#
+# Use a descriptive name for the node:
+#
+node.name: relforge1008-relforge-eqiad-small-alpha
+#
+# Add custom attributes to the node:
+#
+#node.attr.rack: r1
+#
+# Row/rack awareness attributes
+#
+cluster.routing.allocation.awareness.attributes: row
+node.attr.row: eqiad-row-c
+node.attr.rack: C4
+#
+# Other attributes
+#
+node.attr.hostname: relforge1008
+node.attr.fqdn: relforge1008.eqiad.wmnet
+#
+# Allow this node to be eligible as a master node (enabled by default):
+node.master: true
+# Allow this node to store data (enabled by default):
+node.data: true
+#
+# ----------------------------------- Paths ------------------------------------
+#
+# Path to directory where to store the data (separate multiple locations by comma):
+#
+path.data: /srv/opensearch/relforge-eqiad-small-alpha
+#
+# Path to log files:
+#
+path.logs: /var/log/opensearch
+#
+# ----------------------------------- Memory -----------------------------------
+#
+# Lock the memory on startup:
+#
+bootstrap.memory_lock: false # we don't have swap anyway
+#
+# Make sure that the heap size is set to about half the memory available
+# on the system and that the owner of the process is allowed to use this
+# limit.
+#
+# OpenSearch performs poorly when the system is swapping the memory.
+#
+# ---------------------------------- Network -----------------------------------
+#
+# Set the bind address to a specific IP (IPv4 or IPv6):
+#
+#network.host: 192.168.0.1
+#
+# Set a custom port for HTTP:
+#
+http.port: 9400
+#
+# For more information, consult the network module documentation.
+#
+# Set the address other nodes will use to communicate with this node. If not
+# set, it is automatically derived. It must point to an actual IP address.
+#
+network.publish_host: 10.64.32.100
+
+# Set both 'bind_host' and 'publish_host':
+#
+network.host: [ _local_,_site_ ]
+
+# Set a custom port for the node to node communication (9300 by default):
+#
+transport.tcp.port: 9500
+#
+# --------------------------------- Discovery ----------------------------------
+#
+# Pass an initial list of hosts to perform discovery when this node is started:
+# The default list of hosts is ["127.0.0.1", "[::1]"]
+#
+#discovery.seed_hosts: ["host1", "host2"]
+#
+# Bootstrap the cluster using an initial set of master-eligible nodes:
+#
+#cluster.initial_master_nodes: ["node-1", "node-2"]
+#
+# Set to ensure a node sees N other master eligible nodes to be considered
+# operational within the cluster. Its recommended to set it to a higher value
+# than 1 when running more than 2 nodes in the cluster.
+#
+discovery.zen.ping.unicast.hosts: ["relforge1008.eqiad.wmnet", "relforge1009.eqiad.wmnet", "relforge1010.eqiad.wmnet"]
+#
+# For more information, consult the discovery and cluster formation module documentation.
+#
+# ---------------------------------- Gateway -----------------------------------
+#
+# Block initial recovery after a full cluster restart until N nodes are started:
+#
+gateway.recover_after_nodes: 1
+#
+# For more information, consult the gateway module documentation.
+#
+#
+gateway:
+    recover_after_time: 1m
+    expected_nodes: 2
+#
+# ---------------------------------- Various -----------------------------------
+#
+# Require explicit names when deleting indices:
+#
+#action.destructive_requires_name: true
+#
+# ---------------------------------- Custom settings ---------------------------
+#
+# If a plugin listed here is not installed for current node, the node will not start.
+#
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+
+# Specify the pattern of index names allowed to be created automatically
+action.auto_create_index: true
+
+# Protect against accidental close/delete operations on all indices. You can
+# still close/delete individual indices.
+action.destructive_requires_name: true
+
+# Allow up to 1000 shards to be queried at a time. The default
+# 1k is too low to allow mwgrep to operate.
+action.search.shard_count.limit: 1000
+
+# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-cluster.html#disk-based-shard-allocation
+# Enable the disk space aware shard allocator
+cluster.routing.allocation.disk.threshold_enabled: true
+# Stop allocating shards to nodes (default 80%)
+cluster.routing.allocation.disk.watermark.low: 0.75
+# Begin evicting shards off the node (default 90%)
+cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node (default 95%)
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+
+# Check disk utilization every 60 seconds
+cluster.info.update.interval: 60s
+
+# Set the number of concurrent recoveries happening on a node:
+#
+# 1. During the initial recovery
+cluster.routing.allocation.node_initial_primaries_recoveries: 3
+
+# 2. During adding/removing nodes, rebalancing, etc
+cluster.routing.allocation.node_concurrent_recoveries: 2
+
+# Set to throttle throughput when recovering (eg. 100mb, by default 20mb):
+indices.recovery.max_bytes_per_sec: 40mb
+
+##
+# OpenSearch cluster balance configuration
+##
+# This puts much more weight on distributing the shards per index then the defaults.  This will make sure that,
+# for example, each live node gets a copy of enwiki.
+cluster.routing.allocation.balance:
+    shard: 0.195
+    index: 0.8
+
+##
+# Filter cache size
+##
+indices.queries.cache.size: 20%
+
+##
+# Thread pool settings
+##
+thread_pool:
+    write:
+        size: 32
+        queue_size: 1000
+
+
+
+
+##
+# Hosts allowed as data sources for reindexing
+reindex.remote.whitelist: "search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243"
+
+
+##
+# Size the LTR cache to prevent churn under normal and AB testing scenarios
+ltr.caches.max_mem: 100mb
+
+
+
+# Disables the security plugin and the requirement for intra-cluster TLS.
+plugins.security.disabled: true

File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0400

Content differences:

--- /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr.orig
+++ /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr
@@ -0,0 +1,14 @@
+{
+  "CN": "relforge1008.eqiad.wmnet",
+  "hosts": [
+    "relforge.svc.eqiad.wmnet",
+    "relforge1008.eqiad.wmnet"
+  ],
+  "key": {
+    "algo": "ecdsa",
+    "size": 256
+  },
+  "names": [
+
+  ]
+}

File[/lib/systemd/system/opensearch-hot-threads-log.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-hot-threads-log.service].orig
+++ File[/lib/systemd/system/opensearch-hot-threads-log.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-hot-threads-log.service.orig
+++ /lib/systemd/system/opensearch-hot-threads-log.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Archive exception logs of hot opensearch threads
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=opensearch
+ExecStart=/usr/local/bin/opensearch_hot_threads_logger.py

Package[opensearch-madvise]

Parameters differences:

--- Package[opensearch-madvise].orig
+++ Package[opensearch-madvise]

+    ensure   => present
+    provider => apt

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]']

File[/lib/systemd/system/opensearch-disable-readahead.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead.service
+
+[Timer]
+Unit=opensearch-disable-readahead.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

Parameters differences:

--- Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources].orig
+++ Concat_file[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

+    backup         => puppet
+    tag            => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    owner          => root
+    replace        => True
+    ensure_newline => False
+    force          => False
+    order          => alpha
+    show_diff      => True
+    group          => root
+    format         => plain
+    mode           => 0444

Systemd::Timer::Job[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes

Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]
+    unit_type                => timer

Java::Cacert[wmf:Wikimedia_Internal_Root_CA]

Parameters differences:

--- Java::Cacert[wmf:Wikimedia_Internal_Root_CA].orig
+++ Java::Cacert[wmf:Wikimedia_Internal_Root_CA]

+    require   => Alternatives::Java[17]
+    group     => root
+    storepass => changeit
+    owner     => root
+    ensure    => present
+    path      => /usr/share/ca-certificates/wikimedia/Wikimedia_Internal_Root_CA.crt

Motd::Message[cirrus::test]

Parameters differences:

--- Motd::Message[cirrus::test].orig
+++ Motd::Message[cirrus::test]

-    message  => relforge1008 is a Unpuppetised system for testing (cirrus::test)
-    ensure   => present
-    priority => 5

File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]

Parameters differences:

--- File[/etc/rsyslog.d/50-udp-json-logback-compat.conf].orig
+++ File[/etc/rsyslog.d/50-udp-json-logback-compat.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/50-udp-json-logback-compat.conf.orig
+++ /etc/rsyslog.d/50-udp-json-logback-compat.conf
@@ -0,0 +1,43 @@
+# Provide a UDP input to accept JSON payloads on localhost and forward them to logstash via Kakfa.
+
+module(load="imudp")
+module(load="mmjsonparse")
+module(load="omkafka")
+
+template(name="template_udp_json_logback_compat" type="list") {
+  property(name="$!all-json")
+}
+
+# Use the parsed json "level" field value as the kafka topic suffix
+template(name="udp_json_logback_compat_topic" type="string" string="logback-%!level:::lowercase%")
+
+# Use a separate (in memory) queue to limit message processing to this ruleset only.
+ruleset(name="ruleset_udp_json_logback_compat" queue.type="LinkedList") {
+
+  action(type="mmjsonparse" name="mmjsonparse_udp_json_logback_compat" cookie="" useRawMsg="on")
+
+  if $parsesuccess == "OK" then {
+    action(type="omkafka"
+           broker=["kafka-logging1001.eqiad.wmnet:9093","kafka-logging1002.eqiad.wmnet:9093","kafka-logging1003.eqiad.wmnet:9093","kafka-logging1004.eqiad.wmnet:9093","kafka-logging1005.eqiad.wmnet:9093"]
+           topic="udp_json_logback_compat_topic"
+           dynatopic="on"
+           dynatopic.cachesize="1000"
+           partitions.auto="on"
+           template="template_udp_json_logback_compat"
+           queue.type="LinkedList" queue.size="10000" queue.filename="udp_json_logback_compat"
+           queue.highWatermark="7000" queue.lowWatermark="6000"
+           queue.checkpointInterval="5"
+           queue.maxDiskSpace="40960000"
+           confParam=[ "security.protocol=ssl",
+                       "ssl.ca.location=/etc/ssl/certs/wmf-ca-certificates.crt",
+                       "compression.codec=snappy",
+                       "socket.timeout.ms=60000",
+                       "socket.keepalive.enable=true",
+                       "queue.buffering.max.ms=50",
+                       "batch.num.messages=1000" ]
+    )
+  }
+
+}
+
+input(type="imudp" port="11514" address="localhost" ruleset="ruleset_udp_json_logback_compat")

Monitoring::Service[opensearch_shards_9200]

Parameters differences:

--- Monitoring::Service[opensearch_shards_9200].orig
+++ Monitoring::Service[opensearch_shards_9200]

+    migration_task => T371083
+    check_interval => 1
+    contact_group  => admins,team-discovery
+    retries        => 3
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_shards_9200!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch health check for shards on 9200

Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter-9200].orig
+++ Service[prometheus-elasticsearch-exporter-9200]

+    ensure => running
+    enable => True

Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    mode     => 0444
+    require  => File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]
+    ensure   => present
+    priority => 40

Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3].orig
+++ Prometheus::Alert::Rule[check_opensearch_unassigned_shard_check_9200_54ca99505516d5452912b573aa7d35d3]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch unassigned shard check - 9200
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_unassigned_shard_check___9200
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="54ca99505516d5452912b573aa7d35d3",check_name="check_opensearch_unassigned_shard_check_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 720m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_unassigned_shard_check_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch unassigned shard check - 9200

Concat_fragment[main contacts]

Content differences:

--- main contacts.orig
+++ main contacts
@@ -1,3 +1,3 @@
 ---
-role::cirrus::test:
+role::cirrus::relforge:
 - Search Platform

File[/etc/ferm/conf.d/10_opensearch-https-9443]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch-https-9443].orig
+++ File[/etc/ferm/conf.d/10_opensearch-https-9443]

+    require => File[/etc/ferm/conf.d]
+    tag     => ferm
+    group   => root
+    notify  => Service[ferm]
+    owner   => root
+    ensure  => present
+    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_opensearch-https-9443.orig
+++ /etc/ferm/conf.d/10_opensearch-https-9443
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9443, (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS)));
+
+

Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]

Parameters differences:

--- Exec[symlink /etc/opensearch/relforge-eqiad/sudachi].orig
+++ Exec[symlink /etc/opensearch/relforge-eqiad/sudachi]

+    onlyif  => /usr/bin/test -d '/usr/share/opensearch/config/sudachi'
+    creates => /etc/opensearch/relforge-eqiad/sudachi
+    command => /usr/bin/ln -s '/usr/share/opensearch/config/sudachi' '/etc/opensearch/relforge-eqiad/sudachi'

File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for prometheus-wmf-elasticsearch-exporter-9200.service (prometheus-wmf-elasticsearch-exporter-9200)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/prometheus-wmf-elasticsearch-exporter-9200.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Prometheus WMF Elasticsearch Exporter
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/local/bin/prometheus-wmf-elasticsearch-exporter \
+    --listen :9120 \
+    --port 9200 \
+    --indices 
+
+[Install]
+WantedBy=multi-user.target

Systemd::Override[opensearch_2@]

Parameters differences:

--- Systemd::Override[opensearch_2@].orig
+++ Systemd::Override[opensearch_2@]

+    unit    => opensearch_2@
+    ensure  => present
+    restart => False

Class[Profile::Apt]

Parameters differences:

--- Class[Profile::Apt].orig
+++ Class[Profile::Apt]

@@
-    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]']
+    before => ['Package[puppet]', 'Package[facter]', 'Package[augeas-tools]', 'Package[virt-what]', 'Package[puppet-module-puppetlabs-augeas-core]', 'Package[python3-prometheus-client]', 'Package[python3-yaml]', 'Package[ruby-net-ssh]', 'Package[openssl]', 'Package[ssl-cert]', 'Package[ca-certificates]', 'Package[wmf-certificates]', 'Package[ntp]', 'Package[systemd-timesyncd]', 'Package[exim4-config]', 'Package[exim4-daemon-light]', 'Package[logrotate]', 'Package[prometheus-node-exporter]', 'Package[bsdutils]', 'Package[smartmontools]', 'Package[rsyslog]', 'Package[rsyslog-openssl]', 'Package[cadvisor]', 'Package[prometheus-ethtool-exporter]', 'Package[acct]', 'Package[byobu]', 'Package[colordiff]', 'Package[curl]', 'Package[debian-goodies]', 'Package[ethtool]', 'Package[gdb]', 'Package[gdisk]', 'Package[git]', 'Package[htop]', 'Package[httpry]', 'Package[iotop]', 'Package[iperf]', 'Package[jq]', 'Package[libtemplate-perl]', 'Package[lldpd]', 'Package[lshw]', 'Package[molly-guard]', 'Package[moreutils]', 'Package[net-tools]', 'Package[numactl]', 'Package[ncdu]', 'Package[ngrep]', 'Package[pigz]', 'Package[psmisc]', 'Package[pv]', 'Package[python3]', 'Package[screen]', 'Package[strace]', 'Package[sysstat]', 'Package[tcpdump]', 'Package[tmux]', 'Package[tree]', 'Package[vim]', 'Package[vim-addon-manager]', 'Package[vim-scripts]', 'Package[wipe]', 'Package[xfsprogs]', 'Package[zsh]', 'Package[icdiff]', 'Package[linux-perf]', 'Package[bsd-mailx]', 'Package[ack]', 'Package[netcat-openbsd]', 'Package[tshark]', 'Package[fzf]', 'Package[ripgrep]', 'Package[fd-find]', 'Package[kitty-terminfo]', 'Package[mtr-tiny]', 'Package[bat]', 'Package[efibootmgr]', 'Package[bind9-dnsutils]', 'Package[tzdata]', 'Package[python3-wmflib]', 'Package[ruby-sorted-set]', 'Package[btop]', 'Package[dstat]', 'Package[apport]', 'Package[command-not-found]', 'Package[command-not-found-data]', 'Package[ecryptfs-utils]', 'Package[mlocate]', 'Package[os-prober]', 'Package[python3-apport]', 'Package[wpasupplicant]', 'Package[apt-listchanges]', 'Package[isc-dhcp-client]', 'Package[rasdaemon]', 'Package[libicu67]', 'Package[libwsutil12]', 'Package[libwireshark14]', 'Package[libopencsd0]', 'Package[libwiretap11]', 'Package[ruby2.7]', 'Package[python3.9-minimal]', 'Package[python3.9]', 'Package[perl-modules-5.32]', 'Package[libpython3.9]', 'Package[libperl5.32]', 'Package[libpython3.9-minimal]', 'Package[libpython3.9-stdlib]', 'Package[libidn11]', 'Package[libldap-2.4-2]', 'Package[liburing1]', 'Package[libwebp6]', 'Package[libcbor0]', 'Package[libusb-0.1-4]', 'Package[telnet]', 'Package[libruby2.7]', 'Package[libdns-export1110]', 'Package[libisc-export1105]', 'Package[libbpf0]', 'Package[openssh-client]', 'Package[openssh-server]', 'Package[debdeploy-client]', 'Package[python3-dateutil]', 'Package[sudo]', 'Package[golang-cfssl]', 'Package[debmonitor-client]', 'Package[nagios-nrpe-server]', 'Package[monitoring-plugins]', 'Package[monitoring-plugins-basic]', 'Package[monitoring-plugins-standard]', 'Package[liburiparser1]', 'Package[python3-attr]', 'Package[iucode-tool]', 'Package[freeipmi-tools]', 'Package[freeipmi-ipmiseld]', 'Package[rsyslog-kafka]', 'Package[emacs-nox]', 'Package[prometheus-ipmi-exporter]', 'Package[libnet-dns-perl]', 'Package[iptables]', 'Package[ferm]', 'Package[ulogd2]', 'Package[conntrack]', 'Package[edac-utils]', 'Package[opensearch]', 'Package[python3-requests]', 'Package[wmf-opensearch-search-plugins]', 'Package[opensearch-madvise]', 'Package[logstash-oss]', 'Package[logstash-old-name]', 'Package[prometheus-rsyslog-exporter]', 'Package[initramfs-tools]', 'Package[python3-click]', 'Package[python3-box]', 'Package[confd]', 'Package[python3-toml]', 'Package[openjdk-17-jdk]', 'Package[prometheus-elasticsearch-exporter]', 'Package[nginx-light]', 'Package[nginx-common]']

Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]']
+    enable   => False
+    provider => systemd

File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    splay              => 300
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.service

Nrpe::Monitor_service[opensearch_shards_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shards_9200].orig
+++ Nrpe::Monitor_service[opensearch_shards_9200]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4
+    migration_task              => T371083
+    check_interval              => 1
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration
+    contact_group               => admins,team-discovery
+    retries                     => 3
+    ensure                      => present
+    retry_interval              => 1
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch health check for shards on 9200

File[/etc/opensearch/relforge-eqiad]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad].orig
+++ File[/etc/opensearch/relforge-eqiad]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0755

File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch_inter_node_9500].orig
+++ File[/etc/ferm/conf.d/10_opensearch_inter_node_9500]

+    require => File[/etc/ferm/conf.d]
+    tag     => ferm
+    group   => root
+    notify  => Service[ferm]
+    owner   => root
+    ensure  => present
+    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_opensearch_inter_node_9500.orig
+++ /etc/ferm/conf.d/10_opensearch_inter_node_9500
@@ -0,0 +1,8 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9500, (10.64.164.14 10.64.32.100 10.64.48.120 2620:0:861:103:10:64:32:100 2620:0:861:107:10:64:48:120 2620:0:861:12c:10:64:164:14));
+
+
+
+&NO_TRACK(tcp, 9500);

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=720min
+OnActiveSec=1s
+RandomizedDelaySec=43200
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.service (nrpe2nodexp-opensearch_shard_size_check_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shard_size_check_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shard_size_check_9400]=/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4

Monitoring::Service[opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Monitoring::Service[opensearch_unassigned_shard_check_9400].orig
+++ Monitoring::Service[opensearch_unassigned_shard_check_9400]

+    migration_task => T371083
+    check_interval => 720
+    contact_group  => admins,team-discovery
+    retries        => 1
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration
+    ensure         => present
+    passive        => False
+    retry_interval => 120
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_unassigned_shard_check_9400!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch unassigned shard check - 9400

Monitoring::Service[opensearch_shard_size_check_9400]

Parameters differences:

--- Monitoring::Service[opensearch_shard_size_check_9400].orig
+++ Monitoring::Service[opensearch_shard_size_check_9400]

+    migration_task => T371083
+    check_interval => 1440
+    contact_group  => admins,team-discovery
+    retries        => 3
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    ensure         => present
+    passive        => False
+    retry_interval => 180
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_shard_size_check_9400!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch shard size check - 9400

Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9400
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00'}
+    private_tmp               => False
+    description               => Auto restart job: prometheus-elasticsearch-exporter-9400

Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9400]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 1440
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch shard size check - 9400
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    retry_interval         => 180
+    check_command          => nrpe_check!check_opensearch_shard_size_check_9400!10
+    max_check_attempts     => 3
+    host_name              => relforge1008

File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-disable-readahead.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-opensearch-disable-readahead.conf.orig
+++ /etc/rsyslog.d/40-opensearch-disable-readahead.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-disable-readahead" then {
+    action(
+        type="omfile" file="/var/log/opensearch-disable-readahead/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Timer::Job[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Timer::Job[push_cross_cluster_settings_9400].orig
+++ Systemd::Timer::Job[push_cross_cluster_settings_9400]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /bin/bash /usr/local/bin/set-cross-cluster-seeds_9400.sh
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => True
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnUnitActiveSec', 'interval': '15min'}
+    private_tmp               => False
+    description               => Auto set remote cluster seeds for relforge-eqiad-small-alpha

File[/etc/update-motd.d/05-cirrus--test]

Parameters differences:

--- File[/etc/update-motd.d/05-cirrus--test].orig
+++ File[/etc/update-motd.d/05-cirrus--test]

-    owner  => root
-    ensure => present
-    group  => root
-    mode   => 0555

Content differences:

--- /etc/update-motd.d/05-cirrus--test.orig
+++ /etc/update-motd.d/05-cirrus--test
@@ -1,2 +0,0 @@
-#!/bin/sh
-printf "%s\n" "relforge1008 is a Unpuppetised system for testing (cirrus::test)"

File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ File[/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Systemd::Service[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Service[push_cross_cluster_settings_9400].orig
+++ Systemd::Service[push_cross_cluster_settings_9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[push_cross_cluster_settings_9400.service]
+    unit_type                => timer

Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    ensure => present

Systemd::Service[clean_up_dlq]

Parameters differences:

--- Systemd::Service[clean_up_dlq].orig
+++ Systemd::Service[clean_up_dlq]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[clean_up_dlq.service]
+    unit_type                => timer

Motd::Script[cirrus::relforge]

Parameters differences:

--- Motd::Script[cirrus::relforge].orig
+++ Motd::Script[cirrus::relforge]

+    ensure   => present
+    priority => 5

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+    override_filename => puppet-override.conf

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.service (opensearch-disable-readahead-relforge-eqiad.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Service[nginx]

Parameters differences:

--- Service[nginx].orig
+++ Service[nginx]

+    hasrestart => True
+    require    => Package[nginx-light]
+    ensure     => running
+    enable     => True

Concat::Fragment[main contacts]

Logrotate::Conf[push_cross_cluster_settings_9200]

Parameters differences:

--- Logrotate::Conf[push_cross_cluster_settings_9200].orig
+++ Logrotate::Conf[push_cross_cluster_settings_9200]

+    ensure => present

File[/etc/sysctl.d/opensearch.conf]

Parameters differences:

--- File[/etc/sysctl.d/opensearch.conf].orig
+++ File[/etc/sysctl.d/opensearch.conf]

+    notify => Exec[update_sysctl]
+    owner  => root
+    ensure => present
+    group  => root

Content differences:

--- /etc/sysctl.d/opensearch.conf.orig
+++ /etc/sysctl.d/opensearch.conf
@@ -0,0 +1,2 @@
+# sysctl parameters managed by Puppet.
+vm.max_map_count = 1048576

Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]
+    unit_type                => timer

Rsyslog::Conf[push_cross_cluster_settings_9400]

Parameters differences:

--- Rsyslog::Conf[push_cross_cluster_settings_9400].orig
+++ Rsyslog::Conf[push_cross_cluster_settings_9400]

+    mode     => 0444
+    require  => File[/var/log/push_cross_cluster_settings_9400]
+    ensure   => present
+    priority => 40

File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-check-push-cross-cluster-settings-9400-status.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-check_push_cross_cluster_settings_9400_status" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer
+    override_filename => puppet-override.conf

Ferm::Service[opensearch-https-9243]

Parameters differences:

--- Ferm::Service[opensearch-https-9243].orig
+++ Ferm::Service[opensearch-https-9243]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => False
+    port                => 9243
+    desc                => 
+    srange              => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    proto               => tcp

File[/etc/default/logstash]

Parameters differences:

--- File[/etc/default/logstash].orig
+++ File[/etc/default/logstash]

+    require => Package[logstash-oss]
+    group   => root
+    owner   => root
+    notify  => Service[logstash]
+    mode    => 0444

File[/etc/update-motd.d/05-cirrus--relforge]

Parameters differences:

--- File[/etc/update-motd.d/05-cirrus--relforge].orig
+++ File[/etc/update-motd.d/05-cirrus--relforge]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0555

Content differences:

--- /etc/update-motd.d/05-cirrus--relforge.orig
+++ /etc/update-motd.d/05-cirrus--relforge
@@ -0,0 +1,2 @@
+#!/bin/sh
+printf "%s\n" "relforge1008 is a Elasticsearch relforge (cirrus::relforge)"

Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status].orig
+++ Nrpe::Monitor_service[check_push_cross_cluster_settings_9400_status]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400
+    migration_task              => T407130
+    check_interval              => 10
+    contact_group               => admins
+    retries                     => 2
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    ensure                      => present
+    retry_interval              => 1
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => Check unit status of push_cross_cluster_settings_9400

Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad-small-alpha /srv/opensearch
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes

Systemd::Unit[push_cross_cluster_settings_9400.timer]

Parameters differences:

--- Systemd::Unit[push_cross_cluster_settings_9400.timer].orig
+++ Systemd::Unit[push_cross_cluster_settings_9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => push_cross_cluster_settings_9400.timer
+    override_filename => puppet-override.conf

Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]

Parameters differences:

--- Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha].orig
+++ Opensearch::Log::Hot_threads_cluster[relforge-eqiad-small-alpha]

+    cluster_name => relforge-eqiad-small-alpha
+    http_port    => 9400

Nrpe::Check[check_disk_space]

Parameters differences:

--- Nrpe::Check[check_disk_space].orig
+++ Nrpe::Check[check_disk_space]

@@
-    command => /usr/lib/nagios/plugins/check_disk -w 6% -c 3% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" -i "/srv/nvme[0-9]n[0-9]p[0-9]" --exclude-type=fuse --exclude-type=fuse.fuse_dfs --exclude-type=tracefs
+    command => /usr/lib/nagios/plugins/check_disk -w 15% -c 10% -W 6% -K 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs

File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9400.timer].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9400.timer.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9400.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of push_cross_cluster_settings_9400.service
+
+[Timer]
+Unit=push_cross_cluster_settings_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=15min
+OnActiveSec=1s
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => present

File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shards_9200.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ Service[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]']
+    enable   => False
+    provider => systemd

Exec[systemd reload for nginx systemd fragments]

Parameters differences:

--- Exec[systemd reload for nginx systemd fragments].orig
+++ Exec[systemd reload for nginx systemd fragments]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True
+    subscribe   => ['File[/etc/systemd/system/nginx.service.d/numa.conf]', 'File[/etc/systemd/system/nginx.service.d/security.conf]']

Tlsproxy::Localssl[relforge-eqiad-small-alpha]

Parameters differences:

--- Tlsproxy::Localssl[relforge-eqiad-small-alpha].orig
+++ Tlsproxy::Localssl[relforge-eqiad-small-alpha]

+    skip_private       => False
+    only_get_requests  => False
+    cfssl_paths        => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    acme_certname      => relforge-eqiad-small-alpha
+    keepalive_timeout  => 60
+    acme_chief         => False
+    server_name        => relforge1008.eqiad.wmnet
+    enable_http2       => False
+    require            => ['Class[Profile::Tlsproxy::Instance]']
+    read_timeout       => 180
+    keepalive_requests => 100
+    server_aliases     => ['relforge.svc.eqiad.wmnet']
+    tls_ports          => [9443]
+    upstream_ports     => [9400]
+    upstream_ip        => 10.64.32.100
+    default_server     => True
+    access_log         => False

Monitoring::Service[check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Monitoring::Service[check_push_cross_cluster_settings_9400_status].orig
+++ Monitoring::Service[check_push_cross_cluster_settings_9400_status]

+    migration_task => T407130
+    check_interval => 10
+    contact_group  => admins
+    retries        => 2
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_check_push_cross_cluster_settings_9400_status!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => Check unit status of push_cross_cluster_settings_9400

Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad.timer (opensearch-disable-readahead-relforge-eqiad.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[opensearch-disable-readahead-relforge-eqiad.timer]']

File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha].orig
+++ File[/etc/nginx/sites-enabled/relforge-eqiad-small-alpha]

+    group  => root
+    before => ['Service[nginx]']
+    owner  => root
+    ensure => link
+    target => /etc/nginx/sites-available/relforge-eqiad-small-alpha

Class[Opensearch::Packages]

Parameters differences:

--- Class[Opensearch::Packages].orig
+++ Class[Opensearch::Packages]

+    version               => 2.19.5
+    send_logs_to_logstash => True

Service[opensearch_2@relforge-eqiad]

Parameters differences:

--- Service[opensearch_2@relforge-eqiad].orig
+++ Service[opensearch_2@relforge-eqiad]

+    tag      => opensearch_services
+    provider => systemd
+    require  => ['Systemd::Unit[opensearch_2@.service]', 'File[/etc/opensearch/relforge-eqiad/opensearch.yml]', 'File[/etc/opensearch/relforge-eqiad/logging.yml]', 'File[/etc/opensearch/relforge-eqiad/log4j2.properties]', 'File[/etc/opensearch/relforge-eqiad/jvm.options]', 'File[/srv/opensearch/relforge-eqiad]', 'Systemd::Tmpfile[opensearch-relforge-eqiad]']
+    ensure   => running
+    enable   => True

Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Timer[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* 02:12:00'}]
+    unit_name          => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service

Exec[update_java_alternatives_17]

Parameters differences:

--- Exec[update_java_alternatives_17].orig
+++ Exec[update_java_alternatives_17]

+    command => /usr/sbin/update-java-alternatives -s /usr/lib/jvm/java-1.17.0-openjdk-amd64
+    unless  => /usr/bin/update-alternatives --query java | /bin/grep 'Value: /usr/lib/jvm/java-17-openjdk-amd64'

Sysctl::Parameters[TCP Fast Open]

Parameters differences:

--- Sysctl::Parameters[TCP Fast Open].orig
+++ Sysctl::Parameters[TCP Fast Open]

+    ensure             => present
+    values             => {'net.ipv4.tcp_fastopen': 3}
+    priority           => 70
+    no_priority_prefix => False

Ferm::Service[opensearch-https-9443]

Parameters differences:

--- Ferm::Service[opensearch-https-9443].orig
+++ Ferm::Service[opensearch-https-9443]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => False
+    port                => 9443
+    desc                => 
+    srange              => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    proto               => tcp

File[/usr/local/bin/logstash-config-test]

Parameters differences:

--- File[/usr/local/bin/logstash-config-test].orig
+++ File[/usr/local/bin/logstash-config-test]

+    require => Package[logstash-oss]
+    group   => root
+    source  => puppet:///modules/logstash/logstash-config-test
+    owner   => root
+    mode    => 0555

Motd::Message[cirrus::relforge]

Parameters differences:

--- Motd::Message[cirrus::relforge].orig
+++ Motd::Message[cirrus::relforge]

+    message  => relforge1008 is a Elasticsearch relforge (cirrus::relforge)
+    ensure   => present
+    priority => 5

Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Service[prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => running
+    enable => True

File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/java-security.policy]

+    group  => opensearch
+    source => puppet:///modules/profile/opensearch/java-security.policy
+    owner  => opensearch
+    ensure => file
+    mode   => 0440

File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9400]

+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    ensure  => present
+    line    => prometheus-wmf-elasticsearch-exporter-9400
+    path    => /etc/debdeploy-client/autorestarts.conf

Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Nrpe::Monitor_service[opensearch_shard_size_check_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shard_size_check_9200].orig
+++ Nrpe::Monitor_service[opensearch_shard_size_check_9200]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9200 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    migration_task              => T371083
+    check_interval              => 1440
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    contact_group               => admins,team-discovery
+    retries                     => 3
+    ensure                      => present
+    retry_interval              => 180
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch shard size check - 9200

Systemd::Unit[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Unit[prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Unit[prometheus-elasticsearch-exporter-9400]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => prometheus-elasticsearch-exporter-9400
+    override_filename => puppet-override.conf

Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]

Parameters differences:

--- Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet].orig
+++ Cfssl::Cert[discovery2026__relforge1008_eqiad_wmnet]

+    names           => []
+    common_name     => relforge1008.eqiad.wmnet
+    auto_renew      => True
+    environment     => ['GODEBUG=x509ignoreCN=0']
+    owner           => root
+    ensure          => present
+    label           => discovery2026
+    hosts           => ['relforge.svc.eqiad.wmnet']
+    group           => root
+    key             => {'algo': 'ecdsa', 'size': 256}
+    before_services => []
+    notify_services => []
+    renew_seconds   => 952200
+    provide_chain   => True
+    mode            => 0740

File[/lib/systemd/system/opensearch-hot-threads-log.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-hot-threads-log.timer].orig
+++ File[/lib/systemd/system/opensearch-hot-threads-log.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-hot-threads-log.timer (opensearch-hot-threads-log.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-hot-threads-log.timer.orig
+++ /lib/systemd/system/opensearch-hot-threads-log.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of opensearch-hot-threads-log.service
+
+[Timer]
+Unit=opensearch-hot-threads-log.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=*-*-* *:00/5:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shard-size-check-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shard_size_check_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Class[Profile::Rsyslog::Udp_json_logback_compat]

Parameters differences:

--- Class[Profile::Rsyslog::Udp_json_logback_compat].orig
+++ Class[Profile::Rsyslog::Udp_json_logback_compat]

+    logging_kafka_brokers => ['kafka-logging1001.eqiad.wmnet:9093', 'kafka-logging1002.eqiad.wmnet:9093', 'kafka-logging1003.eqiad.wmnet:9093', 'kafka-logging1004.eqiad.wmnet:9093', 'kafka-logging1005.eqiad.wmnet:9093']
+    port                  => 11514
+    queue_enabled_sites   => ['ulsfo', 'esams', 'eqsin', 'eqiad', 'codfw', 'drmrs', 'magru']

Logrotate::Rule[opensearch]

Parameters differences:

--- Logrotate::Rule[opensearch].orig
+++ Logrotate::Rule[opensearch]

+    missing_ok     => True
+    copy_truncate  => True
+    compress       => True
+    ensure         => present
+    date_yesterday => False
+    file_glob      => /var/log/opensearch/*.log
+    post_rotate    => []
+    no_create      => False
+    rotate         => 7
+    date_ext       => False
+    not_if_empty   => True
+    frequency      => daily

File[/var/log/opensearch]

Parameters differences:

--- File[/var/log/opensearch].orig
+++ File[/var/log/opensearch]

+    require => Package[opensearch]
+    group   => opensearch
+    owner   => opensearch
+    ensure  => directory
+    mode    => 0755

File[/etc/elasticsearch]

Parameters differences:

--- File[/etc/elasticsearch].orig
+++ File[/etc/elasticsearch]

+    require => File[/etc/opensearch/instances]
+    group   => root
+    owner   => root
+    ensure  => link
+    target  => /etc/opensearch

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9400.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shards_9400" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

File[/etc/logrotate.d/push_cross_cluster_settings_9400]

Parameters differences:

--- File[/etc/logrotate.d/push_cross_cluster_settings_9400].orig
+++ File[/etc/logrotate.d/push_cross_cluster_settings_9400]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/push_cross_cluster_settings_9400.orig
+++ /etc/logrotate.d/push_cross_cluster_settings_9400
@@ -0,0 +1,12 @@
+# logrotate(8) config for push_cross_cluster_settings_9400
+
+/var/log/push_cross_cluster_settings_9400/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Class[Profile::Opensearch::Cirrus::Server]

Parameters differences:

--- Class[Profile::Opensearch::Cirrus::Server].orig
+++ Class[Profile::Opensearch::Cirrus::Server]

+    cluster                         => relforge
+    expose_http                     => False
+    version                         => 2.19.5
+    base_data_dir                   => /srv/opensearch
+    enable_performance_cpu_governor => False
+    ssl_provider                    => cfssl
+    storage_device                  => md1
+    enable_remote_search            => False
+    ferm_srange                     => (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS))
+    ferm_ro_srange                  => 
+    certificate_domains             => ['relforge.svc.eqiad.wmnet']

File[/srv/opensearch/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/srv/opensearch/relforge-eqiad-small-alpha].orig
+++ File[/srv/opensearch/relforge-eqiad-small-alpha]

+    owner  => opensearch
+    ensure => directory
+    group  => opensearch
+    mode   => 0755

Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Logrotate::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

+    ensure => present

File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]

Parameters differences:

--- File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf].orig
+++ File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf.orig
+++ /etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf
@@ -0,0 +1,24 @@
+# The OpenSearch packages come with a lot of plugins
+# we don't want or need. This override helps us
+# expose only the plugins we explicitly set via
+# $plugins_mandatory var
+[Service]
+# Give OpenSearch its own mount namespace
+PrivateMounts=yes
+
+# Replace plugins dir with an empty tmpfs
+TemporaryFileSystem=/usr/share/opensearch/plugins:rw
+
+BindPaths=
+BindPaths=/usr/share/opensearch/plugins/analysis-hebrew
+BindPaths=/usr/share/opensearch/plugins/analysis-icu
+BindPaths=/usr/share/opensearch/plugins/analysis-smartcn
+BindPaths=/usr/share/opensearch/plugins/analysis-stempel
+BindPaths=/usr/share/opensearch/plugins/analysis-ukrainian
+BindPaths=/usr/share/opensearch/plugins/cirrus-highlighter
+BindPaths=/usr/share/opensearch/plugins/opensearch-analysis-stconvert
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-esperanto
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-serbian
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-slovak
+BindPaths=/usr/share/opensearch/plugins/ltr

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]']

Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer].orig
+++ Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Sysctl::Conffile[opensearch]

Parameters differences:

--- Sysctl::Conffile[opensearch].orig
+++ Sysctl::Conffile[opensearch]

+    ensure             => present
+    priority           => 70
+    no_priority_prefix => True

Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/opensearch_hot_threads.d]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d].orig
+++ File[/etc/opensearch_hot_threads.d]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0755

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
+    override_filename => puppet-override.conf

Systemd::Timer[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Timer[opensearch-hot-threads-log].orig
+++ Systemd::Timer[opensearch-hot-threads-log]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': '*-*-* *:00/5:00'}]
+    unit_name          => opensearch-hot-threads-log.service

Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200].orig
+++ Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9200]

+    ensure => present

File[/var/log/opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead-relforge-eqiad].orig
+++ File[/var/log/opensearch-disable-readahead-relforge-eqiad]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => directory
+    force  => True
+    mode   => 0755

Systemd::Unit[opensearch_2@.service]

Parameters differences:

--- Systemd::Unit[opensearch_2@.service].orig
+++ Systemd::Unit[opensearch_2@.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch_2@.service
+    override_filename => puppet-override.conf

Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_unassigned_shard_check_9400.timer
+    override_filename => puppet-override.conf

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-elasticsearch-exporter-9400
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9400

Rsyslog::Conf[push_cross_cluster_settings_9200]

Parameters differences:

--- Rsyslog::Conf[push_cross_cluster_settings_9200].orig
+++ Rsyslog::Conf[push_cross_cluster_settings_9200]

+    mode     => 0444
+    require  => File[/var/log/push_cross_cluster_settings_9200]
+    ensure   => present
+    priority => 40

File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/opensearch.keystore].orig
+++ File[/etc/opensearch/relforge-eqiad/opensearch.keystore]

+    owner => root
+    group => opensearch
+    mode  => 0640

Systemd::Timer::Job[clean_up_dlq]

Parameters differences:

--- Systemd::Timer::Job[clean_up_dlq].orig
+++ Systemd::Timer::Job[clean_up_dlq]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/cleanup-dlq
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': '1h'}
+    private_tmp               => False
+    description               => Clean up dead letter queue and restart logstash

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.csr]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0440

File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shards_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shards_9400]=/usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4

Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Systemd::Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]
+    unit_type                => timer

File[/etc/default/prometheus-elasticsearch-exporter]

Parameters differences:

--- File[/etc/default/prometheus-elasticsearch-exporter].orig
+++ File[/etc/default/prometheus-elasticsearch-exporter]

+    owner  => root
+    ensure => absent
+    group  => root

File[/etc/nginx/sites-enabled]

Parameters differences:

--- File[/etc/nginx/sites-enabled].orig
+++ File[/etc/nginx/sites-enabled]

+    group   => root
+    before  => ['Service[nginx]']
+    recurse => True
+    owner   => root
+    ensure  => directory
+    force   => True
+    purge   => True

Systemd::Timer::Job[opensearch-hot-threads-log]

Parameters differences:

--- Systemd::Timer::Job[opensearch-hot-threads-log].orig
+++ Systemd::Timer::Job[opensearch-hot-threads-log]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => opensearch
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    require                   => ['Package[opensearch]', 'File[/usr/local/bin/opensearch_hot_threads_logger.py]']
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/opensearch_hot_threads_logger.py
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': '*-*-* *:00/5:00'}
+    private_tmp               => False
+    description               => Archive exception logs of hot opensearch threads

Package[openjdk-17-jdk]

Parameters differences:

--- Package[openjdk-17-jdk].orig
+++ Package[openjdk-17-jdk]

+    ensure   => installed
+    provider => apt

Monitoring::Service[opensearch_shards_9400]

Parameters differences:

--- Monitoring::Service[opensearch_shards_9400].orig
+++ Monitoring::Service[opensearch_shards_9400]

+    migration_task => T371083
+    check_interval => 1
+    contact_group  => admins,team-discovery
+    retries        => 3
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_shards_9400!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch health check for shards on 9400

File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service].orig
+++ File[/lib/systemd/system/prometheus-elasticsearch-exporter-9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9400.service (prometheus-elasticsearch-exporter-9400)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/prometheus-elasticsearch-exporter-9400.service.orig
+++ /lib/systemd/system/prometheus-elasticsearch-exporter-9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus exporter for Elasticsearch
+Documentation=https://prometheus.io/docs/introduction/overview/
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/bin/prometheus-elasticsearch-exporter --es.uri=http://localhost:9400 --web.listen-address=:9109 
+
+[Install]
+WantedBy=multi-user.target

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.service (opensearch-disable-readahead-relforge-eqiad-small-alpha.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad-small-alpha /srv/opensearch

File[/var/log/clean_up_dlq]

Parameters differences:

--- File[/var/log/clean_up_dlq].orig
+++ File[/var/log/clean_up_dlq]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => absent
+    force  => True
+    mode   => 0755

Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shards_9200.timer
+    override_filename => puppet-override.conf

Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
+    override_filename => puppet-override.conf

File[/lib/systemd/system/clean_up_dlq.service]

Parameters differences:

--- File[/lib/systemd/system/clean_up_dlq.service].orig
+++ File[/lib/systemd/system/clean_up_dlq.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/clean_up_dlq.service.orig
+++ /lib/systemd/system/clean_up_dlq.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Clean up dead letter queue and restart logstash
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/cleanup-dlq

Logrotate::Conf[opensearch-disable-readahead]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead].orig
+++ Logrotate::Conf[opensearch-disable-readahead]

+    ensure => absent

File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9200.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Systemd::Syslog[clean_up_dlq]

Parameters differences:

--- Systemd::Syslog[clean_up_dlq].orig
+++ Systemd::Syslog[clean_up_dlq]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => absent
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Java::Cacert[wmf:puppetca.pem]

Parameters differences:

--- Java::Cacert[wmf:puppetca.pem].orig
+++ Java::Cacert[wmf:puppetca.pem]

+    require   => Alternatives::Java[17]
+    group     => root
+    storepass => changeit
+    owner     => root
+    ensure    => present
+    path      => /etc/ssl/certs/Puppet_Internal_CA.pem

File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]

Parameters differences:

--- File[/usr/local/bin/set-cross-cluster-seeds_9400.sh].orig
+++ File[/usr/local/bin/set-cross-cluster-seeds_9400.sh]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0700

Content differences:

--- /usr/local/bin/set-cross-cluster-seeds_9400.sh.orig
+++ /usr/local/bin/set-cross-cluster-seeds_9400.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# set-cross-cluster-seeds.sh. Given a json file with opensearch remote seeds,
+# update OpenSearch cluster with correct settings.
+# Author: Brian King
+
+set -eo pipefail
+
+data="/etc/opensearch/relforge-eqiad-small-alpha/cirrus_check_settings.json"
+
+logger "updating cross cluster seed settings via ${data}"
+
+curl -XPUT \
+  -H 'content-type: application/json' \
+  --data "@${data}" \
+  http://localhost:9400/_cluster/settings

Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_shards_9200.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_shards_9200]=/usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9200 --shards-inactive '>=0.15' --timeout 4

File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]

Parameters differences:

--- File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter].orig
+++ File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]

+    group  => root
+    source => puppet:///modules/prometheus/usr/local/bin/prometheus-wmf-elasticsearch-exporter.py
+    owner  => root
+    ensure => present
+    mode   => 0555

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shard_size_check_9400.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Class[Opensearch]

Parameters differences:

--- Class[Opensearch].orig
+++ Class[Opensearch]

+    version                 => 2.19.5
+    base_data_dir           => /srv/opensearch
+    enable_curator          => False
+    native_lib_path         => /usr/share/opensearch/plugins/opensearch-knn/lib
+    before                  => ['File[/usr/share/opensearch/plugins]']
+    instances               => {'relforge-eqiad': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad', 'short_cluster_name': 'chi', 'heap_memory': '30G', 'http_port': 9200, 'tls_port': 9243, 'transport_tcp_port': 9300, 'certificate_name': 'relforge.svc.eqiad.wmnet'}, 'relforge-eqiad-small-alpha': {'awareness_attributes': 'row', 'recover_after_nodes': 1, 'recover_after_time': '1m', 'search_shard_count_limit': 1000, 'reindex_remote_whitelist': 'search.svc.eqiad.wmnet:9243,search.svc.codfw.wmnet:9243', 'bulk_thread_pool_executors': 32, 'bulk_thread_pool_capacity': 1000, 'ltr_cache_size': '100mb', 'auto_create_index': True, 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'filter_cache_size': '20%', 'script_max_compilations_per_minute': 10000, 'send_logs_to_logstash': True, 'disable_security_plugin': True, 'unicast_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'cluster_hosts': ['relforge1008.eqiad.wmnet', 'relforge1009.eqiad.wmnet', 'relforge1010.eqiad.wmnet'], 'expected_nodes': 2, 'cluster_name': 'relforge-eqiad-small-alpha', 'short_cluster_name': 'omega', 'heap_memory': '4G', 'http_port': 9400, 'tls_port': 9443, 'transport_tcp_port': 9500, 'certificate_name': 'relforge.svc.eqiad.wmnet'}}
+    default_instance_params => {}
+    row                     => eqiad-row-c
+    plugins_mandatory       => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    rack                    => C4
+    logstash_host           => localhost
+    java_home               => /usr/lib/jvm/java-21-openjdk-amd64
+    logstash_logback_port   => 11514

Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]

Parameters differences:

--- Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi].orig
+++ Exec[symlink /etc/opensearch/relforge-eqiad-small-alpha/sudachi]

+    onlyif  => /usr/bin/test -d '/usr/share/opensearch/config/sudachi'
+    creates => /etc/opensearch/relforge-eqiad-small-alpha/sudachi
+    command => /usr/bin/ln -s '/usr/share/opensearch/config/sudachi' '/etc/opensearch/relforge-eqiad-small-alpha/sudachi'

Package[logstash-oss]

Parameters differences:

--- Package[logstash-oss].orig
+++ Package[logstash-oss]

+    require  => Package[openjdk-17-jdk]
+    ensure   => 1:7.16.3-1
+    provider => apt

File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list].orig
+++ File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]

+    owner  => root
+    ensure => absent
+    group  => root

Systemd::Mask[nginx.service]

Parameters differences:

--- Systemd::Mask[nginx.service].orig
+++ Systemd::Mask[nginx.service]

+    unit   => nginx.service
+    before => ['Package[nginx-light]']
+    unless => /usr/bin/dpkg -s nginx-light | /bin/grep -q '^Status: install ok installed$'

Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Rsyslog::Conf[opensearch-relforge-eqiad-gc-log-cleanup]

+    mode     => 0444
+    require  => File[/var/log/opensearch-relforge-eqiad-gc-log-cleanup]
+    ensure   => present
+    priority => 40

File[/etc/opensearch/relforge-eqiad/jvm.options]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/jvm.options].orig
+++ File[/etc/opensearch/relforge-eqiad/jvm.options]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad/jvm.options.orig
+++ /etc/opensearch/relforge-eqiad/jvm.options
@@ -0,0 +1,109 @@
+## JVM configuration
+
+################################################################
+## IMPORTANT: JVM heap size
+################################################################
+##
+## You should always set the min and max JVM heap
+## size to the same value. For example, to set
+## the heap to 4 GB, set:
+##
+## -Xms4g
+## -Xmx4g
+##
+## See https://opensearch.org/docs/opensearch/install/important-settings/
+## for more information
+##
+################################################################
+
+# Xms represents the initial size of total heap space
+# Xmx represents the maximum size of total heap space
+
+-Xms30G
+-Xmx30G
+
+################################################################
+## Numa Awareness
+################################################################
+-XX:+UseNUMA
+
+################################################################
+## Expert settings
+################################################################
+##
+## All settings below this section are considered
+## expert settings. Don't tamper with them unless
+## you understand what you are doing
+##
+################################################################
+
+
+## optimizations
+
+# disable calls to System#gc
+-XX:+DisableExplicitGC
+
+# pre-touch memory pages used by the JVM during initialization
+-XX:+AlwaysPreTouch
+
+## basic
+
+# force the server VM (remove on 32-bit client JVMs)
+-server
+
+# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
+-Xss1m
+
+# log4j 2
+-Dlog4j.skipJansi=true
+-Dlog4j2.formatMsgNoLookups=true
+
+# security policy, needed for OpenSearch to access non-default files/sockets/ports T395571
+
+-Djava.security.policy=/etc/opensearch/relforge-eqiad/java-security.policy
+
+## heap dumps
+
+# generate a heap dump when an allocation from the Java heap fails
+# heap dumps are created in the working directory of the JVM
+-XX:+HeapDumpOnOutOfMemoryError
+
+# specify an alternative path for heap dumps
+# ensure the directory exists and has sufficient space
+-XX:HeapDumpPath=/srv/opensearch/relforge-eqiad
+
+## GC logging
+
+#-XX:+PrintGCDetails
+#-XX:+PrintGCTimeStamps
+#-XX:+PrintGCDateStamps
+#-XX:+PrintClassHistogram
+#-XX:+PrintTenuringDistribution
+#-XX:+PrintGCApplicationStoppedTime
+
+# log GC status to a file with time stamps
+# ensure the directory exists
+#-Xloggc:${loggc}
+
+## Custom Wikimedia configuration
+
+# Defines a limit of proportion of time spent in GC before
+# an OutOfMemory error is thrown.  The goal is to bail and
+# restart the server if throughput is suffering due to
+# GC stalls.
+# TODO: Why 70? why not 60 or 80?
+-XX:GCTimeLimit=70
+
+# Defines minimum percentage of free space after a full GC
+# before an OutOfMemoryError is thrown. Typically a full GC
+# should leave > 50% of heap free. If we are down to 10%
+# something is wrong and bailing is a reasonable solution.
+-XX:GCHeapFreeLimit=10
+
+# Exit on OutOfMemoryError, since in most cases the JVM does not recover
+-XX:+ExitOnOutOfMemoryError
+
+# Custom WMF GC flags
+-Xlog:gc*:file=/var/log/opensearch/relforge-eqiad_jvm_gc.%p.log::filecount=10,filesize=20000
+-Xlog:gc+age=trace
+-Xlog:safepoint

File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_unassigned_shard_check_9400.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Systemd::Timer[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Timer[opensearch-disable-readahead].orig
+++ Systemd::Timer[opensearch-disable-readahead]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    unit_name          => opensearch-disable-readahead.service

Ferm::Service[elastic-main-https-9243]

Parameters differences:

--- Ferm::Service[elastic-main-https-9243].orig
+++ Ferm::Service[elastic-main-https-9243]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => False
+    port                => 9243
+    desc                => 
+    srange              => (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49)
+    proto               => tcp

File[/srv/opensearch/relforge-eqiad]

Parameters differences:

--- File[/srv/opensearch/relforge-eqiad].orig
+++ File[/srv/opensearch/relforge-eqiad]

+    owner  => opensearch
+    ensure => directory
+    group  => opensearch
+    mode   => 0755

Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]

Parameters differences:

--- Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha].orig
+++ Elasticsearch::Tlsproxy[relforge-eqiad-small-alpha]

+    upstream_port     => 9400
+    cfssl_paths       => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    server_aliases    => ['relforge.svc.eqiad.wmnet']
+    acme_chief        => False
+    server_name       => relforge1008.eqiad.wmnet
+    certificate_names => []
+    enable_http2      => False
+    tls_port          => 9443
+    read_only         => False

Mount[/var/lib/nginx]

Parameters differences:

--- Mount[/var/lib/nginx].orig
+++ Mount[/var/lib/nginx]

+    pass    => 0
+    before  => Service[nginx]
+    ensure  => mounted
+    fstype  => tmpfs
+    dump    => 0
+    options => defaults,noatime,uid=0,gid=0,mode=755,size=1g
+    device  => tmpfs
+    require => File[/var/lib/nginx]

Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)].orig
+++ Exec[systemd daemon-reload for opensearch-hot-threads-log.service (opensearch-hot-threads-log.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shards_9400.timer
+    override_filename => puppet-override.conf

File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400].orig
+++ File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9400]

+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    ensure  => present
+    line    => prometheus-elasticsearch-exporter-9400
+    path    => /etc/debdeploy-client/autorestarts.conf

Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shard_size_check_9200]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

File[/etc/logstash/conf.d/README]

Parameters differences:

--- File[/etc/logstash/conf.d/README].orig
+++ File[/etc/logstash/conf.d/README]

+    require => Package[logstash-oss]
+    group   => logstash
+    source  => puppet:///modules/logstash/conf.d/README
+    owner   => logstash
+    ensure  => present

Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad].orig
+++ Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad]

+    ensure => present

File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ File[/etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup.orig
+++ /etc/logrotate.d/opensearch-relforge-eqiad-gc-log-cleanup
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-relforge-eqiad-gc-log-cleanup
+
+/var/log/opensearch-relforge-eqiad-gc-log-cleanup/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ File[/etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha.orig
+++ /etc/logrotate.d/opensearch-disable-readahead-relforge-eqiad-small-alpha
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-disable-readahead-relforge-eqiad-small-alpha
+
+/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]

Parameters differences:

--- File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200].orig
+++ File[/etc/sudoers.d/nrpe-check_opensearch_unassigned_shard_check_9200]

+    require => Package[nagios-nrpe-server]
+    owner   => root
+    ensure  => absent
+    group   => root

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-elasticsearch-exporter-9200
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9200

Concat::Fragment[wikimedia-opensearch-plugins-header]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-plugins-header].orig
+++ Concat::Fragment[wikimedia-opensearch-plugins-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    order  => 01
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources

Monitoring::Service[opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Monitoring::Service[opensearch_unassigned_shard_check_9200].orig
+++ Monitoring::Service[opensearch_unassigned_shard_check_9200]

+    migration_task => T371083
+    check_interval => 720
+    contact_group  => admins,team-discovery
+    retries        => 1
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search#Administration
+    ensure         => present
+    passive        => False
+    retry_interval => 120
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_opensearch_unassigned_shard_check_9200!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => OpenSearch unassigned shard check - 9200

Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Logrotate::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    ensure => present

Service[logstash]

Parameters differences:

--- Service[logstash].orig
+++ Service[logstash]

+    hasstatus  => True
+    provider   => systemd
+    hasrestart => True
+    ensure     => running
+    enable     => True

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9400.timer
+    override_filename => puppet-override.conf

Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Service[prometheus-wmf-elasticsearch-exporter-9400]

+    override                 => False
+    restart                  => True
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    subscribe                => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    require                  => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    unit_type                => service

File[/etc/nginx/sites-available]

Parameters differences:

--- File[/etc/nginx/sites-available].orig
+++ File[/etc/nginx/sites-available]

+    group   => root
+    before  => ['Service[nginx]']
+    recurse => True
+    owner   => root
+    ensure  => directory
+    force   => True
+    purge   => True

Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shard_size_check_9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]
+    unit_type                => timer

File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-gc-log-cleanup.service (opensearch-relforge-eqiad-gc-log-cleanup.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-gc-log-cleanup.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cleanup GC logs
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/find /var/log/opensearch -name 'relforge-eqiad_jvm_gc.*.log*' -mtime +30 -delete

Systemd::Service[nrpe2nodexp-opensearch_shards_9400]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shards_9400].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shards_9400]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]
+    unit_type                => timer

Logstash::Conf[output-udp-gelf_relay]

Parameters differences:

--- Logstash::Conf[output-udp-gelf_relay].orig
+++ Logstash::Conf[output-udp-gelf_relay]

+    ensure   => present
+    priority => 90

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.service (nrpe2nodexp-opensearch_shards_9400.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shards_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shards_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "2a63eb7dcbe368b7f538118765d629d6" --timeout 10 --check-command "check_opensearch_shards_9400"

Class[Logstash]

Parameters differences:

--- Class[Logstash].orig
+++ Class[Logstash]

+    version              => 7.16.3-1
+    gc_log               => True
+    manage_service       => True
+    log_format           => json
+    enable_dlq           => False
+    dlq_max_bytes        => 1024mb
+    dlq_hosts            => []
+    java_package         => openjdk-17-jdk
+    pipeline_batch_delay => 50
+    pipeline_workers     => 2
+    pipeline_batch_size  => 125
+    heap_memory          => 192m

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-shards-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_shards_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

File[/etc/default/opensearch]

Parameters differences:

--- File[/etc/default/opensearch].orig
+++ File[/etc/default/opensearch]

+    require => Package[opensearch]
+    group   => root
+    owner   => root
+    ensure  => file
+    mode    => 0444

Content differences:

--- /etc/default/opensearch.orig
+++ /etc/default/opensearch
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# Default env file provided by upstream always reset OPENSEARCH_PATH_CONF
+# when called from /usr/share/opensearch/bin/opensearch-env
+# only set it when not defined.
+# This file is still required to set OPENSEARCH_PATH_CONF by postinst scripts
+OPENSEARCH_PATH_CONF=${OPENSEARCH_PATH_CONF-/etc/opensearch}

File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shards_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shards_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shards_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=1min
+OnActiveSec=1s
+RandomizedDelaySec=60
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]

Parameters differences:

--- Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400].orig
+++ Profile::Prometheus::Elasticsearch_exporter[relforge1008:9400]

+    extra_config       => 
+    elasticsearch_port => 9400
+    prometheus_port    => 9109

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Nrpe::Monitor_service[opensearch_shards_9400]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_shards_9400].orig
+++ Nrpe::Monitor_service[opensearch_shards_9400]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4
+    migration_task              => T371083
+    check_interval              => 1
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration
+    contact_group               => admins,team-discovery
+    retries                     => 3
+    ensure                      => present
+    retry_interval              => 1
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch health check for shards on 9400

Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)].orig
+++ Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9200.timer
+    override_filename => puppet-override.conf

Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Service[prometheus-elasticsearch-exporter-9400].orig
+++ Service[prometheus-elasticsearch-exporter-9400]

+    ensure => running
+    enable => True

Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ Service[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shards_9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shards_9400.service
+    override_filename => puppet-override.conf

Opensearch::Keystore[s3-credentials-user-relforge-eqiad]

Parameters differences:

--- Opensearch::Keystore[s3-credentials-user-relforge-eqiad].orig
+++ Opensearch::Keystore[s3-credentials-user-relforge-eqiad]

+    value        => search:platform
+    cluster_name => relforge-eqiad
+    key          => s3.client.default.access_key

File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

+    owner => root
+    group => opensearch
+    mode  => 0640

Package[nginx-light]

Parameters differences:

--- Package[nginx-light].orig
+++ Package[nginx-light]

+    notify   => ['Systemd::Unmask[nginx.service]']
+    ensure   => installed
+    provider => apt

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Unit[opensearch-hot-threads-log.service]

Parameters differences:

--- Systemd::Unit[opensearch-hot-threads-log.service].orig
+++ Systemd::Unit[opensearch-hot-threads-log.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-hot-threads-log.service
+    override_filename => puppet-override.conf

File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/prometheus-elasticsearch-exporter-9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for prometheus-elasticsearch-exporter-9200.service (prometheus-elasticsearch-exporter-9200)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/prometheus-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/prometheus-elasticsearch-exporter-9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=Prometheus exporter for Elasticsearch
+Documentation=https://prometheus.io/docs/introduction/overview/
+
+[Service]
+Restart=always
+User=prometheus
+ExecStart=/usr/bin/prometheus-elasticsearch-exporter --es.uri=http://localhost:9200 --web.listen-address=:9108 
+
+[Install]
+WantedBy=multi-user.target

File[/etc/opensearch/relforge-eqiad/java-security.policy]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/java-security.policy].orig
+++ File[/etc/opensearch/relforge-eqiad/java-security.policy]

+    group  => opensearch
+    source => puppet:///modules/profile/opensearch/java-security.policy
+    owner  => opensearch
+    ensure => file
+    mode   => 0440

Logrotate::Conf[push_cross_cluster_settings_9400]

Parameters differences:

--- Logrotate::Conf[push_cross_cluster_settings_9400].orig
+++ Logrotate::Conf[push_cross_cluster_settings_9400]

+    ensure => present

Concat::Fragment[wikimedia-opensearch-plugins]

Parameters differences:

--- Concat::Fragment[wikimedia-opensearch-plugins].orig
+++ Concat::Fragment[wikimedia-opensearch-plugins]

+    order  => 10
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources

Class[Profile::Java]

Parameters differences:

--- Class[Profile::Java].orig
+++ Class[Profile::Java]

+    java_packages   => []
+    trust_puppet_ca => True
+    extra_args      => {}
+    enable_dbg      => False
+    egd_source      => /dev/random
+    hardened_tls    => False

File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+
+[Timer]
+Unit=nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=5min
+OnActiveSec=1s
+RandomizedDelaySec=300
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/usr/lib/nagios/plugins/check_elasticsearch]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch]

+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch
+    owner  => root
+    group  => root
+    mode   => 0755

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Nginx::Site[relforge-eqiad]

Parameters differences:

--- Nginx::Site[relforge-eqiad].orig
+++ Nginx::Site[relforge-eqiad]

+    require => Exec[tlsproxy localssl default_server on ports [9243]]
+    ensure  => present

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.timer]']

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer
+    override_filename => puppet-override.conf

Prometheus::Elasticsearch_exporter[localhost:9200]

Parameters differences:

--- Prometheus::Elasticsearch_exporter[localhost:9200].orig
+++ Prometheus::Elasticsearch_exporter[localhost:9200]

+    extra_config       => 
+    elasticsearch_port => 9200
+    prometheus_port    => 9108

Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status].orig
+++ Systemd::Timer[nrpe2nodexp-check_push_cross_cluster_settings_9400_status]

+    splay              => 300
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '5min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-check_push_cross_cluster_settings_9400_status.service

Apt::Repository[wikimedia-opensearch-plugins]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch-plugins].orig
+++ Apt::Repository[wikimedia-opensearch-plugins]

+    bin                      => True
+    components               => component/opensearch2
+    uri                      => http://apt.wikimedia.org/wikimedia
+    source                   => True
+    ensure                   => present
+    trust_repo               => False
+    dist                     => bookworm-wikimedia
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    allow_releaseinfo_change => False

File[/etc/systemd/system/nginx.service.d]

Parameters differences:

--- File[/etc/systemd/system/nginx.service.d].orig
+++ File[/etc/systemd/system/nginx.service.d]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0555

File[/etc/opensearch]

Parameters differences:

--- File[/etc/opensearch].orig
+++ File[/etc/opensearch]

+    group   => root
+    recurse => True
+    mode    => 0755
+    owner   => root
+    ensure  => directory
+    force   => True
+    purge   => True

File[/etc/opensearch/instances]

Parameters differences:

--- File[/etc/opensearch/instances].orig
+++ File[/etc/opensearch/instances]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/instances.orig
+++ /etc/opensearch/instances
@@ -0,0 +1,2 @@
+opensearch_2@relforge-eqiad
+opensearch_2@relforge-eqiad-small-alpha

Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer (nrpe2nodexp-check_push_cross_cluster_settings_9400_status.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Service[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Service[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Service[nrpe2nodexp-opensearch_shards_9200]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => absent
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[nrpe2nodexp-opensearch_shards_9200.service]
+    unit_type                => timer

Ferm::Service[opensearch_http_9200]

Parameters differences:

--- Ferm::Service[opensearch_http_9200].orig
+++ Ferm::Service[opensearch_http_9200]

-    unrestricted_access => False
-    prio                => 10
-    ensure              => present
-    notrack             => False
-    port                => [9200]
-    src_sets            => ['ANALYTICS_NETWORKS']
-    desc                => 
-    proto               => tcp

File[/etc/opensearch/relforge-eqiad/log4j2.properties]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/log4j2.properties].orig
+++ File[/etc/opensearch/relforge-eqiad/log4j2.properties]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/opensearch/relforge-eqiad/log4j2.properties.orig
+++ /etc/opensearch/relforge-eqiad/log4j2.properties
@@ -0,0 +1,200 @@
+status = error
+
+# log action execution errors for easier debugging
+logger.action.name = org.opensearch.action
+logger.action.level = debug
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+######## Server JSON ############################
+appender.rolling.type = RollingFile
+appender.rolling.name = rolling
+appender.rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_server.json
+appender.rolling.layout.type = OpenSearchJsonLayout
+appender.rolling.layout.type_name = server
+
+appender.rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
+appender.rolling.policies.type = Policies
+appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling.policies.time.interval = 1
+appender.rolling.policies.time.modulate = true
+appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling.policies.size.size = 128MB
+appender.rolling.strategy.type = DefaultRolloverStrategy
+appender.rolling.strategy.fileIndex = nomax
+appender.rolling.strategy.action.type = Delete
+appender.rolling.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling.strategy.action.condition.type = IfFileName
+appender.rolling.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+######## Server -  old style pattern ###########
+appender.rolling_old.type = RollingFile
+appender.rolling_old.name = rolling_old
+appender.rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}.log
+appender.rolling_old.layout.type = PatternLayout
+appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
+appender.rolling_old.policies.type = Policies
+appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
+appender.rolling_old.policies.time.interval = 1
+appender.rolling_old.policies.time.modulate = true
+appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.rolling_old.policies.size.size = 128MB
+appender.rolling_old.strategy.type = DefaultRolloverStrategy
+appender.rolling_old.strategy.fileIndex = nomax
+appender.rolling_old.strategy.action.type = Delete
+appender.rolling_old.strategy.action.basepath = ${sys:opensearch.logs.base_path}
+appender.rolling_old.strategy.action.condition.type = IfFileName
+appender.rolling_old.strategy.action.condition.glob = ${sys:opensearch.logs.cluster_name}-*
+appender.rolling_old.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
+appender.rolling_old.strategy.action.condition.nested_condition.exceeds = 2GB
+################################################
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
+rootLogger.appenderRef.rolling.ref = rolling
+rootLogger.appenderRef.rolling_old.ref = rolling_old
+
+######## Deprecation JSON #######################
+appender.deprecation_rolling.type = RollingFile
+appender.deprecation_rolling.name = deprecation_rolling
+appender.deprecation_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.json
+appender.deprecation_rolling.layout.type = OpenSearchJsonLayout
+appender.deprecation_rolling.layout.type_name = deprecation
+appender.deprecation_rolling.layout.opensearchmessagefields=x-opaque-id
+
+appender.deprecation_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation-%i.json.gz
+appender.deprecation_rolling.policies.type = Policies
+appender.deprecation_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling.policies.size.size = 1GB
+appender.deprecation_rolling.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling.strategy.max = 4
+#################################################
+######## Deprecation -  old style pattern #######
+appender.deprecation_rolling_old.type = RollingFile
+appender.deprecation_rolling_old.name = deprecation_rolling_old
+appender.deprecation_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}_deprecation.log
+appender.deprecation_rolling_old.layout.type = PatternLayout
+appender.deprecation_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.deprecation_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _deprecation-%i.log.gz
+appender.deprecation_rolling_old.policies.type = Policies
+appender.deprecation_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.deprecation_rolling_old.policies.size.size = 1GB
+appender.deprecation_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.deprecation_rolling_old.strategy.max = 4
+#################################################
+logger.deprecation.name = org.opensearch.deprecation
+logger.deprecation.level = warn
+logger.deprecation.appenderRef.deprecation_rolling.ref = deprecation_rolling
+logger.deprecation.appenderRef.deprecation_rolling_old.ref = deprecation_rolling_old
+logger.deprecation.additivity = false
+
+######## Search slowlog JSON ####################
+appender.index_search_slowlog_rolling.type = RollingFile
+appender.index_search_slowlog_rolling.name = index_search_slowlog_rolling
+appender.index_search_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog.json
+appender.index_search_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_search_slowlog_rolling.layout.type_name = index_search_slowlog
+appender.index_search_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,total_hits,types,stats,search_type,total_shards,source,id
+
+appender.index_search_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs\
+  .cluster_name}_index_search_slowlog-%i.json.gz
+appender.index_search_slowlog_rolling.policies.type = Policies
+appender.index_search_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling.policies.size.size = 1GB
+appender.index_search_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling.strategy.max = 4
+#################################################
+######## Search slowlog -  old style pattern ####
+appender.index_search_slowlog_rolling_old.type = RollingFile
+appender.index_search_slowlog_rolling_old.name = index_search_slowlog_rolling_old
+appender.index_search_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog.log
+appender.index_search_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_search_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_search_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_search_slowlog-%i.log.gz
+appender.index_search_slowlog_rolling_old.policies.type = Policies
+appender.index_search_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_search_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_search_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_search_slowlog_rolling_old.strategy.max = 4
+#################################################
+logger.index_search_slowlog_rolling.name = index.search.slowlog
+logger.index_search_slowlog_rolling.level = trace
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling.ref = index_search_slowlog_rolling
+logger.index_search_slowlog_rolling.appenderRef.index_search_slowlog_rolling_old.ref = index_search_slowlog_rolling_old
+logger.index_search_slowlog_rolling.additivity = false
+
+######## Indexing slowlog JSON ##################
+appender.index_indexing_slowlog_rolling.type = RollingFile
+appender.index_indexing_slowlog_rolling.name = index_indexing_slowlog_rolling
+appender.index_indexing_slowlog_rolling.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.json
+appender.index_indexing_slowlog_rolling.layout.type = OpenSearchJsonLayout
+appender.index_indexing_slowlog_rolling.layout.type_name = index_indexing_slowlog
+appender.index_indexing_slowlog_rolling.layout.opensearchmessagefields=message,took,took_millis,doc_type,id,routing,source
+
+appender.index_indexing_slowlog_rolling.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.json.gz
+appender.index_indexing_slowlog_rolling.policies.type = Policies
+appender.index_indexing_slowlog_rolling.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling.strategy.max = 4
+#################################################
+######## Indexing slowlog -  old style pattern ##
+appender.index_indexing_slowlog_rolling_old.type = RollingFile
+appender.index_indexing_slowlog_rolling_old.name = index_indexing_slowlog_rolling_old
+appender.index_indexing_slowlog_rolling_old.fileName = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog.log
+appender.index_indexing_slowlog_rolling_old.layout.type = PatternLayout
+appender.index_indexing_slowlog_rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
+
+appender.index_indexing_slowlog_rolling_old.filePattern = ${sys:opensearch.logs.base_path}${sys:file.separator}${sys:opensearch.logs.cluster_name}\
+  _index_indexing_slowlog-%i.log.gz
+appender.index_indexing_slowlog_rolling_old.policies.type = Policies
+appender.index_indexing_slowlog_rolling_old.policies.size.type = SizeBasedTriggeringPolicy
+appender.index_indexing_slowlog_rolling_old.policies.size.size = 1GB
+appender.index_indexing_slowlog_rolling_old.strategy.type = DefaultRolloverStrategy
+appender.index_indexing_slowlog_rolling_old.strategy.max = 4
+#################################################
+
+logger.index_indexing_slowlog.name = index.indexing.slowlog.index
+logger.index_indexing_slowlog.level = trace
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling.ref = index_indexing_slowlog_rolling
+logger.index_indexing_slowlog.appenderRef.index_indexing_slowlog_rolling_old.ref = index_indexing_slowlog_rolling_old
+logger.index_indexing_slowlog.additivity = false
+
+
+### Custom settings
+monitorInterval=600
+
+# Add references to Logstash appender for interesting loggers
+
+rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
+logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
+
+# Define Logstash appender
+appender.ship_to_logstash.name=ship_to_logstash
+appender.ship_to_logstash.type=syslog
+appender.ship_to_logstash.protocol=udp
+appender.ship_to_logstash.facility=LOCAL0
+appender.ship_to_logstash.layout.type=OpenSearchJsonLayout
+appender.ship_to_logstash.layout.compact=true
+appender.ship_to_logstash.layout.complete=true
+appender.ship_to_logstash.host=localhost
+appender.ship_to_logstash.port=11514
+appender.ship_to_logstash.layout.type_name=opensearch
+

File[/etc/logrotate.d/nginx]

Parameters differences:

--- File[/etc/logrotate.d/nginx].orig
+++ File[/etc/logrotate.d/nginx]

+    group  => root
+    before => ['Service[nginx]']
+    source => puppet:///modules/profile/tlsproxy/logrotate
+    owner  => root
+    ensure => present
+    mode   => 0444

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-disable-readahead-relforge-eqiad-small-alpha.timer
+    override_filename => puppet-override.conf

File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf].orig
+++ File[/etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf.orig
+++ /etc/rsyslog.d/40-wmf-auto-restart-prometheus-wmf-elasticsearch-exporter-9200.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200" then {
+    action(
+        type="omfile" file="/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom].orig
+++ File[/var/lib/prometheus/node.d/check_opensearch_shards_9400.prom]

+    owner  => root
+    ensure => absent
+    group  => root

Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Logrotate::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    ensure => present

Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ Systemd::Syslog[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Service[opensearch]

Parameters differences:

--- Service[opensearch].orig
+++ Service[opensearch]

+    require => Package[opensearch]
+    ensure  => stopped
+    enable  => False

Motd::Script[cluster_memberships]

Parameters differences:

--- Motd::Script[cluster_memberships].orig
+++ Motd::Script[cluster_memberships]

+    source   => puppet:///modules/opensearch/opensearch.motd
+    ensure   => present
+    priority => 96

File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]

Parameters differences:

--- File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem].orig
+++ File[/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem]

+    backup    => False
+    show_diff => False
+    group     => root
+    notify    => ['Exec[nginx-reload]', 'Exec[nginx-reload]']
+    owner     => root
+    ensure    => file
+    mode      => 0440

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9200.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-elasticsearch-exporter-9200
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 7:36:00'}
+    private_tmp               => False
+    description               => Auto restart job: prometheus-elasticsearch-exporter-9200

Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ Service[opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    ensure   => running
+    enable   => True
+    provider => systemd

Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

Parameters differences:

--- Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources].orig
+++ Concat[/etc/apt/sources.list.d/wikimedia-opensearch.sources]

+    warn           => False
+    backup         => puppet
+    owner          => root
+    notify         => Exec[apt_repository_wikimedia-opensearch]
+    ensure         => present
+    replace        => True
+    ensure_newline => False
+    force          => False
+    order          => alpha
+    show_diff      => True
+    group          => root
+    format         => plain
+    path           => /etc/apt/sources.list.d/wikimedia-opensearch.sources
+    mode           => 0444

Concat_fragment[wikimedia-opensearch-header]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-header].orig
+++ Concat_fragment[wikimedia-opensearch-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    order  => 01
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch.sources
+    target => /etc/apt/sources.list.d/wikimedia-opensearch.sources

Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Syslog[opensearch-disable-readahead-relforge-eqiad]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]

Parameters differences:

--- Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh].orig
+++ Exec[Generate cert discovery2026__relforge1008_eqiad_wmnet refresh]

+    environment => ['GODEBUG=x509ignoreCN=0']
+    command     => /usr/bin/cfssl gencert -config /etc/cfssl/client-cfssl.conf -tls-remote-ca /etc/ssl/certs/wmf-ca-certificates.crt -mutual-tls-client-cert /etc/cfssl/mutual_tls_client_cert.pem -mutual-tls-client-key /var/lib/puppet/ssl/private_keys/relforge1008.eqiad.wmnet.pem -label discovery2026  /etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr | /usr/bin/cfssljson -bare /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet

+    refreshonly => True
+    subscribe   => File[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py]

+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch_unassigned_shards.py
+    owner  => root
+    group  => root
+    mode   => 0755

Nrpe::Check[check_opensearch_shards_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_shards_9400].orig
+++ Nrpe::Check[check_opensearch_shards_9400]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch.py --ignore-status --url http://localhost:9400 --shards-inactive '>=0.15' --timeout 4
+    before  => Monitoring::Service[opensearch_shards_9400]

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9400.timer
@@ -0,0 +1,14 @@
+[Unit]
+Description=Periodic execution of nrpe2nodexp-opensearch_shard_size_check_9400.service
+
+[Timer]
+Unit=nrpe2nodexp-opensearch_shard_size_check_9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitInactiveSec=600min
+OnActiveSec=1s
+RandomizedDelaySec=36000
+FixedRandomDelay=true
+
+[Install]
+WantedBy=multi-user.target

File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg.orig
+++ /etc/nagios/nrpe.d/check_check_push_cross_cluster_settings_9400_status.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_check_push_cross_cluster_settings_9400_status]=/usr/local/lib/nagios/plugins/check_systemd_unit_status push_cross_cluster_settings_9400

Ferm::Service[elastic-small-alpha-https-9443]

Parameters differences:

--- Ferm::Service[elastic-small-alpha-https-9443].orig
+++ Ferm::Service[elastic-small-alpha-https-9443]

+    unrestricted_access => False
+    prio                => 10
+    ensure              => present
+    notrack             => False
+    port                => 9443
+    desc                => 
+    srange              => (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49)
+    proto               => tcp

Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Profile::Auto_restarts::Service[prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => present

Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad].orig
+++ Monitoring::Exported_nagios_service[relforge1008 elasticsearch-https-relforge-eqiad]

+    check_period           => 24x7
+    contact_groups         => team-data-platform,team-discovery
+    check_interval         => 1
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => Elasticsearch HTTPS for relforge-eqiad
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search
+    retry_interval         => 1
+    check_command          => check_ssl_on_port!relforge1008.eqiad.wmnet!9243
+    max_check_attempts     => 3
+    host_name              => relforge1008

Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_unassigned_shard_check_9200]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 720
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch unassigned shard check - 9200
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#Administration
+    retry_interval         => 120
+    check_command          => nrpe_check!check_opensearch_unassigned_shard_check_9200!10
+    max_check_attempts     => 1
+    host_name              => relforge1008

Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008].orig
+++ Prometheus::Alert::Rule[check_opensearch_shard_size_check_9400_aefe2d8b10db3bc84662052af1e20008]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch shard size check - 9400
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_shard_size_check___9400
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="aefe2d8b10db3bc84662052af1e20008",check_name="check_opensearch_shard_size_check_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 1800m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shard_size_check_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch shard size check - 9400

File[/lib/systemd/system/push_cross_cluster_settings_9200.service]

Parameters differences:

--- File[/lib/systemd/system/push_cross_cluster_settings_9200.service].orig
+++ File[/lib/systemd/system/push_cross_cluster_settings_9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for push_cross_cluster_settings_9200.service (push_cross_cluster_settings_9200.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/push_cross_cluster_settings_9200.service.orig
+++ /lib/systemd/system/push_cross_cluster_settings_9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto set remote cluster seeds for relforge-eqiad
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/bin/bash /usr/local/bin/set-cross-cluster-seeds_9200.sh

Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Service[prometheus-wmf-elasticsearch-exporter-9200]

+    ensure => running
+    enable => True

Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shards_9200]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 60
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_shards_9200
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "fd6eadf747c645a72e11bd74109a8156" --timeout 10 --check-command "check_opensearch_shards_9200"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '1min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_shards_9200 command.

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer (nrpe2nodexp-opensearch_unassigned_shard_check_9200.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/etc/ferm/conf.d/10_opensearch-https-9243]

Parameters differences:

--- File[/etc/ferm/conf.d/10_opensearch-https-9243].orig
+++ File[/etc/ferm/conf.d/10_opensearch-https-9243]

+    require => File[/etc/ferm/conf.d]
+    tag     => ferm
+    group   => root
+    notify  => Service[ferm]
+    owner   => root
+    ensure  => present
+    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_opensearch-https-9243.orig
+++ /etc/ferm/conf.d/10_opensearch-https-9243
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9243, (($CUMIN_MASTERS $ANALYTICS_NETWORKS $STAGING_KUBEPODS_NETWORKS)));
+
+

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+    override_filename => puppet-override.conf

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9400.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9400.service
+    override_filename => puppet-override.conf

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9200.service (nrpe2nodexp-opensearch_unassigned_shard_check_9200.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_unassigned_shard_check_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "54ca99505516d5452912b573aa7d35d3" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9200"

Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service].orig
+++ Systemd::Unit[nrpe2nodexp-opensearch_shard_size_check_9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-opensearch_shard_size_check_9200.service
+    override_filename => puppet-override.conf

File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]

Parameters differences:

--- File[/usr/local/bin/set-cross-cluster-seeds_9200.sh].orig
+++ File[/usr/local/bin/set-cross-cluster-seeds_9200.sh]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0700

Content differences:

--- /usr/local/bin/set-cross-cluster-seeds_9200.sh.orig
+++ /usr/local/bin/set-cross-cluster-seeds_9200.sh
@@ -0,0 +1,16 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# set-cross-cluster-seeds.sh. Given a json file with opensearch remote seeds,
+# update OpenSearch cluster with correct settings.
+# Author: Brian King
+
+set -eo pipefail
+
+data="/etc/opensearch/relforge-eqiad/cirrus_check_settings.json"
+
+logger "updating cross cluster seed settings via ${data}"
+
+curl -XPUT \
+  -H 'content-type: application/json' \
+  --data "@${data}" \
+  http://localhost:9200/_cluster/settings

Systemd::Syslog[opensearch-disable-readahead]

Parameters differences:

--- Systemd::Syslog[opensearch-disable-readahead].orig
+++ Systemd::Syslog[opensearch-disable-readahead]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => absent
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]

Parameters differences:

--- File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg].orig
+++ File[/etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg]

+    require => Package[nagios-nrpe-server]
+    tag     => nrpe::check
+    group   => root
+    owner   => root
+    notify  => Service[nagios-nrpe-server]
+    ensure  => present
+    mode    => 0444

Content differences:

--- /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg.orig
+++ /etc/nagios/nrpe.d/check_opensearch_unassigned_shard_check_9400.cfg
@@ -0,0 +1,2 @@
+# File generated by puppet. DO NOT edit by hand
+command[check_opensearch_unassigned_shard_check_9400]=/usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9400 --timeout 4

Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9200]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 36000
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_shard_size_check_9200
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "c62725841c61e20ac3f6819b4c0ad800" --timeout 10 --check-command "check_opensearch_shard_size_check_9200"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_shard_size_check_9200 command.

Package[wmf-opensearch-search-plugins]

Parameters differences:

--- Package[wmf-opensearch-search-plugins].orig
+++ Package[wmf-opensearch-search-plugins]

+    require  => ['Class[Java]', 'Package[opensearch]']
+    ensure   => present
+    before   => ['Service[opensearch_2@relforge-eqiad]', 'Service[opensearch_2@relforge-eqiad-small-alpha]']
+    provider => apt

Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]

Parameters differences:

--- Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup].orig
+++ Systemd::Syslog[opensearch-relforge-eqiad-gc-log-cleanup]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200].orig
+++ Nrpe::Monitor_service[opensearch_unassigned_shard_check_9200]

+    enable_nrpe2nodexp          => False
+    enable_icinga_check         => True
+    nrpe_command                => /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py --url http://localhost:9200 --timeout 4
+    migration_task              => T371083
+    check_interval              => 720
+    notes_url                   => https://wikitech.wikimedia.org/wiki/Search#Administration
+    contact_group               => admins,team-discovery
+    retries                     => 1
+    ensure                      => present
+    retry_interval              => 120
+    nrpe2nodexp_parse_perf_data => False
+    timeout                     => 10
+    critical                    => False
+    alertmanager_team           => observability
+    description                 => OpenSearch unassigned shard check - 9200

File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]

Parameters differences:

--- File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf].orig
+++ File[/etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf.orig
+++ /etc/rsyslog.d/25-nrpe2nodexp-opensearch-unassigned-shard-check-9200.conf
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+if $programname contains "nrpe2nodexp-opensearch_unassigned_shard_check_9200" then {
+    if ($msg contains "\"ecs.version\": \"1.7.0\"") then {
+        # Send logs to kafka
+        set $.log_outputs = "kafka ecs_170 local";
+    } else {
+        # Filter out non-relevant nrpe2nodexp messages
+        stop
+    }
+}

Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]

Parameters differences:

--- Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad].orig
+++ Systemd::Timer::Job[opensearch-disable-readahead-relforge-eqiad]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/opensearch-disable-readahead.sh relforge-eqiad /srv/opensearch
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitActiveSec', 'interval': '30min'}, {'start': 'OnBootSec', 'interval': '1min'}]
+    private_tmp               => False
+    description               => Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes

File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer (wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Periodic execution of wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+
+[Timer]
+Unit=wmf_auto_restart_prometheus-elasticsearch-exporter-9400.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnCalendar=Mon,Tue,Wed,Thu,Fri *-*-* 14:27:00
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Systemd::Syslog[push_cross_cluster_settings_9200]

Parameters differences:

--- Systemd::Syslog[push_cross_cluster_settings_9200].orig
+++ Systemd::Syslog[push_cross_cluster_settings_9200]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Logstash::Input::Gelf[gelf_relay]

Parameters differences:

--- Logstash::Input::Gelf[gelf_relay].orig
+++ Logstash::Input::Gelf[gelf_relay]

+    tags      => ['input-gelf-relay']
+    priority  => 10
+    host      => localhost
+    plugin_id => input/gelf/12201
+    ensure    => present
+    port      => 12201

Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400].orig
+++ Profile::Auto_restarts::Service[prometheus-elasticsearch-exporter-9400]

+    ensure => present

Monitoring::Service[check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Monitoring::Service[check_push_cross_cluster_settings_9200_status].orig
+++ Monitoring::Service[check_push_cross_cluster_settings_9200_status]

+    migration_task => T407130
+    check_interval => 10
+    contact_group  => admins
+    retries        => 2
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    config_dir     => /etc/nagios
+    check_command  => nrpe_check!check_check_push_cross_cluster_settings_9200_status!10
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => Check unit status of push_cross_cluster_settings_9200

Java::Package[openjdk-jdk-17]

Parameters differences:

--- Java::Package[openjdk-jdk-17].orig
+++ Java::Package[openjdk-jdk-17]

+    package_info => {'version': '17', 'variant': 'jdk'}
+    egd_source   => /dev/random
+    hardened_tls => False

Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

Parameters differences:

--- Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer].orig
+++ Systemd::Unit[nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => nrpe2nodexp-check_push_cross_cluster_settings_9200_status.timer
+    override_filename => puppet-override.conf

Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)].orig
+++ Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shards_9400.timer (nrpe2nodexp-opensearch_shards_9400.timer)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Package[python3-requests]

Parameters differences:

--- Package[python3-requests].orig
+++ Package[python3-requests]

+    ensure   => installed
+    provider => apt

File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200].orig
+++ File_line[auto_restart_file_presence_prometheus-elasticsearch-exporter-9200]

+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    ensure  => present
+    line    => prometheus-elasticsearch-exporter-9200
+    path    => /etc/debdeploy-client/autorestarts.conf

File[/etc/nginx/sites-available/relforge-eqiad]

Parameters differences:

--- File[/etc/nginx/sites-available/relforge-eqiad].orig
+++ File[/etc/nginx/sites-available/relforge-eqiad]

+    group  => root
+    before => ['Service[nginx]']
+    owner  => root
+    notify => Exec[nginx-reload]
+    ensure => present

Content differences:

--- /etc/nginx/sites-available/relforge-eqiad.orig
+++ /etc/nginx/sites-available/relforge-eqiad
@@ -0,0 +1,59 @@
+# Proxy site configuration file for relforge-eqiad
+# This file is managed by Puppet!
+upstream local_fe_relforge-eqiad {
+    server 10.64.32.100:9200 max_fails=0;
+}
+
+# SSL proxying
+server {
+    # Enabling TCP Fast Open is safe for HTTP over TLS. There is no idempotency
+    # concern replaying TLS Client Hello.
+    # https://tools.ietf.org/html/rfc7413#section-6.3.2
+    listen [::]:9243 default_server deferred backlog=16384 reuseport ipv6only=on fastopen=150 ssl ;
+    listen 9243 default_server deferred backlog=16384 reuseport fastopen=150 ssl ;
+    server_name relforge1008.eqiad.wmnet relforge.svc.eqiad.wmnet;
+
+    error_log   /var/log/nginx/relforge-eqiad.error.log;
+    access_log   off;
+
+    ssl_certificate /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem;
+    ssl_certificate_key /etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem;
+
+    keepalive_timeout 60;
+    keepalive_requests 100;
+
+    location / {
+        proxy_pass http://local_fe_relforge-eqiad;
+        proxy_http_version 1.1;
+        proxy_set_header Connection close; # should be default
+
+        # this should be in sync with Varnish's first_byte_timeout
+        # and PHP's max_execution_time
+        proxy_read_timeout 180s;
+
+        proxy_set_header Host $host;
+        proxy_set_header X-Client-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto https;
+        proxy_set_header X-Connection-Properties "H2=$h2; SSR=$session_reused; SSL=$ssl_protocol; C=$ssl_cipher;";
+
+        # Don't clobber the Server header from the backend.
+        proxy_pass_header Server;
+
+        proxy_redirect off;
+        proxy_buffering off;
+        proxy_request_buffering on;
+
+        # With proxy buffering off, proxy_buffer_size is still used to
+        # define the singular chunk of buffer the content passes
+        # through.  Apparently if nginx can't read all the response
+        # headers in the first such chunk, it fails with 502.  Upping
+        # this to 8k (from def 4k) solves the one reported case in
+        # https://phabricator.wikimedia.org/T161819
+        # --
+        # Raising this further from 8K to 32K, at least temporarily, due to T226840
+        # Note 32KB is also the similar default limit in our varnishes (http_resp_size)
+        proxy_buffer_size       32k;
+        proxy_buffers         4 32k;
+    }
+}

File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]

Parameters differences:

--- File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf].orig
+++ File[/etc/logstash/conf.d/10-input-gelf-gelf_relay.conf]

+    group        => logstash
+    validate_cmd => /usr/local/bin/logstash-config-test %
+    owner        => logstash
+    notify       => Service[logstash]
+    ensure       => present
+    mode         => 0440

Content differences:

--- /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf.orig
+++ /etc/logstash/conf.d/10-input-gelf-gelf_relay.conf
@@ -0,0 +1,11 @@
+input {
+  # Graylog extended logging format
+  gelf {
+    type  => "gelf"
+    port  => 12201
+    remap => false
+    host  => "localhost"
+    id   => "input/gelf/12201"
+    tags => ["input-gelf-relay"]
+  }
+}

File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]

Parameters differences:

--- File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443].orig
+++ File[/etc/ferm/conf.d/10_elastic-small-alpha-https-9443]

+    require => File[/etc/ferm/conf.d]
+    tag     => ferm
+    group   => root
+    notify  => Service[ferm]
+    owner   => root
+    ensure  => present
+    mode    => 0400

Content differences:

--- /etc/ferm/conf.d/10_elastic-small-alpha-https-9443.orig
+++ /etc/ferm/conf.d/10_elastic-small-alpha-https-9443
@@ -0,0 +1,6 @@
+# Autogenerated by puppet. DO NOT EDIT BY HAND!
+#
+# 
+&R_SERVICE(tcp, 9443, (10.64.16.154 2620:0:861:102:10:64:16:154 10.192.32.49 2620:0:860:103:10:192:32:49));
+
+

File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

Parameters differences:

--- File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service].orig
+++ File[/lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service.orig
+++ /lib/systemd/system/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Auto restart job: prometheus-wmf-elasticsearch-exporter-9200
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9200

Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200].orig
+++ Monitoring::Exported_nagios_service[relforge1008 opensearch_shard_size_check_9200]

+    check_period           => 24x7
+    contact_groups         => admins,team-discovery
+    check_interval         => 1440
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => OpenSearch shard size check - 9200
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    retry_interval         => 180
+    check_command          => nrpe_check!check_opensearch_shard_size_check_9200!10
+    max_check_attempts     => 3
+    host_name              => relforge1008

Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    mode     => 0444
+    require  => File[/var/log/wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]
+    ensure   => present
+    priority => 40

Ferm::Service[opensearch_inter_node_9300]

Parameters differences:

--- Ferm::Service[opensearch_inter_node_9300].orig
+++ Ferm::Service[opensearch_inter_node_9300]

@@
-    notrack => False
+    notrack => True
@@
-    port    => [9300]
+    port    => 9300

Apt::Repository[wikimedia-opensearch]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch].orig
+++ Apt::Repository[wikimedia-opensearch]

+    bin                      => True
+    components               => thirdparty/opensearch2
+    uri                      => http://apt.wikimedia.org/wikimedia
+    before                   => Class[Opensearch]
+    source                   => True
+    ensure                   => present
+    trust_repo               => False
+    dist                     => bookworm-wikimedia
+    keyfile                  => puppet:///modules/install_server/autoinstall/keyring/wikimedia-archive-keyring.gpg
+    allow_releaseinfo_change => False

Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]

Parameters differences:

--- Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)].orig
+++ Exec[systemd daemon-reload for clean_up_dlq.service (clean_up_dlq.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

File[/srv/opensearch]

Parameters differences:

--- File[/srv/opensearch].orig
+++ File[/srv/opensearch]

+    require => Package[opensearch]
+    group   => opensearch
+    owner   => opensearch
+    ensure  => directory
+    mode    => 0755

File[/etc/nginx/sites-enabled/relforge-eqiad]

Parameters differences:

--- File[/etc/nginx/sites-enabled/relforge-eqiad].orig
+++ File[/etc/nginx/sites-enabled/relforge-eqiad]

+    group  => root
+    before => ['Service[nginx]']
+    owner  => root
+    ensure => link
+    target => /etc/nginx/sites-available/relforge-eqiad

Systemd::Unit[opensearch-hot-threads-log.timer]

Parameters differences:

--- Systemd::Unit[opensearch-hot-threads-log.timer].orig
+++ Systemd::Unit[opensearch-hot-threads-log.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-hot-threads-log.timer
+    override_filename => puppet-override.conf

Exec[tlsproxy localssl default_server on ports [9443]]

Parameters differences:

--- Exec[tlsproxy localssl default_server on ports [9443]].orig
+++ Exec[tlsproxy localssl default_server on ports [9443]]

+    command     => /bin/true
+    refreshonly => True
+    onlyif      => /bin/false

Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6].orig
+++ Prometheus::Alert::Rule[check_opensearch_shards_9400_2a63eb7dcbe368b7f538118765d629d6]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch health check for shards on 9400
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_health_check_for_shards_on_9400
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#Administration
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="2a63eb7dcbe368b7f538118765d629d6",check_name="check_opensearch_shards_9400", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 3m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shards_9400))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch health check for shards on 9400

Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Rsyslog::Conf[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    mode     => 0444
+    require  => File[/var/log/opensearch-disable-readahead-relforge-eqiad-small-alpha]
+    ensure   => present
+    priority => 40

File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ File_line[auto_restart_file_presence_prometheus-wmf-elasticsearch-exporter-9200]

+    require => File[/etc/debdeploy-client/autorestarts.conf]
+    ensure  => present
+    line    => prometheus-wmf-elasticsearch-exporter-9200
+    path    => /etc/debdeploy-client/autorestarts.conf

File[/etc/nginx/conf.d]

Parameters differences:

--- File[/etc/nginx/conf.d].orig
+++ File[/etc/nginx/conf.d]

+    group   => root
+    before  => ['Service[nginx]']
+    recurse => True
+    owner   => root
+    ensure  => directory
+    force   => True
+    purge   => True

Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_shards_9200]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]

Parameters differences:

--- Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800].orig
+++ Prometheus::Alert::Rule[check_opensearch_shard_size_check_9200_c62725841c61e20ac3f6819b4c0ad800]

+    team               => observability
+    summary            => NRPE CHECK: OpenSearch shard size check - 9200
+    site               => eqiad
+    alert_name         => nrpe_OpenSearch_shard_size_check___9200
+    dashboard          => TODO
+    ensure             => absent
+    runbook            => https://wikitech.wikimedia.org/wiki/Search#If_it_has_been_indexed
+    instance           => ops
+    expr               => (nagios_nrpe_check_result{alert_rule_hash="c62725841c61e20ac3f6819b4c0ad800",check_name="check_opensearch_shard_size_check_9200", status=~"(WARNING|CRITICAL)", severity=~"(warning|critical)"} > 0) * on (instance) group_left (team) role_owner
+    severity           => info
+    for                => 1800m
+    group              => nrpechecks
+    logs               => https://logstash.wikimedia.org/app/dashboards#/view/2d343ac0-6df8-11f0-8e08-7fab0da52b33?_g=(filters:!((query:(match_phrase:(event.module:check_opensearch_shard_size_check_9200))),(query:(match_phrase:(host.name:{{$labels.instance|stripPort}})))))
+    def_label_whitelst => ['team', 'severity']
+    description        => NRPE CHECK: OpenSearch shard size check - 9200

File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]

Parameters differences:

--- File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py].orig
+++ File[/usr/lib/nagios/plugins/check_elasticsearch_shard_size.py]

+    source => puppet:///modules/icinga/elasticsearch/check_elasticsearch_shard_size.py
+    owner  => root
+    group  => root
+    mode   => 0755

Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer].orig
+++ Service[nrpe2nodexp-opensearch_shard_size_check_9400.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9400.timer (nrpe2nodexp-opensearch_shard_size_check_9400.timer)]']
+    enable   => False
+    provider => systemd

File[/etc/nginx]

Parameters differences:

--- File[/etc/nginx].orig
+++ File[/etc/nginx]

+    require => Package[nginx-light]
+    group   => root
+    before  => ['Service[nginx]']
+    owner   => root
+    ensure  => directory

Exec[apt_repository_wikimedia-opensearch-plugins]

Parameters differences:

--- Exec[apt_repository_wikimedia-opensearch-plugins].orig
+++ Exec[apt_repository_wikimedia-opensearch-plugins]

+    command     => /usr/bin/apt-get update 
+    refreshonly => True

Systemd::Unit[prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Unit[prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Unit[prometheus-elasticsearch-exporter-9200]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => prometheus-elasticsearch-exporter-9200
+    override_filename => puppet-override.conf

File[/etc/opensearch/relforge-eqiad/scripts]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/scripts].orig
+++ File[/etc/opensearch/relforge-eqiad/scripts]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0444

Exec[java__cacert_wmf:puppetca.pem]

Parameters differences:

--- Exec[java__cacert_wmf:puppetca.pem].orig
+++ Exec[java__cacert_wmf:puppetca.pem]

+    command => /usr/bin/keytool -import -trustcacerts -noprompt -cacerts     -file /etc/ssl/certs/Puppet_Internal_CA.pem -storepass changeit -alias wmf:puppetca.pem

+    user    => root
+    group   => root
+    unless  => /usr/bin/keytool -list -cacerts -noprompt -storepass changeit -alias wmf:puppetca.pem

Exec[Refresh tmpfile opensearch-relforge-eqiad]

Parameters differences:

--- Exec[Refresh tmpfile opensearch-relforge-eqiad].orig
+++ Exec[Refresh tmpfile opensearch-relforge-eqiad]

+    command     => /bin/systemd-tmpfiles --create --remove '/etc/tmpfiles.d/opensearch-relforge-eqiad.conf'
+    user        => root
+    refreshonly => True
+    subscribe   => File[/etc/tmpfiles.d/opensearch-relforge-eqiad.conf]

Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]

Parameters differences:

--- Exec[opensearch-create-keystore-relforge-eqiad-small-alpha].orig
+++ Exec[opensearch-create-keystore-relforge-eqiad-small-alpha]

+    environment => ['OPENSEARCH_PATH_CONF=/etc/opensearch/relforge-eqiad-small-alpha']
+    creates     => /etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore
+    command     => /usr/share/opensearch/bin/opensearch-keystore create
+    before      => File[/etc/opensearch/relforge-eqiad-small-alpha/opensearch.keystore]

File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_unassigned_shard_check_9400.service (nrpe2nodexp-opensearch_unassigned_shard_check_9400.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_unassigned_shard_check_9400.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_unassigned_shard_check_9400 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_unassigned_shard_check_9400
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "a0a4892478c289e50484ffd2fb4c3202" --timeout 10 --check-command "check_opensearch_unassigned_shard_check_9400"

Systemd::Monitor[push_cross_cluster_settings_9400]

Parameters differences:

--- Systemd::Monitor[push_cross_cluster_settings_9400].orig
+++ Systemd::Monitor[push_cross_cluster_settings_9400]

+    migration_task => T407130
+    check_interval => 10
+    contact_group  => admins
+    retries        => 2
+    critical       => False
+    ensure         => present
+    notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state

Nrpe::Check[check_opensearch_shard_size_check_9400]

Parameters differences:

--- Nrpe::Check[check_opensearch_shard_size_check_9400].orig
+++ Nrpe::Check[check_opensearch_shard_size_check_9400]

+    ensure  => present
+    command => /usr/lib/nagios/plugins/check_elasticsearch_shard_size.py --url http://localhost:9400 --shard-size-warning 110 --shard-size-critical 140 --timeout 4
+    before  => Monitoring::Service[opensearch_shard_size_check_9400]

Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)].orig
+++ Exec[systemd daemon-reload for push_cross_cluster_settings_9400.timer (push_cross_cluster_settings_9400.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[push_cross_cluster_settings_9400.timer]']

File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

Parameters differences:

--- File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup].orig
+++ File[/etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup]

+    owner  => root
+    ensure => present
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.orig
+++ /etc/logrotate.d/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup
@@ -0,0 +1,12 @@
+# logrotate(8) config for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup
+
+/var/log/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

File[/etc/opensearch/relforge-eqiad-small-alpha]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad-small-alpha].orig
+++ File[/etc/opensearch/relforge-eqiad-small-alpha]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0755

Elasticsearch::Tlsproxy[relforge-eqiad]

Parameters differences:

--- Elasticsearch::Tlsproxy[relforge-eqiad].orig
+++ Elasticsearch::Tlsproxy[relforge-eqiad]

+    upstream_port     => 9200
+    cfssl_paths       => {'cert': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.pem', 'key': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet-key.pem', 'chain': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chain.pem', 'chained': '/etc/cfssl/ssl/discovery2026__relforge1008_eqiad_wmnet/discovery2026__relforge1008_eqiad_wmnet.chained.pem'}
+    server_aliases    => ['relforge.svc.eqiad.wmnet']
+    acme_chief        => False
+    server_name       => relforge1008.eqiad.wmnet
+    certificate_names => []
+    enable_http2      => False
+    tls_port          => 9243
+    read_only         => False

File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]

Parameters differences:

--- File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml].orig
+++ File[/etc/opensearch_hot_threads.d/relforge-eqiad.yml]

+    owner  => root
+    ensure => file
+    group  => root
+    mode   => 0755

Content differences:

--- /etc/opensearch_hot_threads.d/relforge-eqiad.yml.orig
+++ /etc/opensearch_hot_threads.d/relforge-eqiad.yml
@@ -0,0 +1,3 @@
+---
+port: 9200
+log_file: "/var/log/opensearch/opensearch_hot_threads-relforge-eqiad.log"

Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

Parameters differences:

--- Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200].orig
+++ Systemd::Timer[nrpe2nodexp-opensearch_unassigned_shard_check_9200]

+    splay              => 43200
+    fixed_random_delay => True
+    accuracy           => 15sec
+    ensure             => absent
+    timer_intervals    => [{'start': 'OnUnitInactiveSec', 'interval': '720min'}, {'interval': '1s', 'start': 'OnActiveSec'}]
+    unit_name          => nrpe2nodexp-opensearch_unassigned_shard_check_9200.service

File[/etc/udev/rules.d/40-opensearch-readahead.rules]

Parameters differences:

--- File[/etc/udev/rules.d/40-opensearch-readahead.rules].orig
+++ File[/etc/udev/rules.d/40-opensearch-readahead.rules]

+    group  => root
+    owner  => root
+    notify => Exec[udev_reload]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/udev/rules.d/40-opensearch-readahead.rules.orig
+++ /etc/udev/rules.d/40-opensearch-readahead.rules
@@ -0,0 +1 @@
+SUBSYSTEM=="block", KERNEL=="md1", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="16"

Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer].orig
+++ Systemd::Unit[opensearch-disable-readahead-relforge-eqiad.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-disable-readahead-relforge-eqiad.timer
+    override_filename => puppet-override.conf

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9400.timer
+    override_filename => puppet-override.conf

Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

Parameters differences:

--- Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer].orig
+++ Service[nrpe2nodexp-opensearch_shard_size_check_9200.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.timer (nrpe2nodexp-opensearch_shard_size_check_9200.timer)]']
+    enable   => False
+    provider => systemd

File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]

Parameters differences:

--- File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service].orig
+++ File[/lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for nrpe2nodexp-opensearch_shard_size_check_9200.service (nrpe2nodexp-opensearch_shard_size_check_9200.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service.orig
+++ /lib/systemd/system/nrpe2nodexp-opensearch_shard_size_check_9200.service
@@ -0,0 +1,11 @@
+[Unit]
+Description=execution of nrpe2nodexp for the check_opensearch_shard_size_check_9200 command.
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=nagios
+
+Group=prometheus-node-exporter
+SyslogIdentifier=nrpe2nodexp-opensearch_shard_size_check_9200
+ExecStart=-/usr/local/bin/nrpe2nodexp --alert-rule-hash "c62725841c61e20ac3f6819b4c0ad800" --timeout 10 --check-command "check_opensearch_shard_size_check_9200"

Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00'}]
+    unit_name          => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400.service

File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]

Parameters differences:

--- File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf].orig
+++ File[/etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf]

+    group  => root
+    owner  => root
+    notify => Service[rsyslog]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf.orig
+++ /etc/rsyslog.d/40-opensearch-relforge-eqiad-gc-log-cleanup.conf
@@ -0,0 +1,10 @@
+# rsyslog.conf(5) configuration file for services.
+# This file is managed by Puppet.
+if $programname startswith "opensearch-relforge-eqiad-gc-log-cleanup" then {
+    action(
+        type="omfile" file="/var/log/opensearch-relforge-eqiad-gc-log-cleanup/syslog.log"
+        fileOwner="root" fileGroup="root"
+        fileCreateMode="0644"
+    )
+    & stop
+}

Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Service[prometheus-wmf-elasticsearch-exporter-9200]

+    override                 => False
+    restart                  => True
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    subscribe                => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    require                  => File[/usr/local/bin/prometheus-wmf-elasticsearch-exporter]
+    unit_type                => service

Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status].orig
+++ Rsyslog::Conf[nrpe2nodexp-check_push_cross_cluster_settings_9200_status]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service].orig
+++ File[/lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service.orig
+++ /lib/systemd/system/opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Cleanup GC logs
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/bin/find /var/log/opensearch -name 'relforge-eqiad-small-alpha_jvm_gc.*.log*' -mtime +30 -delete

Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]

Parameters differences:

--- Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)].orig
+++ Exec[systemd daemon-reload for wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service (wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service)]

+    command     => /bin/systemctl daemon-reload
+    refreshonly => True

Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]

Parameters differences:

--- Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400].orig
+++ Systemd::Timer::Job[nrpe2nodexp-opensearch_shard_size_check_9400]

+    logging_enabled           => False
+    environment               => {}
+    ensure                    => absent
+    syslog_force_stop         => True
+    logfile_perms             => all
+    splay                     => 36000
+    user                      => nagios
+    group                     => prometheus-node-exporter
+    logfile_name              => syslog.log
+    fixed_random_delay        => True
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    syslog_identifier         => nrpe2nodexp-opensearch_shard_size_check_9400
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/bin/nrpe2nodexp --alert-rule-hash "aefe2d8b10db3bc84662052af1e20008" --timeout 10 --check-command "check_opensearch_shard_size_check_9400"
+    send_mail_only_on_error   => True
+    ignore_errors             => True
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => [{'start': 'OnUnitInactiveSec', 'interval': '600min'}]
+    private_tmp               => False
+    description               => execution of nrpe2nodexp for the check_opensearch_shard_size_check_9400 command.

Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200].orig
+++ Systemd::Timer[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200]

+    splay              => 0
+    fixed_random_delay => False
+    accuracy           => 15sec
+    ensure             => present
+    timer_intervals    => [{'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 10:55:00'}]
+    unit_name          => wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9200.service

Exec[unmask_nginx.service]

Parameters differences:

--- Exec[unmask_nginx.service].orig
+++ Exec[unmask_nginx.service]

+    command     => /bin/systemctl unmask nginx.service
+    refreshonly => True
+    onlyif      => /bin/readlink -f /etc/systemd/system/nginx.service | grep -q /dev/null

Motd::Script[cirrus::test]

Parameters differences:

--- Motd::Script[cirrus::test].orig
+++ Motd::Script[cirrus::test]

-    ensure   => present
-    priority => 5

Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

Parameters differences:

--- Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200].orig
+++ Systemd::Syslog[wmf_auto_restart_prometheus-elasticsearch-exporter-9200]

+    programname_comparison => startswith
+    owner                  => root
+    ensure                 => present
+    readable_by            => all
+    group                  => root
+    force_stop             => True
+    base_dir               => /var/log
+    log_filename           => syslog.log

Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]

Parameters differences:

--- Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha].orig
+++ Systemd::Service[opensearch-disable-readahead-relforge-eqiad-small-alpha]

+    override                 => False
+    restart                  => False
+    monitoring_contact_group => admins
+    monitoring_critical      => False
+    migration_task           => T407130
+    ensure                   => present
+    service_params           => {}
+    monitoring_enabled       => False
+    require                  => Systemd::Unit[opensearch-disable-readahead-relforge-eqiad-small-alpha.service]
+    unit_type                => timer

Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

Parameters differences:

--- Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr].orig
+++ Cfssl::Csr[/etc/cfssl/csr/discovery2026__relforge1008_eqiad_wmnet.csr]

+    names       => []
+    common_name => relforge1008.eqiad.wmnet
+    hosts       => ['relforge.svc.eqiad.wmnet']
+    key         => {'algo': 'ecdsa', 'size': 256}
+    ensure      => present

Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

Parameters differences:

--- Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400].orig
+++ Rsyslog::Conf[nrpe2nodexp-opensearch_unassigned_shard_check_9400]

+    ensure   => absent
+    mode     => 0444
+    priority => 25

Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

Parameters differences:

--- Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400].orig
+++ Systemd::Timer::Job[wmf_auto_restart_prometheus-wmf-elasticsearch-exporter-9400]

+    logging_enabled           => True
+    environment               => {}
+    ensure                    => present
+    syslog_force_stop         => True
+    logfile_perms             => all
+    user                      => root
+    logfile_name              => syslog.log
+    fixed_random_delay        => False
+    send_mail_to              => root@relforge1008.eqiad.wmnet
+    monitoring_contact_groups => admins
+    monitoring_notes_url      => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    require                   => File[/usr/local/sbin/wmf-auto-restart]
+    success_exit_status       => []
+    logfile_group             => root
+    syslog_match_startswith   => True
+    command                   => /usr/local/sbin/wmf-auto-restart -s prometheus-wmf-elasticsearch-exporter-9400
+    send_mail_only_on_error   => True
+    ignore_errors             => False
+    monitoring_enabled        => False
+    send_mail                 => False
+    logfile_basedir           => /var/log
+    interval                  => {'start': 'OnCalendar', 'interval': 'Mon,Tue,Wed,Thu,Fri *-*-* 11:6:00'}
+    private_tmp               => False
+    description               => Auto restart job: prometheus-wmf-elasticsearch-exporter-9400

File[/var/log/opensearch-disable-readahead]

Parameters differences:

--- File[/var/log/opensearch-disable-readahead].orig
+++ File[/var/log/opensearch-disable-readahead]

+    backup => False
+    group  => root
+    owner  => root
+    ensure => absent
+    force  => True
+    mode   => 0755

Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)].orig
+++ Exec[systemd daemon-reload for opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer (opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload
+    before      => ['Service[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.timer]']

File[/etc/systemd/system/opensearch_2@.service.d]

Parameters differences:

--- File[/etc/systemd/system/opensearch_2@.service.d].orig
+++ File[/etc/systemd/system/opensearch_2@.service.d]

+    owner  => root
+    ensure => directory
+    group  => root
+    mode   => 0555

Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

Parameters differences:

--- Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service].orig
+++ Systemd::Unit[opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => opensearch-relforge-eqiad-small-alpha-gc-log-cleanup.service
+    override_filename => puppet-override.conf

Concat_fragment[wikimedia-opensearch-plugins-header]

Parameters differences:

--- Concat_fragment[wikimedia-opensearch-plugins-header].orig
+++ Concat_fragment[wikimedia-opensearch-plugins-header]

+    source => puppet:///modules/apt/sources-deb822-header.txt
+    order  => 01
+    tag    => _etc_apt_sources.list.d_wikimedia-opensearch-plugins.sources
+    target => /etc/apt/sources.list.d/wikimedia-opensearch-plugins.sources

File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]

Parameters differences:

--- File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom].orig
+++ File[/var/lib/prometheus/node.d/check_check_push_cross_cluster_settings_9200_status.prom]

+    owner  => root
+    ensure => absent
+    group  => root

File[/etc/opensearch/relforge-eqiad/logging.yml]

Parameters differences:

--- File[/etc/opensearch/relforge-eqiad/logging.yml].orig
+++ File[/etc/opensearch/relforge-eqiad/logging.yml]

+    owner  => root
+    ensure => absent
+    group  => root

Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]

Parameters differences:

--- Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status].orig
+++ Monitoring::Exported_nagios_service[relforge1008 check_push_cross_cluster_settings_9200_status]

+    check_period           => 24x7
+    contact_groups         => admins
+    check_interval         => 10
+    ensure                 => present
+    active_checks_enabled  => 1
+    passive_checks_enabled => 1
+    service_description    => Check unit status of push_cross_cluster_settings_9200
+    notifications_enabled  => 0
+    servicegroups          => relforge_eqiad
+    notification_period    => 24x7
+    is_volatile            => 0
+    check_freshness        => 0
+    notification_options   => c,r,f
+    notification_interval  => 0
+    notes_url              => https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+    retry_interval         => 1
+    check_command          => nrpe_check!check_check_push_cross_cluster_settings_9200_status!10
+    max_check_attempts     => 2
+    host_name              => relforge1008

Systemd::Unit[opensearch-disable-readahead.service]

Parameters differences:

--- Systemd::Unit[opensearch-disable-readahead.service].orig
+++ Systemd::Unit[opensearch-disable-readahead.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => absent
+    unit              => opensearch-disable-readahead.service
+    override_filename => puppet-override.conf

Monitoring::Service[elasticsearch-https-relforge-eqiad]

Parameters differences:

--- Monitoring::Service[elasticsearch-https-relforge-eqiad].orig
+++ Monitoring::Service[elasticsearch-https-relforge-eqiad]

+    migration_task => T384998
+    check_interval => 1
+    notes_url      => https://wikitech.wikimedia.org/wiki/Search
+    retries        => 3
+    config_dir     => /etc/nagios
+    ensure         => present
+    passive        => False
+    retry_interval => 1
+    check_command  => check_ssl_on_port!relforge1008.eqiad.wmnet!9243
+    freshness      => 36000
+    host           => relforge1008
+    critical       => False
+    description    => Elasticsearch HTTPS for relforge-eqiad

File[/etc/apt/sources.list.d/wikimedia-opensearch.list]

Parameters differences:

--- File[/etc/apt/sources.list.d/wikimedia-opensearch.list].orig
+++ File[/etc/apt/sources.list.d/wikimedia-opensearch.list]

+    owner  => root
+    ensure => absent
+    group  => root

File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead-relforge-eqiad-small-alpha.timer (opensearch-disable-readahead-relforge-eqiad-small-alpha.timer)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer.orig
+++ /lib/systemd/system/opensearch-disable-readahead-relforge-eqiad-small-alpha.timer
@@ -0,0 +1,13 @@
+[Unit]
+Description=Periodic execution of opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+
+[Timer]
+Unit=opensearch-disable-readahead-relforge-eqiad-small-alpha.service
+# Accuracy sets the maximum time interval around the execution time we want to allow
+AccuracySec=15sec
+OnUnitActiveSec=30min
+OnBootSec=1min
+RandomizedDelaySec=0
+
+[Install]
+WantedBy=multi-user.target

Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

Parameters differences:

--- Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400].orig
+++ Rsyslog::Conf[wmf_auto_restart_prometheus-elasticsearch-exporter-9400]

+    mode     => 0444
+    require  => File[/var/log/wmf_auto_restart_prometheus-elasticsearch-exporter-9400]
+    ensure   => present
+    priority => 40

File[/etc/logrotate.d/clean_up_dlq]

Parameters differences:

--- File[/etc/logrotate.d/clean_up_dlq].orig
+++ File[/etc/logrotate.d/clean_up_dlq]

+    owner  => root
+    ensure => absent
+    group  => root
+    mode   => 0444

Content differences:

--- /etc/logrotate.d/clean_up_dlq.orig
+++ /etc/logrotate.d/clean_up_dlq
@@ -0,0 +1,12 @@
+# logrotate(8) config for clean_up_dlq
+
+/var/log/clean_up_dlq/*.log {
+    daily
+    copytruncate
+    missingok
+    compress
+    delaycompress
+    notifempty
+    rotate 15
+    size 256M
+}

Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

Parameters differences:

--- Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service].orig
+++ Systemd::Unit[wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service]

+    override          => False
+    restart           => False
+    require           => ['Class[Systemd]']
+    ensure            => present
+    unit              => wmf_auto_restart_prometheus-elasticsearch-exporter-9200.service
+    override_filename => puppet-override.conf

File[/lib/systemd/system/opensearch-disable-readahead.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch-disable-readahead.service].orig
+++ File[/lib/systemd/system/opensearch-disable-readahead.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch-disable-readahead.service (opensearch-disable-readahead.service)]
+    ensure => absent
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch-disable-readahead.service.orig
+++ /lib/systemd/system/opensearch-disable-readahead.service
@@ -0,0 +1,8 @@
+[Unit]
+Description=Disables readahead on all open files every 30 minutes to alleviate Cirrussearch / opensearch IO load spikes
+Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
+
+[Service]
+Type=oneshot
+User=root
+ExecStart=/usr/local/bin/opensearch-disable-readahead.sh

Service[opensearch-disable-readahead.timer]

Parameters differences:

--- Service[opensearch-disable-readahead.timer].orig
+++ Service[opensearch-disable-readahead.timer]

+    ensure   => stopped
+    before   => ['Exec[systemd daemon-reload for opensearch-disable-readahead.timer (opensearch-disable-readahead.timer)]']
+    enable   => False
+    provider => systemd

Compilation results for relforge1008.eqiad.wmnet: System changes detected

Catalog differences

Summary

Resources only in the new catalog

Resources only in the old catalog

Resources modified

Relevant files