Compilation results for cloudelastic1011.eqiad.wmnet: System changes detected

You can retrieve this result from host.json.

Catalog differences

Summary

Total Resources:	3790
Resources added:	24
Resources removed:	18
Resources modified:	85
Change percentage:	3.35%

Resources only in the new catalog

File[/usr/local/bin/pool-nginx]
Systemd::Override[opensearch_2@]
Systemd::Unit[opensearch_2@.service]
Systemd::Unit[opensearch_2@-opensearch_2@]
File[/usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad]
File[/usr/local/sbin/restart-nginx]
Service[opensearch_2@cloudelastic-psi-eqiad]
File[/usr/local/bin/pool-opensearch_2@production-search-omega-eqiad]
File[/usr/local/bin/depool-nginx]
Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-eqiad]
Service[opensearch_2@cloudelastic-chi-eqiad]
File[/usr/local/bin/depool-opensearch_2@production-search-omega-eqiad]
Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]
File[/usr/local/sbin/restart-opensearch_2@production-search-eqiad]
Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]
File[/usr/local/bin/pool-opensearch_2@production-search-eqiad]
File[/usr/local/bin/depool-opensearch_2@production-search-eqiad]
Exec[install-opensearch]
Conftool::Scripts::Safe_service_restart[nginx]
File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]
Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-omega-eqiad]
Service[opensearch_2@cloudelastic-omega-eqiad]
File[/etc/systemd/system/opensearch_2@.service.d]
File[/lib/systemd/system/opensearch_2@.service]

Resources only in the old catalog

File[/usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service]
Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-psi-eqiad.service]
File[/usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service]
File[/usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service]
Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-omega-eqiad.service]
Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]
Service[opensearch_1@cloudelastic-psi-eqiad]
File[/lib/systemd/system/opensearch_1@.service]
Service[opensearch_1@cloudelastic-omega-eqiad]
File[/usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service]
File[/usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service]
Service[opensearch_1@cloudelastic-chi-eqiad]
File[/usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service]
Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-chi-eqiad.service]
File[/usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service]
File[/usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service]
File[/usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service]
Systemd::Unit[opensearch_1@.service]

Resources modified

Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-omega-eqiad.service]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-omega-eqiad.service].orig
+++ Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-omega-eqiad.service]

-    max_concurrency => 1
-    lvs_pools       => ['cloudelastic-omega-https']
-    require         => ['Class[Conftool::Scripts]']

Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)].orig
+++ Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Opensearch::Cross_cluster_settings[cloudelastic-psi-eqiad]

Parameters differences:

--- Opensearch::Cross_cluster_settings[cloudelastic-psi-eqiad].orig
+++ Opensearch::Cross_cluster_settings[cloudelastic-psi-eqiad]

@@
-    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}
+    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}

File[/lib/systemd/system/opensearch_1@.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch_1@.service].orig
+++ File[/lib/systemd/system/opensearch_1@.service]

-    group  => root
-    owner  => root
-    notify => Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]
-    ensure => present
-    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch_1@.service.orig
+++ /lib/systemd/system/opensearch_1@.service
@@ -1,73 +0,0 @@
-[Unit]
-Description=OpenSearch (cluster %i)
-PartOf=opensearch.service
-Documentation=https://opensearch.org/docs
-Wants=network-online.target
-After=network-online.target
-
-[Service]
-Type=notify
-RuntimeDirectory=opensearch
-PrivateTmp=true
-Environment=OPENSEARCH_HOME=/usr/share/opensearch
-Environment=OPENSEARCH_PATH_CONF=/etc/opensearch/%i
-Environment=OPENSEARCH_JVM_OPTIONS=/etc/opensearch/%i/jvm.options
-Environment=PID_DIR=/run/opensearch-%i
-Environment=OPENSEARCH_SD_NOTIFY=true
-
-Environment=LD_LIBRARY_PATH=/usr/share/opensearch/plugins/opensearch-knn/lib
-Environment=JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
-
-WorkingDirectory=/usr/share/opensearch
-
-User=opensearch
-Group=opensearch
-
-# Force creation of /run/opensearch (see T276198)
-ExecStartPre=/bin/systemd-tmpfiles --create /etc/tmpfiles.d/opensearch-%i.conf
-ExecStart=/usr/share/opensearch/bin/opensearch \
-        -p ${PID_DIR}/%i.pid \
-        --quiet \
-        -Epath.logs=/var/log/opensearch \
-        -Epath.data=/srv/opensearch/%i
-
-# StandardOutput is configured to redirect to journalctl since
-# some error messages may be logged in standard output before
-# opensearch logging system is initialized. OpenSearch
-# stores its logs in /var/log/opensearch and does not use
-# journalctl by default. If you also want to enable journalctl
-# logging, you can simply remove the "quiet" option from ExecStart.
-StandardOutput=journal
-StandardError=inherit
-
-# Specifies the maximum file descriptor number that can be opened by this process
-LimitNOFILE=65535
-
-# Specifies the maximum number of processes
-LimitNPROC=4096
-
-# Specifies the maximum size of virtual memory
-LimitAS=infinity
-
-# Specifies the maximum file size
-LimitFSIZE=infinity
-
-# Disable timeout logic and wait until process is stopped
-TimeoutStopSec=0
-
-# SIGTERM signal is used to stop the Java process
-KillSignal=SIGTERM
-
-# Send the signal only to the JVM rather than its control group
-KillMode=process
-
-# Java process is never killed
-SendSIGKILL=no
-
-# When a JVM receives a SIGTERM signal it exits with code 143
-SuccessExitStatus=143
-
-[Install]
-WantedBy=multi-user.target
-
-# Built for packages-7.4.1 (packages)

Systemd::Timer::Job[prometheus_ferm_mss]

Parameters differences:

--- Systemd::Timer::Job[prometheus_ferm_mss].orig
+++ Systemd::Timer::Job[prometheus_ferm_mss]

@@
-    command => /usr/local/bin/prometheus-ferm-mss -o /var/lib/prometheus/node.d/ferm-mss.prom -e 208.80.154.241:9243 -e 208.80.154.241:9443 -e 208.80.154.241:9643 -e [2620:0:861:ed1a::3:241]:9243 -e [2620:0:861:ed1a::3:241]:9443 -e [2620:0:861:ed1a::3:241]:9643
+    command => /usr/local/bin/prometheus-ferm-mss -o /var/lib/prometheus/node.d/ferm-mss.prom -e 10.2.2.30:9200 -e 10.2.2.30:9243 -e 10.2.2.30:9443

Conftool::Scripts::Safe_service_restart[nginx]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[nginx].orig
+++ Conftool::Scripts::Safe_service_restart[nginx]

+    max_concurrency => 5
+    lvs_pools       => ['search-https', 'search-omega-https']
+    require         => ['Class[Conftool::Scripts]']

Systemd::Override[opensearch_2@]

Parameters differences:

--- Systemd::Override[opensearch_2@].orig
+++ Systemd::Override[opensearch_2@]

+    restart => False
+    ensure  => present
+    unit    => opensearch_2@

File[/etc/ferm/conf.d/10_clamp-mss-ipv6]

Content differences:

--- /etc/ferm/conf.d/10_clamp-mss-ipv6.orig
+++ /etc/ferm/conf.d/10_clamp-mss-ipv6
@@ -5,7 +5,7 @@
 domain (ip6) {
 	table filter {
 		chain OUTPUT {
-			outerface (enp152s0f0np0 lo) saddr @ipfilter((208.80.154.241 2620:0:861:ed1a::3:241])) proto tcp sport (9243 9443 9643) tcp-flags (SYN) SYN TCPMSS set-mss 1400;
+			outerface (enp152s0f0np0 lo) saddr @ipfilter(10.2.2.30) proto tcp sport (9200 9243 9443) tcp-flags (SYN) SYN TCPMSS set-mss 1400;
 		}
 	}
 }

File[/usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service].orig
+++ File[/usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service.orig
+++ /usr/local/bin/depool-opensearch_1@cloudelastic-psi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-psi-https --depool --retries 10 --wait 5

Opensearch::Cross_cluster_settings[cloudelastic-omega-eqiad]

Parameters differences:

--- Opensearch::Cross_cluster_settings[cloudelastic-omega-eqiad].orig
+++ Opensearch::Cross_cluster_settings[cloudelastic-omega-eqiad]

@@
-    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}
+    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}

File[/usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service]

Parameters differences:

--- File[/usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service].orig
+++ File[/usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service.orig
+++ /usr/local/sbin/restart-opensearch_1@cloudelastic-omega-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-omega-https --services opensearch_1@cloudelastic-omega-eqiad.service --retries 10 --wait 5 --max-concurrency 1 $@

File[/usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service].orig
+++ File[/usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service.orig
+++ /usr/local/bin/depool-opensearch_1@cloudelastic-omega-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-omega-https --depool --retries 10 --wait 5

File[/usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service].orig
+++ File[/usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service.orig
+++ /usr/local/bin/pool-opensearch_1@cloudelastic-psi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-psi-https --pool --retries 10 --wait 5

Class[Opensearch::Packages]

Parameters differences:

--- Class[Opensearch::Packages].orig
+++ Class[Opensearch::Packages]

@@
-    version => 1.3.20
+    version => 2.19.5

File[/etc/opensearch/cloudelastic-chi-eqiad/opensearch.yml]

Content differences:

--- /etc/opensearch/cloudelastic-chi-eqiad/opensearch.yml.orig
+++ /etc/opensearch/cloudelastic-chi-eqiad/opensearch.yml
@@ -100,6 +100,10 @@
 #
 #cluster.initial_master_nodes: ["node-1", "node-2"]
 #
+# Set to ensure a node sees N other master eligible nodes to be considered
+# operational within the cluster. Its recommended to set it to a higher value
+# than 1 when running more than 2 nodes in the cluster.
+#
 discovery.zen.ping.unicast.hosts: ["cloudelastic1009.eqiad.wmnet", "cloudelastic1010.eqiad.wmnet", "cloudelastic1011.eqiad.wmnet"]
 #
 # For more information, consult the discovery and cluster formation module documentation.
@@ -127,7 +131,7 @@
 #
 # If a plugin listed here is not installed for current node, the node will not start.
 #
-plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,opensearch-ltr,repository-s3
 
 # Specify the pattern of index names allowed to be created automatically
 action.auto_create_index: false
@@ -140,12 +144,16 @@
 # 1k is too low to allow mwgrep to operate.
 action.search.shard_count.limit: 5000
 
+# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-cluster.html#disk-based-shard-allocation
 # Enable the disk space aware shard allocator
 cluster.routing.allocation.disk.threshold_enabled: true
-# Stop allocating new shards to nodes with 75% disk utilization
+# Stop allocating shards to nodes (default 80%)
 cluster.routing.allocation.disk.watermark.low: 0.75
-# Move shards off of nodes with 80% disk utilization
+# Begin evicting shards off the node (default 90%)
 cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node (default 95%)
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+
 # Check disk utilization every 60 seconds
 cluster.info.update.interval: 60s
 
@@ -191,3 +199,4 @@
 ltr.caches.max_mem: 100mb
 
 
+

Exec[install-opensearch]

Parameters differences:

--- Exec[install-opensearch].orig
+++ Exec[install-opensearch]

+    unless      => /usr/bin/dpkg-query -W -f='${Status} ${Version}\n' opensearch 2>/dev/null | /bin/grep -Fxq 'install ok installed 2.19.5'
+    command     => /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install opensearch=2.19.5
+    logoutput   => False
+    before      => Package[opensearch]
+    environment => ['OPENSEARCH_INITIAL_ADMIN_PASSWORD=OpensearchTemp1!', 'DEBIAN_FRONTEND=noninteractive']
+    timeout     => 300

File[/etc/systemd/system/opensearch_2@.service.d]

Parameters differences:

--- File[/etc/systemd/system/opensearch_2@.service.d].orig
+++ File[/etc/systemd/system/opensearch_2@.service.d]

+    group  => root
+    owner  => root
+    ensure => directory
+    mode   => 0555

Ferm::Rule[clamp-mss-ipv4]

Parameters differences:

--- Ferm::Rule[clamp-mss-ipv4].orig
+++ Ferm::Rule[clamp-mss-ipv4]

@@
-    rule => outerface (enp152s0f0np0 lo) saddr @ipfilter((208.80.154.241 2620:0:861:ed1a::3:241])) proto tcp sport (9243 9443 9643) tcp-flags (SYN) SYN TCPMSS set-mss 1440;
+    rule => outerface (enp152s0f0np0 lo) saddr @ipfilter(10.2.2.30) proto tcp sport (9200 9243 9443) tcp-flags (SYN) SYN TCPMSS set-mss 1440;

File[/usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service].orig
+++ File[/usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service.orig
+++ /usr/local/bin/pool-opensearch_1@cloudelastic-omega-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-omega-https --pool --retries 10 --wait 5

Opensearch::Instance[cloudelastic-chi-eqiad]

Parameters differences:

--- Opensearch::Instance[cloudelastic-chi-eqiad].orig
+++ Opensearch::Instance[cloudelastic-chi-eqiad]

@@
-    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3']
@@
-    version           => 1.3.20
+    version           => 2.19.5

Class[Lvs::Realserver]

Parameters differences:

--- Class[Lvs::Realserver].orig
+++ Class[Lvs::Realserver]

@@
-    realserver_ips => ['208.80.154.241', '2620:0:861:ed1a::3:241']
+    realserver_ips => ['10.2.2.30']

File[/usr/local/sbin/restart-nginx]

Parameters differences:

--- File[/usr/local/sbin/restart-nginx].orig
+++ File[/usr/local/sbin/restart-nginx]

+    group  => root
+    owner  => root
+    before => ['Service[nginx]']
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-nginx.orig
+++ /usr/local/sbin/restart-nginx
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-https search-omega-https --services nginx --retries 10 --wait 5 --max-concurrency 5 $@

Package[wmf-opensearch-search-plugins]

Parameters differences:

--- Package[wmf-opensearch-search-plugins].orig
+++ Package[wmf-opensearch-search-plugins]

@@
-    before => ['Service[opensearch_1@cloudelastic-chi-eqiad]', 'Service[opensearch_1@cloudelastic-omega-eqiad]', 'Service[opensearch_1@cloudelastic-psi-eqiad]']
+    before => ['Service[opensearch_2@cloudelastic-chi-eqiad]', 'Service[opensearch_2@cloudelastic-omega-eqiad]', 'Service[opensearch_2@cloudelastic-psi-eqiad]']

Prometheus::Node_ferm_mss[ferm_clamped_ipport]

Parameters differences:

--- Prometheus::Node_ferm_mss[ferm_clamped_ipport].orig
+++ Prometheus::Node_ferm_mss[ferm_clamped_ipport]

@@
-    clamped_ipport => ['208.80.154.241:9243', '208.80.154.241:9443', '208.80.154.241:9643', '[2620:0:861:ed1a::3:241]:9243', '[2620:0:861:ed1a::3:241]:9443', '[2620:0:861:ed1a::3:241]:9643']
+    clamped_ipport => ['10.2.2.30:9200', '10.2.2.30:9243', '10.2.2.30:9443']

File[/usr/local/bin/pool-opensearch_2@production-search-eqiad]

Parameters differences:

--- File[/usr/local/bin/pool-opensearch_2@production-search-eqiad].orig
+++ File[/usr/local/bin/pool-opensearch_2@production-search-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/pool-opensearch_2@production-search-eqiad.orig
+++ /usr/local/bin/pool-opensearch_2@production-search-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search search-https --pool --retries 10 --wait 5

File[/etc/conftool/local_services.yaml]

Content differences:

--- /etc/conftool/local_services.yaml.orig
+++ /etc/conftool/local_services.yaml
@@ -1,19 +1,19 @@
 ---
-cloudelastic-chi-https:
-  cluster: cloudelastic
-  service: cloudelastic-chi-ssl
+search:
+  cluster: elasticsearch
+  service: elasticsearch
   servers:
-  - pybal-high-traffic2-eqiad.wikimedia.org
+  - pybal-low-traffic.svc.eqiad.wmnet
+  port: 9200
+search-https:
+  cluster: elasticsearch
+  service: elasticsearch-ssl
+  servers:
+  - pybal-low-traffic.svc.eqiad.wmnet
   port: 9243
-cloudelastic-omega-https:
-  cluster: cloudelastic
-  service: cloudelastic-omega-ssl
+search-omega-https:
+  cluster: elasticsearch
+  service: elasticsearch-omega-ssl
   servers:
-  - pybal-high-traffic2-eqiad.wikimedia.org
+  - pybal-low-traffic.svc.eqiad.wmnet
   port: 9443
-cloudelastic-psi-https:
-  cluster: cloudelastic
-  service: cloudelastic-psi-ssl
-  servers:
-  - pybal-high-traffic2-eqiad.wikimedia.org
-  port: 9643

File[/lib/systemd/system/opensearch_2@.service]

Parameters differences:

--- File[/lib/systemd/system/opensearch_2@.service].orig
+++ File[/lib/systemd/system/opensearch_2@.service]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /lib/systemd/system/opensearch_2@.service.orig
+++ /lib/systemd/system/opensearch_2@.service
@@ -0,0 +1,73 @@
+[Unit]
+Description=OpenSearch (cluster %i)
+PartOf=opensearch.service
+Documentation=https://opensearch.org/docs
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=notify
+RuntimeDirectory=opensearch
+PrivateTmp=true
+Environment=OPENSEARCH_HOME=/usr/share/opensearch
+Environment=OPENSEARCH_PATH_CONF=/etc/opensearch/%i
+Environment=OPENSEARCH_JVM_OPTIONS=/etc/opensearch/%i/jvm.options
+Environment=PID_DIR=/run/opensearch-%i
+Environment=OPENSEARCH_SD_NOTIFY=true
+
+Environment=LD_LIBRARY_PATH=/usr/share/opensearch/plugins/opensearch-knn/lib
+Environment=JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
+
+WorkingDirectory=/usr/share/opensearch
+
+User=opensearch
+Group=opensearch
+
+# Force creation of /run/opensearch (see T276198)
+ExecStartPre=/bin/systemd-tmpfiles --create /etc/tmpfiles.d/opensearch-%i.conf
+ExecStart=/usr/share/opensearch/bin/opensearch \
+        -p ${PID_DIR}/%i.pid \
+        --quiet \
+        -Epath.logs=/var/log/opensearch \
+        -Epath.data=/srv/opensearch/%i
+
+# StandardOutput is configured to redirect to journalctl since
+# some error messages may be logged in standard output before
+# opensearch logging system is initialized. OpenSearch
+# stores its logs in /var/log/opensearch and does not use
+# journalctl by default. If you also want to enable journalctl
+# logging, you can simply remove the "quiet" option from ExecStart.
+StandardOutput=journal
+StandardError=inherit
+
+# Specifies the maximum file descriptor number that can be opened by this process
+LimitNOFILE=65535
+
+# Specifies the maximum number of processes
+LimitNPROC=4096
+
+# Specifies the maximum size of virtual memory
+LimitAS=infinity
+
+# Specifies the maximum file size
+LimitFSIZE=infinity
+
+# Disable timeout logic and wait until process is stopped
+TimeoutStopSec=0
+
+# SIGTERM signal is used to stop the Java process
+KillSignal=SIGTERM
+
+# Send the signal only to the JVM rather than its control group
+KillMode=process
+
+# Java process is never killed
+SendSIGKILL=no
+
+# When a JVM receives a SIGTERM signal it exits with code 143
+SuccessExitStatus=143
+
+[Install]
+WantedBy=multi-user.target
+
+# Built for packages-7.4.1 (packages)

Class[Profile::Opensearch::Server]

Parameters differences:

--- Class[Profile::Opensearch::Server].orig
+++ Class[Profile::Opensearch::Server]

@@
-    common_settings => {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '30G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%'}
+    common_settings => {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '30G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%'}
@@
-    java_home       => /usr/lib/jvm/java-11-openjdk-amd64
+    java_home       => /usr/lib/jvm/java-21-openjdk-amd64
@@
-    version         => 1.3.20
+    version         => 2.19.5

Apt::Repository[wikimedia-opensearch]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch].orig
+++ Apt::Repository[wikimedia-opensearch]

@@
-    components => thirdparty/opensearch1
+    components => thirdparty/opensearch2

File[/usr/local/bin/depool-opensearch_2@production-search-omega-eqiad]

Parameters differences:

--- File[/usr/local/bin/depool-opensearch_2@production-search-omega-eqiad].orig
+++ File[/usr/local/bin/depool-opensearch_2@production-search-omega-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/depool-opensearch_2@production-search-omega-eqiad.orig
+++ /usr/local/bin/depool-opensearch_2@production-search-omega-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-omega-https --depool --retries 10 --wait 5

Class[Opensearch]

Parameters differences:

--- Class[Opensearch].orig
+++ Class[Opensearch]

@@
-    java_home         => /usr/lib/jvm/java-11-openjdk-amd64
+    java_home         => /usr/lib/jvm/java-21-openjdk-amd64
@@
-    instances         => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}
+    instances         => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}
@@
-    version           => 1.3.20
+    version           => 2.19.5
@@
-    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3']

Package[opensearch]

Parameters differences:

--- Package[opensearch].orig
+++ Package[opensearch]

@@
-    ensure => 1.3.20
+    ensure => 2.19.5

File[/usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service]

Parameters differences:

--- File[/usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service].orig
+++ File[/usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service.orig
+++ /usr/local/sbin/restart-opensearch_1@cloudelastic-psi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-psi-https --services opensearch_1@cloudelastic-psi-eqiad.service --retries 10 --wait 5 --max-concurrency 1 $@

Opensearch::Cross_cluster_settings[cloudelastic-chi-eqiad]

Parameters differences:

--- Opensearch::Cross_cluster_settings[cloudelastic-chi-eqiad].orig
+++ Opensearch::Cross_cluster_settings[cloudelastic-chi-eqiad]

@@
-    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}
+    settings => {'cloudelastic-chi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '55G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-chi-eqiad', 'short_cluster_name': 'chi', 'http_port': 9200, 'tls_port': 9243, 'tls_ro_port': 8243, 'transport_tcp_port': 9300, 'tune_gc_new_size_ratio': 3}, 'cloudelastic-omega-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-omega-eqiad', 'short_cluster_name': 'omega', 'http_port': 9400, 'tls_port': 9443, 'tls_ro_port': 8443, 'transport_tcp_port': 9500}, 'cloudelastic-psi-eqiad': {'awareness_attributes': 'row', 'plugins_mandatory': ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3'], 'heap_memory': '14G', 'bind_networks': ['0.0.0.0'], 'certificate_name': 'cloudelastic.wikimedia.org', 'recover_after_time': '5m', 'search_shard_count_limit': 5000, 'ltr_cache_size': '100mb', 'bulk_thread_pool_executors': 18, 'bulk_thread_pool_capacity': 1000, 'filter_cache_size': '20%', 'unicast_hosts': ['cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet'], 'cluster_hosts': ['cloudelastic1007.eqiad.wmnet', 'cloudelastic1008.eqiad.wmnet', 'cloudelastic1009.eqiad.wmnet', 'cloudelastic1010.eqiad.wmnet', 'cloudelastic1011.eqiad.wmnet', 'cloudelastic1012.eqiad.wmnet'], 'expected_nodes': 6, 'recover_after_nodes': 3, 'cluster_name': 'cloudelastic-psi-eqiad', 'short_cluster_name': 'psi', 'http_port': 9600, 'tls_port': 9643, 'tls_ro_port': 8643, 'transport_tcp_port': 9700}}

File[/etc/apt/sources.list.d/wikimedia-opensearch-plugins.list]

Content differences:

--- /etc/apt/sources.list.d/wikimedia-opensearch-plugins.list.orig
+++ /etc/apt/sources.list.d/wikimedia-opensearch-plugins.list
@@ -1,2 +1,2 @@
-deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/opensearch13
-deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/opensearch13
+deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/opensearch2
+deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/opensearch2

File[/usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad]

Parameters differences:

--- File[/usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad].orig
+++ File[/usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad.orig
+++ /usr/local/sbin/restart-opensearch_2@production-search-omega-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-omega-https --services opensearch_2@production-search-omega-eqiad --retries 10 --wait 5 --max-concurrency 5 $@

File[/lib/systemd/system/tcp-mss-clamper.service]

Content differences:

--- /lib/systemd/system/tcp-mss-clamper.service.orig
+++ /lib/systemd/system/tcp-mss-clamper.service
@@ -7,5 +7,5 @@
 
 [Service]
 LimitMEMLOCK=infinity
-ExecStart=/usr/bin/tcp-mss-clamper --ipv4-mss 1440 --ipv6-mss 1400 -p :2200 -s "208.80.154.241:9243,208.80.154.241:9443,208.80.154.241:9643,[2620:0:861:ed1a::3:241]:9243,[2620:0:861:ed1a::3:241]:9443,[2620:0:861:ed1a::3:241]:9643" -i enp152s0f0np0,lo
+ExecStart=/usr/bin/tcp-mss-clamper --ipv4-mss 1440 --ipv6-mss 1400 -p :2200 -s "10.2.2.30:9200,10.2.2.30:9243,10.2.2.30:9443" -i enp152s0f0np0,lo
 Restart=on-failure

Service[opensearch_2@cloudelastic-psi-eqiad]

Parameters differences:

--- Service[opensearch_2@cloudelastic-psi-eqiad].orig
+++ Service[opensearch_2@cloudelastic-psi-eqiad]

+    tag      => opensearch_services
+    require  => ['Systemd::Unit[opensearch_2@.service]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-psi-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-psi-eqiad]']
+    provider => systemd
+    enable   => True
+    ensure   => running

Systemd::Unit[opensearch_2@-opensearch_2@]

Parameters differences:

--- Systemd::Unit[opensearch_2@-opensearch_2@].orig
+++ Systemd::Unit[opensearch_2@-opensearch_2@]

+    override_filename => opensearch_2@
+    require           => ['Class[Systemd]']
+    override          => True
+    restart           => False
+    ensure            => present
+    unit              => opensearch_2@

Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-omega-eqiad]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-omega-eqiad].orig
+++ Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-omega-eqiad]

+    max_concurrency => 5
+    lvs_pools       => ['search-omega-https']
+    require         => ['Class[Conftool::Scripts]']

File[/usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service].orig
+++ File[/usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service.orig
+++ /usr/local/bin/pool-opensearch_1@cloudelastic-chi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-chi-https --pool --retries 10 --wait 5

File[/usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service]

Parameters differences:

--- File[/usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service].orig
+++ File[/usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service.orig
+++ /usr/local/bin/depool-opensearch_1@cloudelastic-chi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-chi-https --depool --retries 10 --wait 5

File[/usr/local/bin/pool-nginx]

Parameters differences:

--- File[/usr/local/bin/pool-nginx].orig
+++ File[/usr/local/bin/pool-nginx]

+    group  => root
+    owner  => root
+    before => ['Service[nginx]']
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/pool-nginx.orig
+++ /usr/local/bin/pool-nginx
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-https search-omega-https --pool --retries 10 --wait 5

File[/lib/systemd/system/prometheus_lvs_realserver_mss.service]

Content differences:

--- /lib/systemd/system/prometheus_lvs_realserver_mss.service.orig
+++ /lib/systemd/system/prometheus_lvs_realserver_mss.service
@@ -5,4 +5,4 @@
 [Service]
 Type=oneshot
 User=root
-ExecStart=/usr/local/bin/prometheus-lvs-realserver-mss -o /var/lib/prometheus/node.d/lvs-realserver-mss.prom -e 208.80.154.241:9243 -e 208.80.154.241:9443 -e 208.80.154.241:9643 -e [2620:0:861:ed1a::3:241]:9243 -e [2620:0:861:ed1a::3:241]:9443 -e [2620:0:861:ed1a::3:241]:9643
+ExecStart=/usr/local/bin/prometheus-lvs-realserver-mss -o /var/lib/prometheus/node.d/lvs-realserver-mss.prom -e 10.2.2.30:9200 -e 10.2.2.30:9243 -e 10.2.2.30:9443

Class[Profile::Lvs::Realserver]

Parameters differences:

--- Class[Profile::Lvs::Realserver].orig
+++ Class[Profile::Lvs::Realserver]

@@
-    pools => {'cloudelastic-chi-https': {'services': ['opensearch_1@cloudelastic-chi-eqiad.service']}, 'cloudelastic-psi-https': {'services': ['opensearch_1@cloudelastic-psi-eqiad.service']}, 'cloudelastic-omega-https': {'services': ['opensearch_1@cloudelastic-omega-eqiad.service']}}
+    pools => {'search': {'services': ['opensearch_2@production-search-eqiad']}, 'search-https': {'services': ['nginx', 'opensearch_2@production-search-eqiad']}, 'search-omega-https': {'services': ['nginx', 'opensearch_2@production-search-omega-eqiad']}}

File[/usr/local/bin/depool-nginx]

Parameters differences:

--- File[/usr/local/bin/depool-nginx].orig
+++ File[/usr/local/bin/depool-nginx]

+    group  => root
+    owner  => root
+    before => ['Service[nginx]']
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/depool-nginx.orig
+++ /usr/local/bin/depool-nginx
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-https search-omega-https --depool --retries 10 --wait 5

Class[Profile::Lvs::Realserver::Ipip]

Parameters differences:

--- Class[Profile::Lvs::Realserver::Ipip].orig
+++ Class[Profile::Lvs::Realserver::Ipip]

@@
-    pools => {'cloudelastic-chi-https': {'services': ['opensearch_1@cloudelastic-chi-eqiad.service']}, 'cloudelastic-psi-https': {'services': ['opensearch_1@cloudelastic-psi-eqiad.service']}, 'cloudelastic-omega-https': {'services': ['opensearch_1@cloudelastic-omega-eqiad.service']}}
+    pools => {'search': {'services': ['opensearch_2@production-search-eqiad']}, 'search-https': {'services': ['nginx', 'opensearch_2@production-search-eqiad']}, 'search-omega-https': {'services': ['nginx', 'opensearch_2@production-search-omega-eqiad']}}

Systemd::Unit[tcp-mss-clamper]

File[/etc/default/wikimedia-lvs-realserver]

Content differences:

--- /etc/default/wikimedia-lvs-realserver.orig
+++ /etc/default/wikimedia-lvs-realserver
@@ -7,4 +7,4 @@
 
 # LVS service IPs to be bound to the loopback interface,
 # separate using spaces
-LVS_SERVICE_IPS="208.80.154.241 2620:0:861:ed1a::3:241"
+LVS_SERVICE_IPS="10.2.2.30"

Systemd::Unit[prometheus_lvs_realserver_mss.service]

Apt::Repository[wikimedia-opensearch-plugins]

Parameters differences:

--- Apt::Repository[wikimedia-opensearch-plugins].orig
+++ Apt::Repository[wikimedia-opensearch-plugins]

@@
-    components => component/opensearch13
+    components => component/opensearch2

Service[opensearch_2@cloudelastic-omega-eqiad]

Parameters differences:

--- Service[opensearch_2@cloudelastic-omega-eqiad].orig
+++ Service[opensearch_2@cloudelastic-omega-eqiad]

+    tag      => opensearch_services
+    require  => ['Systemd::Unit[opensearch_2@.service]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-omega-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-omega-eqiad]']
+    provider => systemd
+    enable   => True
+    ensure   => running

Opensearch::Instance[cloudelastic-omega-eqiad]

Parameters differences:

--- Opensearch::Instance[cloudelastic-omega-eqiad].orig
+++ Opensearch::Instance[cloudelastic-omega-eqiad]

@@
-    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3']
@@
-    version           => 1.3.20
+    version           => 2.19.5

Class[Profile::Opensearch::Cirrus::Server]

Parameters differences:

--- Class[Profile::Opensearch::Cirrus::Server].orig
+++ Class[Profile::Opensearch::Cirrus::Server]

@@
-    version => 1.3.20
+    version => 2.19.5

File[/etc/opensearch/cloudelastic-psi-eqiad/opensearch.yml]

Content differences:

--- /etc/opensearch/cloudelastic-psi-eqiad/opensearch.yml.orig
+++ /etc/opensearch/cloudelastic-psi-eqiad/opensearch.yml
@@ -100,6 +100,10 @@
 #
 #cluster.initial_master_nodes: ["node-1", "node-2"]
 #
+# Set to ensure a node sees N other master eligible nodes to be considered
+# operational within the cluster. Its recommended to set it to a higher value
+# than 1 when running more than 2 nodes in the cluster.
+#
 discovery.zen.ping.unicast.hosts: ["cloudelastic1009.eqiad.wmnet", "cloudelastic1010.eqiad.wmnet", "cloudelastic1011.eqiad.wmnet"]
 #
 # For more information, consult the discovery and cluster formation module documentation.
@@ -127,7 +131,7 @@
 #
 # If a plugin listed here is not installed for current node, the node will not start.
 #
-plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,opensearch-ltr,repository-s3
 
 # Specify the pattern of index names allowed to be created automatically
 action.auto_create_index: false
@@ -140,12 +144,16 @@
 # 1k is too low to allow mwgrep to operate.
 action.search.shard_count.limit: 5000
 
+# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-cluster.html#disk-based-shard-allocation
 # Enable the disk space aware shard allocator
 cluster.routing.allocation.disk.threshold_enabled: true
-# Stop allocating new shards to nodes with 75% disk utilization
+# Stop allocating shards to nodes (default 80%)
 cluster.routing.allocation.disk.watermark.low: 0.75
-# Move shards off of nodes with 80% disk utilization
+# Begin evicting shards off the node (default 90%)
 cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node (default 95%)
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+
 # Check disk utilization every 60 seconds
 cluster.info.update.interval: 60s
 
@@ -191,3 +199,4 @@
 ltr.caches.max_mem: 100mb
 
 
+

Ferm::Rule[clamp-mss-ipv6]

Parameters differences:

--- Ferm::Rule[clamp-mss-ipv6].orig
+++ Ferm::Rule[clamp-mss-ipv6]

@@
-    rule => outerface (enp152s0f0np0 lo) saddr @ipfilter((208.80.154.241 2620:0:861:ed1a::3:241])) proto tcp sport (9243 9443 9643) tcp-flags (SYN) SYN TCPMSS set-mss 1400;
+    rule => outerface (enp152s0f0np0 lo) saddr @ipfilter(10.2.2.30) proto tcp sport (9200 9243 9443) tcp-flags (SYN) SYN TCPMSS set-mss 1400;

File[/usr/local/bin/pool-opensearch_2@production-search-omega-eqiad]

Parameters differences:

--- File[/usr/local/bin/pool-opensearch_2@production-search-omega-eqiad].orig
+++ File[/usr/local/bin/pool-opensearch_2@production-search-omega-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/pool-opensearch_2@production-search-omega-eqiad.orig
+++ /usr/local/bin/pool-opensearch_2@production-search-omega-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search-omega-https --pool --retries 10 --wait 5

File[/etc/ferm/conf.d/10_clamp-mss-ipv4]

Content differences:

--- /etc/ferm/conf.d/10_clamp-mss-ipv4.orig
+++ /etc/ferm/conf.d/10_clamp-mss-ipv4
@@ -5,7 +5,7 @@
 domain (ip) {
 	table filter {
 		chain OUTPUT {
-			outerface (enp152s0f0np0 lo) saddr @ipfilter((208.80.154.241 2620:0:861:ed1a::3:241])) proto tcp sport (9243 9443 9643) tcp-flags (SYN) SYN TCPMSS set-mss 1440;
+			outerface (enp152s0f0np0 lo) saddr @ipfilter(10.2.2.30) proto tcp sport (9200 9243 9443) tcp-flags (SYN) SYN TCPMSS set-mss 1440;
 		}
 	}
 }

File[/usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service]

Parameters differences:

--- File[/usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service].orig
+++ File[/usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service]

-    group  => root
-    owner  => root
-    ensure => present
-    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service.orig
+++ /usr/local/sbin/restart-opensearch_1@cloudelastic-chi-eqiad.service
@@ -1,2 +0,0 @@
-#!/bin/bash
-/usr/local/bin/safe-service-restart --pools cloudelastic-chi-https --services opensearch_1@cloudelastic-chi-eqiad.service --retries 10 --wait 5 --max-concurrency 1 $@

File[/lib/systemd/system/prometheus_ferm_mss.service]

Content differences:

--- /lib/systemd/system/prometheus_ferm_mss.service.orig
+++ /lib/systemd/system/prometheus_ferm_mss.service
@@ -5,4 +5,4 @@
 [Service]
 Type=oneshot
 User=root
-ExecStart=/usr/local/bin/prometheus-ferm-mss -o /var/lib/prometheus/node.d/ferm-mss.prom -e 208.80.154.241:9243 -e 208.80.154.241:9443 -e 208.80.154.241:9643 -e [2620:0:861:ed1a::3:241]:9243 -e [2620:0:861:ed1a::3:241]:9443 -e [2620:0:861:ed1a::3:241]:9643
+ExecStart=/usr/local/bin/prometheus-ferm-mss -o /var/lib/prometheus/node.d/ferm-mss.prom -e 10.2.2.30:9200 -e 10.2.2.30:9243 -e 10.2.2.30:9443

File[/etc/apt/sources.list.d/wikimedia-opensearch.list]

Content differences:

--- /etc/apt/sources.list.d/wikimedia-opensearch.list.orig
+++ /etc/apt/sources.list.d/wikimedia-opensearch.list
@@ -1,2 +1,2 @@
-deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia thirdparty/opensearch1
-deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia thirdparty/opensearch1
+deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia thirdparty/opensearch2
+deb-src http://apt.wikimedia.org/wikimedia bullseye-wikimedia thirdparty/opensearch2

Opensearch::Instance[cloudelastic-psi-eqiad]

Parameters differences:

--- Opensearch::Instance[cloudelastic-psi-eqiad].orig
+++ Opensearch::Instance[cloudelastic-psi-eqiad]

@@
-    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'ltr']
+    plugins_mandatory => ['analysis-hebrew', 'analysis-icu', 'analysis-smartcn', 'analysis-stempel', 'analysis-ukrainian', 'cirrus-highlighter', 'opensearch-analysis-stconvert', 'opensearch-extra', 'opensearch-extra-analysis-esperanto', 'opensearch-extra-analysis-serbian', 'opensearch-extra-analysis-slovak', 'opensearch-ltr', 'repository-s3']
@@
-    version           => 1.3.20
+    version           => 2.19.5

Prometheus::Node_lvs_realserver_mss[lvs_clamped_ipport]

Parameters differences:

--- Prometheus::Node_lvs_realserver_mss[lvs_clamped_ipport].orig
+++ Prometheus::Node_lvs_realserver_mss[lvs_clamped_ipport]

@@
-    clamped_ipport => ['208.80.154.241:9243', '208.80.154.241:9443', '208.80.154.241:9643', '[2620:0:861:ed1a::3:241]:9243', '[2620:0:861:ed1a::3:241]:9443', '[2620:0:861:ed1a::3:241]:9643']
+    clamped_ipport => ['10.2.2.30:9200', '10.2.2.30:9243', '10.2.2.30:9443']

Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)].orig
+++ Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@.service)]

+    refreshonly => True
+    command     => /bin/systemctl daemon-reload

Service[opensearch_1@cloudelastic-psi-eqiad]

Parameters differences:

--- Service[opensearch_1@cloudelastic-psi-eqiad].orig
+++ Service[opensearch_1@cloudelastic-psi-eqiad]

-    tag      => opensearch_services
-    require  => ['Systemd::Unit[opensearch_1@.service]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-psi-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-psi-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-psi-eqiad]']
-    provider => systemd
-    enable   => True
-    ensure   => running

File[/etc/opensearch/instances]

Content differences:

--- /etc/opensearch/instances.orig
+++ /etc/opensearch/instances
@@ -1,3 +1,3 @@
-opensearch_1@cloudelastic-chi-eqiad
-opensearch_1@cloudelastic-omega-eqiad
-opensearch_1@cloudelastic-psi-eqiad
+opensearch_2@cloudelastic-chi-eqiad
+opensearch_2@cloudelastic-omega-eqiad
+opensearch_2@cloudelastic-psi-eqiad

File[/usr/local/bin/depool-opensearch_2@production-search-eqiad]

Parameters differences:

--- File[/usr/local/bin/depool-opensearch_2@production-search-eqiad].orig
+++ File[/usr/local/bin/depool-opensearch_2@production-search-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/bin/depool-opensearch_2@production-search-eqiad.orig
+++ /usr/local/bin/depool-opensearch_2@production-search-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search search-https --depool --retries 10 --wait 5

File[/etc/opensearch/cloudelastic-psi-eqiad/log4j2.properties]

Content differences:

--- /etc/opensearch/cloudelastic-psi-eqiad/log4j2.properties.orig
+++ /etc/opensearch/cloudelastic-psi-eqiad/log4j2.properties
@@ -185,7 +185,6 @@
 rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
-logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
 
 # Define Logstash appender
 appender.ship_to_logstash.name=ship_to_logstash

File[/etc/opensearch/cloudelastic-omega-eqiad/log4j2.properties]

Content differences:

--- /etc/opensearch/cloudelastic-omega-eqiad/log4j2.properties.orig
+++ /etc/opensearch/cloudelastic-omega-eqiad/log4j2.properties
@@ -185,7 +185,6 @@
 rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
-logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
 
 # Define Logstash appender
 appender.ship_to_logstash.name=ship_to_logstash

File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]

Parameters differences:

--- File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf].orig
+++ File[/etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf]

+    group  => root
+    owner  => root
+    notify => Exec[systemd daemon-reload for opensearch_2@.service (opensearch_2@-opensearch_2@)]
+    ensure => present
+    mode   => 0444

Content differences:

--- /etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf.orig
+++ /etc/systemd/system/opensearch_2@.service.d/opensearch_2@.conf
@@ -0,0 +1,25 @@
+# The OpenSearch packages come with a lot of plugins
+# we don't want or need. This override helps us
+# expose only the plugins we explicitly set via
+# $plugins_mandatory var
+[Service]
+# Give OpenSearch its own mount namespace
+PrivateMounts=yes
+
+# Replace plugins dir with an empty tmpfs
+TemporaryFileSystem=/usr/share/opensearch/plugins:rw
+
+BindPaths=
+BindPaths=/usr/share/opensearch/plugins/analysis-hebrew
+BindPaths=/usr/share/opensearch/plugins/analysis-icu
+BindPaths=/usr/share/opensearch/plugins/analysis-smartcn
+BindPaths=/usr/share/opensearch/plugins/analysis-stempel
+BindPaths=/usr/share/opensearch/plugins/analysis-ukrainian
+BindPaths=/usr/share/opensearch/plugins/cirrus-highlighter
+BindPaths=/usr/share/opensearch/plugins/opensearch-analysis-stconvert
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-esperanto
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-serbian
+BindPaths=/usr/share/opensearch/plugins/opensearch-extra-analysis-slovak
+BindPaths=/usr/share/opensearch/plugins/opensearch-ltr
+BindPaths=/usr/share/opensearch/plugins/repository-s3

Systemd::Unit[opensearch_2@.service]

Parameters differences:

--- Systemd::Unit[opensearch_2@.service].orig
+++ Systemd::Unit[opensearch_2@.service]

+    override_filename => puppet-override.conf
+    require           => ['Class[Systemd]']
+    override          => False
+    restart           => False
+    ensure            => present
+    unit              => opensearch_2@.service

Service[opensearch_1@cloudelastic-omega-eqiad]

Parameters differences:

--- Service[opensearch_1@cloudelastic-omega-eqiad].orig
+++ Service[opensearch_1@cloudelastic-omega-eqiad]

-    tag      => opensearch_services
-    require  => ['Systemd::Unit[opensearch_1@.service]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-omega-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-omega-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-omega-eqiad]']
-    provider => systemd
-    enable   => True
-    ensure   => running

Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-psi-eqiad.service]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-psi-eqiad.service].orig
+++ Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-psi-eqiad.service]

-    max_concurrency => 1
-    lvs_pools       => ['cloudelastic-psi-https']
-    require         => ['Class[Conftool::Scripts]']

Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-chi-eqiad.service]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-chi-eqiad.service].orig
+++ Conftool::Scripts::Safe_service_restart[opensearch_1@cloudelastic-chi-eqiad.service]

-    max_concurrency => 1
-    lvs_pools       => ['cloudelastic-chi-https']
-    require         => ['Class[Conftool::Scripts]']

Systemd::Unit[prometheus_ferm_mss.service]

File[/etc/opensearch/cloudelastic-omega-eqiad/opensearch.yml]

Content differences:

--- /etc/opensearch/cloudelastic-omega-eqiad/opensearch.yml.orig
+++ /etc/opensearch/cloudelastic-omega-eqiad/opensearch.yml
@@ -100,6 +100,10 @@
 #
 #cluster.initial_master_nodes: ["node-1", "node-2"]
 #
+# Set to ensure a node sees N other master eligible nodes to be considered
+# operational within the cluster. Its recommended to set it to a higher value
+# than 1 when running more than 2 nodes in the cluster.
+#
 discovery.zen.ping.unicast.hosts: ["cloudelastic1009.eqiad.wmnet", "cloudelastic1010.eqiad.wmnet", "cloudelastic1011.eqiad.wmnet"]
 #
 # For more information, consult the discovery and cluster formation module documentation.
@@ -127,7 +131,7 @@
 #
 # If a plugin listed here is not installed for current node, the node will not start.
 #
-plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,ltr
+plugin.mandatory: analysis-hebrew,analysis-icu,analysis-smartcn,analysis-stempel,analysis-ukrainian,cirrus-highlighter,opensearch-analysis-stconvert,opensearch-extra,opensearch-extra-analysis-esperanto,opensearch-extra-analysis-serbian,opensearch-extra-analysis-slovak,opensearch-ltr,repository-s3
 
 # Specify the pattern of index names allowed to be created automatically
 action.auto_create_index: false
@@ -140,12 +144,16 @@
 # 1k is too low to allow mwgrep to operate.
 action.search.shard_count.limit: 5000
 
+# https://www.elastic.co/guide/en/elasticsearch/reference/7.12/modules-cluster.html#disk-based-shard-allocation
 # Enable the disk space aware shard allocator
 cluster.routing.allocation.disk.threshold_enabled: true
-# Stop allocating new shards to nodes with 75% disk utilization
+# Stop allocating shards to nodes (default 80%)
 cluster.routing.allocation.disk.watermark.low: 0.75
-# Move shards off of nodes with 80% disk utilization
+# Begin evicting shards off the node (default 90%)
 cluster.routing.allocation.disk.watermark.high: 0.80
+# Set indexes read-only with allocated shards on affected node (default 95%)
+cluster.routing.allocation.disk.watermark.flood_stage: 0.95
+
 # Check disk utilization every 60 seconds
 cluster.info.update.interval: 60s
 
@@ -191,3 +199,4 @@
 ltr.caches.max_mem: 100mb
 
 
+

File[/usr/local/sbin/restart-opensearch_2@production-search-eqiad]

Parameters differences:

--- File[/usr/local/sbin/restart-opensearch_2@production-search-eqiad].orig
+++ File[/usr/local/sbin/restart-opensearch_2@production-search-eqiad]

+    group  => root
+    owner  => root
+    ensure => present
+    mode   => 0555

Content differences:

--- /usr/local/sbin/restart-opensearch_2@production-search-eqiad.orig
+++ /usr/local/sbin/restart-opensearch_2@production-search-eqiad
@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/local/bin/safe-service-restart --pools search search-https --services opensearch_2@production-search-eqiad --retries 10 --wait 5 --max-concurrency 5 $@

Service[opensearch_2@cloudelastic-chi-eqiad]

Parameters differences:

--- Service[opensearch_2@cloudelastic-chi-eqiad].orig
+++ Service[opensearch_2@cloudelastic-chi-eqiad]

+    tag      => opensearch_services
+    require  => ['Systemd::Unit[opensearch_2@.service]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-chi-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-chi-eqiad]']
+    provider => systemd
+    enable   => True
+    ensure   => running

Systemd::Unit[opensearch_1@.service]

Parameters differences:

--- Systemd::Unit[opensearch_1@.service].orig
+++ Systemd::Unit[opensearch_1@.service]

-    override_filename => puppet-override.conf
-    require           => ['Class[Systemd]']
-    override          => False
-    restart           => False
-    ensure            => present
-    unit              => opensearch_1@.service

Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]

Parameters differences:

--- Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)].orig
+++ Exec[systemd daemon-reload for opensearch_1@.service (opensearch_1@.service)]

-    refreshonly => True
-    command     => /bin/systemctl daemon-reload

Systemd::Service[tcp-mss-clamper]

Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-eqiad]

Parameters differences:

--- Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-eqiad].orig
+++ Conftool::Scripts::Safe_service_restart[opensearch_2@production-search-eqiad]

+    max_concurrency => 5
+    lvs_pools       => ['search', 'search-https']
+    require         => ['Class[Conftool::Scripts]']

File[/etc/opensearch/cloudelastic-chi-eqiad/log4j2.properties]

Content differences:

--- /etc/opensearch/cloudelastic-chi-eqiad/log4j2.properties.orig
+++ /etc/opensearch/cloudelastic-chi-eqiad/log4j2.properties
@@ -185,7 +185,6 @@
 rootLogger.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_indexing_slowlog.appenderRef.ship_to_logstash.ref = ship_to_logstash
 logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
-logger.index_search_slowlog_rolling.appenderRef.ship_to_logstash.ref = ship_to_logstash
 
 # Define Logstash appender
 appender.ship_to_logstash.name=ship_to_logstash

Systemd::Timer::Job[prometheus_lvs_realserver_mss]

Parameters differences:

--- Systemd::Timer::Job[prometheus_lvs_realserver_mss].orig
+++ Systemd::Timer::Job[prometheus_lvs_realserver_mss]

@@
-    command => /usr/local/bin/prometheus-lvs-realserver-mss -o /var/lib/prometheus/node.d/lvs-realserver-mss.prom -e 208.80.154.241:9243 -e 208.80.154.241:9443 -e 208.80.154.241:9643 -e [2620:0:861:ed1a::3:241]:9243 -e [2620:0:861:ed1a::3:241]:9443 -e [2620:0:861:ed1a::3:241]:9643
+    command => /usr/local/bin/prometheus-lvs-realserver-mss -o /var/lib/prometheus/node.d/lvs-realserver-mss.prom -e 10.2.2.30:9200 -e 10.2.2.30:9243 -e 10.2.2.30:9443

Service[opensearch_1@cloudelastic-chi-eqiad]

Parameters differences:

--- Service[opensearch_1@cloudelastic-chi-eqiad].orig
+++ Service[opensearch_1@cloudelastic-chi-eqiad]

-    tag      => opensearch_services
-    require  => ['Systemd::Unit[opensearch_1@.service]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/opensearch.yml]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/logging.yml]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/log4j2.properties]', 'File[/etc/opensearch/cloudelastic-chi-eqiad/jvm.options]', 'File[/srv/opensearch/cloudelastic-chi-eqiad]', 'Systemd::Tmpfile[opensearch-cloudelastic-chi-eqiad]']
-    provider => systemd
-    enable   => True
-    ensure   => running

Compilation results for cloudelastic1011.eqiad.wmnet: System changes detected

Catalog differences

Summary

Resources only in the new catalog

Resources only in the old catalog

Resources modified

Relevant files