Compilation results for an-presto1006.eqiad.wmnet: System changes detected

Catalog differences

Summary

Resources only in the new catalog

• File[/etc/presto/resource-groups.json]
• Presto::Properties[resource-groups]
• File[/etc/presto/resource-groups.properties]
• File[/srv/presto/var/spill]

Resources modified

• Presto::Properties[config]

  Parameters differences:

  --- Presto::Properties[config].orig
  +++ Presto::Properties[config]
  
  @@
  -    properties => {'coordinator': False, 'node-scheduler.include-coordinator': False, 'http-server.http.port': 8280, 'jmx.rmiregistry.port': 8279, 'discovery.uri': 'https://analytics-presto.eqiad.wmnet:8281', 'experimental.spill-enabled': False, 'node-scheduler.network-topology': 'legacy', 'http-server.http.enabled': True, 'query.max-memory': '200GB', 'query.max-memory-per-node': '20GB', 'query.max-total-memory-per-node': '40GB', 'join-distribution-type': 'AUTOMATIC', 'http-server.https.keystore.path': '/etc/presto/ssl/server.p12', 'http-server.https.keystore.key': 'batman', 'internal-communication.https.required': True, 'internal-communication.https.keystore.path': '/etc/presto/ssl/server.p12', 'internal-communication.https.keystore.key': 'batman', 'internal-communication.https.trust-store-path': '/etc/ssl/certs/java/cacerts', 'internal-communication.https.trust-store-password': 'changeit', 'http-server.https.port': 8281, 'http-server.https.enabled': True, 'internal-communication.kerberos.enabled': True, 'http-server.authentication.type': 'KERBEROS', 'http.authentication.krb5.config': '/etc/krb5.conf', 'http.server.authentication.krb5.keytab': '/etc/security/keytabs/presto/presto.keytab', 'http.server.authentication.krb5.service-name': 'presto'}
  +    properties => {'coordinator': False, 'node-scheduler.include-coordinator': False, 'http-server.http.port': 8280, 'jmx.rmiregistry.port': 8279, 'discovery.uri': 'https://analytics-presto.eqiad.wmnet:8281', 'experimental.spill-enabled': True, 'node-scheduler.network-topology': 'legacy', 'http-server.http.enabled': True, 'memory.heap-headroom-per-node': '30GB', 'query.max-execution-time': '60m', 'query.max-memory': '600GB', 'query.max-total-memory': '900GB', 'query.max-memory-per-node': '40GB', 'query.max-total-memory-per-node': '60GB', 'experimental.max-spill-per-node': '100GB', 'experimental.spiller-spill-path': '/srv/presto/var/spill', 'task.concurrency': 4, 'task.max-worker-threads': 32, 'join-distribution-type': 'AUTOMATIC', 'http-server.https.keystore.path': '/etc/presto/ssl/server.p12', 'http-server.https.keystore.key': 'batman', 'internal-communication.https.required': True, 'internal-communication.https.keystore.path': '/etc/presto/ssl/server.p12', 'internal-communication.https.keystore.key': 'batman', 'internal-communication.https.trust-store-path': '/etc/ssl/certs/java/cacerts', 'internal-communication.https.trust-store-password': 'changeit', 'http-server.https.port': 8281, 'http-server.https.enabled': True, 'internal-communication.kerberos.enabled': True, 'http-server.authentication.type': 'KERBEROS', 'http.authentication.krb5.config': '/etc/krb5.conf', 'http.server.authentication.krb5.keytab': '/etc/security/keytabs/presto/presto.keytab', 'http.server.authentication.krb5.service-name': 'presto'}
  

• Class[Profile::Presto::Server]

  Parameters differences:

  --- Class[Profile::Presto::Server].orig
  +++ Class[Profile::Presto::Server]
  
  @@
  -    resource_groups_enabled => False
  +    resource_groups_enabled => True
  @@
  -    spill_enabled           => False
  +    spill_enabled           => True
  @@
  -    heap_max                => 100G
  +    heap_max                => 140G
  @@
  -    config_properties       => {'node-scheduler.network-topology': 'legacy', 'query.max-memory': '200GB', 'query.max-memory-per-node': '20GB', 'query.max-total-memory-per-node': '40GB', 'join-distribution-type': 'AUTOMATIC'}
  +    config_properties       => {'node-scheduler.network-topology': 'legacy', 'memory.heap-headroom-per-node': '30GB', 'query.max-execution-time': '60m', 'query.max-memory': '600GB', 'query.max-total-memory': '900GB', 'query.max-memory-per-node': '40GB', 'query.max-total-memory-per-node': '60GB', 'experimental.max-spill-per-node': '100GB', 'experimental.spiller-spill-path': '/srv/presto/var/spill', 'task.concurrency': 4, 'task.max-worker-threads': 32, 'join-distribution-type': 'AUTOMATIC'}
  

• Presto::Properties[resource-groups]

  Parameters differences:

  --- Presto::Properties[resource-groups].orig
  +++ Presto::Properties[resource-groups]
  
  +    group                 => presto
  +    may_contain_passwords => False
  +    properties            => {'resource-groups.configuration-manager': 'file', 'resource-groups.config-file': '/etc/presto/resource-groups.json'}
  +    owner                 => presto
  

• Service[presto-server]

  Parameters differences:

  --- Service[presto-server].orig
  +++ Service[presto-server]
  
  @@
  -    require => ['Presto::Properties[config]', 'Presto::Properties[node]', 'Presto::Properties[log]', 'File[/etc/presto/jvm.config]', 'File[/var/log/presto]', 'Rsyslog::Conf[presto-server]', 'Systemd::Override[presto-umask]']
  +    require => ['Presto::Properties[config]', 'Presto::Properties[node]', 'Presto::Properties[log]', 'File[/etc/presto/jvm.config]', 'File[/var/log/presto]', 'Rsyslog::Conf[presto-server]', 'Systemd::Override[presto-umask]', 'Presto::Properties[resource-groups]', 'File[/etc/presto/resource-groups.json]']
  

• Class[Presto::Server]

  Parameters differences:

  --- Class[Presto::Server].orig
  +++ Class[Presto::Server]
  
  @@
  -    resource_groups_enabled => False
  +    resource_groups_enabled => True
  @@
  -    spill_enabled           => False
  +    spill_enabled           => True
  @@
  -    heap_max                => 100G
  +    heap_max                => 140G
  @@
  -    config_properties       => {'jmx.rmiregistry.port': 8279, 'discovery.uri': 'https://analytics-presto.eqiad.wmnet:8281', 'node-scheduler.network-topology': 'legacy', 'http-server.http.port': 8280, 'http-server.http.enabled': True, 'query.max-memory': '200GB', 'query.max-memory-per-node': '20GB', 'query.max-total-memory-per-node': '40GB', 'join-distribution-type': 'AUTOMATIC', 'http-server.https.keystore.path': '/etc/presto/ssl/server.p12', 'http-server.https.keystore.key': 'batman', 'internal-communication.https.required': True, 'internal-communication.https.keystore.path': '/etc/presto/ssl/server.p12', 'internal-communication.https.keystore.key': 'batman', 'internal-communication.https.trust-store-path': '/etc/ssl/certs/java/cacerts', 'internal-communication.https.trust-store-password': 'changeit', 'http-server.https.port': 8281, 'http-server.https.enabled': True, 'internal-communication.kerberos.enabled': True, 'http-server.authentication.type': 'KERBEROS', 'http.authentication.krb5.config': '/etc/krb5.conf', 'http.server.authentication.krb5.keytab': '/etc/security/keytabs/presto/presto.keytab', 'http.server.authentication.krb5.service-name': 'presto'}
  +    config_properties       => {'jmx.rmiregistry.port': 8279, 'discovery.uri': 'https://analytics-presto.eqiad.wmnet:8281', 'node-scheduler.network-topology': 'legacy', 'http-server.http.port': 8280, 'http-server.http.enabled': True, 'memory.heap-headroom-per-node': '30GB', 'query.max-execution-time': '60m', 'query.max-memory': '600GB', 'query.max-total-memory': '900GB', 'query.max-memory-per-node': '40GB', 'query.max-total-memory-per-node': '60GB', 'experimental.max-spill-per-node': '100GB', 'experimental.spiller-spill-path': '/srv/presto/var/spill', 'task.concurrency': 4, 'task.max-worker-threads': 32, 'join-distribution-type': 'AUTOMATIC', 'http-server.https.keystore.path': '/etc/presto/ssl/server.p12', 'http-server.https.keystore.key': 'batman', 'internal-communication.https.required': True, 'internal-communication.https.keystore.path': '/etc/presto/ssl/server.p12', 'internal-communication.https.keystore.key': 'batman', 'internal-communication.https.trust-store-path': '/etc/ssl/certs/java/cacerts', 'internal-communication.https.trust-store-password': 'changeit', 'http-server.https.port': 8281, 'http-server.https.enabled': True, 'internal-communication.kerberos.enabled': True, 'http-server.authentication.type': 'KERBEROS', 'http.authentication.krb5.config': '/etc/krb5.conf', 'http.server.authentication.krb5.keytab': '/etc/security/keytabs/presto/presto.keytab', 'http.server.authentication.krb5.service-name': 'presto'}
  

• File[/etc/presto/config.properties]

  Content differences:

  --- /etc/presto/config.properties.orig
  +++ /etc/presto/config.properties
  @@ -2,7 +2,9 @@
   
   coordinator=false
   discovery.uri=https://analytics-presto.eqiad.wmnet:8281
  -experimental.spill-enabled=false
  +experimental.max-spill-per-node=100GB
  +experimental.spill-enabled=true
  +experimental.spiller-spill-path=/srv/presto/var/spill
   http-server.authentication.type=KERBEROS
   http-server.http.enabled=true
   http-server.http.port=8280
  @@ -21,8 +23,13 @@
   internal-communication.kerberos.enabled=true
   jmx.rmiregistry.port=8279
   join-distribution-type=AUTOMATIC
  +memory.heap-headroom-per-node=30GB
   node-scheduler.include-coordinator=false
   node-scheduler.network-topology=legacy
  -query.max-memory=200GB
  -query.max-memory-per-node=20GB
  -query.max-total-memory-per-node=40GB
  +query.max-execution-time=60m
  +query.max-memory=600GB
  +query.max-memory-per-node=40GB
  +query.max-total-memory=900GB
  +query.max-total-memory-per-node=60GB
  +task.concurrency=4
  +task.max-worker-threads=32

• File[/etc/presto/jvm.config]

  Content differences:

  --- /etc/presto/jvm.config.orig
  +++ /etc/presto/jvm.config
  @@ -1,5 +1,5 @@
   -server
  --Xmx100G
  +-Xmx140G
   -XX:+UseG1GC
   -XX:G1HeapRegionSize=32M
   -XX:+UseGCOverheadLimit

• File[/etc/presto/resource-groups.properties]

  Parameters differences:

  --- File[/etc/presto/resource-groups.properties].orig
  +++ File[/etc/presto/resource-groups.properties]
  
  +    group => presto
  +    mode  => 0444
  +    owner => presto
  

  Content differences:

  --- /etc/presto/resource-groups.properties.orig
  +++ /etc/presto/resource-groups.properties
  @@ -0,0 +1,4 @@
  +# NOTE: This file is managed by Puppet.
  +
  +resource-groups.config-file=/etc/presto/resource-groups.json
  +resource-groups.configuration-manager=file

• File[/srv/presto/var/spill]

  Parameters differences:

  --- File[/srv/presto/var/spill].orig
  +++ File[/srv/presto/var/spill]
  
  +    group   => presto
  +    ensure  => directory
  +    require => Package[presto-server]
  +    mode    => 0755
  +    before  => Service[presto-server]
  +    owner   => presto
  

• File[/etc/presto/resource-groups.json]

  Parameters differences:

  --- File[/etc/presto/resource-groups.json].orig
  +++ File[/etc/presto/resource-groups.json]
  
  +    group => root
  +    owner => root
  

  Content differences:

  --- /etc/presto/resource-groups.json.orig
  +++ /etc/presto/resource-groups.json
  @@ -0,0 +1,53 @@
  +{
  +  "rootGroups": [
  +    {
  +      "name": "global",
  +      "softMemoryLimit": "100%",
  +      "hardConcurrencyLimit": 40,
  +      "maxQueued": 200,
  +      "schedulingPolicy": "weighted",
  +      "subGroups": [
  +        {
  +          "name": "high_priority",
  +          "softMemoryLimit": "35%",
  +          "hardConcurrencyLimit": 12,
  +          "maxQueued": 40,
  +          "schedulingWeight": 6
  +        },
  +        {
  +          "name": "standard",
  +          "softMemoryLimit": "45%",
  +          "hardConcurrencyLimit": 20,
  +          "maxQueued": 100,
  +          "schedulingWeight": 3
  +        },
  +        {
  +          "name": "heavy",
  +          "softMemoryLimit": "20%",
  +          "hardConcurrencyLimit": 5,
  +          "maxQueued": 60,
  +          "schedulingWeight": 1
  +        }
  +      ]
  +    }
  +  ],
  +  "selectors": [
  +    {
  +      "principal": "superset/[^@]+@WIKIMEDIA",
  +      "group": "global.high_priority"
  +    },
  +    {
  +      "principal": "HTTP/growthbook[^@]*@WIKIMEDIA",
  +      "group": "global.high_priority"
  +    },
  +    {
  +      "clientTags": [
  +        "heavy"
  +      ],
  +      "group": "global.heavy"
  +    },
  +    {
  +      "group": "global.standard"
  +    }
  +  ]
  +}

Relevant files

• Production catalog
• Change catalog
• Production errors/warnings
• Change errors/warnings
• Core Diff
• Standard Diff