Manager YARN capacity scheduler file

[rpignolet]

Context

Currently, YARN ResourceManager capacity scheduler file is manage by the yarn_resourcemanager_config.yml playbook (which call config.yml from role tosit.tdp.yarn.resourcemanager). The playbook yarn_resourcemanager_capacityscheduler.yml modify the capacity scheduler file too but this playbook is not in the DAG so it must be call manually.

Problem

We don't have a process for updating YARN's capacity scheduler. Currently, if we modify the capacity_scheduler variable in tdp_vars/yarn/yarn.yml and run a tdp reconfigure, it will reconfigure several elements that do not need to be reconfigured like yarn_ranger_config, yarn_client_config, yarn_resourcemanager_restart.yml, yarn_nodemanager_restart.yml.

Analysis

After you modify the Capacity Scheduler configuration, YARN enables you to refresh the queues without restarting your ResourceManager, if you have made no destructive changes (such as completely removing a queue) to your configuration. The Refresh operation will fail with the following message: Failed to re-init queues if you attempt to refresh queues in a case where you performed a destructive change, such as removing a queue. In cases where you have made destructive changes, you must perform a ResourceManager restart for the capacity scheduler change to take effect.

The yarn rmadmin -refreshQueues command allows to read the capacity-scheduler.xml file and to apply the configuration of the queues except in the case of deletion of a queue, a restart of ResourceManager is necessary.

Question

1. Should we keep the capacity_scheduler variable in the tdp_vars? This implies that after modifying this variable, a tdp reconfigure must be launched. In other words, do we keep the configuration of the capacity scheduler in the DAG? In the case where you have to do a reconfigure for the management of the YARN queues, this means that as long as a deployment is in progress, you cannot change the configuration of the YARN queues.
2. Should TDP carry refresh or restart logic in case of queue deletion? If so, should we change the capacity_scheduler variable to an abstraction of queue management to facilitate queue deletion detection code?

References

• https://docs.cloudera.com/runtime/7.2.10/yarn-allocate-resources/topics/yarn-cluster-management.html
• https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.3.4/bk_performance_tuning/content/section_refresh_yarn_queue_with_changed_settings.html
• https://docs.cloudera.com/HDPDocuments/Ambari-latest/managing-and-monitoring-ambari/content/amb_refresh_yarn_capacity_scheduler.html