從這個例子可以看到,用戶不斷提交任務,有可能鎖定其它隊列的資源,直到資源釋放或獲得搶占。為了避免這種情況,Capacity Scheduler 支持在隊列設(shè)置彈性增長限制。例如,以限制 root.Engineering.Development 隊列,管理員可以設(shè)置 Max Capacity 為 40%。一旦設(shè)置,root.Engineering.Development 隊列的用戶仍然可以超越其 1.8 GB 的容量,但他們不會被分配超過 40% 的 root.Engineering 父隊列的能力,即 9 GB 的 40% = 3.6 GB。
通過 Capacity 屬性和 Max Capacity 屬性的設(shè)置,可用于控制組織和子機構(gòu)資源彈性共享。管理員應當平衡這些特性,以避免嚴格的限制導致降低集群利用率,也要避免過度的跨組織資源共享。
隊列的映射
管理員可以定義一個默認的映射策略,指定用戶提交的申請將被自動提交到隊列。有了一個默認的映射策略,用戶無需遞交申請時指定的隊列名稱。默認映射策略可以被配置成如果所提交的應用程序指定的隊列名被覆蓋。
隊列映射是用逗號分隔的映射分配的列表中定義,隊列映射分配可以為用戶(使用 u)或一組用戶(使用 g)來定義。
圖 12. 隊列映射分配


完成配置
IBM BigInsights 通過圖形化界面配置,產(chǎn)生了新的 Capacity Scheduler 配置,極大簡化了管理,同時也避免了錯誤。保存配置后可以不重啟服務,直接在 Capacity-Scheduler 視圖 保存并刷新隊列 Refresh Queues ResourceManager,當然也可以重新啟動YARN 服務使配置生效。
清單 2. Capacity Scheduler 修改后的配置
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.queue-mappings=u:dev1:Development,u:qa1:QA,u:adv1:Advertising,u:sal1:Sales,u:serv1:Services,u:tr1:Training
yarn.scheduler.capacity.queue-mappings-override.enable=false
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=10
yarn.scheduler.capacity.root.default.maximum-capacity=10
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.Engineering.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.capacity=30
yarn.scheduler.capacity.root.Engineering.Development.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.Development.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.Development.capacity=20
yarn.scheduler.capacity.root.Engineering.Development.maximum-capacity=100
yarn.scheduler.capacity.root.Engineering.Development.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.Engineering.Development.ordering-policy=fifo
yarn.scheduler.capacity.root.Engineering.Development.state=RUNNING
yarn.scheduler.capacity.root.Engineering.Development.user-limit-factor=1
yarn.scheduler.capacity.root.Engineering.maximum-capacity=30
yarn.scheduler.capacity.root.Engineering.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.Engineering.ordering-policy=fifo
yarn.scheduler.capacity.root.Engineering.QA.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.QA.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.QA.capacity=80
yarn.scheduler.capacity.root.Engineering.QA.maximum-capacity=80
yarn.scheduler.capacity.root.Engineering.QA.minimum-user-limit-percent=100
yarn.scheduler.capacity.root.Engineering.QA.ordering-policy=fifo
yarn.scheduler.capacity.root.Engineering.QA.state=RUNNING
yarn.scheduler.capacity.root.Engineering.QA.user-limit-factor=1
yarn.scheduler.capacity.root.Engineering.queues=Development,QA
yarn.scheduler.capacity.root.Engineering.state=RUNNING
yarn.scheduler.capacity.root.Engineering.user-limit-factor=1
yarn.scheduler.capacity.root.Marketing.acl_administer_queue=*