工作流调度系统 Azkaban介绍与安装（一）

mkdir -p /usr/local/azkaban
cd /usr/local/azkaban
rz

tar -zxvf azkaban-db-3.84.4.tar.gz
tar -zxvf azkaban-exec-server-3.84.4.tar.gz
tar -zxvf azkaban-web-server-3.84.4.tar.gz

docker run -d \
-p 3322:3306 \
-v /usr/local/docker/mysql-api/data:/etc/mysql/conf.d \
-v /usr/local/docker/mysql-api/data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=123456* \
--name azkaban mysql:5.7

mysql -u root -p root

create database azkaban;

# 设置密码有效程度 4 为及以上
set global validate_password_length=4;
# 设置密码策略最低级别
set global validate_password_policy=0;

create user 'azkaban'@'%' identified by'000000';

grant select,insert,update,delete on azkaban.* to 'azkaban'@'%' with grant option;

use azkaban;
source /usr/local/azkaban/azkaban-db-3.84.4/create-all-sql-3.84.4.sql

vim /etc/my.cnf

[mysqld]
# 默认1M
max_allowed_packet=1024M

systemctl restart mysqld

vim /usr/local/azkaban/azkaban-exec-server-3.84.4/conf/azkaban.properties

default.timezone.id=Asia/Shanghai

azkaban.webserver.url=http://192.168.56.20:8085

mysql.host=192.168.56.20

mysql.password=root

# 指定端口
executor.port=12321

scp -r /usr/local/azkaban root@slave1:/usr/local
scp -r /usr/local/azkaban root@slave2:/usr/local

cd /usr/local/azkaban/azkaban-exec-server-3.84.4
./bin/start-exec.sh

shutdown-exec.sh

curl -G "192.168.56.20:12321/executor?action=activate" && echo
curl -G "192.168.56.21:12321/executor?action=activate" && echo
curl -G "192.168.56.22:12321/executor?action=activate" && echo

cd /usr/local/azkaban/azkaban-web-server-3.84.4

vim /usr/local/azkaban/azkaban-web-server-3.84.4/conf/azkaban.properties

default.timezone.id=Asia/Shanghai

# Web Server端口
jetty.port=8085

mysql.host=master

mysql.password=000000

azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus

vim ./confg/azkaban-users.xml

<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <user password="123456" roles="admin" username="charles"/>

  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

cd /usr/local/azkaban/azkaban-web-server-3.84.4
bin/start-web.sh

bin/shutdown-web.sh

<h2><a id="_0"></a>前言</h2> <h3><a id="1_2"></a>1、为什么要用工作流调度系统</h3> <p>（1）一个完整的数据分析系统通常都是由大量任务单元组成：shell脚本程序、java程序、MapReduce程序、Hive脚本等。<br /> （2）各个单元之间存在时间先后及前后依赖关系<br /> （3）为了很好地组织这样的复杂执行计划，需要一个工作流调度系统来执行。</p> <h3><a id="2_8"></a>2、常见的工作流调度系统</h3> <p>（1）简单任务调度：直接只用Linux的Crnotab。<br /> （2）复杂任务调度：Ooize、Azkaban、Airflow、DolphinScheduler等</p> <h2><a id="1__14"></a>1 集群规划</h2> <p><code>192.168.56.20</code>: exec、web<br /> <code>192.168.56.20</code>：exec<br /> <code>192.168.56.20</code>：exec</p> <p>准备文件</p> <pre><div class="hljs"><code class="lang-shell">mkdir -p /usr/local/azkaban cd /usr/local/azkaban rz </code></div></pre> <pre><div class="hljs"><code class="lang-shell">tar -zxvf azkaban-db-3.84.4.tar.gz tar -zxvf azkaban-exec-server-3.84.4.tar.gz tar -zxvf azkaban-web-server-3.84.4.tar.gz </code></div></pre> <h2><a id="2__MySQL_33"></a>2 配置 MySQL</h2> <p>1、安装</p> <pre><div class="hljs"><code class="lang-shell">docker run -d \ -p 3322:3306 \ -v /usr/local/docker/mysql-api/data:/etc/mysql/conf.d \ -v /usr/local/docker/mysql-api/data:/var/lib/mysql \ -e MYSQL_ROOT_PASSWORD=123456* \ --name azkaban mysql:5.7 </code></div></pre> <p>2、启动</p> <pre><div class="hljs"><code class="lang-sql">mysql <span class="hljs-operator">-</span>u root <span class="hljs-operator">-</span>p root </code></div></pre> <p>3、创建 azkaban 数据库，并赋予权限</p> <pre><div class="hljs"><code class="lang-shell">create database azkaban; </code></div></pre> <p>配置允许简单密码</p> <pre><div class="hljs"><code class="lang-shell"><span class="hljs-meta"># </span><span class="language-bash">设置密码有效程度 4 为及以上</span> set global validate_password_length=4; <span class="hljs-meta"># </span><span class="language-bash">设置密码策略最低级别</span> set global validate_password_policy=0; </code></div></pre> <p>创建 azkaban用户，任何主机都可以访问 Azkaban，密码是000000</p> <pre><div class="hljs"><code class="lang-shell">create user &#x27;azkaban&#x27;@&#x27;%&#x27; identified by&#x27;000000&#x27;; </code></div></pre> <p>赋予 azkaban用户 增删改查权限</p> <pre><div class="hljs"><code class="lang-shell">grant select,insert,update,delete on azkaban.* to &#x27;azkaban&#x27;@&#x27;%&#x27; with grant option; </code></div></pre> <p>4、创建 azkaban表</p> <pre><div class="hljs"><code class="lang-shell">use azkaban; source /usr/local/azkaban/azkaban-db-3.84.4/create-all-sql-3.84.4.sql </code></div></pre> <p>5、更改 MySql 包大小，防止 Azkaban 链接 MySql阻塞</p> <pre><div class="hljs"><code class="lang-shell">vim /etc/my.cnf </code></div></pre> <p>在<code>[mysqld]</code>下面家以上 <code>max_allowed_pocked=1024M</code></p> <pre><code class="lang-conf">[mysqld] # 默认1M max_allowed_packet=1024M </code></pre> <p>6、重启 MySQL</p> <pre><div class="hljs"><code class="lang-shell">systemctl restart mysqld </code></div></pre> <h2><a id="3__Executor_Server_98"></a>3 配置 Executor Server</h2> <h3><a id="31__azkabanproperties_100"></a>3.1 修改 azkaban.properties</h3> <pre><div class="hljs"><code class="lang-shell">vim /usr/local/azkaban/azkaban-exec-server-3.84.4/conf/azkaban.properties </code></div></pre> <p>修改如下内容</p> <pre><div class="hljs"><code class="lang-shell">default.timezone.id=Asia/Shanghai azkaban.webserver.url=http://192.168.56.20:8085 mysql.host=192.168.56.20 mysql.password=root <span class="hljs-meta"> # </span><span class="language-bash">指定端口</span> executor.port=12321 </code></div></pre> <p>分发<br /> xsync</p> <pre><div class="hljs"><code class="lang-shell">scp -r /usr/local/azkaban root@slave1:/usr/local scp -r /usr/local/azkaban root@slave2:/usr/local </code></div></pre> <h3><a id="32__127"></a>3.2 启动</h3> <p>所有机器</p> <pre><div class="hljs"><code class="lang-shell">cd /usr/local/azkaban/azkaban-exec-server-3.84.4 ./bin/start-exec.sh </code></div></pre> <p>关闭</p> <pre><div class="hljs"><code class="lang-shell">shutdown-exec.sh </code></div></pre> <h3><a id="33__140"></a>3.3 激活</h3> <p>所有机器</p> <pre><div class="hljs"><code class="lang-shell">curl -G &quot;192.168.56.20:12321/executor?action=activate&quot; &amp;&amp; echo curl -G &quot;192.168.56.21:12321/executor?action=activate&quot; &amp;&amp; echo curl -G &quot;192.168.56.22:12321/executor?action=activate&quot; &amp;&amp; echo </code></div></pre> <h2><a id="4__Web_Server_150"></a>4 配置 Web Server</h2> <p>Azkaban Web Server 处理项目管理，身份验证，计划和执行触发。</p> <h3><a id="41__azkabanproperties_154"></a>4.1 修改 azkaban.properties</h3> <pre><div class="hljs"><code class="lang-shell">cd /usr/local/azkaban/azkaban-web-server-3.84.4 vim /usr/local/azkaban/azkaban-web-server-3.84.4/conf/azkaban.properties </code></div></pre> <p>修改如下属性</p> <pre><div class="hljs"><code class="lang-shell">default.timezone.id=Asia/Shanghai <span class="hljs-meta"> # </span><span class="language-bash">Web Server端口</span> jetty.port=8085 mysql.host=master mysql.password=000000 azkaban.executorselector.filters=StaticRemainingFlowSize,CpuStatus </code></div></pre> <h3><a id="42_azkabanusersxml_atguigu__176"></a>4.2 修改azkaban-users.xml文件，添加 atguigu 用户</h3> <p>用户/权利管理文件</p> <pre><div class="hljs"><code class="lang-shell">vim ./confg/azkaban-users.xml </code></div></pre> <pre><div class="hljs"><code class="lang-shell">&lt;azkaban-users&gt; &lt;user groups=&quot;azkaban&quot; password=&quot;azkaban&quot; roles=&quot;admin&quot; username=&quot;azkaban&quot;/&gt; &lt;user password=&quot;metrics&quot; roles=&quot;metrics&quot; username=&quot;metrics&quot;/&gt; &lt;user password=&quot;123456&quot; roles=&quot;admin&quot; username=&quot;charles&quot;/&gt; &lt;role name=&quot;admin&quot; permissions=&quot;ADMIN&quot;/&gt; &lt;role name=&quot;metrics&quot; permissions=&quot;METRICS&quot;/&gt; &lt;/azkaban-users&gt; </code></div></pre> <h3><a id="43__195"></a>4.3 启动</h3> <pre><div class="hljs"><code class="lang-shell">cd /usr/local/azkaban/azkaban-web-server-3.84.4 bin/start-web.sh </code></div></pre> <p>关闭</p> <pre><div class="hljs"><code class="lang-shell">bin/shutdown-web.sh </code></div></pre> <p>访问：http://192.168.56.20:8085<br /> 账号：charles<br /> 密码：123456<br /> <img src="https://static.couragesteak.com/article/40ec788364156a324fb87073e25db41d.png" alt="image.png" /></p> <p>https://www.bilibili.com/video/BV1y54y18713</p>