一. 部署读写分离的契机
目前公司整体项目稳定运行在 CDH5.6 版本上, 与其搭配的 Hbase1.0.0 无法正确运行 Kylin, 原因是 Kylin 只满足 Hbase1.1.x + 版本. 解决方案如下
1. 升级整体 CDH 版本, 从而获得高版本 Hbase(方案风险太大)
2. 把 Hbase 从 CDH 单独剥离出来, 用原生的 Hbase 高版本替代 (方案缺点是管理 Hbase 不方便, 原有的应用难迁移)
3. Kylin 读写分离 (经验证, CDH5.6 的 Hbase 支持 Kylin 建 CUBE, 但无法读 (API 不兼容), 所以只需在另一个集群配置高版本的 Hbase 即可解决问题, 方案高可行, 因为既不影响现有的应用, 也提高了 Kylin 的高可用性, 一举两得)
二. 环境说明
从上图可看出, Kylin 支持读写分离, 但其设计的初衷是为了分离集群压力, 读和写分离, 实现高速稳定可用.
当我们在前段发现建 CUBE 请求时, Build 操作在计算集群实现, 计算 CUBE 之后把它 load 到 Hbase 集群, 最后转成 HFILE 到 Hbase, 从而提供前端读. 具体到目前我的环境, 可把上图抽象为:
Kylin 版本: apache-kylin-2.4.0-bin-cdh57
集群 机器 IP 机器名称 备注
CDH5.6 10.5.8.10 see-data-pre-master-01 集群 A 主 (CDH5.6)
CDH5.6 10.5.8.17 see-data-pre-slave-1 集群 A 从
CDH5.15.0 10.5.8.12 test-data-master-1 集群 B 主 (CDH5.15.0)
CDH5.15.0 10.5.8.6 test-data-slave-1 集群 B 从
CDH5.15.0 10.5.8.7 test-data-slave-2 集群 B 从
后面我们把 CDH5.6 集群简述为集群 A,CDH5.15.0 简述为集群 B
三. 部署思路
部署 Kylin 的读写分离, 顾名思义是把写的操作指向集群 A, 读操作指向集群 B, 反映到配置上, 其实就是:
1. 把集群 A 中的 Hadooo\MR\Hive\Yarn 配置复制到部署 Kylin 的配置目录
2. 把集群 B 中的 Hbase 配置文件复制到 Kylin 的配置目录
3. 配置 Kylin.property 文件中对集群 A 和集群 B 的指针属性
四. 部署过程
1. 首先保证两个集群的所有机器都配置完域名映射, 可免密访问, 保证两集群可正常运行.
2. Kylin 下载解压后放在集群 B 机器 test-data-slave-2 的 / home/hadoop/kylin/apache-kylin-2.4.0-bin-cdh57 目录下 =$KYLIN_HOME
3. 所有配置文件复制到 $KYLIN_HOME(CDH 的配置文件都默认放在 / etc/hadoop/conf; /etc/hive/conf; ....)
把集群 A 的 / etc/hadoop/conf 下的 core-site.xml,hdfs-site.xml,yarn-site.xml,mapred-site.xml 复制到 $ YLIN_HOME 的 conf 目录下
把集群 A 的 / etc/hive/conf 下的 hive-site.xml 复制到 $KYLIN_HOME 的 conf 目录下
把集群 B 的 / etc/hbase/conf 下的 hbase-site.xml 复制到 $KYLIN_HOME 的 conf 目录下
原则上, 这些从集群拷贝的配置文件都不需要改, 但是如果 hdfs 或者 hive 的指向地址为本地地址, 就需要改成远程访问地址!
- [hadoop@test-data-slave-2 conf]$ ll
- total 76
- -rw-r--r-- 1 hadoop data 3865 Dec 11 15:37 core-site.xml
- -rw-r--r-- 1 hadoop data 2926 Dec 11 15:42 hbase-site.xml
- -rw-r--r-- 1 hadoop data 1748 Dec 11 15:37 hdfs-site.xml
- -rw-r--r-- 1 hadoop data 5517 Dec 11 15:41 hive-site.xml
- -rw-r--r-- 1 hadoop data 3605 Jun 20 15:53 kylin_hive_conf.xml
- -rw-r--r-- 1 hadoop data 3807 Jun 20 15:53 kylin_job_conf_inmem.xml
- -rw-r--r-- 1 hadoop data 3386 Dec 12 11:08 kylin_job_conf.xml
- -rw-r--r-- 1 hadoop data 1156 Jun 20 15:53 kylin-kafka-consumer.xml
- -rw-r--r-- 1 hadoop data 13112 Dec 11 20:35 kylin.properties
- -rw-r--r-- 1 hadoop data 1339 Jun 20 15:53 kylin-server-log4j.properties
- -rw-r--r-- 1 hadoop data 1656 Jun 20 15:53 kylin-tools-log4j.properties
- -rw-r--r-- 1 hadoop data 4563 Dec 11 15:40 mapred-site.xml
- -rwxr-xr-x 1 hadoop data 3649 Jun 20 15:53 setenv.sh
- -rw-r--r-- 1 hadoop data 3828 Dec 11 15:39 yarn-site.xml
以下是各个主要文件的配置信息:
- core-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>fs.defaultFS</name>
- <value>hdfs://see-data-pre-master-01:8020</value>
- </property>
- <property>
- <name>fs.trash.interval</name>
- <value>1</value>
- </property>
- <property>
- <name>io.compression.codecs</name>
- <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
- </property>
- <property>
- <name>hadoop.security.authentication</name>
- <value>simple</value>
- </property>
- <property>
- <name>hadoop.security.authorization</name>
- <value>false</value>
- </property>
- <property>
- <name>hadoop.rpc.protection</name>
- <value>authentication</value>
- </property>
- <property>
- <name>hadoop.security.auth_to_local</name>
- <value>DEFAULT</value>
- </property>
- <property>
- <name>hadoop.proxyuser.oozie.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.oozie.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.mapred.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.mapred.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.flume.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.flume.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.HTTP.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.HTTP.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hive.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hive.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hue.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hue.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.httpfs.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.httpfs.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hdfs.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.hdfs.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.yarn.hosts</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.proxyuser.yarn.groups</name>
- <value>*</value>
- </property>
- <property>
- <name>hadoop.security.group.mapping</name>
- <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
- </property>
- <property>
- <name>hadoop.security.instrumentation.requires.admin</name>
- <value>false</value>
- </property>
- <property>
- <name.NET.topology.script.file.name</name>
- <value>/etc/hadoop/conf.cloudera.yarn/topology.py</value>
- </property>
- <property>
- <name>io.file.buffer.size</name>
- <value>65536</value>
- </property>
- <property>
- <name>hadoop.ssl.enabled</name>
- <value>false</value>
- </property>
- <property>
- <name>hadoop.ssl.require.client.cert</name>
- <value>false</value>
- <final>true</final>
- </property>
- <property>
- <name>hadoop.ssl.keystores.factory.class</name>
- <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
- <final>true</final>
- </property>
- <property>
- <name>hadoop.ssl.server.conf</name>
- <value>ssl-server.xml</value>
- <final>true</final>
- </property>
- <property>
- <name>hadoop.ssl.client.conf</name>
- <value>ssl-client.xml</value>
- <final>true</final>
- </property>
- </configuration>
- hbase-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>hbase.rootdir</name>
- <value>hdfs://test-data-master-1:8020/hbase_test</value>
- </property>
- <property>
- <name>hbase.client.write.buffer</name>
- <value>2097152</value>
- </property>
- <property>
- <name>hbase.client.pause</name>
- <value>100</value>
- </property>
- <property>
- <name>hbase.client.retries.number</name>
- <value>35</value>
- </property>
- <property>
- <name>hbase.client.scanner.caching</name>
- <value>100</value>
- </property>
- <property>
- <name>hbase.client.keyvalue.maxsize</name>
- <value>10485760</value>
- </property>
- <property>
- <name>hbase.ipc.client.allowsInterrupt</name>
- <value>true</value>
- </property>
- <property>
- <name>hbase.client.primaryCallTimeout.get</name>
- <value>10</value>
- </property>
- <property>
- <name>hbase.client.primaryCallTimeout.multiget</name>
- <value>10</value>
- </property>
- <property>
- <name>hbase.fs.tmp.dir</name>
- <value>/user/${user.name}/hbase-staging</value>
- </property>
- <property>
- <name>hbase.client.scanner.timeout.period</name>
- <value>60000</value>
- </property>
- <property>
- <name>hbase.coprocessor.region.classes</name>
- <value>org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint</value>
- </property>
- <property>
- <name>hbase.regionserver.thrift.http</name>
- <value>false</value>
- </property>
- <property>
- <name>hbase.thrift.support.proxyuser</name>
- <value>false</value>
- </property>
- <property>
- <name>hbase.rpc.timeout</name>
- <value>60000</value>
- </property>
- <property>
- <name>hbase.snapshot.enabled</name>
- <value>true</value>
- </property>
- <property>
- <name>hbase.snapshot.master.timeoutMillis</name>
- <value>60000</value>
- </property>
- <property>
- <name>hbase.snapshot.region.timeout</name>
- <value>60000</value>
- </property>
- <property>
- <name>hbase.snapshot.master.timeout.millis</name>
- <value>60000</value>
- </property>
- <property>
- <name>hbase.security.authentication</name>
- <value>simple</value>
- </property>
- <property>
- <name>hbase.rpc.protection</name>
- <value>authentication</value>
- </property>
- <property>
- <name>zookeeper.session.timeout</name>
- <value>60000</value>
- </property>
- <property>
- <name>zookeeper.znode.parent</name>
- <value>/hbase_test</value>
- </property>
- <property>
- <name>zookeeper.znode.rootserver</name>
- <value>root-region-server-test</value>
- </property>
- <property>
- <name>hbase.zookeeper.quorum</name>
- <value>test-data-master-1,test-data-slave-2,test-data-slave-1</value>
- </property>
- <property>
- <name>hbase.zookeeper.property.clientPort</name>
- <value>2181</value>
- </property>
- <property>
- <name>hbase.REST.ssl.enabled</name>
- <value>false</value>
- </property>
- </configuration>
- hdfs-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>dfs.namenode.name.dir</name>
- <value>file:///dfs/nn</value>
- </property>
- <property>
- <name>dfs.namenode.servicerpc-address</name>
- <value>see-data-pre-master-01:8022</value>
- </property>
- <property>
- <name>dfs.https.address</name>
- <value>see-data-pre-master-01:50470</value>
- </property>
- <property>
- <name>dfs.https.port</name>
- <value>50470</value>
- </property>
- <property>
- <name>dfs.namenode.http-address</name>
- <value>see-data-pre-master-01:50070</value>
- </property>
- <property>
- <name>dfs.replication</name>
- <value>3</value>
- </property>
- <property>
- <name>dfs.blocksize</name>
- <value>134217728</value>
- </property>
- <property>
- <name>dfs.client.use.datanode.hostname</name>
- <value>false</value>
- </property>
- <property>
- <name>fs.permissions.umask-mode</name>
- <value>022</value>
- </property>
- <property>
- <name>dfs.namenode.acls.enabled</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.client.use.legacy.blockreader</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.client.read.shortcircuit</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.domain.socket.path</name>
- <value>/var/run/hdfs-sockets/dn</value>
- </property>
- <property>
- <name>dfs.client.read.shortcircuit.skip.checksum</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.client.domain.socket.data.traffic</name>
- <value>false</value>
- </property>
- <property>
- <name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
- <value>true</value>
- </property>
- </configuration>
- hive-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>hive.metastore.uris</name>
- <value>thrift://see-data-pre-master-01:9083</value>
- </property>
- <property>
- <name>hive.metastore.client.socket.timeout</name>
- <value>300</value>
- </property>
- <property>
- <name>hive.metastore.warehouse.dir</name>
- <value>/user/hive/warehouse</value>
- </property>
- <property>
- <name>hive.warehouse.subdir.inherit.perms</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.enable.spark.execution.engine</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.conf.restricted.list</name>
- <value>hive.enable.spark.execution.engine</value>
- </property>
- <property>
- <name>hive.auto.convert.join</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.auto.convert.join.noconditionaltask.size</name>
- <value>20971520</value>
- </property>
- <property>
- <name>hive.optimize.bucketmapjoin.sortedmerge</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.smbjoin.cache.rows</name>
- <value>10000</value>
- </property>
- <property>
- <name>mapred.reduce.tasks</name>
- <value>-1</value>
- </property>
- <property>
- <name>hive.exec.reducers.bytes.per.reducer</name>
- <value>67108864</value>
- </property>
- <property>
- <name>hive.exec.copyfile.maxsize</name>
- <value>33554432</value>
- </property>
- <property>
- <name>hive.exec.reducers.max</name>
- <value>1099</value>
- </property>
- <property>
- <name>hive.vectorized.groupby.checkinterval</name>
- <value>4096</value>
- </property>
- <property>
- <name>hive.vectorized.groupby.flush.percent</name>
- <value>0.1</value>
- </property>
- <property>
- <name>hive.compute.query.using.stats</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.vectorized.execution.enabled</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.vectorized.execution.reduce.enabled</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.merge.mapfiles</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.merge.mapredfiles</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.cbo.enable</name>
- <value>false</value>
- </property>
- <property>
- <name>hive.fetch.task.conversion</name>
- <value>minimal</value>
- </property>
- <property>
- <name>hive.fetch.task.conversion.threshold</name>
- <value>268435456</value>
- </property>
- <property>
- <name>hive.limit.pushdown.memory.usage</name>
- <value>0.1</value>
- </property>
- <property>
- <name>hive.merge.sparkfiles</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.merge.smallfiles.avgsize</name>
- <value>16777216</value>
- </property>
- <property>
- <name>hive.merge.size.per.task</name>
- <value>268435456</value>
- </property>
- <property>
- <name>hive.optimize.reducededuplication</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.optimize.reducededuplication.min.reducer</name>
- <value>4</value>
- </property>
- <property>
- <name>hive.map.aggr</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.map.aggr.hash.percentmemory</name>
- <value>0.5</value>
- </property>
- <property>
- <name>hive.optimize.sort.dynamic.partition</name>
- <value>false</value>
- </property>
- <property>
- <name>spark.executor.memory</name>
- <value>268435456</value>
- </property>
- <property>
- <name>spark.driver.memory</name>
- <value>268435456</value>
- </property>
- <property>
- <name>spark.executor.cores</name>
- <value>1</value>
- </property>
- <property>
- <name>spark.yarn.driver.memoryOverhead</name>
- <value>26</value>
- </property>
- <property>
- <name>spark.yarn.executor.memoryOverhead</name>
- <value>26</value>
- </property>
- <property>
- <name>spark.dynamicAllocation.enabled</name>
- <value>true</value>
- </property>
- <property>
- <name>spark.dynamicAllocation.initialExecutors</name>
- <value>1</value>
- </property>
- <property>
- <name>spark.dynamicAllocation.minExecutors</name>
- <value>1</value>
- </property>
- <property>
- <name>spark.dynamicAllocation.maxExecutors</name>
- <value>2147483647</value>
- </property>
- <property>
- <name>hive.metastore.execute.setugi</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.support.concurrency</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.zookeeper.quorum</name>
- <value>see-data-pre-master-01</value>
- </property>
- <property>
- <name>hive.zookeeper.client.port</name>
- <value>2181</value>
- </property>
- <property>
- <name>hive.zookeeper.namespace</name>
- <value>hive_zookeeper_namespace_hive</value>
- </property>
- <property>
- <name>hbase.zookeeper.quorum</name>
- <value>see-data-pre-master-01</value>
- </property>
- <property>
- <name>hbase.zookeeper.property.clientPort</name>
- <value>2181</value>
- </property>
- <property>
- <name>hive.cluster.delegation.token.store.class</name>
- <value>org.apache.hadoop.hive.thrift.MemoryTokenStore</value>
- </property>
- <property>
- <name>hive.server2.enable.doAs</name>
- <value>true</value>
- </property>
- <property>
- <name>hive.server2.use.SSL</name>
- <value>false</value>
- </property>
- <property>
- <name>spark.shuffle.service.enabled</name>
- <value>true</value>
- </property>
- </configuration>
- mapred-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>mapreduce.job.split.metainfo.maxsize</name>
- <value>10000000</value>
- </property>
- <property>
- <name>mapreduce.job.counters.max</name>
- <value>120</value>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress</name>
- <value>false</value>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.type</name>
- <value>BLOCK</value>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.DefaultCodec</value>
- </property>
- <property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- </property>
- <property>
- <name>mapreduce.map.output.compress</name>
- <value>true</value>
- </property>
- <property>
- <name>zlib.compress.level</name>
- <value>DEFAULT_COMPRESSION</value>
- </property>
- <property>
- <name>mapreduce.task.io.sort.factor</name>
- <value>64</value>
- </property>
- <property>
- <name>mapreduce.map.sort.spill.percent</name>
- <value>0.8</value>
- </property>
- <property>
- <name>mapreduce.reduce.shuffle.parallelcopies</name>
- <value>10</value>
- </property>
- <property>
- <name>mapreduce.task.timeout</name>
- <value>600000</value>
- </property>
- <property>
- <name>mapreduce.client.submit.file.replication</name>
- <value>1</value>
- </property>
- <property>
- <name>mapreduce.job.reduces</name>
- <value>5</value>
- </property>
- <property>
- <name>mapreduce.task.io.sort.mb</name>
- <value>256</value>
- </property>
- <property>
- <name>mapreduce.map.speculative</name>
- <value>false</value>
- </property>
- <property>
- <name>mapreduce.reduce.speculative</name>
- <value>false</value>
- </property>
- <property>
- <name>mapreduce.job.reduce.slowstart.completedmaps</name>
- <value>0.8</value>
- </property>
- <property>
- <name>mapreduce.jobhistory.address</name>
- <value>see-data-pre-master-01:10020</value>
- </property>
- <property>
- <name>mapreduce.jobhistory.webapp.address</name>
- <value>see-data-pre-master-01:19888</value>
- </property>
- <property>
- <name>mapreduce.jobhistory.webapp.https.address</name>
- <value>see-data-pre-master-01:19890</value>
- </property>
- <property>
- <name>mapreduce.jobhistory.admin.address</name>
- <value>see-data-pre-master-01:10033</value>
- </property>
- <property>
- <name>mapreduce.framework.name</name>
- <value>yarn</value>
- </property>
- <property>
- <name>yarn.App.mapreduce.am.staging-dir</name>
- <value>/user</value>
- </property>
- <property>
- <name>mapreduce.am.max-attempts</name>
- <value>2</value>
- </property>
- <property>
- <name>yarn.App.mapreduce.am.resource.mb</name>
- <value>1024</value>
- </property>
- <property>
- <name>yarn.App.mapreduce.am.resource.CPU-vcores</name>
- <value>1</value>
- </property>
- <property>
- <name>mapreduce.job.ubertask.enable</name>
- <value>false</value>
- </property>
- <property>
- <name>yarn.App.mapreduce.am.command-opts</name>
- <value>-Djava.NET.preferIPv4Stack=true -Xmx825955249</value>
- </property>
- <property>
- <name>mapreduce.map.java.opts</name>
- <value>-Djava.NET.preferIPv4Stack=true</value>
- </property>
- <property>
- <name>mapreduce.reduce.java.opts</name>
- <value>-Djava.NET.preferIPv4Stack=true</value>
- </property>
- <property>
- <name>yarn.App.mapreduce.am.admin.user.env</name>
- <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value>
- </property>
- <property>
- <name>mapreduce.map.memory.mb</name>
- <value>0</value>
- </property>
- <property>
- <name>mapreduce.map.CPU.vcores</name>
- <value>1</value>
- </property>
- <property>
- <name>mapreduce.reduce.memory.mb</name>
- <value>0</value>
- </property>
- <property>
- <name>mapreduce.reduce.CPU.vcores</name>
- <value>1</value>
- </property>
- <property>
- <name>mapreduce.job.heap.memory-mb.ratio</name>
- <value>0.8</value>
- </property>
- <property>
- <name>mapreduce.application.classpath</name>
- <value>$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$MR2_CLASSPATH</value>
- </property>
- <property>
- <name>mapreduce.admin.user.env</name>
- <value>LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH</value>
- </property>
- <property>
- <name>mapreduce.shuffle.max.connections</name>
- <value>80</value>
- </property>
- </configuration>
- yarn-site.xml
- <?xml version="1.0" encoding="UTF-8"?>
- <!--Autogenerated by Cloudera Manager-->
- <configuration>
- <property>
- <name>yarn.acl.enable</name>
- <value>true</value>
- </property>
- <property>
- <name>yarn.admin.acl</name>
- <value>*</value>
- </property>
- <property>
- <name>yarn.resourcemanager.address</name>
- <value>see-data-pre-master-01:8032</value>
- </property>
- <property>
- <name>yarn.resourcemanager.admin.address</name>
- <value>see-data-pre-master-01:8033</value>
- </property>
- <property>
- <name>yarn.resourcemanager.scheduler.address</name>
- <value>see-data-pre-master-01:8030</value>
- </property>
- <property>
- <name>yarn.resourcemanager.resource-tracker.address</name>
- <value>see-data-pre-master-01:8031</value>
- </property>
- <property>
- <name>yarn.resourcemanager.webapp.address</name>
- <value>see-data-pre-master-01:8088</value>
- </property>
- <property>
- <name>yarn.resourcemanager.webapp.https.address</name>
- <value>see-data-pre-master-01:8090</value>
- </property>
- <property>
- <name>yarn.resourcemanager.client.thread-count</name>
- <value>50</value>
- </property>
- <property>
- <name>yarn.resourcemanager.scheduler.client.thread-count</name>
- <value>50</value>
- </property>
- <property>
- <name>yarn.resourcemanager.admin.client.thread-count</name>
- <value>1</value>
- </property>
- <property>
- <name>yarn.scheduler.minimum-allocation-mb</name>
- <value>1024</value>
- </property>
- <property>
- <name>yarn.scheduler.increment-allocation-mb</name>
- <value>512</value>
- </property>
- <property>
- <name>yarn.scheduler.maximum-allocation-mb</name>
- <value>3374</value>
- </property>
- <property>
- <name>yarn.scheduler.minimum-allocation-vcores</name>
- <value>1</value>
- </property>
- <property>
- <name>yarn.scheduler.increment-allocation-vcores</name>
- <value>1</value>
- </property>
- <property>
- <name>yarn.scheduler.maximum-allocation-vcores</name>
- <value>8</value>
- </property>
- <property>
- <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name>
- <value>1000</value>
- </property>
- <property>
- <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
- <value>600000</value>
- </property>
- <property>
- <name>yarn.resourcemanager.am.max-attempts</name>
- <value>2</value>
- </property>
- <property>
- <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
- <value>600000</value>
- </property>
- <property>
- <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
- <value>1000</value>
- </property>
- <property>
- <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
- <value>600000</value>
- </property>
- <property>
- <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
- <value>50</value>
- </property>
- <property>
- <name>yarn.application.classpath</name>
- <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
- </property>
- <property>
- <name>yarn.resourcemanager.scheduler.class</name>
- <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
- </property>
- <property>
- <name>yarn.scheduler.fair.user-as-default-queue</name>
- <value>true</value>
- </property>
- <property>
- <name>yarn.scheduler.fair.preemption</name>
- <value>false</value>
- </property>
- <property>
- <name>yarn.scheduler.fair.sizebasedweight</name>
- <value>false</value>
- </property>
- <property>
- <name>yarn.scheduler.fair.assignmultiple</name>
- <value>false</value>
- </property>
- <property>
- <name>yarn.resourcemanager.max-completed-applications</name>
- <value>10000</value>
- </property>
- </configuration>
4. 在 $KYLIN_HOME/conf / 下的 kylin.properties 中追加以下设置
- kylin.source.hive.beeline-shell=beeline
- kylin.source.hive.beeline-params=-n hadoop --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*' -u jdbc:hive2://see-data-pre-master-01:10000
- # 重要: 这是通过 beeline 向集群 A 的 Hive 指定 Kylin 计算过程中产生的中间表存储的数据库
- kylin.source.hive.database-for-flat-table=kylin
- kylin.source.hive.redistribute-flat-table=true
- kylin.storage.url=hbase
- kylin.storage.hbase.cluster-fs=hdfs://test-data-master-1:8020
- # 重要: 这是集群 B 的 zookeeper 节点, Hbase 要依赖 zk, 需要加上
- kylin.env.zookeeper-connect-string=test-data-master-1,test-data-slave-2,test-data-slave-1
5. 配置环境变量
在安装 Kylin 的机器上配置 ~/.bashrc 文件, 追加以下内容
- # hadoop
- export CONF_HOME=/home/hadoop/kylin/apache-kylin-2.4.0-bin-cdh57/conf
- export HBASE_CONF=$CONF_HOME
- export HBASE_CONF_DIR=$CONF_HOME
- export HADOOP_CONF_DIR=$CONF_HOME
- export HIVE_CONF=$CONF_HOME
- export HIVE_CONF_DIR=$CONF_HOME
- #added by Hive hcatalog
- export HCAT_HOME=/opt/cloudera/parcels/CDH/lib/hive-hcatalog
- #add by KYLIN
- export KYLIN_HOME=/home/hadoop/kylin/apache-kylin-2.4.0-bin-cdh57
- export PATH=$KYLIN_HOME/bin:$PATH
这个环境变量告诉 Kylin 不取本机的 hadoop 计算, 重要!
编辑完后 source ~/.bashrc 以下让其生效!
6. 单服务验证以上配置是否正确
在集群 B, 安装 Kylin 的机器下执行以下操作以确定是否都指向了集群 A
验证 HDFS, 以下的结果是集群 A 上的 HDFS 目录
- [hadoop@test-data-slave-2 conf]$ hdfs dfs -ls /user/hive/warehouse/
- Found 2 items
- drwxrwxrwt - hadoop hive 0 2018-12-11 19:23 /user/hive/warehouse/kylin.db
- drwxrwxrwt - superuser hive 0 2018-12-12 10:53 /user/hive/warehouse/test_default
验证 HIVE, 打开 HIVE CLI, 是集群 A 的 hive 数据库
- hive> show databases;
- OK
- default
- kylin
- Time taken: 1.78 seconds, Fetched: 2 row(s)
验证 YARN,Running 列表里头的两台机器是集群 A 的
- 2018-12-12 18:14:55,885 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(123)) - Connecting to ResourceManager at see-data-pre-master-01/10.5.8.10:8032
- Total Nodes:2
- Node-Id Node-State Node-Http-Address Number-of-Running-Containers
- see-data-pre-slave-1:8041 RUNNING see-data-pre-slave-1:8042 0
- see-data-pre-master-01:8041 RUNNING see-data-pre-master-01:8042 0
7. 到 $KYLIN_HOME 下运行 Kylin
./kylin.sh start
8. 打开 Kylin UI, 执行 demo 建 cube , 到集群 A 的 CDH Yarn 页面查看建 CUBE 的 MR 任务
在 Kylin UI 中查看 CUBE 保存后的 Kylin 表
到集群 B 的 Hbase 中验证
至此搭建 Kylin 读写分离完成.
来源: https://www.cnblogs.com/yongjian/p/10110119.html