2025莞纺大数据竞赛队
CentOS基础配置(Ver.8)
Linux虚拟机IP分配
Hadoop+JDK配置
Zookeeper集群配置
ChenJiaYing-第一周工作汇报
Kafka配置
Hadoop搭建
数据可视化总结
ChenJiaYing-第三周工作汇报
数据分析总结
本文档使用 MrDoc 发布
-
+
首页
Hadoop搭建
### IP固定 nmtui ### 主机名修改 ``` hostnamectl set-hostname master ``` ### 关闭防火墙 ``` systemctl stop firewalld systemctl disable firewalld ``` ### 映射文件修改 ``` /etc/hosts 10.155.151.99 a01 ``` ### 用户创建 ``` useradd hadoop passwd hadoop ``` ### 用户提权、用户组提权 ``` Hadoop:root sudo visudo hadoop ALL=(ALL) NOPASSWD: ALL ``` ### 免密登录 ``` ssh-keygen -t rsa -b 4096 ssh-sopy-id a01 ``` ### 创建文件夹 ``` /export/server ``` ### 解压 ``` Hadoop、jdk ``` ### 创建软连接 ``` ln -s ``` ### 基础环境变量搭建 ``` $HADOOP_HOME、$JAVA_HOME ``` ### HDFS服务搭建 ### 配置环境变量 ##### 配置HDFS集群,我们主要涉及到如下文件的修改: ``` workers hadoop-env.sh core-site.xml hdfs-site.xml ``` ###### 均存在$HADOOP_HOME/etc/hadoop ``` workers: a01 a02 a03 ``` ### 各节点主机名 ```hadoop-env.sh: ###vim hadoop-env.sh### export JAVA_HOME=/export/server/jdk export HADOOP_HOME=/export/server/hadoop export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export HADOOP_LOG_DIR=$HADOOP_HOME/logs ``` ```core-site.xml: <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://node1:8020</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> </configuration> ``` ```hdfs-site.xml: ###vim hdfs-site.xml### <configuration> <property> <name>dfs.datanode.data.dir.perm</name> <value>700</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/data/nn</value> </property> <property> <name>dfs.namenode.hosts</name> <value>node1,node2,node3</value> </property> <property> <name>dfs.blocksize</name> <value>268435456</value> </property> <property> <name>dfs.namenode.handler.count</name> <value>100</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/data/dn</value> </property> </configuration> ``` ### 运行dfs服务 start-dfs.sh ### yarn服务搭建 ### 基础配置 #### yarn相关配置 ```yarn-env.sh JAVA_HOEM HADOOP_HOME HADOOP_CONF_DIR HADOOP_LOG_DIR ``` ```yarn-site.xml <configuration> <property> <name>yarn.resourcemanager.hostname</name> <value>a01</value> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/data/nm-local</value> </property> <property> <name>yarn.nodemanager,log-dirs</name> <value>/data/nm-log</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce</value> </property> <property> <name>yarn.log.server.url</name> <value>http://a01:19888/jobhistory/logs</value> </property> <property> <name>yarn.wed-proxy.address</name> <value>a01:8089</value> </property> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/tmp/logs</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchduler</value> </property </configuration> ``` #### mapreduce相关配置 ```mapreduce-env.sh export JAVA_HOME=/ export/server/jdk.* export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000 export HADOOP_MADRED_ROOT_LOGGER=INFO,RFA ``` ```mapreduce-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>a01:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>a01:19888</value> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/data/mr-history/tmp</value> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/data/mr-history/done</value> </property> <property> <name>yarn.app.mapreduce.am.env</name> <value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.map.env</name> <value>HADOOP_MADRED_HOME=$HADOOP_HOME</value> </property> <property> <name>mapreduce.reduce.env</name> <value>HADOOP_MADRED_HOME=$HADOOP_HOME</value> </property> ``` ### hive服务搭建 ### MySQL服务搭建 ### 解压MySQL-*.rpm包 #### 严格按照依赖顺序安装 ### 启动MySQL服务 ### 启动hive服务
MARKJY
2025年10月31日 13:50
转发文档
收藏文档
上一篇
下一篇
手机扫码
复制链接
手机扫一扫转发分享
复制链接
Markdown文件
PDF文档(打印)
分享
链接
类型
密码
更新密码