Hadoop搭建

### IP固定
nmtui
### 主机名修改
```
hostnamectl set-hostname master
```
### 关闭防火墙
```
systemctl stop firewalld
systemctl disable firewalld
```
### 映射文件修改
```
/etc/hosts
10.155.151.99 a01
```
### 用户创建
```
useradd hadoop
passwd hadoop
```
### 用户提权、用户组提权
```
Hadoop：root
sudo visudo
hadoop ALL=(ALL) NOPASSWD: ALL
```
### 免密登录
```
ssh-keygen -t rsa -b 4096
ssh-sopy-id a01
```
### 创建文件夹
```
/export/server
```
### 解压
```
Hadoop、jdk
```
### 创建软连接
```
ln -s
```
### 基础环境变量搭建
```
$HADOOP_HOME、$JAVA_HOME
```
### HDFS服务搭建
### 配置环境变量
##### 配置HDFS集群，我们主要涉及到如下文件的修改：
```
workers
hadoop-env.sh
core-site.xml
hdfs-site.xml
```
###### 均存在$HADOOP_HOME/etc/hadoop
```
workers：
a01
a02
a03
```
### 各节点主机名
```hadoop-env.sh：
###vim hadoop-env.sh###
export JAVA_HOME=/export/server/jdk
export HADOOP_HOME=/export/server/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
```
```core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://node1:8020</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
```
```hdfs-site.xml:
###vim hdfs-site.xml###
<configuration>
<property>
<name>dfs.datanode.data.dir.perm</name>
<value>700</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/data/nn</value>
</property>
<property>
<name>dfs.namenode.hosts</name>
<value>node1,node2,node3</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>268435456</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data/dn</value>
</property>
</configuration>
```
### 运行dfs服务
start-dfs.sh
### yarn服务搭建
### 基础配置
#### yarn相关配置
```yarn-env.sh
JAVA_HOEM
HADOOP_HOME
HADOOP_CONF_DIR
HADOOP_LOG_DIR
```
```yarn-site.xml
<configuration>
<property>
                <name>yarn.resourcemanager.hostname</name>
<value>a01</value>
</property>

<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/data/nm-local</value>
</property>

<property>
                <name>yarn.nodemanager,log-dirs</name>
<value>/data/nm-log</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce</value>
</property>

<property>
<name>yarn.log.server.url</name>
<value>http://a01:19888/jobhistory/logs</value>
</property>

<property>
<name>yarn.wed-proxy.address</name>
<value>a01:8089</value>
</property>

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>

<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchduler</value>
</property
</configuration>
```
#### mapreduce相关配置
```mapreduce-env.sh
export JAVA_HOME=/ export/server/jdk.*
export HADOOP_JOB_HISTORYSERVER_HEAPSIZE=1000
export HADOOP_MADRED_ROOT_LOGGER=INFO,RFA
```
```mapreduce-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>a01:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>a01:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/data/mr-history/tmp</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/data/mr-history/done</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MADRED_HOME=$HADOOP_HOME</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MADRED_HOME=$HADOOP_HOME</value>
</property>
```
### hive服务搭建
### MySQL服务搭建
### 解压MySQL-*.rpm包

#### 严格按照依赖顺序安装
### 启动MySQL服务

### 启动hive服务