安装JDK
选用jdk8
wget -c https://download.oracle.com/otn/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jdk-8u202-linux-x64.tar.gz?AuthParam=1740798857_e761b4f4445ed1554d08ad450f4786c8
然后解压拷贝到目标放置目录下,并在~/.bashrc中配置路径
#JAVA_HOME
JAVA_HOME=/usr/local/tools/jdk1.8.0_202
export PATH=${JAVA_HOME}/bin:$PATH
下载Hadoop
此处选用3.2.4版本
wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gztar -xzf hadoop-3.2.4.tar.gz
配置hadoop环境变量
#HADOOP_HOME
HADOOP_HOME=/usr/local/tools/hadoop-3.2.4
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH
进入 $HADOOP_HOME/etc/hadoop/目录下,执行命令 vi hadoop-env.sh
,按“A”或“I”进入编辑内容,在文件中配置JDK环境变量:
export JAVA_HOME=/usr/local/tools/jdk1.8.0_202
在 $HADOOP_HOME/etc/hadoop/目录下,执行命令vi core-site.xml
,修改配置文件core-site.xml,内容如下:
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property><property><name>hadoop.tmp.dir</name><value>/usr/local/tools/hadoop-3.2.4/data</value></property>
</configuration>
在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi mapred-site.xml
,修改配置文件mapred-site.xml,内容如下:
<configuration><property><name>mapreduce.framework.name</name> <value>yarn</value> </property>
</configuration>
在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi yarn-site.xml
,修改配置文件yarn-site.xml,内容如下:
<configuration> <!-- Site specific YARN configuration properties --><property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name> <value>org.apache.hadoop.mapred.ShuffleHandler</value> </property></configuration>
在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi hdfs-site.xml
,修改配置文件hdfs-site.xml,内容如下:
<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.name.dir</name><value>file:/usr/local/tools/hadoop-3.2.4/namenode</value></property><property><name>dfs.datanode.data.dir</name><value>file:/usr/local/tools/hadoop-3.2.4/datanode</value></property>
</configuration>
配置ssh免密登录
在终端执行命令ssh-keygen -t rsa
,然后一直回车,生成密钥
-
执行命令
ssh-copy-id localhost
,在提示信息中输入yes
,然后输入本机密码。 -
执行命令
ssh localhost
,出现登录时间,说明ssh配置成功。
启动Hadoop服务
-
进入目录 $HADOOP_HOME,执行命令
hdfs namenode -format
,初始化节点。 -
进入目录 $HADOOP_HOME/sbin/,执行命令
start-all.sh
,然后执行命令jps
,出现以下6个线程说配置成功:
fhe@node0:~$ jps
45665 NodeManager
43442 ResourceManager
44902 NameNode
45050 DataNode
45291 SecondaryNameNode
45886 Jps
Hbase下载、编译与配置
我因为需要修改代码,所以从github上克隆
git clone https://github.com/apache/hbase.git
配置Hbase环境变量
#HBASE_HOME
HBASE_HOME=/opt/workspace/hbase
export PATH=$PATH:$HBASE_HOME/bin
配置hbase/conf/hbase-site.xml,添加配置
<configuration><!--The following properties are set for running HBase as a single process on adeveloper workstation. With this configuration, HBase is running in"stand-alone" mode and without a distributed file system. In this mode, andwithout further configuration, HBase and ZooKeeper data are stored on thelocal filesystem, in a path under the value configured for `hbase.tmp.dir`.This value is overridden from its default value of `/tmp` because manysystems clean `/tmp` on a regular basis. Instead, it points to a path withinthis HBase installation directory.Running against the `LocalFileSystem`, as opposed to a distributedfilesystem, runs the risk of data integrity issues and data loss. NormallyHBase will refuse to run in such an environment. Setting`hbase.unsafe.stream.capability.enforce` to `false` overrides this behavior,permitting operation. This configuration is for the developer workstationonly and __should not be used in production!__See also https://hbase.apache.org/book.html#standalone_dist--><property><name>hbase.zookeeper.property.dataDir</name><value>/opt/workspace/hbase/zookeeper</value></property><property><name>hbase.zookeeper.quorum</name><value>localhost</value></property><property><name>hbase.rootdir</name><value>hdfs://localhost:9000/hbase</value></property><property><name>hbase.zookeeper.property.clientPort</name><value>2181</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.master.port</name><value>16000</value></property><property><name>hbase.regionserver.port</name><value>16020</value></property> <property><name>hbase.tmp.dir</name><value>./tmp</value></property><property><name>hbase.unsafe.stream.capability.enforce</name><value>false</value></property>
</configuration>
因为我需要编译hbase代码,所以下载maven,并mv到需要放置的目录中
wget -c https://dlcdn.apache.org/maven/maven-3/3.9.9/binaries/apache-maven-3.9.9-bin.tar.gz
配置maven path
#MAVEN_HOME
MAVEN_HOME=/usr/local/tools/apache-maven-3.9.9
export PATH=$MAVEN_HOME/bin:$PATH
在conf/hbase-env.sh配置JAVA_HOME
# The java implementation to use. Java 1.7+ required.export JAVA_HOME=/usr/local/tools/jdk1.8.0_202
编译
清理之前的构建,编译并安装到本地 Maven 仓库。
mvn clean install -DskipTests
启动hbase
start-hbase.sh