当前位置: 首页> 文旅> 艺术 > 上海市建设工程信息网_郑州开发软件公司_搜狐财经峰会直播_软文自动发布软件

上海市建设工程信息网_郑州开发软件公司_搜狐财经峰会直播_软文自动发布软件

时间:2025/8/25 20:13:04来源:https://blog.csdn.net/qq_53286237/article/details/146044266 浏览次数:0次
上海市建设工程信息网_郑州开发软件公司_搜狐财经峰会直播_软文自动发布软件

安装JDK

选用jdk8

wget -c https://download.oracle.com/otn/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jdk-8u202-linux-x64.tar.gz?AuthParam=1740798857_e761b4f4445ed1554d08ad450f4786c8

然后解压拷贝到目标放置目录下,并在~/.bashrc中配置路径

#JAVA_HOME
JAVA_HOME=/usr/local/tools/jdk1.8.0_202
export PATH=${JAVA_HOME}/bin:$PATH

下载Hadoop

此处选用3.2.4版本

wget https://archive.apache.org/dist/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gztar -xzf hadoop-3.2.4.tar.gz

配置hadoop环境变量

#HADOOP_HOME
HADOOP_HOME=/usr/local/tools/hadoop-3.2.4
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

进入 $HADOOP_HOME/etc/hadoop/目录下,执行命令 vi hadoop-env.sh,按“A”或“I”进入编辑内容,在文件中配置JDK环境变量:

export JAVA_HOME=/usr/local/tools/jdk1.8.0_202

在 $HADOOP_HOME/etc/hadoop/目录下,执行命令vi core-site.xml,修改配置文件core-site.xml,内容如下:

<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property><property><name>hadoop.tmp.dir</name><value>/usr/local/tools/hadoop-3.2.4/data</value></property>
</configuration>

在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi mapred-site.xml,修改配置文件mapred-site.xml,内容如下:

<configuration><property><name>mapreduce.framework.name</name>       <value>yarn</value>   </property>
</configuration>

在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi yarn-site.xml,修改配置文件yarn-site.xml,内容如下:

<configuration>   <!-- Site specific YARN configuration properties --><property>       <name>yarn.nodemanager.aux-services</name>       <value>mapreduce_shuffle</value>   </property>   <property>       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>       <value>org.apache.hadoop.mapred.ShuffleHandler</value>   </property></configuration>

在$HADOOP_HOME/etc/hadoop/目录下,执行命令vi hdfs-site.xml,修改配置文件hdfs-site.xml,内容如下:

<configuration><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.namenode.name.dir</name><value>file:/usr/local/tools/hadoop-3.2.4/namenode</value></property><property><name>dfs.datanode.data.dir</name><value>file:/usr/local/tools/hadoop-3.2.4/datanode</value></property>
</configuration>

配置ssh免密登录

在终端执行命令ssh-keygen -t rsa,然后一直回车,生成密钥

  • 执行命令ssh-copy-id localhost,在提示信息中输入yes,然后输入本机密码。

  • 执行命令ssh localhost,出现登录时间,说明ssh配置成功。

启动Hadoop服务

  • 进入目录 $HADOOP_HOME,执行命令hdfs namenode -format,初始化节点。

  • 进入目录 $HADOOP_HOME/sbin/,执行命令start-all.sh,然后执行命令jps,出现以下6个线程说配置成功:

fhe@node0:~$ jps
45665 NodeManager
43442 ResourceManager
44902 NameNode
45050 DataNode
45291 SecondaryNameNode
45886 Jps

Hbase下载、编译与配置

我因为需要修改代码,所以从github上克隆

git clone https://github.com/apache/hbase.git

配置Hbase环境变量

#HBASE_HOME
HBASE_HOME=/opt/workspace/hbase
export PATH=$PATH:$HBASE_HOME/bin

配置hbase/conf/hbase-site.xml,添加配置

<configuration><!--The following properties are set for running HBase as a single process on adeveloper workstation. With this configuration, HBase is running in"stand-alone" mode and without a distributed file system. In this mode, andwithout further configuration, HBase and ZooKeeper data are stored on thelocal filesystem, in a path under the value configured for `hbase.tmp.dir`.This value is overridden from its default value of `/tmp` because manysystems clean `/tmp` on a regular basis. Instead, it points to a path withinthis HBase installation directory.Running against the `LocalFileSystem`, as opposed to a distributedfilesystem, runs the risk of data integrity issues and data loss. NormallyHBase will refuse to run in such an environment. Setting`hbase.unsafe.stream.capability.enforce` to `false` overrides this behavior,permitting operation. This configuration is for the developer workstationonly and __should not be used in production!__See also https://hbase.apache.org/book.html#standalone_dist--><property><name>hbase.zookeeper.property.dataDir</name><value>/opt/workspace/hbase/zookeeper</value></property><property><name>hbase.zookeeper.quorum</name><value>localhost</value></property><property><name>hbase.rootdir</name><value>hdfs://localhost:9000/hbase</value></property><property><name>hbase.zookeeper.property.clientPort</name><value>2181</value></property><property><name>hbase.cluster.distributed</name><value>true</value></property><property><name>hbase.master.port</name><value>16000</value></property><property><name>hbase.regionserver.port</name><value>16020</value></property>  <property><name>hbase.tmp.dir</name><value>./tmp</value></property><property><name>hbase.unsafe.stream.capability.enforce</name><value>false</value></property>
</configuration>

因为我需要编译hbase代码,所以下载maven,并mv到需要放置的目录中

wget -c https://dlcdn.apache.org/maven/maven-3/3.9.9/binaries/apache-maven-3.9.9-bin.tar.gz

配置maven path

#MAVEN_HOME
MAVEN_HOME=/usr/local/tools/apache-maven-3.9.9
export PATH=$MAVEN_HOME/bin:$PATH

在conf/hbase-env.sh配置JAVA_HOME

# The java implementation to use.  Java 1.7+ required.export JAVA_HOME=/usr/local/tools/jdk1.8.0_202

编译

清理之前的构建,编译并安装到本地 Maven 仓库。

mvn clean install -DskipTests

启动hbase

start-hbase.sh

关键字:上海市建设工程信息网_郑州开发软件公司_搜狐财经峰会直播_软文自动发布软件

版权声明:

本网仅为发布的内容提供存储空间,不对发表、转载的内容提供任何形式的保证。凡本网注明“来源:XXX网络”的作品,均转载自其它媒体,著作权归作者所有,商业转载请联系作者获得授权,非商业转载请注明出处。

我们尊重并感谢每一位作者,均已注明文章来源和作者。如因作品内容、版权或其它问题,请及时与我们联系,联系邮箱:809451989@qq.com,投稿邮箱:809451989@qq.com

责任编辑: