HADOOP-基本操作

接上篇

基本操作

上传文件

首先创建目录

[zyi@h102 sbin]$ hadoop fs -mkdir /input

在Hadoop：http://h102:9870查看

上传文件：

[zyi@h102 hadoop-3.1.3]$ hadoop fs -put /tmp/word.txt /input

2021-06-02 22:28:37,950 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
[zyi@h102 hadoop-3.1.3]$ hadoop fs -put /opt/software/
hadoop-3.1.3.tar.gz         jdk-8u212-linux-x64.tar.gz
[zyi@h102 hadoop-3.1.3]$ hadoop fs -put /opt/software/jdk-8u212-linux-x64.tar.gz /input
2021-06-02 22:33:18,080 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-02 22:33:29,127 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted =

两个文件在web页面都可以看到

Block Size默认为128M 副本数Replication 3，点击可以调整

实际文件存储的地方：

$hadoop_home/data开始找到文件存储的地方：
[zyi@h102 dfs]$ cd data/current/BP-1568640629-192.168.110.82-1622624666933/current/finalized/
[zyi@h102 finalized]$ ll
total 0
drwxrwxr-x. 3 zyi zyi 21 Jun  2 22:28 subdir0
[zyi@h102 finalized]$ cd subdir0/
[zyi@h102 subdir0]$ ll
total 0
drwxrwxr-x. 2 zyi zyi 168 Jun  2 22:33 subdir0
[zyi@h102 subdir0]$ cd subdir0/
[zyi@h102 subdir0]$ ll
total 191944
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:28 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:28 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:33 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:33 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:33 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:33 blk_1073741827_1003.meta
[zyi@h102 subdir0]$ pwd
/opt/module/hadoop-3.1.3/data/dfs/data/current/BP-1568640629-192.168.110.82-1622624666933/current/finalized/subdir0/subdir0

以上我们看到一堆文件到底是什么呢

[zyi@h102 subdir0]$ cat blk_1073741825
Creative Commons is not a party to this License, and makes no warranty
        whatsoever in connection with the Work. Creative Commons will not be
        liable to You or any party on any legal theory for any damages
        whatsoever, including without limitation any general, special,
        incidental or consequential damages arising in connection to this
        license. Notwithstanding the foregoing two (2) sentences, if Creative
        Commons has expressly identified itself as the Licensor hereunder, it
        shall have all rights and obligations of Licensor.

        Except for the limited purpose of indicating to the public that the
        Work is licensed under the CCPL, Creative Commons does not authorize
        the use by either party of the trademark "Creative Commons" or any
        related trademark or logo of Creative Commons without the prior
        written consent of Creative Commons. Any permitted use will be in
        compliance with Creative Commons' then-current trademark usage
        guidelines, as may be published on its website or otherwise made
        available upon request from time to time. For the avoidance of doubt,
        this trademark restriction does not form part of this License.

这个文件就是word.txt 另外两个文件，其中 -rw-rw-r–. 1 zyi zyi 134217728 Jun 2 22:33 blk_1073741826 为128M，符合前面看到的block size

把这两个文件组合起来：

[zyi@h102 subdir0]$ cat blk_1073741826 >> tmp.tar.gz
[zyi@h102 subdir0]$ cat blk_1073741827 >> tmp.tar.gz
[zyi@h102 subdir0]$ ll
total 585160
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:28 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:28 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:33 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:33 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:33 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:33 blk_1073741827_1003.meta
-rw-rw-r--. 1 zyi zyi 195013152 Jun  2 22:39 tmp.tar.gz
[zyi@h102 subdir0]$ tar -zxvf tmp.tar.gz
jdk1.8.0_212/
jdk1.8.0_212/README.html
jdk1.8.0_212/LICENSE
jdk1.8.0_212/include/
jdk1.8.0_212/include/jawt.h
jdk1.8.0_212/include/linux/
jdk1.8.0_212/include/linux/jawt_md.h
jdk1.8.0_212/include/linux/jni_md.h
jdk1.8.0_212/include/classfile_constants.h
jdk1.8.0_212/include/jvmticmlr.h
jdk1.8.0_212/include/jni.h
jdk1.8.0_212/include/jdwpTransport.h
jdk1.8.0_212/include/jvmti.h
。。。。。。
[zyi@h102 subdir0]$ ll
total 585160
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:28 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:28 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:33 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:33 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:33 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:33 blk_1073741827_1003.meta
drwxr-xr-x. 7 zyi zyi       245 Apr  2  2019 jdk1.8.0_212
-rw-rw-r--. 1 zyi zyi 195013152 Jun  2 22:39 tmp.tar.gz

这个就是jdk文件

验证3副本

删除jdk以后，查看集群里的三台主机存储情况

[zyi@h102 subdir0]$ ll
total 191944
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:28 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:28 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:33 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:33 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:33 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:33 blk_1073741827_1003.meta
[zyi@h102 subdir0]$ ssh h103
Last login: Wed Jun  2 22:22:07 2021 from h102
[zyi@h103 ~]$ ll /opt/module/hadoop-3.1.3/data/dfs/data/current/BP-1568640629-192.168.110.82-1622624666933/current/finalized/subdir0/subdir0
total 191944
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:29 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:29 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:34 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:34 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:34 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:34 blk_1073741827_1003.meta
[zyi@h103 ~]$ exit
logout
Connection to h103 closed.
[zyi@h102 subdir0]$ ssh h104
Last login: Wed Jun  2 06:16:42 2021 from h102
[zyi@h104 ~]$ ll /opt/module/hadoop-3.1.3/data/dfs/data/current/BP-1568640629-192.168.110.82-1622624666933/current/finalized/subdir0/subdir0/
total 191944
-rw-rw-r--. 1 zyi zyi      1248 Jun  2 22:28 blk_1073741825
-rw-rw-r--. 1 zyi zyi        19 Jun  2 22:28 blk_1073741825_1001.meta
-rw-rw-r--. 1 zyi zyi 134217728 Jun  2 22:33 blk_1073741826
-rw-rw-r--. 1 zyi zyi   1048583 Jun  2 22:33 blk_1073741826_1002.meta
-rw-rw-r--. 1 zyi zyi  60795424 Jun  2 22:33 blk_1073741827
-rw-rw-r--. 1 zyi zyi    474975 Jun  2 22:33 blk_1073741827_1003.meta

验证成功

Mapreduce任务

删除jdk文件执行wordcount任务

[zyi@h102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output

完整过程

2021-06-02 23:26:05,512 INFO client.RMProxy: Connecting to ResourceManager at h103/192.168.110.83:8032
2021-06-02 23:26:07,332 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/zyi/.staging/job_1622628704820_0001
2021-06-02 23:26:07,593 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-02 23:26:08,046 INFO input.FileInputFormat: Total input files to process : 1
2021-06-02 23:26:08,137 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-02 23:26:08,343 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-02 23:26:08,421 INFO mapreduce.JobSubmitter: number of splits:1
2021-06-02 23:26:09,047 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-02 23:26:09,127 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1622628704820_0001
2021-06-02 23:26:09,127 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-06-02 23:26:09,585 INFO conf.Configuration: resource-types.xml not found
2021-06-02 23:26:09,586 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-06-02 23:26:10,934 INFO impl.YarnClientImpl: Submitted application application_1622628704820_0001
2021-06-02 23:26:11,455 INFO mapreduce.Job: The url to track the job: http://h103:8088/proxy/application_1622628704820_0001/
2021-06-02 23:26:11,456 INFO mapreduce.Job: Running job: job_1622628704820_0001
2021-06-02 23:26:28,137 INFO mapreduce.Job: Job job_1622628704820_0001 running in uber mode : false
2021-06-02 23:26:28,138 INFO mapreduce.Job:  map 0% reduce 0%
2021-06-02 23:26:42,562 INFO mapreduce.Job: Task Id : attempt_1622628704820_0001_m_000000_0, Status : FAILED
[2021-06-02 23:26:40.533]Container [pid=32136,containerID=container_1622628704820_0001_01_000002] is running 273295872B beyond the 'VIRTUAL' memory limit. Current usage: 91.3 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1622628704820_0001_01_000002 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 32136 32134 32136 32136 (bash) 0 3 9797632 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/zyi/appcache/application_1622628704820_0001/container_1622628704820_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.110.83 39249 attempt_1622628704820_0001_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000002/stderr
        |- 32145 32136 32136 32136 (java) 315 288 2518355968 23074 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/zyi/appcache/application_1622628704820_0001/container_1622628704820_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.110.83 39249 attempt_1622628704820_0001_m_000000_0 2

[2021-06-02 23:26:40.760]Container killed on request. Exit code is 143
[2021-06-02 23:26:40.781]Container exited with a non-zero exit code 143.

2021-06-02 23:26:50,784 INFO mapreduce.Job:  map 100% reduce 0%
2021-06-02 23:26:57,022 INFO mapreduce.Job: Task Id : attempt_1622628704820_0001_r_000000_0, Status : FAILED
[2021-06-02 23:26:55.995]Container [pid=32233,containerID=container_1622628704820_0001_01_000004] is running 315304448B beyond the 'VIRTUAL' memory limit. Current usage: 166.6 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1622628704820_0001_01_000004 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 32243 32233 32233 32233 (java) 545 47 2560364544 42352 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/zyi/appcache/application_1622628704820_0001/container_1622628704820_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.110.83 39249 attempt_1622628704820_0001_r_000000_0 4
        |- 32233 32232 32233 32233 (bash) 0 0 9797632 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/zyi/appcache/application_1622628704820_0001/container_1622628704820_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Dyarn.app.mapreduce.shuffle.logger=INFO,shuffleCLA -Dyarn.app.mapreduce.shuffle.logfile=syslog.shuffle -Dyarn.app.mapreduce.shuffle.log.filesize=0 -Dyarn.app.mapreduce.shuffle.log.backups=0 org.apache.hadoop.mapred.YarnChild 192.168.110.83 39249 attempt_1622628704820_0001_r_000000_0 4 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000004/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1622628704820_0001/container_1622628704820_0001_01_000004/stderr

[2021-06-02 23:26:56.036]Container killed on request. Exit code is 143
[2021-06-02 23:26:56.071]Container exited with a non-zero exit code 143.

2021-06-02 23:27:05,170 INFO mapreduce.Job:  map 100% reduce 100%
2021-06-02 23:27:05,189 INFO mapreduce.Job: Job job_1622628704820_0001 completed successfully
2021-06-02 23:27:05,323 INFO mapreduce.Job: Counters: 56
        File System Counters
                FILE: Number of bytes read=1462
                FILE: Number of bytes written=437045
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=1344
                HDFS: Number of bytes written=1013
                HDFS: Number of read operations=8
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Failed map tasks=1
                Failed reduce tasks=1
                Launched map tasks=2
                Launched reduce tasks=2
                Other local map tasks=1
                Data-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=14362
                Total time spent by all reduces in occupied slots (ms)=8549
                Total time spent by all map tasks (ms)=14362
                Total time spent by all reduce tasks (ms)=8549
                Total vcore-milliseconds taken by all map tasks=14362
                Total vcore-milliseconds taken by all reduce tasks=8549
                Total megabyte-milliseconds taken by all map tasks=14706688
                Total megabyte-milliseconds taken by all reduce tasks=8754176
        Map-Reduce Framework
                Map input records=19
                Map output records=176
                Map output bytes=1822
                Map output materialized bytes=1462
                Input split bytes=96
                Combine input records=176
                Combine output records=111
                Reduce input groups=111
                Reduce shuffle bytes=1462
                Reduce input records=111
                Reduce output records=111
                Spilled Records=222
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=258
                CPU time spent (ms)=2270
                Physical memory (bytes) snapshot=476758016
                Virtual memory (bytes) snapshot=5136683008
                Total committed heap usage (bytes)=435683328
                Peak Map Physical memory (bytes)=291237888
                Peak Map Virtual memory (bytes)=2565316608
                Peak Reduce Physical memory (bytes)=185520128
                Peak Reduce Virtual memory (bytes)=2571366400
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=1248
        File Output Format Counters
                Bytes Written=1013

在yarn的web页面上可以看到任务：

执行完成以后：

count的结果

其他–集群误删等，恢复

jps查看进程并停止
删除每台上的/datahe /logs
格式化namenode
启动fs和yarn

启动历史服务器

在Yarn查看任务结束后，可以看到页面有一个链接到历史默认地址为：http://h103:8088/proxy/application_1622701983047_0001/，系统会查找mapred-site.xml的配置从定向到配置的服务器地址。

修改mapred-site.xml

位置：$hadoop_home/etc/hadoop/

[zyi@h102 hadoop-3.1.3]$ vim etc/hadoop/mapred-site.xml

增加下面内容：

  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>h102:10020</value>
  </property>


  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>h102:19888</value>
  </property>

同步集群内配置文件

[zyi@h102 hadoop-3.1.3]$ xsync etc/hadoop/
==============================h102==============================
sending incremental file list

sent 888 bytes  received 18 bytes  604.00 bytes/sec
total size is 107,834  speedup is 119.02
==============================h103==============================
sending incremental file list
hadoop/
hadoop/mapred-site.xml

sent 1,440 bytes  received 52 bytes  994.67 bytes/sec
total size is 107,834  speedup is 72.27
==============================h104==============================
sending incremental file list
hadoop/
hadoop/mapred-site.xml

sent 1,440 bytes  received 52 bytes  994.67 bytes/sec
total size is 107,834  speedup is 72.27

重启Yarn

到h103上重启Yarn

[zyi@h103 ~]$ stop-yarn.sh
Stopping nodemanagers
Stopping resourcemanager
[zyi@h103 ~]$  start-yarn.sh
Starting resourcemanager
Starting nodemanagers

手动启用historyserver进程

[zyi@h102 hadoop-3.1.3]$ mapred --daemon start historyserver
[zyi@h102 hadoop-3.1.3]$ jps
19681 NameNode
3766 Jps
2839 JobHistoryServer
19821 DataNode
2589 NodeManager

重新执行mapreduce任务

删除前面任务的目录

[zyi@h102 hadoop-3.1.3]$ hadoop fs -rm -r /output
Deleted /output

执行mapreduce–wordcount任务

[zyi@h102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
2021-06-03 02:35:31,696 INFO client.RMProxy: Connecting to ResourceManager at h103/192.168.110.83:8032
2021-06-03 02:35:32,926 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/zyi/.staging/job_1622701983047_0001
······

点击history进入h102:19888页面

日志汇总

完成应用运行后，将日志文件都传到HDFS保存初始情况下在logs后面提示：

开启此功能，要重启NodeName、ResourceManager和historyServer

配置yarn-site.xml

增加以下内容

  <property>
    <description>Log aggregation</description>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <description>Log server  url</description>
    <name>yarn.log.server.url</name>
    <value>http://h102:19888/jobhistory/logs</value>
  </property>

  <property>
    <description>Log retains by seconds</description>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>

分发给集群所有主机

[zyi@h102 hadoop-3.1.3]$ xsync etc/hadoop/
==============================h102==============================
sending incremental file list

sent 888 bytes  received 18 bytes  604.00 bytes/sec
total size is 108,309  speedup is 119.55
==============================h103==============================
sending incremental file list
hadoop/
hadoop/yarn-site.xml

sent 2,022 bytes  received 52 bytes  4,148.00 bytes/sec
total size is 108,309  speedup is 52.22
==============================h104==============================
sending incremental file list
hadoop/
hadoop/yarn-site.xml

sent 2,022 bytes  received 52 bytes  1,382.67 bytes/sec
total size is 108,309  speedup is 52.22

重启NodeName、ResourceManager和historyServer

h102重启NodeName、historyServer，h103重启Yarn ResourceManager

[zyi@h102 hadoop-3.1.3]$ !220
mapred --daemon stop historyserver
[zyi@h102 hadoop-3.1.3]$ stop-dfs.sh
Stopping namenodes on [h102]
Stopping datanodes
Stopping secondary namenodes [h104]
[zyi@h102 hadoop-3.1.3]$ ssh h103
Last login: Thu Jun  3 02:32:17 2021 from h102
[zyi@h103 ~]$ stop-yarn.sh
Stopping nodemanagers
h103: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
h102: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
h104: WARNING: nodemanager did not stop gracefully after 5 seconds: Trying to kill with kill -9
Stopping resourcemanager
[zyi@h103 ~]$  start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[zyi@h103 ~]$ exit
logout
Connection to h103 closed.
[zyi@h102 hadoop-3.1.3]$ start-dfs.sh
Starting namenodes on [h102]
Starting datanodes
Starting secondary namenodes [h104]
[zyi@h102 hadoop-3.1.3]$ mapred --daemon start historyserver

重新做Mapreduce任务

[zyi@h102 hadoop-3.1.3]$ !226
hadoop fs -rm -r /output
Deleted /output
[zyi@h102 hadoop-3.1.3]$ !213
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /input /output
2021-06-03 03:45:03,861 INFO client.RMProxy: Connecting to ResourceManager at h103/192.168.110.83:8032
2021-06-03 03:45:05,880 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/zyi/.staging/job_1622706200490_0001
2021-06-03 03:45:06,116 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-06-03 03:45:07,055 INFO input.FileInputFormat: Total input files to process : 1
···

查看Job History，可以看到logs链接点击logs

以上