하둡을 실행 하기 전에 namenode를 포멧 해야 한다.
|
1
|
[hadoop@master ~]$ hdfs namenode -format
|
하둡 실행 명령어
|
1
|
[hadoop@master ~]$ start-all.sh
|
잘 실행됬는지 확인
[master서버]
|
1
2
3
4
5
|
[hadoop@master ~]$ jps
42274 NameNode
42787 ResourceManager
42555 SecondaryNameNode
49102 Jps
|
[slave 서버]
|
1
2
3
4
|
[hadoop@slave1 ~]$ jps
26948 NodeManager
26821 DataNode
31374 Jps
|
HDFS 확인
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
|
[hadoop@master ~]$ hdfs dfsadmin -report
Configured Capacity: 112675270656 (104.94 GB)
Present Capacity: 96419569664 (89.80 GB)
DFS Remaining: 96407281664 (89.79 GB)
DFS Used: 12288000 (11.72 MB)
DFS Used%: 0.01%
Under replicated blocks: 4
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 37558423552 (34.98 GB)
DFS Used: 4096000 (3.91 MB)
Non DFS Used: 5424627712 (5.05 GB)
DFS Remaining: 32129699840 (29.92 GB)
DFS Used%: 0.01%
DFS Remaining%: 85.55%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu May 09 15:46:40 KST 2019
Hostname: slave3
Decommission Status : Normal
Configured Capacity: 37558423552 (34.98 GB)
DFS Used: 4096000 (3.91 MB)
Non DFS Used: 5360566272 (4.99 GB)
DFS Remaining: 32193761280 (29.98 GB)
DFS Used%: 0.01%
DFS Remaining%: 85.72%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu May 09 15:46:41 KST 2019
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 37558423552 (34.98 GB)
DFS Used: 4096000 (3.91 MB)
Non DFS Used: 5470507008 (5.09 GB)
DFS Remaining: 32083820544 (29.88 GB)
DFS Used%: 0.01%
DFS Remaining%: 85.42%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu May 09 15:46:40 KST 2019
|
웹 UI 확인
 |
 |
테스트
hdfs에 input 파일 경로를 만든다.
|
1
2
3
|
[hadoop@master ~]$ hdfs dfs -mkdir /user
[hadoop@master ~]$ hdfs dfs -mkdir /user/hadoop
[hadoop@master ~]$ hdfs dfs -mkdir /user/hadoop/input
|
/usr/local/hadoop 위치에 test.txt 파일을 만들고 파일에 다음과 같이 작성하고 저장한다.
|
1
2
3
4
|
그리고 다음과 같은 명령어로 hdfs에 test.txt파일을 저장한다.
|
1
|
|
hdfs web ui에서 위 이미지와 같이 'Browse the file system'을 클릭하고 /user/hadoop/input 경로에 가보면
test.txt 파일이 저장되어있는것으 확인 할 수 있다.
이후 yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount input output 명령어로 test.txt 파일을 mapreduce wordcount하겠습니다.
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
|
[hadoop@master hadoop]$ yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.6.jar wordcount input output
19/05/09 16:11:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/05/09 16:11:37 INFO client.RMProxy: Connecting to ResourceManager at master/xxx.xxx.xxx.35:8032
19/05/09 16:11:37 INFO input.FileInputFormat: Total input paths to process : 1
19/05/09 16:11:37 INFO mapreduce.JobSubmitter: number of splits:1
19/05/09 16:11:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1557362273260_0002
19/05/09 16:11:37 INFO impl.YarnClientImpl: Submitted application application_1557362273260_0002
19/05/09 16:11:37 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1557362273260_0002/
File System Counters
FILE: Number of bytes read=98
FILE: Number of bytes written=247351
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=186
HDFS: Number of bytes written=60
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4734
Total time spent by all reduces in occupied slots (ms)=7605
Total time spent by all map tasks (ms)=2367
Total time spent by all reduce tasks (ms)=2535
Total vcore-milliseconds taken by all map tasks=2367
Total vcore-milliseconds taken by all reduce tasks=2535
Total megabyte-milliseconds taken by all map tasks=3635712
Total megabyte-milliseconds taken by all reduce tasks=7787520
Map-Reduce Framework
Map input records=4
Map output records=12
Map output bytes=124
Map output materialized bytes=98
Input split bytes=110
Combine input records=12
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=98
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=96
CPU time spent (ms)=1050
Physical memory (bytes) snapshot=514449408
Virtual memory (bytes) snapshot=7718776832
Total committed heap usage (bytes)=686817280
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=76
File Output Format Counters
Bytes Written=60
|
결과 확인
|
1
2
3
4
5
6
7
8
9
|
[hadoop@master hadoop]$ hdfs dfs -cat /user/hadoop/output/part-r-00000
Mr.smith 3
handsome 2
he 1
hello 1
is 1
like 1
look 2
you 1
|
'Centos7 > Hadoop' 카테고리의 다른 글
| HDFS에 분산되어 저장됬는지 확인하는 명령어 (0) | 2019.05.15 |
|---|---|
| 하둡 설치 4. ssh 키 설정 및 하둡 설치 (0) | 2019.05.09 |
| 하둡 설치 3. 가상서버 복제 및 주키퍼 설치 (0) | 2019.05.09 |
| 하둡 설치 2. 자바 설치 및 설정 (0) | 2019.05.09 |
| 하둡설치 1. 네트워크 설정 및 host설정 (0) | 2019.05.09 |