Interacting with HDFS

Transcription

1 HADOOP Interacting with HDFS For University Program on Apache Hadoop & Apache Apex 1

2 2 What's the Need? Big data Ocean Expensive hardware Frequent Failures and Difficult recovery Scaling up with more machines

3 3 Hadoop Open source software - a Java framework - initial release: December 10, 2011 It provides both, Storage [HDFS] Processing [MapReduce] HDFS: Hadoop Distributed File System

4 4 How Hadoop addresses the need? Big data Ocean Have multiple machines. Each will store some portion of data, not the entire data. Expensive hardware Use commodity hardware. Simple and cheap. Frequent Failures and Difficult recovery Have multiple copies of data. Have the copies in different machines. Scaling up with more machines If more processing is needed, add new machines on the fly

5 5 HDFS Runs on Commodity hardware: Doesn't require expensive machines Large Files; Write-once, Read-many (WORM) Files are split into blocks Actual blocks go to DataNodes The metadata is stored at NameNode Replicate blocks to different node Default configuration: Block size = 128MB Replication Factor = 3

6 6

7 7

8 8

9 9 Where NOT TO use HDFS Low latency data access HDFS is optimized for high throughput of data at the expense of latency. Large number of small files Namenode has the entire file-system metadata in memory. Too much metadata as compared to actual data. Multiple writers / Arbitrary file modifications No support for multiple writers for a file Always append to end of a file

10 10 Some Key Concepts NameNode DataNodes JobTracker TaskTrackers ResourceManager (MRv2) NodeManager (MRv2) ApplicationMaster (MRv2)

11 11 NameNode & DataNodes NameNode: DataNode: Centerpiece of HDFS: The Master Only stores the block metadata: block-name, block-location etc. Critical component; When down, whole cluster is considered down; Single point of failure Should be configured with higher RAM Stores the actual data: The Slave In constant communication with NameNode When down, it does not affect the availability of data/cluster Should be configured with higher disk space SecondaryNameNode: Doesn't actually act as a NameNode Stores the image of primary NameNode at certain checkpoint Used as backup to restore NameNode

12 12

13 13 JobTracker & TaskTrackers JobTracker: Talks to the NameNode to determine location of the data Monitors all TaskTrackers and submits status of the job back to the client When down, HDFS is still functional; no new MR job; existing jobs halted Replaced by ResourceManager/ApplicationMaster in MRv2 TaskTracker: Runs on all DataNodes TaskTracker communicates with JobTracker signaling the task progress TaskTracker failure is not considered fatal Replaced by NodeManager in MRv2

14 14 ResourceManager & NodeManager Present in Hadoop v2.0 Equivalent of JobTracker & TaskTracker in v1.0 ResourceManager (RM): Runs usually at NameNode; Distributes resources among applications. Two main components: Scheduler and ApplicationsManager (AM) NodeManager (NM): Per-node framework agent Responsible for containers Monitors their resource usage Reports the stats to RM Central ResourceManager and Node specific Manager together is called YARN

15 15

16 16 Hadoop 1.0 vs. 2.0 HDFS 1.0: Single point of failure Horizontal scaling performance issue HDFS 2.0: HDFS High Availability HDFS Snapshot Improved performance HDFS Federation

17 17 HDFS Federation

18 18 Interacting with HDFS Command prompt: Similar to Linux terminal commands Unix is the model, POSIX is the API Web Interface: Similar to browsing a FTP site on web

19 Interacting With HDFS On Command Prompt 19

20 20 Notes File Paths on HDFS: hdfs:// :8020/user/username/demo/data/file.txt hdfs://localhost:8020/user/username/demo/data/file.txt /user/username/demo/file.txt demo/file.txt File System: Local: local file system (linux) HDFS: hadoop file system At some places: The terms file and directory has the same meaning.

21 21 Before we start Command: Usage: hdfs hdfs [--config confdir] COMMAND Example: hdfs dfs hdfs dfsadmin hdfs fsck hdfs namenode hdfs datanode

22 hdfs `dfs` commands 22

23 23 In general Syntax for `dfs` commands hdfs dfs -<COMMAND> -[OPTIONS] <PARAMETERS> e.g. hdfs dfs -ls -R /user/username/demo/data/

24 24 0. Do It yourself Syntax: hdfs dfs -help [COMMAND ] hdfs dfs -usage [COMMAND ] Example: hdfs dfs -help cat hdfs dfs -usage cat

25 25 1. List the file/directory Syntax: hdfs dfs -ls [-d] [-h] [-R] <hdfs-dir-path> Example: hdfs dfs -ls hdfs dfs -ls / hdfs dfs -ls /user/username/demo/list-dir-example hdfs dfs -ls -R /user/username/demo/list-dir-example

26 26 2. Creating a directory Syntax: hdfs dfs -mkdir [-p] <hdfs-dir-path> Example: hdfs dfs -mkdir /user/username/demo/create-dir-example hdfs dfs -mkdir -p /user/username/demo/create-direxample/dir1/dir2/dir3

27 27 3. Create a file on local & put it on HDFS Syntax: vi filename.txt hdfs dfs -put [options] <local-file-path> <hdfs-dir-path> Example: vi file-copy-to-hdfs.txt hdfs dfs -put file-copy-to-hdfs.txt /user/username/demo/putexample/

28 28 4. Get a file from HDFS to local Syntax: hdfs dfs -get <hdfs-file-path> [local-dir-path] Example: hdfs dfs -get /user/username/demo/get-example/file-copy-fromhdfs.txt ~/demo/

29 29 5. Copy From LOCAL To HDFS Syntax: hdfs dfs -copyfromlocal <local-file-path> <hdfs-file-path> Example: hdfs dfs -copyfromlocal file-copy-to-hdfs.txt /user/username/demo/copyfromlocal-example/

30 30 6. Copy To LOCAL From HDFS Syntax: hdfs dfs -copytolocal <hdfs-file-path> <local-file-path> Example: hdfs dfs -copytolocal /user/username/demo/copytolocalexample/file-copy-from-hdfs.txt ~/demo/

31 31 7. Move a file from local to HDFS Syntax: hdfs dfs -movefromlocal <local-file-path> <hdfs-dir-path> Example: hdfs dfs -movefromlocal /path/to/file.txt /user/username/demo/movefromlocal-example/

32 32 8. Copy a file within HDFS Syntax: hdfs dfs -cp <hdfs-source-file-path> <hdfs-dest-file-path> Example: hdfs dfs -cp /user/username/demo/copy-within-hdfs/file-copy.txt /user/username/demo/data/

33 33 9. Move a file within HDFS Syntax: hdfs dfs -mv <hdfs-source-file-path> <hdfs-dest-file-path> Example: hdfs dfs -mv /user/username/demo/move-within-hdfs/file-move.txt /user/username/demo/data/

34 Merge files on HDFS Syntax: hdfs dfs -getmerge [-nl] <hdfs-dir-path> <local-file-path> Examples: hdfs dfs -getmerge -nl /user/username/demo/merge-example/ /path/to/all-files.txt

35 View file contents Syntax: hdfs dfs -cat <hdfs-file-path> hdfs dfs -tail <hdfs-file-path> hdfs dfs -text <hdfs-file-path> Examples: hdfs dfs -cat /user/username/demo/data/cat-example.txt hdfs dfs -cat /user/username/demo/data/cat-example.txt head

36 Remove files/dirs from HDFS Syntax: hdfs dfs -rm [options] <hdfs-file-path> Examples: hdfs dfs -rm /user/username/demo/remove-example/remove-file.txt hdfs dfs -rm -R /user/username/demo/remove-example/ hdfs dfs -rm -R -skiptrash /user/username/demo/remove-example/

37 Change file/dir properties Syntax: hdfs dfs -chgrp [-R] <NewGroupName> <hdfs-file-path> hdfs dfs -chmod [-R] <permissions> <hdfs-file-path> hdfs dfs -chown [-R] <NewOwnerName> <hdfs-file-path> Examples: hdfs dfs -chmod -R 777 /user/username/demo/data/file-changeproperties.txt

38 Check the file size Syntax: hdfs dfs -du <hdfs-file-path> Examples: hdfs dfs -du /user/username/demo/data/file.txt hdfs dfs -du -s -h /user/username/demo/data/

39 Create a zero byte file in HDFS Syntax: hdfs dfs -touchz <hdfs-file-path> Examples: hdfs dfs -touchz /user/username/demo/data/zero-byte-file.txt

40 File test operations Syntax: hdfs dfs -test -[defsz] <hdfs-file-path> Examples: hdfs dfs -test -e /user/username/demo/data/file.txt echo $?

41 Get FileSystem Statistics Syntax: hdfs dfs -stat [format] <hdfs-file-path> Format Options: %b - file size in blocks, %n - filename %r - replication %y - modification date %g - group name of owner %o - block size %u - user name of owner

42 Get File/Dir Counts Syntax: hdfs dfs -count [-q] [-h] [-v] <hdfs-file-path> Example: hdfs dfs -count -v /user/username/demo/

43 Set replication factor Syntax: hdfs dfs -setrep -w -R n <hdfs-file-path> Examples: hdfs dfs -setrep -w -R 2 /user/username/demo/data/file.txt

44 Set Block Size Syntax: hdfs dfs -D dfs.blocksize=blocksize -copyfromlocal <local-file-path> <hdfs-file-path> Examples: hdfs dfs -D dfs.blocksize= copyfromlocal /path/to/file.txt /user/username/demo/block-example/

45 Empty the HDFS trash Syntax: hdfs dfs -expunge Location:

46 Other hdfs commands (admin) 46

47 HDFS Admin Commands: fsck Syntax: hdfs fsck <hdfs-file-path> Options: [-list-corruptfileblocks [-move -delete -openforwrite] [-files [-blocks [-locations -racks]]] [-includesnapshots]

48 48

49 HDFS Admin Commands: dfsadmin Syntax: hdfs dfsadmin Options: [-report [-live] [-dead] [-decommissioning]] [-safemode enter leave get wait] [-refreshnodes] [-refresh <host:ipc_port> <key> [arg1..argn]] [-shutdowndatanode <datanode:port> [upgrade]] [-getdatanodeinfo <datanode_host:ipc_port>] [-help [cmd]] Examples: hdfs dfsadmin -report -live

50 50

51 HDFS Admin Commands: namenode Syntax: hdfs namenode Options: [-checkpoint] [-format [-clusterid cid ] [-force] [-noninteractive] ] [-upgrade [-clusterid cid] ] [-rollback] [-recover [-force] ] [-metadataversion ] Examples: hdfs namenode -help

52 HDFS Admin Commands: getconf Syntax: hdfs getconf [-options] Options: [ -namenodes ] [ -backupnodes ] [ -excludefile ] [ -confkey [key] ] [ -secondarynamenodes ] [ -includefile ] [ -nnrpcaddresses ]

53 53 Again,,, THE most important command!! Syntax: hdfs dfs -help [options] hdfs dfs -usage [options] Examples: hdfs dfs -help help hdfs dfs -usage usage

54 Interacting With HDFS In Web Browser 54

55 55 Web HDFS URL: Examples:

56 56 References

57 57 Thank You!! Please send your questions at:

58 Resources Apache Apex website - Subscribe - Download - Twitter Follow - Facebook - Meetup - Startup Program Free Enterprise License for Startups, Educational Institutions, Non-Profits - Cloud Trial DataTorrent

59 We Are Hiring Developers/Architects QA Automation Developers Information Developers Build and Release DataTorrent

60 Upcoming Events March 15th March 17th 6pm PST Title March 24th 9am PST Title DataTorrent

61 APPENDIX 61

62 62 Copy data from one node to another node in HDFS Description: Copy data between clusters Syntax: hadoop distcp hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo hadoop distcp hdfs://nn1:8020/foo/a hdfs://nn1:8020/foo/b hdfs: //nn2:8020/bar/foo hadoop distcp -f hdfs://nn1:8020/srclist.file hdfs://nn2:8020/bar/foo Where srclist.file contains hdfs://nn1:8020/foo/a hdfs://nn1:8020/foo/b