Scheme and Authority in Hadoop URI

Hadoop URI (uniform resource identifier) consist of Scheme, Authority and Path. In other words, fully qualified name of directory/file is determined by these three parameters. The URI format is scheme://autority/path.FS shell command (./bin/hdfs dfs) uses scheme and authority for determining which file system it has to refer. Collectively both scheme and authority determine the FileSystem implementation.
Scheme value for hadoop file system (HDFS) is "hdfs" and for local file system scheme value is "file".URI scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class.
Authority is host name/top level domain and port number associated with a file system.
The scheme and authority are optional. If not specified, the default scheme specified in the configuration is used.
Question:- Where do we configure default scheme ?
In core-site.xml with property name "fs.default.name" and default value for it is
hdfs://<URL/toplevelDomain:port> and same can be verified from core-site.xml present in following location:-  <HADOOP_INSTALATION_DIR>/etc/hadoop/core-site.xml.
Below is the sample node property for fs.default.name and it's value is hdf://localhost:543310
where hdfs is scheme and localhost:54310 is namenodehost.
<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
 </property>

Similarly, for local file system, scheme name changes too "file" and URI looks like
file:///file3 /user/hadoop/filename
Path is a valid name of a file or directory which specifies a unique location in a file system. It looks like : /user./local/<parent_dir>/<Sub_dir>/<Chile_dir/file>

FS shell command execution for HDFS and Local file system based on scheme values :- 

As stated earlier : scheme and authority determine the FileSystem implementation. Lets execute FS Shell command for HDFS and Local file system and spot the fundamental differences and understand how does scheme determine file system implementation it should refer.
For HDFS:- Here scheme name is "hdfs" and namenode host value is localhost:54310 then following command execution refers specified location as Hadoop file system and list out all the files and directory at given location.
hduser1@ubuntu:/usr/local/hadoop2.6.1$ ./bin/hdfs dfs -ls hdfs://localhost:54310/user/hduser1/hdfsdata
Found 2 items
drwxr-xr-x   - hduser1 supergroup          0 2015-10-04 11:51 hdfs://localhost:54310/user/hduser1/hdfsdata/hadoop_data
drwxr-xr-x   - hduser1 supergroup          0 2015-10-07 08:14 hdfs://localhost:54310/user/hduser1/hdfsdata/input
Note:- As mentioned earlier, scheme and authority are optional fields. So, if we have configured <HADOOP_Installation>/etc/hadoop/core-site.xml with property fs.default.name(as shown above), we do not need to qualify path(/user/hduser1/hdfsdata) with scheme and authority .Lets execute following command without scheme and authority (assuming configuration file is set-up with valid value for property name "fs.default.name")
hduser1@ubuntu:/usr/local/hadoop2.6.1$ ./bin/hdfs dfs -ls /user/hduser1/hdfsdata
Found 2 items
drwxr-xr-x   - hduser1 supergroup          0 2015-10-04 11:51 hdfs://localhost:54310/user/hduser1/hdfsdata/hadoop_data
drwxr-xr-x   - hduser1 supergroup          0 2015-10-07 08:14 hdfs://localhost:54310/user/hduser1/hdfsdata/input

For Local file system:-  List out all files or directory inside the given directory using FS shell command.
hduser1@ubuntu:/usr/local/hadoop2.6.1$ ./bin/hdfs dfs -ls file:///usr/local/
Found 15 items
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/bin
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/etc
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/games
drwxr-xr-x   - hduser  hadoop       4096 2015-10-02 09:05 file:///usr/local/hadoop
drwxr-xr-x   - root    root         4096 2013-07-22 15:26 file:///usr/local/hadoop-1.2.1
-rw-r--r--   1 root    root     63851630 2015-10-02 06:26 file:///usr/local/hadoop-1.2.1.tar.gz
drwxr-xr-x   - hduser1 hadoop       4096 2015-10-04 14:45 file:///usr/local/hadoop2.6.1
drwxr-xr-x   - hduser1 hadoop       4096 2015-10-04 02:33 file:///usr/local/hadoop_store
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/include
drwxr-xr-x   - root    root         4096 2015-10-02 04:29 file:///usr/local/java
drwxr-xr-x   - root    root         4096 2013-04-24 10:04 file:///usr/local/lib
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/man
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/sbin
drwxr-xr-x   - root    root         4096 2013-04-24 10:05 file:///usr/local/share
drwxr-xr-x   - root    root         4096 2013-04-24 10:01 file:///usr/local/src
Here scheme name is "file" then FS shell command refers to local file system and list down all files/directories at given location(/usr/local/).

Now we are in position to conclude that, scheme and authority collectively determines file system implementation to be referenced and uniquely identify file or directory in the file system.

18 Comments

  1. Great Article. As I read the blog I felt a tug on the heartstrings. it exhibits how much effort has been put into this.
    IEEE Projects for CSE in Big Data

    Spring Framework Corporate TRaining

    Final Year Project Centers in Chennai

    JavaScript Training in Chennai

    ReplyDelete
  2. New slot website, free credit giveaway, many privileges
    New slot website, free credit giveaway ตารางคะแนน

    ReplyDelete
  3. Ready to win huge prizes Within a very short time, get ready to be rich and become a millionaire. pgslot

    ReplyDelete
  4. Anyone can play It is also highly secure. Reliable, real payment, fast transfer, no cheating history pgslot เว็บตรง

    ReplyDelete
  5. One web slot including all camps give away unlimited money new game giveaway free credit bonus ดูบอลออนไลน์

    ReplyDelete
  6. Hello all friends, our website, ambbet, the number one website. The best online slots right now ผลบอลสด

    ReplyDelete
  7. บริการเกมสล็อตออนไลน์ปี 2022 เกมให้เลือกเล่นมากกว่า 1,000 เกม เว็บพนันออนไลน์ไม่ผ่านเอเย่นต์2022 สล็อตเล่นผ่านเว็บ ก็ได้เช่นกัน สมัครเล่น SLOT สล็อตออนไลน์ superslot.li

    ReplyDelete
  8. His game is quite easy to play and there are no rules that are too complicated and messy that we can't understand it with just a little study. pgslot เว็บตรง

    ReplyDelete
  9. each other. Play a fun game. Enjoy every language. Chopping board and pgslot เว็บตรง

    ReplyDelete
  10. on your mobile phone without downloading best cracked slot games Easy to play, comfortable at any time pgslot

    ReplyDelete
  11. Each game is very fun, fun, including slots for all camps. Ready for everyone to bet 24 hours a day. pgslot

    ReplyDelete
  12. AMBBET.BAR because it is a website that meets high standards. Web slots that Thai people like to play pgslot

    ReplyDelete
  13. Lock the result, there is no blocking any bonus at all. You can play PG SLOT camp games through the number 1 slot website. ดูบอลสด

    ReplyDelete
  14. that definitely meets the needs of all gamblers because we have gathered Direct web slot games บอลโลก 2022

    ReplyDelete
  15. to win prizes that come without getting tired. because of bonuses and jackpots that will be AMBKING

    ReplyDelete
  16. Amazing article. Your blog helped me to improve myself in many ways thanks for sharing this kind of wonderful informative blogs in live.
    bapehoodies

    ReplyDelete
Previous Post Next Post