It is most likely just the id-rsa.pub filename. cat $HOME/.ssh/id-rsa.pub > $HOME/.ssh/authorized_keys Your public key has been saved in /home/hduser/.ssh/id_rsa.pub. Your identification has been saved in /home/hduser/.ssh/id_rsa. You will get the below example as well as the fingerprint and randomart image.Įnter file in which to save the key (/home/hduser/.ssh/id_rsa):
If not already logged in as the user we created in the Hadoop user section. If asked for a filename just leave it blank and confirm with enter.
#How to install selenium ide 2.9.0 password
We need to eliminate the password requirement by setting up ssh certificates. That is because Hadoop uses ssh to access its nodes. You might be wondering why we installed ssh at the beginning. Lsof wc: How many files are currently open ulimit -aįile-max: Current open file descriptor limitįile-nr: How many file descriptors are currently being used You can now test the limits you applied to make sure they took. sudo nano ~/.bashrcĮxport PATH=$PATH:/sbin/ nano /etc/nf If you get “cannot stat /proc/sys/-p: No such file or directory”. For this example I have set some values but you will have to figure out the best numbers for your specific case. If you do encounter this issue you might have to set the ulimit and descriptor limit. Sometimes you can reach open file descriptor limit and open file limit. If all the above disabling IPv6 configuration was successful you should get “1” returned. nano /etc/nfĪdd the following to the end and save .disable_ipv6 = 1Ĭat /proc/sys/net/ipv6/conf/all/disable_ipv6 bashrc is a script that is executed when a terminal session is started.Īdd the following line to the end and save because Hadoop uses IPv4.Įxport _JAVA_OPTIONS=’-XX:+UseCompressedOops 4Stack=true’ source ~/.bashrcĭisable ipv6 as it causes issues in getting your server up and running. OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode) You use this to verify that java installed correctly and will return something like the following. These two commands will check that ssh installed correctly and will return “/usr/bin/ssh” and “/usr/bin/sshd” java -version The “-a” on usermod is for appending to a group used with –G for which groups addgroup hduser Rack Awareness: Fault tolerance to ensure blocks are placed as evenly as possible on different racks if they are available.Īpply the following to all NameNode and DataNodes unless otherwise directed:įor this example we will just use hduser as our group and user for simplicity sake. Secondary NameNode: This should be done on a seperate server and it’s function is to take checkpoints of the namenodes file system. Zookeeper: It is also a good idea to use ZooKeeper to synchronize your configuration For all purposes here Hadoop will be open without having to login, etc.Īdditional Setup/Configurations to Consider: Also this article does not take into consideration any SSL, kerberos, etc.
I also use nano for this article for beginners but you can use any editor you prefer (ie: vi). NOTE: Sometimes you may have to use “sudo” in front of the command. I will put all commands used in this tutorial right down to the very basics for those that are new to Ubuntu. The deployment I have done is to have a Name Node and 1-* DataNodes on Ubuntu 16.04 assuming 5 cpu and 13GB RAM. I would like to share what I have learned and applied in the hopes that it will help someone else configure their system. I have been working with Hadoop 2.9.1 for over a year and have learned much on the installation of Hadoop in a multi node cluster environment.