Once you have created your XSEDE account, have set up DUO authentication, and have verified that you can connect to the XSEDE's Single Sign On (SSO) login hub, execute the following instructions to generate your personal virtual three node Hadoop cluster.
To start a Hadoop session with a persistent hdfs:
1. start an interactive job with at least 3 nodes
interact -N 3
2. load the hadoop module when the job starts
module load hadoop
WARNING! Once the module is loaded it will instruct you to run "start-hadoop.sh" This will create a NON-PERSISTENT HDFS (filesystem) nothing will be saved.
-----------------------Almost there... one more step.Now please runstart-hadoop.shto set up your environment-----------------------
3. Instead, for a persistent file system, set up your environment with the following script:
/opt/packages/hadoop-testing/persist-start.sh
4. Finally, load up the Hive module
source /opt/packages/hive/hive.sh
5. Run Hive
hive
For more information on Bridges see https://portal.xsede.org/psc-bridges.