Once you have created your XSEDE account, have set up DUO authentication, and have verified that you can connect to the XSEDE's Single Sign On (SSO) login hub, and have connected to Bridges, execute the following instructions to generate your personal virtual three node Hadoop cluster. If you have not yet connected to Bridges via the Single Sign On hub, click here for instructions on how to do so.
To start a Hadoop session with a persistent hdfs:
1. start an interactive job with at least 3 nodes
interact -N 3
Depending on the time of day and the number of jobs running on Bridges, if you attempt to access Bridges outside of the reserved time slots for your course, it may be several minutes before you are assigned a virtual cluster.
2. load the hadoop module when the job starts
...
/opt/packages/hadoop-testing/persist-start.sh
This command creates a persistent HDFS file system that can hold about 5 GB, this file-system will remain in tact for the duration of the course, after which it may be cleared.
4. Finally, load up the Hive module
...