Using SCP or SFTP
After obtaining an account one of the things that you may wish to do is get data transferred to or from the HPC system. Two of the most widely available tools for secure file transfer are SCP (Secure Copy) and SFTP (Secure File Transfer Protocol) which utilize the SSH protocol to implement security. While SCP is a simple, non-interactive command line tool, it is often a more suitable choice when setting up unattended file transfers using scripts. Alternatively, SFTP is an interactive tool which offers features like remote file listing, copying, deleting, directory creation, etc.
While SCP and SFTP are available out of the box on Linux and Mac systems, ITS also offers downloads of Fetch (login required) for MacOS and WS_FTP Professional (login required) for Windows from the ITS Software Download Site - ITS Help Desk - Information Technology Services at The University of Iowa site. In addition, SecureCRT also has SFTP capability. Below are some basic examples showing syntax for both tools.
Off-campus access to the default ssh/scp/sftp port of 22 has been blocked at the campus border. Therefore, users must specify an alternate port in order for these tools to work from off campus. The ITS-RS HPC systems use port 40 for off campus, non-VPN access. Some examples below show port specification.
DUO Dual Facter
Duo dual factor authentication is required for SFTP and SCP to work. Ensure that you have a default request method for duo to authenticate.
Non-interactive shell output
Your client software will first try to establish a non-interactive session on the server. It will fail with an error if the non-interactive portion of your shell's startup prints anything. For the common case of bash, your ~/.bashrc configures non-interactive sessions, so make sure it contains no modifications which cause output. It's safe to place such modifications in ~/.bash_profile which only configures interactive logins.
SCP
[user@local ~]$ scp -P 40 file1 hawkid@argon.hpc.uiowa.edu:file1
[user@local~]$ sftp -o port=40 hawkid@argon.hpc.uiowa.eduConnecting to argon.hpc.uiowa.edu...sftp> put file1Uploading file1 to /Users/hawkid/file1file1 100% 840 0.8KB/s 00:00 sftp>[user@local~]$ sftp hawkid@argon.hpc.uiowa.eduConnecting to argon.hpc.uiowa.edu...sftp> mkdir remotesftp> put local/* remote/Uploading local/file1Uploading local/file2Uploading local/file3sftp> ls remote/remote/file1remote/file2remote/file3This process is more involved in that one must create the directory remotely first and then transfer the contents from one to the other – wildcards are helpful for this.