Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

After logging on obtaining an account one of the things that you may wish to do is get data transferred to ( or from ) the HPC system. Two of the most widely available tools for secure file transfer are SCP (Secure Copy) and SFTP (Secure File Transfer Protocol) which utilize the SSH protocol to implement security. While SCP is a simple, non-interactive command line tool, it is often a more suitable choice when setting up unattended file transfers using scripts. Alternatively, SFTP is an interactive tool which offers features like remote file listing, copying, deleting, directory creation, etc.

While SCP and SFTP are available out of the box on Linux and Mac systems, ITS also offers downloads of Fetch (login required) for MacOS and WS_FTP (login required) for Windows from the ITS Software Download Site - ITS Help Desk - Information Technology Services at The University of Iowa site. In addition, SecureCRT also has SFTP capability. Below are some basic examples showing syntax for both tools. Note that while the examples are specific to Helium, they are broadly applicable and will thus work for Neon as well, and both commands have extensive man pages available.

Info

Off-campus access to the default ssh/scp/sftp port of 22 has been blocked at the campus border. Therefore, users must specify an alternate port in order for these tools to work from off campus. Helium and Neon The ITS-RS HPC systems use port 40 for off campus, non-VPN access. Some examples below show port specification.

Info
titleDUO Dual Facter

Duo dual facter factor authentication is required for SFTP and SCP to work. Ensure that you have a default request method for duo to authenticate.

...

Panel
titleCopy file from local to the HPC system (note alternate port specification)

[user@local ~]$ scp -P 40 file1 hawkid@heliumhawkid@neon.hpc.uiowa.edu:file1

Panel
titleCopy file1 and file2 to your home directory on the HPC system

[user@local ~]$ scp file1 file2 hawkid@heliumhawkid@neon.hpc.uiowa.edu:file:~

Panel
titleCopy directory from local to the HPC system

[user@local ~]$scp -r dir1 hawkid@heliumhawkid@neon.hpc.uiowa.edu:dir1

SFTP
Panel
titleCopy file from local to HPC system (note alternate port specification)
[user@local~]$ sftp -o port=40 hawkid@heliumhawkid@neon.hpc.uiowa.edu
Connecting to heliumneon.hpc.uiowa.edu...
sftp> put file1
Uploading file1 to /Users/hawkid/file1
file1 100% 840 0.8KB/s 00:00
sftp>
Panel
titleCopy directory from local to HPC system
[user@local~]$ sftp hawkid@heliumhawkid@neon.hpc.uiowa.edu
Connecting to heliumneon.hpc.uiowa.edu...
sftp> mkdir remote
sftp> put local/* remote/
Uploading local/file1
Uploading local/file2
Uploading local/file3
 
sftp> ls remote/
remote/file1
remote/file2
remote/file3

 

Info

This process is more involved in that one must create the directory remotely first and then transfer the contents from one to the other – wildcards are helpful for this.