Note - As of July 1st, 2016, files on scratch filesystems are subject to deletion 60 days after they were created.
User Scratch Space
Each compute node has its own local scratch filesystem. Users may read from and write to this using their own exclusive directory at /localscratch/Users/<HawkID>.
In addition to local storage, the HPC cluster system has its own large, shared filesystem mounted across all its nodes via NFS. Analogously, users can read from and write to this using their own exclusive directory at /nfsscratch/Users/<HawkID>.
Cleanup Policy
Scratch filesystems are a shared resource available for the convenience of all users. Therefore, files on these filesystems are subject to deletion after a certain lifespan as determined by the HPC policy committee. Home account storage and purchased storage are not subject to this policy.
/localscratch
On /localscratch, the allowed file lifespan is 30 days after the file was last accessed, and a file's age is the time elapsed since its access timestamp ("atime"). An automated cleanup process will run periodically on each node to delete files whose atime has reached the maximum lifespan. It is also expected that large amounts of data put on /localscratch are cleaned up at the end of a job. Otherwise, the node can become unavailable and will have its /localscratch cleaned entirely, irrespective of time stamps, to make it available again.
/nfsscratch
On /nfsscratch, the allowed file lifespan is 60 days after first being written filesystem, and a file's age is the time elapsed since its creation timestamp ("crtime") which is tracked on the fileserver. An automated cleanup process will run periodically on the server to delete files whose crtime has reached the maximum lifespan.
Altering or duplicating files solely to circumvent the scratch cleanup process is against policy. Please make legitimate use of scratch filesystems, then move your intermediate and final results to stable storage in accordance with policy.
Please contact research-computing@uiowa.edu with any questions or for assistance with this.
/nfsscratch_itf
During the last few months Argon system performance on /nfsscratch has at times been poorer than expected.
The HPC team is in the early stages of exploring alternative storage architectures that may provide improved performance and better scaling. The team continues to look for opportunities to optimize and improve the current architecture but options to do this are decreasing.
In the meantime, /nfsscratch_itf has been deployed to help relieve some of the I/O on scratch. Please feel free to start using this new file system while we continue to work towards a solution.
Please contact research-computing@uiowa.edu with any questions or for assistance with this.
Local or Shared Scratch?
- Multiple jobs might be running on your job's node. These jobs can compete for local storage I/O, causing contention independent of /nfsscratch. Only a job with exclusive access to a node can expect the full performance potential of the node's local storage.
- A parallel job running on multiple nodes typically shouldn't use filesystems local to any of its nodes. Even if you're writing your own MPI instead of using an off-the-shelf application, you can expect better performance if you collate results in memory via message passing and write your result to the shared filesystem. Consider local disk primarily as a structured alternative to swap.
- If your job places partial results on /localscratch but fails to handle them for any reason (logic error, eviction, crash, etc.), you won't have access to these anywhere else and they will be difficult to recover.
- As always, please test a few jobs if you are unsure.
File Timestamps
- modification time (mtime): This is the time the contents of the file were last modified, for example, by editing it. The modification time can be seen with
ls -l file
- change time (ctime): This is the time the metadata of the file was last changed. An example of this would be moving a file to a different directory. The change time can be seen with
ls -lc file
- access time (atime): This is the time the contents of the file were last accessed; for example, by viewing with 'less'. The access time can be seen with
ls -lu file
- creation time (crtime): This is the time the contents of the file were first written to the filesystem. This attribute is part of the underlying ZFS filesystem and is not accessible via NFS or standard Linux utilities.
All of these timestamps can be different for a single file. Most file and archive utilities will maintain the first 3 timestamps, either by default or optionally. This includes using archive mode ('-a') with either 'cp' or 'rsync'. However, note that no utility can affect a file's crtime at all over NFS.