User scratch space
All users have /nfsscratch/Users/HawkID available to them via our ZFS/NFS file systems. It is set to be read/write/executable for the user only. This user space is located on /nfsscratch/$Username on all nodes. There is also a local scratch space available (/localscratch/$USER) on all nodes if local disk scratch is needed. Note that using /localscratch is not usually the best choice. While there is no potential for network contention it is only a single disk on a node. If you are running a single job on a node, and have exclusive access to the node, then /localscratch should perform well. However, if you have multiple jobs on a node accessing /localscratch then the disk IO contention will be high and job performance will suffer. If the IO load is light then /localscratch may still be a viable option but heavy IO load could create a disk bottleneck. If you are not sure then test a job or small set of jobs and compare the performance of /localscratch vs. /nfsscratch. If the job is a parallel job and all nodes need access to the files then /localscratch is not an option as the files are only available to a single node.
Cleanup policy
The cleanup script currently runs every day looking for files that have not been accessed in the last 30 days. Basing this on access time works well in the case where files are being generated by computations on scratch and left there. However, in the case where files are moved to the scratch system from another source, the time stamps of the files from the source need to be considered.
There are 3 time stamps that get put on a file:
modification time (mtime): This is the time the contents of the file were last modified, for example, by editing it. The modification time can be seen with
ls -l file
change time (ctime): This is the time the metadata of the file was last changed. An example of this would be moving a file to a different directory. The change time can be seen with
ls -lc file
access time (atime): This is the time the contents of the file were last accessed, for example, by viewing with 'less'. The access time can be seen with
ls -lu file
It is possible for the 3 timestamps to be different for the file. The most important point to remember, in light of the cleanup script using access time, is that the timestamps (including access time) may be carried over from an archive if the files are being extracted from one. Most archive utilities will maintain the timestamps, either by default or optionally. This includes using archive mode ('-a') with either 'cp' or 'rsync'. If the timestamps from the source are carried over then it could lead to a situation where your extracted files are deleted by the clean up script before 30 days have elapsed. This is because the access time stamp (carried over from the archive) is older than what you might be expecting.
There are several ways to update the timestamps but the easiest would be to use the 'touch' command. Even using that, there could be several ways to call it depending on the directory structure of the files you are extracting/moving/copying. One example would be to run the 'find' command in the top level directory of your newly extracted files.
Example of updating timestamps of extracted files
find /nfsscratch/Users/gpjohnsn/extracted/ -type file -print0 | xargs -0 touch
After running the above, the files will have the current timestamp. It is possible, if desired, to update only the access timestamp by using the '-a' option to touch in the above command.
find /nfsscratch/Users/gpjohnsn/extracted/ -type file -print0 | xargs -0 touch -a
Touching files as described above should only be done when extracting archived data. Using 'touch' to circumvent the scratch cleanup policy is against policy.
Please contact hpc-sysadmins@iowa.uiowa.edu if you need assistance with this.