Storage Locations on SCG¶
Each user has access to several storage types and locations. The following table lists the storage locations, limits and the use for which each is best suited.
(Can not be expanded)
|Config files, important scripts, private software/data|
||128 GB (Free Tier)
7 TB and up (Full Tier)
|Working data, lab shared software, results, etc.|
||Oak Storage||Working data, lab shared software, results, etc.|
||no quota, limited by node capacity||Per-job temporary files, data read many times during a job|
Best Usage of the Filesystem: Stage your data to
/tmp. Start by copying input data into
/tmp. Your jobs should write (or read) from
/tmp. Once you have the results you can finalize by moving the output into
/labs. This method gives you the best performance and prevents I/O bottlenecks.
$TMPDIR is a convenient variable assigned by SLURM that can be used instead of
/tmp. It is auto removed once your job has completed. Use of
$TMPDIR is recommended for jobs that use many small files. Example job script section:
# Copy the input to local storage cp /path-to/inputfile $TMPDIR # Run the code/application from $TMPDIR cd $TMPDIR ./myapp < inputfile > outputfile # Move the result cp outputfile /labs/path-to/your-lab-space/myapp_result # SLURM will auto-remove $TMPDIR
Storage in home directories, lab space, and Oak is all limited by quota. When you reach quota, all operations will start to fail, except for actions that only read or delete. For example, you can open, copy, and delete files; but not add to them, or make changes; until that space is under quota.
Checking Home directory quota¶
To check quota on your home directory, run this command:
du -hs ~
The command will take a little while to run, depending on how many items are in your home directory. You will eventually get back a single result, like so:
[akkornel@smsh11dsu-srcf-d15-36 ~]$ du -hs ~ 30G /home/akkornel
In the above example, user akkornel is using 30 GB. Home directory quotas as 32 GB, so this user is close to hitting quota.
Checking Lab quota¶
To check quota on your lab(s), run the
[ta5@smsh11dsu-srcf-d15-35 ~]$ checkquota Your current disk quotas are: Disk Disk Usage Limit /labs/ruthm 35.29G 128G
Details on how Oak based Lab quotas work:
Lab and Oak quotas have two limits: A limit on storage space, and a limit on “inodes”. An inode is a single file or directory. The same process is used to check both quotas.
Quota is applied based on group, so before checking quota, you first need to know which group is associated with your lab space. To do this, run this command:
ls -l /labs/PI_SUNetID/
PI_SUNetID with the PI’s SUNetID, and remember to keep the forward-slash at the end. You will eventually get back a single line, like so:
[akkornel@smsh11dsu-srcf-d15-36 ~]$ ls -ld /labs/ruthm/ drwxrws--x+ 5 ruthm scg_lab_ruthm 4096 Jul 9 11:37 /labs/ruthm/
In the above example, for PI “ruthm”, the group name is
You can now check the quota, using the following command:
lfs quota -h -g GROUP /labs/PI_SUNetID. Again,
PI_SUNetID is the PI’s SUNetID, and also
GROUP is the name of the group you identified earlier.
You will get back a multi-line result like the following:
[akkornel@smsh11dsu-srcf-d15-36 ~]$ lfs quota -h -g scg_lab_ruthm /labs/ruthm Disk quotas for grp scg_lab_ruthm (gid 6022): Filesystem used quota limit grace files quota limit grace /labs/ruthm 17.2G 128G 128G - 45 20615 20615 -
In the above example, the “ruthm” group has a storage quota of 128 GiB (the normal quota for a lab on the Free Tier), of which 17.2 GiB is being used. Also, the group has 45 inodes in use, out of a quota of 20,615.