Storage Locations on SCG¶

Each user has access to several storage types and locations. The following table lists the storage locations, limits and the use for which each is best suited.

Location	Quota	Usage
`/home/SUNetID`	32 GB (Can not be expanded)	Config files, important scripts, private software/data
`/labs/PI_SUNetID`	128 GB (Free Tier) 7 TB and up (Full Tier)	Working data, lab shared software, results, etc.
`/oak/stanford`	Oak Storage	Working data, lab shared software, results, etc.
`/tmp`	no quota, limited by node capacity	Per-job temporary files, data read many times during a job

Best Usage of the Filesystem: Stage your data to /tmp. Start by copying input data into /tmp. Your jobs should write (or read) from /tmp. Once you have the results you can finalize by moving the output into /labs. This method gives you the best performance and prevents I/O bottlenecks.

$TMPDIR¶

$TMPDIR is a convenient variable assigned by SLURM that can be used instead of /tmp. It is auto removed once your job has completed. Use of $TMPDIR is recommended for jobs that use many small files. Example job script section:

# Copy the input to local storage
cp /path-to/inputfile $TMPDIR
# Run the code/application from $TMPDIR
cd $TMPDIR
./myapp < inputfile > outputfile
# Move the result
cp outputfile /labs/path-to/your-lab-space/myapp_result
# SLURM will auto-remove $TMPDIR

Checking Quota¶

Storage in home directories, lab space, and Oak is all limited by quota. When you reach quota, all operations will start to fail, except for actions that only read or delete. For example, you can open, copy, and delete files; but not add to them, or make changes; until that space is under quota.

Checking Home directory quota¶

To check quota on your home directory, run this command: du -hs ~

The command will take a little while to run, depending on how many items are in your home directory. You will eventually get back a single result, like so:

[akkornel@smsh11dsu-srcf-d15-36 ~]$ du -hs ~
30G    /home/akkornel

In the above example, user akkornel is using 30 GB. Home directory quotas as 32 GB, so this user is close to hitting quota.

Checking Lab quota¶

To check quota on your lab(s), run the checkquota command:

[ta5@smsh11dsu-srcf-d15-35 ~]$ checkquota 
Your current disk quotas are: 
Disk            Disk Usage      Limit
/labs/ruthm         35.29G       128G

Details on how Oak based Lab quotas work:

Lab and Oak quotas have two limits: A limit on storage space, and a limit on “inodes”. An inode is a single file or directory. The same process is used to check both quotas.

Quota is applied based on group, so before checking quota, you first need to know which group is associated with your lab space. To do this, run this command: ls -l /labs/PI_SUNetID/

Replace PI_SUNetID with the PI’s SUNetID, and remember to keep the forward-slash at the end. You will eventually get back a single line, like so:

[akkornel@smsh11dsu-srcf-d15-36 ~]$ ls -ld /labs/ruthm/
drwxrws--x+ 5 ruthm scg_lab_ruthm 4096 Jul  9 11:37 /labs/ruthm/

In the above example, for PI “ruthm”, the group name is scg_lab_ruthm.

You can now check the quota, using the following command: lfs quota -h -g GROUP /labs/PI_SUNetID. Again, PI_SUNetID is the PI’s SUNetID, and also GROUP is the name of the group you identified earlier.

You will get back a multi-line result like the following:

[akkornel@smsh11dsu-srcf-d15-36 ~]$ lfs quota -h -g scg_lab_ruthm /labs/ruthm
Disk quotas for grp scg_lab_ruthm (gid 6022):
     Filesystem    used   quota   limit   grace   files   quota   limit   grace
    /labs/ruthm   17.2G    128G    128G       -      45   20615   20615       -

In the above example, the “ruthm” group has a storage quota of 128 GiB (the normal quota for a lab on the Free Tier), of which 17.2 GiB is being used. Also, the group has 45 inodes in use, out of a quota of 20,615.