Data Management
As an HPC user you have access to several different storage spaces:
|
Location
|
|
|---|---|
Home space |
This is your default working space on the HPC systems. Here you have an allocation of 1TB (and up to 10 million files). This allocation will remain as long as you are a member of the college and will be deleted when you leave. To refer to this location in job scripts use the environment variable $HOME. |
Ephemeral space |
This is additional individual working space where you can store unlimited amounts of working data. Any file in this space are automatically deleted after 30 days. You can refer to this location using the environment variable $EPHEMERAL. Home and Ephemeral are accessible on HPC interactive systems and within batch jobs. They can also be accessed from your personal computer as described below. |
Job temporary space |
Each running batch job is allocated temporary working space. This is accessible only to the job and deleted as soon as the job completes. The location of this temporary space is found in the environment variable $TMPDIR. It is also the default working directory of a job. Job temporary storage is normally on a disk directly attached to the compute node assigned to the job. It is best used for jobs that create lots of intermediate files or do random access to files, and so need very fast storage. There is typically between 10-100GB of space available in TMPDIR. |
Research Data Store project allocation |
You may also have been granted access to a Research Data Store project allocation. If so you will find these in the location specified in the environment variable $RDS_PROJECT. |
Data management options
Accessing data from your personal computer
The Research Data Store is accessible as a network share, at the location \\rds.imperial.ac.uk\RDS\user\ (macOS instructions). There you will have read/write access to your home and ephemeral spaces, along with any projects you are a member of. On Windows, you map any folder to a driver letter.
This is only accessible from Windows 10, macOS (or OS X 10.10 or later) or Linux machines on the college network. If you are off-site you must first connect to the OpenVPN VPN.
Sharing data
By default, only you have access to your Home and Ephemeral spaces. However, it is possible to share access with other members of the research group you are registered with. See who you can share data with. To do this, simply change the group permissions on the files you wish to share with chmod g+r file-list (read only) or chmod g+rw file-list (read-write). Additionally, all of the folders from $HOME down to the the one containing the files to be shared should be set with chmod g+rx directory-list .
Note: If you write any files into another user's home space it will still count against your 1TB quota.
Seeing how much data you have stored
When you log in to the HPC systems you will automatically be shown how much data you have in Home and Ephemeral, along with how much is in any project you are a member of. If To check this at any other time type the command quota.
Recovering deleted files
Every evening we take a snapshot of all of the files in Home and Project spaces (but not Ephemeral spaces) which we keep for a month. You can use these to to recover an accidentally-deleted file, or an earlier version of one.
From Windows, right-click on the folder that contained the deleted file (or file you want to retrieve an earlier version of) and select "Restore previous versions".
If working interactively on the HPC systems, do cd .snapshots in the relevant directory to see all snapshots arranged by date. These are read-only directories that you can copy files from.
Note for users registered before 12 October 2018
If you were first registered as an HPC users before 12 October 2018, you may also have been using WORK or SCRATCH storage. These areas are now retired. Any data you held there has been moved into your $HOME, under WORK or SCRATCH.