Skip to content
GitLab
  • Explore
  • Sign in
  • EDA Guides
  • eda-servers-guide
  • Wiki
  • storage

storage · Changes

Page history
Update storage authored May 06, 2021 by Daniele Jahier Pagliari's avatar Daniele Jahier Pagliari
Show whitespace changes
Inline Side-by-side
storage.md
View page @ bc3eefef
...@@ -42,35 +42,49 @@ The (hard) quota settings on philae are as follows: ...@@ -42,35 +42,49 @@ The (hard) quota settings on philae are as follows:
As specified [here](/servers), thesis students do not have access to `icaro.polito.it`. Therefore, the information below is only of interest for other group members. The storage space on icaro is much more limited with respect to philae. Therefore, this server should **not** be used for permanent storage of data. As specified [here](/servers), thesis students do not have access to `icaro.polito.it`. Therefore, the information below is only of interest for other group members. The storage space on icaro is much more limited with respect to philae. Therefore, this server should **not** be used for permanent storage of data.
The appropriate workflow for icaro is composed of the following steps: The appropriate workflow for icaro is composed of the following steps:
1. Transfer data from your local machine or from philae 1. Transfer input data from your local machine or from philae
2. Run your jobs 2. Run your jobs
3. Save the final results to your local machine or to philae and **remove** all other data. 3. Save the final results to your local machine or to philae and **remove** all other data.
### Accessing Philae Files from Icaro ### Accessing Philae Files from Icaro
To make the aforementioned workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from icaro as: To make the aforementioned workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from icaro under the path:
``` ```
icaro.polito.it:/philae/<user> icaro.polito.it:/philae/<user>
``` ```
This allows you to easily read/write files from/to your main storage location (on philae), as long as you don't exceed the 500GB quota mentioned above. This allows you to easily read/write files from/to your main storage location (which should be your home directory on philae), as long as you don't exceed the 500GB quota mentioned above.
Clearly, reading and writing data from another server is slower than from a local disk. However, the two servers have a dedicated Gbit Ethernet connection, so the I/O bandwidth should be sufficient, for copying inputs and outputs at the beginning and at the end of your jobs, respectively.
### Local Storage on Icaro ### Local Storage on Icaro
For temporary local storage, each user has access to two directories: For temporary local storage, each user has access to two directories on icaro:
``` ```
icaro.polito.it:/home/<user> icaro.polito.it:/home/<user>
icaro.polito.it:/space/<user> icaro.polito.it:/space/<user>
``` ```
The quota settings for icaro are as follows. The quota settings for these two folders are as follows.
##### /home/ #### /home/
In `/home/<user>`, each user has a (hard) quota of 10GB. This directory should be used only for configuration files (e.g. RC files) and data that should remain permanently on that server (e.g., [Python Virtual Environments](/virtualenv)). In `/home/<user>`, each user has a (hard) quota of 10GB. This directory should be used only for configuration files (e.g. RC files) and data that should remain permanently on that server (e.g., [Python Virtual Environments](/virtualenv)).
##### /space/ #### /space/
In `/space/<user>` each user has a (soft) quota of 100GB and a (hard) quota of 5T with a grace period of 7 days. This means that users will be able to occupy a large disk space (more than half of the total 8TB available), but only for short periods of time (less than 7 days). After that, the user will not be able to write to disk anymore until he frees all space exceeding 100GB. In `/space/<user>` each user has a (soft) quota of 100GB and a (hard) quota of 5T with a grace period of 7 days. This means that users will be able to occupy a large disk space (more than half of the total 8TB available), but only for short periods of time (less than 7 days). After that, the user will not be able to write to disk anymore until he frees all space exceeding 100GB.
This is intended to allow users to have a large temporary storage (e.g., to store This is intended to allow all users to have a large temporary storage (e.g., to accomodate a large input dataset), but at the same time forbid them from keeping a large portion of the (limited) total space for themselves for a long time.
##### Automatic Free of /space/
Importantly, a script running on icaro will **automatically remove** all data in `/space/` for users 7 days after their last login to that server. So:
- You should always remember to login periodically if you have long jobs running
- You should always remember to copy your final results to a permanent location after your jobs terminate. The best option is to copy them to `/philae/<user>`.
Sysadmins are **not** responsible for any data loss caused by this automatic script.
Clone repository
Home

Server Information
Account
Connecting to the Servers
  • Maintaining a Permanent Shell
  • File Transfers
Storage Management and Quotas
Monitoring Resources
  • Monitoring CPU and RAM
  • Monitoring GPU
  • Monitoring Storage
Gitlab
Software and Libraries
  • Additional Software on Philae
  • Remote Code Deployment
  • Python Virtual Environments
  • EDA Technology Libraries
  • Shared Datasets
Miscellaneous
  • Setting CUDA Drivers on Icaro

Sidebar