... | @@ -42,35 +42,49 @@ The (hard) quota settings on philae are as follows: |
... | @@ -42,35 +42,49 @@ The (hard) quota settings on philae are as follows: |
|
As specified [here](/servers), thesis students do not have access to `icaro.polito.it`. Therefore, the information below is only of interest for other group members. The storage space on icaro is much more limited with respect to philae. Therefore, this server should **not** be used for permanent storage of data.
|
|
As specified [here](/servers), thesis students do not have access to `icaro.polito.it`. Therefore, the information below is only of interest for other group members. The storage space on icaro is much more limited with respect to philae. Therefore, this server should **not** be used for permanent storage of data.
|
|
|
|
|
|
The appropriate workflow for icaro is composed of the following steps:
|
|
The appropriate workflow for icaro is composed of the following steps:
|
|
1. Transfer data from your local machine or from philae
|
|
1. Transfer input data from your local machine or from philae
|
|
2. Run your jobs
|
|
2. Run your jobs
|
|
3. Save the final results to your local machine or to philae and **remove** all other data.
|
|
3. Save the final results to your local machine or to philae and **remove** all other data.
|
|
|
|
|
|
### Accessing Philae Files from Icaro
|
|
### Accessing Philae Files from Icaro
|
|
|
|
|
|
To make the aforementioned workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from icaro as:
|
|
To make the aforementioned workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from icaro under the path:
|
|
```
|
|
```
|
|
icaro.polito.it:/philae/<user>
|
|
icaro.polito.it:/philae/<user>
|
|
```
|
|
```
|
|
This allows you to easily read/write files from/to your main storage location (on philae), as long as you don't exceed the 500GB quota mentioned above.
|
|
This allows you to easily read/write files from/to your main storage location (which should be your home directory on philae), as long as you don't exceed the 500GB quota mentioned above.
|
|
|
|
|
|
|
|
Clearly, reading and writing data from another server is slower than from a local disk. However, the two servers have a dedicated Gbit Ethernet connection, so the I/O bandwidth should be sufficient, for copying inputs and outputs at the beginning and at the end of your jobs, respectively.
|
|
|
|
|
|
|
|
|
|
### Local Storage on Icaro
|
|
### Local Storage on Icaro
|
|
|
|
|
|
For temporary local storage, each user has access to two directories:
|
|
For temporary local storage, each user has access to two directories on icaro:
|
|
```
|
|
```
|
|
icaro.polito.it:/home/<user>
|
|
icaro.polito.it:/home/<user>
|
|
icaro.polito.it:/space/<user>
|
|
icaro.polito.it:/space/<user>
|
|
```
|
|
```
|
|
|
|
|
|
The quota settings for icaro are as follows.
|
|
The quota settings for these two folders are as follows.
|
|
|
|
|
|
##### /home/
|
|
#### /home/
|
|
|
|
|
|
In `/home/<user>`, each user has a (hard) quota of 10GB. This directory should be used only for configuration files (e.g. RC files) and data that should remain permanently on that server (e.g., [Python Virtual Environments](/virtualenv)).
|
|
In `/home/<user>`, each user has a (hard) quota of 10GB. This directory should be used only for configuration files (e.g. RC files) and data that should remain permanently on that server (e.g., [Python Virtual Environments](/virtualenv)).
|
|
|
|
|
|
##### /space/
|
|
#### /space/
|
|
|
|
|
|
In `/space/<user>` each user has a (soft) quota of 100GB and a (hard) quota of 5T with a grace period of 7 days. This means that users will be able to occupy a large disk space (more than half of the total 8TB available), but only for short periods of time (less than 7 days). After that, the user will not be able to write to disk anymore until he frees all space exceeding 100GB.
|
|
In `/space/<user>` each user has a (soft) quota of 100GB and a (hard) quota of 5T with a grace period of 7 days. This means that users will be able to occupy a large disk space (more than half of the total 8TB available), but only for short periods of time (less than 7 days). After that, the user will not be able to write to disk anymore until he frees all space exceeding 100GB.
|
|
|
|
|
|
This is intended to allow users to have a large temporary storage (e.g., to store
|
|
This is intended to allow all users to have a large temporary storage (e.g., to accomodate a large input dataset), but at the same time forbid them from keeping a large portion of the (limited) total space for themselves for a long time.
|
|
|
|
|
|
|
|
##### Automatic Free of /space/
|
|
|
|
|
|
|
|
Importantly, a script running on icaro will **automatically remove** all data in `/space/` for users 7 days after their last login to that server. So:
|
|
|
|
- You should always remember to login periodically if you have long jobs running
|
|
|
|
- You should always remember to copy your final results to a permanent location after your jobs terminate. The best option is to copy them to `/philae/<user>`.
|
|
|
|
|
|
|
|
Sysadmins are **not** responsible for any data loss caused by this automatic script.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|