|
# Storage Management and Quotas
|
|
# Storage Management and Quotas
|
|
|
|
|
|
|
|
The servers have quite different storage capabilities and settings.
|
|
|
|
|
|
The servers have quite different storage capabilities and settings. However, all servers have quotas activated to limit the disk occupation of single users. If you don't know what different types of quota mean, please check [online](https://en.wikipedia.org/wiki/Disk_quota).
|
|
Some of the servers also have _quotas_ activated to limit the disk occupation of single users. If you don't know what different types of quota mean, please check [online](https://en.wikipedia.org/wiki/Disk_quota).
|
|
|
|
|
|
Users can check their quota limits and current occupied space usage with the following command:
|
|
Users can check their quota limits and current occupied space usage with the following command:
|
|
|
|
|
|
```
|
|
```plaintext
|
|
quota -s
|
|
quota -s
|
|
```
|
|
```
|
|
|
|
|
|
Group quotas (currently not used) are checked with:
|
|
Group quotas (currently not used) are checked with:
|
|
|
|
|
|
```
|
|
```plaintext
|
|
quota -s -g
|
|
quota -s -g
|
|
```
|
|
```
|
|
|
|
|
|
The minimum between the user and group quotas is what determines the available space for a user.
|
|
The minimum between the user and group quotas is what determines the available space for a user.
|
|
|
|
|
|
**IMPORTANT:** sysadmins are *not* allowed to modify user quotas for any reason. All requests for a temporary increase of the quota (for legitimate work reasons) should be forwarded directly to Prof. Enrico Macii, who will decide whether to authorize them or not, and inform the sysadmins accordingly.
|
|
**IMPORTANT:** sysadmins are _not_ allowed to modify user quotas for any reason. All requests for a temporary increase of the quota (for legitimate work reasons) should be forwarded directly to Prof. Enrico Macii, who will decide whether to authorize them or not, and inform the sysadmins accordingly.
|
|
|
|
|
|
Below you can find further details on the quotas enforced in each server and on how to transfer data between servers.
|
|
Below you can find further details on the quotas enforced in each server and on how to transfer data between servers.
|
|
|
|
|
|
|
|
|
|
## Storage on Philae
|
|
## Storage on Philae
|
|
|
|
|
|
Currently, `philae.polito.it` is where most of the storage resources are. Therefore, the main location for storing your work data is:
|
|
Currently, `philae.polito.it` is where most of the storage resources are. Therefore, the main location for storing your work data is:
|
|
|
|
|
|
```
|
|
```plaintext
|
|
philae.polito.it:/home/<user>
|
|
philae.polito.it:/home/<user>
|
|
```
|
|
```
|
|
|
|
|
|
The (hard) quota settings on philae are as follows:
|
|
The (hard) quota settings on philae are as follows:
|
|
|
|
|
|
| Role | Quota |
|
|
| Role | Quota |
|
|
| ------ | ------ |
|
|
|------|-------|
|
|
| Thesis Students | 10 GB |
|
|
| Thesis Students | 10 GB |
|
|
| Other Group Members | 500 GB |
|
|
| Other Group Members | 500 GB |
|
|
|
|
|
|
## Storage on Icaro
|
|
## Storage on Other Servers
|
|
|
|
|
|
> :warning: Thesis students only have access to `philae.polito.it` (see ["Servers Information"](/servers)). So, this sub-section is only for other group members.
|
|
Other servers have much more limited disk space with respect to `philae`. Therefore, they should **not** be used for permanent storage of large amounts of data.
|
|
|
|
|
|
The storage space on icaro is much more limited with respect to philae. Therefore, this server should **not** be used for permanent storage of large amounts of data.
|
|
The appropriate workflow to manage large amounts of data on the other servers is composed of the following steps:
|
|
|
|
|
|
The appropriate workflow for icaro is composed of the following steps:
|
|
|
|
1. Transfer input data from your local machine or from philae
|
|
1. Transfer input data from your local machine or from philae
|
|
2. Run your jobs
|
|
2. Run your jobs
|
|
3. Save the final results to your local machine or to philae and **remove** all other data.
|
|
3. Save the final results to your local machine or to philae and **remove** all other data.
|
|
|
|
|
|
### Accessing Philae Files from Icaro
|
|
### Accessing Philae Files from Other Servers
|
|
|
|
|
|
To make the aforementioned workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from icaro under the path:
|
|
> :warning: This mechanism is currently implemented only on `icaro`. It is planned to add it also for other servers.
|
|
```
|
|
|
|
icaro.polito.it:/philae/<user>
|
|
To make the above workflow easier, all users' home directories on `philae.polito.it:/home/<user>` are also visible from other servers under the path:
|
|
|
|
|
|
|
|
```plaintext
|
|
|
|
<servername>.polito.it:/philae/<user>
|
|
```
|
|
```
|
|
|
|
|
|
This allows you to easily read/write files from/to your main storage location (which should be your home directory on philae), as long as you don't exceed the 500GB quota mentioned above.
|
|
This allows you to easily read/write files from/to your main storage location (which should be your home directory on philae), as long as you don't exceed the 500GB quota mentioned above.
|
|
|
|
|
|
Clearly, reading and writing data from another server is slower than from a local disk. However, the two servers have a dedicated Gbit Ethernet connection, so the I/O bandwidth should be sufficient, for copying inputs and outputs at the beginning and at the end of your jobs, respectively.
|
|
Clearly, reading and writing data from another server is slower than from a local disk. However, the communicate with a direct Gbit Ethernet connection, so the I/O bandwidth should be sufficient, for copying inputs and outputs at the beginning and at the end of your jobs, respectively.
|
|
|
|
|
|
|
|
### Local Storage on Other Servers
|
|
|
|
|
|
### Local Storage on Icaro
|
|
For temporary local storage, each user has access to two directories on servers different from `philae`:
|
|
|
|
|
|
For temporary local storage, each user has access to two directories on icaro:
|
|
```plaintext
|
|
```
|
|
<servername>.polito.it:/home/<user>
|
|
icaro.polito.it:/home/<user>
|
|
<servername>.polito.it:/space/<user>
|
|
icaro.polito.it:/space/<user>
|
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
The first folder is created automatically at login, and for newer servers (`corinna`, `stanis`, `biascica`, `karin`) it resides on the faster but smaller SSD drive.
|
|
|
|
The user directory on space can be created by the local server administrator (see [Home](https://philae.polito.it/gitlab/guides/eda-servers-guide/-/wikis/home) for the contact information of the local sysadmin for each server).
|
|
|
|
|
|
|
|
<!-- Commented out as it was never actually implemented
|
|
The quota settings for these two folders are as follows.
|
|
The quota settings for these two folders are as follows.
|
|
|
|
|
|
#### /home
|
|
#### /home
|
... | @@ -82,11 +88,10 @@ This is intended to allow all users to have a large temporary storage (e.g., to |
... | @@ -82,11 +88,10 @@ This is intended to allow all users to have a large temporary storage (e.g., to |
|
#### Automatic Cleaning of /space
|
|
#### Automatic Cleaning of /space
|
|
|
|
|
|
**IMPORTANT**: a script running on icaro will **automatically remove** all data in `/space/<user>` 7 days after the user's last login to that server. So:
|
|
**IMPORTANT**: a script running on icaro will **automatically remove** all data in `/space/<user>` 7 days after the user's last login to that server. So:
|
|
|
|
|
|
- You should always remember to login periodically if you have long jobs running
|
|
- You should always remember to login periodically if you have long jobs running
|
|
- You should always remember to copy your final results to a permanent location after your jobs terminate. The best option is to copy them to `/philae/<user>`.
|
|
- You should always remember to copy your final results to a permanent location after your jobs terminate. The best option is to copy them to `/philae/<user>`.
|
|
|
|
|
|
Sysadmins are **not** responsible for any data loss caused by this automatic script.
|
|
Sysadmins are **not** responsible for any data loss caused by this automatic script.
|
|
|
|
|
|
|
|
--> |
|
|
|
\ No newline at end of file |
|
|
|
|