Skip to content

Handle your data

Storage#

Several volumes of data storage are available on the ABiMS Cluster. Use it knowingly.

Usage Quota (default) Backup policy
/shared/home/ Home directory (personal data) 100GB No
/shared/projects/ Scientific and project (common data) 250GB Yes
/shared/scratch/ For temporary processing data. File older than 90 days are automatically deleted. 500GB No
/shared/bank Read-Only. Common banks (UniProt, RefSeq, ...)

If you need or plan to use more storage space, please contact us: support.abims@sb-roscoff.fr

Backup#

ABiMS provides a backup on your project directory located in /shared/projects/

⚠️ To take advantage of this process, you have to follow some rules:

  • Only the subdirectories archive, script and finalresult are backed up.
  • You must place these subdirectories at the root of your project folder.
  • Please be smart in your backups for our finances and the planet.

Example:

/shared/projects/<my_project>/
├── [...]
├── finalresult
├── script
├── archive
├── [...]

Snapshots#

ABiMS also provides snapshots for short term recovery of your data to protect against deletion by error.

Project#

A project is a directory on the storage associated with users who can access it.

In order to have your folder and space on /shared/projects/<myproject\>, you have to request a new project on the IFB Core Cluster Account Manager: https://my.sb-roscoff.fr/manager2/project

You can have multiple projects (and it's recommended).

More information on The project concept at ABiMS

Transfer#

SSH protocol (or SFTP - SSH File Transfer Protocol) is the only protocol available. But, you can also use clients on the cluster to download your data (wget, curl, scp, rsync, wget, ftp, etc.).

You can get/put your data as described below.

Command line#

⚠️ For the ABiMS infrastructure, replace the URL in the following video by ssh.sb-roscoff.fr

On your workstation or on the cluster, you can use command line with your favorite client: scp, rsync, etc.

SFTP Graphic client#

Use a graphic client like FileZilla on your workstation.

Or use your file manager directly if it's possible (SFTP support).

SSHFS Client#

⚠️ For the ABiMS infrastructure, replace the URL in the following video by ssh.sb-roscoff.fr

Use your file manager with SSHFS

Editors#

Several editors and tools are available on the cluster:

  • vim
  • emacs
  • nano

Or graphical editors. Graphic tools need some configuration: see Export display.

  • xemacs
  • geany
  • gedit

Or use web interface:

Export display#

Linux / Mac#

Simply use -Y option:

ssh -Y <username>@ssh.sb-roscoff.fr

Windows#

1- Install a X server (for example: VcXsrv Windows X Server)
2- Launch the X server
3- Activate X11 forwarding

> Example with PuTTY
X11 Forwarding with PuTTY

Once connected, you can launch your program with a graphical user interface.

Manage#

Several possibilities to browse, manage and edit or view your data/scripts

  • Directly on the cluster with the command line

  • Get the data back on your workstation (see Transfer) and work locally.

  • Use SSHFS on your workstation (see above: SSHFS)

Analysis of disk space used#

Status_bars#

When you connect to ssh, you should have a view of your project spaces and their quotas (size, number of files) with the status_bars.

project1 [##------------------]      36 /     250 GB
project2 [#################---]     390 /     450 GB
Update: 2024-07-31 14:00 - Your current default account is admin - More info: status_bars --help

⚠️ status_bar is updated hourly. So it may still display the old quota though you've freed up space.

Disk usage analyzer: NCurses Disk Usage ncdu#

ncdu (NCurses Disk Usage) is a command-line tool that lets you quickly and efficiently analyse disk space usage on your system. It lists the size of each folder in a simplified interface. It is based on the du command but is much more user-friendly.

cd /shared/project/project1/
ncdu

Some major keys (press the key when ncdu is opened):

  • c Display the number of items/files
  • n Sort by name (ascending/descending)
  • s Sort by size (ascending/descending)
  • C Sort by items (ascending/descending)
  • M Sort by mtime (-e flag)
  • ? Open help screen

Some complement documentations: https://dev.yorhel.nl/ncdu