.. _rdv_principle:


Design principles of Research Datavault
=======================================

Research DataVault is designed to robustly store data across multiple tape libraries, ensuring replication of data across multiple sites and provide capacity to access this data interactively. It does this by using a large cache filesystem, accessible from several data servers, such that user data can be directly added to this storage. Once present, this data is then sent to at least two physically disparate libraries before being allowed to be 'released' - that is, deleted from the cache location and allowed to reside on tape only.
In order for this data to be made available to users, this filesystem is shared over NFS to several locations where users may interact with it - here, the SFTP gateways (allowing filesystem-only access) and the CSD3 login nodes (allowing use of the filesystem, plus interactive use of samnfs binary).

.. figure:: ../img/rdv/rdv_blocks.svg
    :align: center
    :alt: Block diagram for RDV
    :scale: 60%

Design considerations on operation
==================================

Tape drives have several unusual characteristics that are not familiar to users of spinning disk or solid state media. They:

  * Are completely linear in design and cannot do multiple reads and writes per tape without needing to reposition
  * Typically append only, and
  * Need feeding with a constant data rate, otherwise suffer an event called 'shoe-shining', where the tape has to be rewound back and forth over the tape head. This is considered a bad event not just from underflowing the write speed, but also because more movement of a tape can cause physical damage.

Thus, we advise users to use large files in order to maintain this consistent write speed and reduce the chances of shoe-shining happening.

Versity, the software used for the caching layer, applies an additional benefit to the data. It will automatically attempt to bundle files up transparently that are smaller than its anticipated 'block' size (currently 100Gb), and break files down that are larger than this, so that the number of segment operations on the tape is minimised.

As a result of the fact that tapes are usually append-only, data updates and deletions can be challenging. When a file is replaced, the content of it must be fully rewritten onto tape to reflect the new data. However, the old data still persists. This allows us the capability to 'roll back' files for a while should this be needed, however many times this is not a requirement and this data acts as a waste of capacity, reducing our storage efficiency.
Thankfully, Versity provides tooling to be able to 'stage' data out onto the cache, and then archive it to another tape, thus clearing an entire tape of any referenced data. This does, however, work on a per-tape basis, thus this process is only effective when a tape is almost empty, otherwise the effort needed to clear a tape out is not efficient.