RFS Migration 2022

In June and July 2022, the RFS storage platform is being upgraded to a new solution built on top of the Dell Powerscale platform (previously known as Isilon).

Dell Powerscale is the industry leader in scale-out file storage. This solution is a significant advance over the previous platform in both features and performance, enabling us to make several improvements to the RFS service.

Enhancements to the RFS service:
  • Encryption at rest - All data on the platform is now encrypted using self-encrypting drives

  • Enhanced performance - Powerscale is a scale-out, clustered solution for Research File Storage. We now have 6 times the server resources. This will improve capacity for concurrent users, and provide superior responsiveness.

  • File-level auditing and ransomware protection - the platform includes a software suite that provides detailed auditing capabilities and automated alerts for ransomware attacks.

All existing RFS features are still available including:
  • Hourly, daily, weekly snapshots - Providing restore window up to one month

  • Data replicated to second data centre - 30 minute restore point objective.

  • SMBv2, v3 support

What is Changing?

Share location

As a part of this migration, all SMB shares will now be provided through the UIS-run DFS (Distributed File System) service. This has two major benefits:

  • Discoverable shares, consistent namespace - DFS provides a simple, hierarchical namespace of shares. This namespace is separate from the underlying platform. This means that it provides a simple, stable, and easily modifiable path to your RFS share. It also means that all RFS shares you have access to are visible in a single place.

  • Improved latency - DFS also removes the need for our previous load-balancer solution. This removes a step, in the data-path, improving storage latency for users.

This change is already active. The new share location can already be accessed for all current RFS projects at:

\\blue.cam.ac.uk\RFS

All current RFS projects are visible at this location. We encourage all users to convert to this path as soon as possible, so that you are ready for the future migration of projects to the new platform.

The old share locations at the paths, \\rfs.uis.private.cam.ac.uk\rfs-<your project name> will remain active until your RFS project is migrated to the new platform. After a project is migrated to the new platform, the old share locations will no longer be active. All users of the RFS project must update the share location on their computers in advance of this migration to continue using the service.

Guidance on how to make this change on your computer is provided by the guides linked below: [IMPORTANT] - What you need to do before 18th July

Snapshots

One major improvement we are making to the platform with the migration that will benefit all projects is:

  • Snapshot data no longer included in usage - From now on, all snapshot data usage will no longer count towards your quota usage. A 5TB project will be able to store 5TB, irrespective of how much data is stored in snapshots.

    This has some immediate user usability benefits:

    • Deleted data now immediately shows as freed space - Previously, when you deleted data, your storage usage did not reduce immediately due to that data still being held in snapshots. Now, this will immediately show as free space, so you can use this space immediately for something else.

    • Clearly see how much data you are using - The size of your share and it’s free space as shown in Windows Explorer and Mac OS Finder will now properly reflect the real capacity you have purchased and how much data you are using, so this is more transparent.

Quota / Storage Usage

Moving to a new storage platform will have consequences for how storage utilisation is accounted for. The migration is to a new filesystem technology, with different capabilities around data protection. We have found that the reported data usage for individual projects differs between the old and new storage systems.

Some projects will show on the new platform as using more data than they did on the old, some less. This difference depends on the type of data, for example how compressible it is, or how much overhead there is due to many small files.

Our goal with this platform change is to ensure that no group feels they are left worse off by the changes. We will, therefore discuss with projects that are particularly impacted about how we can mitigate this.

If you would like to discuss your project’s data usage please contact us at support@hpc.cam.ac.uk. We can provide the latest figures on the impact of the change on your project.

Migration Timeline

All projects will be migrated individually. We aim to carry this out in coordination with Data Managers on a date that is most convenient to you.

Where it is not possible to find a convenient date, or in cases where we are not able to contact Data Managers for a project, these projects will be migrated in bulk, starting on the evening of Monday 18th July. Projects will be inaccessible while being migrated.

Data Manager Scheduled Migrations

Prior to the scheduled day of Migration

  • Copies of RFS projects will be made on the new storage platform by RFS support staff

  • All RFS users to update their mount locations on personal computers to use the new \\blue.cam.ac.uk\RFS location, following instructions provided in [IMPORTANT] - What you need to do before 18th July

  • RFS Data managers to contact RFS support by emailing support@hpc.cam.ac.uk to schedule a convenient migration date

On the Day of Migration Chosen

  • Your RFS project will be stopped at the agreed upon time and date, after which it will be inaccessible to users until the migration is complete

  • RFS support staff will perform a final sync of the data. We are aiming to complete migrations within 24 hours or less for most projects. this timetable is dependent on the number of files in the project: it may not be achievable for a small number of the very largest projects. RFS support staff will keep you updated on progress throughout the day.

  • Once complete, the project will be accessible under the \\blue.cam.ac.uk\RFS location only.

Unscheduled Migrations

If we do not hear from Data Managers, or if we cannot find a mutually convenient migration date, we will begin your RFS migration on the evening of Monday 18th July.

18:00 Monday 18th July

  • All outstanding RFS projects still on the old storage platform, will be stopped at 18:00 on Monday 18th July and migrated to the new storage platform.

  • Access to these projects will not be possible until the final synchronisation of the data to the new platform is completed. This may take up until the end of Tuesday 19th July.

Tuesday 19th July

  • Data synchronisation to the new storage platform will continue until it is complete.

  • Once synchronisation is complete, the project will be accessible under the \\blue.cam.ac.uk\RFS location only.

[IMPORTANT] - What you need to do before 18th July

Important

This is a disruptive change. All RFS users need to undertake the following steps to maintain access to RFS post-migration to the new platform.

Please read the following guidance on what you need to do to prepare for your particular operating system.

All RFS Users

  • Update your RFS mount path on your computer to use the new location, \\blue.cam.ac.uk\RFS to ensure RFS does not stop working for you after migration.

RFS Data Managers

  • Discuss with your users what date would be most convenient to perform the migration of your RFS project to the new platform.

  • Contact support@hpc.cam.ac.uk to schedule your migration with our support team

  • Communicate with your RFS user group to coordinate the day of migration and notify them of the disruption to their access during this time.

Questions

If you have any questions about this process or any issues before or after the migration, please contact us at support@hpc.cam.ac.uk