[gpfsug-discuss] Hardware refresh

Aaron Knister aaron.s.knister at nasa.gov
Tue Oct 11 23:41:36 BST 2016


Thanks Yuri.

I'm asking for my own purposes but I think it's still relevant here: 
we're still at GPFS 3.5 and will be adding dataOnly NSDs with 4K sectors 
in the near future. We're planning to update to 4.1 before we format 
these NSDs, though. If I understand you correctly we can't bring these 
4K NSDv2 NSDs into a filesystem with 512b-based NSDv1 NSDs? That's a 
pretty big deal :(

Reformatting every few years with 10's of petabytes of data is not 
realistic for us (it would take years to move the data around). It also 
goes against my personal preachings about GPFS's storage virtualization 
capabilities: the ability to perform upgrades/make underlying storage 
infrastructure changes with behind-the-scenes data migration, 
eliminating much of the manual hassle of storage administrators doing 
rsync dances. I guess it's RFE time? It also seems as though AFM could 
help with automating the migration, although many of our filesystems do 
not have filesets on them so we would have to re-think how we lay out 
our filesystems.

This is also curious to me with IBM pitching GPFS as a filesystem for 
cloud services (the cloud *never* goes down, right?). Granted I believe 
this pitch started after the NSDv2 format was defined, but if somebody 
is building a large cloud with GPFS as the underlying filesystem for an 
object or an image store one might think the idea of having to re-format 
the filesystem to gain access to critical new features is inconsistent 
with this pitch. It would be hugely impactful. Just my $.02.

As you can tell, I'm frustrated there's no online conversion tool :) Not 
that there couldn't be... you all are brilliant developers.

-Aaron

On 10/11/16 1:22 PM, Yuri L Volobuev wrote:
> This depends on the committed cluster version level (minReleaseLevel)
> and file system format. Since NFSv2 is an on-disk format change, older
> code wouldn't be able to understand what it is, and thus if there's a
> possibility of a downlevel node looking at the NSD, the NFSv1 format is
> going to be used. The code does NSDv1<->NSDv2 conversions under the
> covers as needed when adding an empty NSD to a file system.
>
> I'd strongly recommend getting a fresh start by formatting a new file
> system. Many things have changed over the course of the last few years.
> In particular, having a 4K-aligned file system can be a pretty big deal,
> depending on what hardware one is going to deploy in the future, and
> this is something that can't be bolted onto an existing file system.
> Having 4K inodes is very handy for many reasons. New directory format
> and NSD format changes are attractive, too. And disks generally tend to
> get larger with time, and at some point you may want to add a disk to an
> existing storage pool that's larger than the existing allocation map
> format allows. Obviously, it's more hassle to migrate data to a new file
> system, as opposed to extending an existing one. In a perfect world,
> GPFS would offer a conversion tool that seamlessly and robustly converts
> old file systems, making them as good as new, but in the real world such
> a tool doesn't exist. Getting a clean slate by formatting a new file
> system every few years is a good long-term investment of time, although
> it comes front-loaded with extra work.
>
> yuri
>
> Inactive hide details for Aaron Knister ---10/10/2016 04:45:31 PM---Can
> one format NSDv2 NSDs and put them in a filesystem withAaron Knister
> ---10/10/2016 04:45:31 PM---Can one format NSDv2 NSDs and put them in a
> filesystem with NSDv1 NSD's? -Aaron
>
> From: Aaron Knister <aaron.s.knister at nasa.gov>
> To: <gpfsug-discuss at spectrumscale.org>,
> Date: 10/10/2016 04:45 PM
> Subject: Re: [gpfsug-discuss] Hardware refresh
> Sent by: gpfsug-discuss-bounces at spectrumscale.org
>
> ------------------------------------------------------------------------
>
>
>
> Can one format NSDv2 NSDs and put them in a filesystem with NSDv1 NSD's?
>
> -Aaron
>
> On 10/10/16 7:40 PM, Luis Bolinches wrote:
>> Hi
>>
>> Creating a new FS sounds like a best way to go. NSDv2 being a very good
>> reason to do so.
>>
>> AFM for migrations is quite good, latest versions allows to use NSD
>> protocol for mounts as well. Olaf did a great job explaining this
>> scenario on the redbook chapter 6
>>
>> http://www.redbooks.ibm.com/abstracts/sg248254.html?Open
>>
>> --
>> Cheers
>>
>> On 10 Oct 2016, at 23.05, Buterbaugh, Kevin L
>> <Kevin.Buterbaugh at Vanderbilt.Edu
>> <mailto:Kevin.Buterbaugh at Vanderbilt.Edu>> wrote:
>>
>>> Hi Mark,
>>>
>>> The last time we did something like this was 2010 (we’re doing rolling
>>> refreshes now), so there are probably lots of better ways to do this
>>> than what we did, but we:
>>>
>>> 1) set up the new hardware
>>> 2) created new filesystems (so that we could make adjustments we
>>> wanted to make that can only be made at FS creation time)
>>> 3) used rsync to make a 1st pass copy of everything
>>> 4) coordinated a time with users / groups to do a 2nd rsync when they
>>> weren’t active
>>> 5) used symbolic links during the transition (i.e. rm -rvf
>>> /gpfs0/home/joeuser; ln -s /gpfs2/home/joeuser /gpfs0/home/joeuser)
>>> 6) once everybody was migrated, updated the symlinks (i.e. /home
>>> became a symlink to /gpfs2/home)
>>>
>>> HTHAL…
>>>
>>> Kevin
>>>
>>>> On Oct 10, 2016, at 2:56 PM, Mark.Bush at siriuscom.com
>>>> <mailto:Mark.Bush at siriuscom.com> wrote:
>>>>
>>>> Have a very old cluster built on IBM X3650’s and DS3500.  Need to
>>>> refresh hardware.  Any lessons learned in this process?  Is it
>>>> easiest to just build new cluster and then use AFM?  Add to existing
>>>> cluster then decommission nodes?  What is the recommended process for
>>>> this?
>>>>
>>>>
>>>> Mark
>>>>
>>>> This message (including any attachments) is intended only for the use
>>>> of the individual or entity to which it is addressed and may contain
>>>> information that is non-public, proprietary, privileged,
>>>> confidential, and exempt from disclosure under applicable law. If you
>>>> are not the intended recipient, you are hereby notified that any use,
>>>> dissemination, distribution, or copying of this communication is
>>>> strictly prohibited. This message may be viewed by parties at Sirius
>>>> Computer Solutions other than those named in the message header. This
>>>> message does not contain an official representation of Sirius
>>>> Computer Solutions. If you have received this communication in error,
>>>> notify Sirius Computer Solutions immediately and (i) destroy this
>>>> message if a facsimile or (ii) delete this message immediately if
>>>> this is an electronic communication. Thank you.
>>>>
>>>> Sirius Computer Solutions <http://www.siriuscom.com/>
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org/>
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>>>>> Kevin Buterbaugh - Senior System Administrator
>>> Vanderbilt University - Advanced Computing Center for Research and
>>> Education
>>> Kevin.Buterbaugh at vanderbilt.edu
>>> <mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633
>>>
>>>
>>>
>>
>> Ellei edellä ole toisin mainittu: / Unless stated otherwise above:
>> Oy IBM Finland Ab
>> PL 265, 00101 Helsinki, Finland
>> Business ID, Y-tunnus: 0195876-3
>> Registered in Finland
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>



More information about the gpfsug-discuss mailing list