Last Friday (25th Sept) Rick Welp and I held an informal "Spectrum Scale Performance" roundtable (although for the sake of transparency I must admit that the table was more rectangular than round). The chosen venue was IBM Southbank which neither of us had previously had the pleasure of visiting. The idea was to get together with a small number of GPFS/Spectrum Scale power users
and take advantage of their years of experience to gain insight into the kind of real world issues they deal with and more importantly what we can do to help mitigate them.
To add to that the idea of this post is to share some of the topics we discussed with the wider community and hopefully generate further discussion/ideas. We will look to hold another session, probably in Manchester, in the not too distant future and will be looking for willing participants.
So without further ado I'll share what we took home from the meeting so if anyone has any strong feelings on any of these topics or just wants to share their experiences please drop me an email. Because of the nature of the meeting and the hopes that this blog will spark further input from the community this is going to be a bit more of a list than I'd usually prefer from a blog so please forgive me for the lack of sparkling prose.
(These are in no particular order other than the order they came up)
Token limit management is a dark art - how do you know how many tokens you need for different scenarios? The auto management of token limits in Spectrum Scale 4.1.1 is still not good enough usually requiring the limit to be manually tuned upwards considerably.
This is a direct quote:
"Also practical experience would tend to suggest that the automatic deadlock detection is, by default, enabled too globally. It tends to kick in one one machine (sometime for a valid reason, sometimes, perhaps not so).
The load then shoots up on that machine, higher level services fail (such as CTDB) or impact user experience with respect to NFS and CIFS traffic. The node is then failed, traffic moves to another nodes, and the process repeats as a domino effect. We now turn it off by default, which means you miss your data, which was the whole reason for you shipping with it enabled. You need to publish the whitelist information (and recommend what those should be) and remove deadlock detection for things such as disk checks."
It seems that there is room for improvement with AFM (Active File Management) especially with the fact that it uses NFS somewhere in the process which can lead to an NFS bottleneck when moving large amounts of data.
Why does AFM use a single threaded prefetch? Can we use parallel NFS through AFM to increase the throughput across clusters?
If a 40gig network loses connectivity and so everything is rerouted to the back-up 1gig network it'd be great to have it automatically reroute back to the 40gig network if/when it comes back.
Yuri wrote 2 whitepapers and these were excellent. Can there be more of the same please maybe covering ILM (Information Lifecycle Management) or mmfsck.
Can we track updates to the filesystem at the - who - what - when level? Can we see what folders are getting written to, who is doing it and how much data is being written. This needs to be done at a real-time interval so they issues can be dealt with as they come up.
Provide an rpm containing a generator for several types of workload and then runs through a standard performance test to show performance stats that would be agreeable for GPFS support to validate client performance issues and help triage.
The ability to track NFS operations through the node, through the vfs, to the disk, and back again.
Metrics for SMB and NFS shares.
Can we have a gpfs snaps --light option? So that we get logs from dark accounts? No password files, root files, user/group names, file paths or file names, binary data or IPs. Also only snap the relevant nodes not every node in the system.
All in all the meeting felt like a great success; we got what we wanted out of it - a better idea of the real issues being faced by Spectrum Scale customers and hopefully they got what they wanted out of it - the ability to speak face to face with developers in a relaxed, sales speak free, environment with the hope of effecting real product development. Thanks a lot to Jez, Mark and Dave for giving us both their time and their input.