[gpfsug-discuss] Quota and hardlimit enforcement

Jaime Pinto pinto at scinet.utoronto.ca
Mon Jul 31 21:03:53 BST 2017


In addition, the in_doubt column is a function of the data turn-over  
and the internal gpfs accounting synchronization period (beyond root  
control). The higher the in_doubt values the less accurate the real  
amount of space/inodes a user/group/fileset has in the filesystem.

What I noticed in practice is the the in_doubt values only get worst  
overtime, and work against the quotas, making them hit the limits  
sooner. Therefore, you may wish to run a 'mmcheckquota' crontab job  
once or twice a day, to reset the in_doubt column to zero mover often.  
GPFS has a very high lag to do this on its own in the most recent  
versions, and seldom really catches up on a very active filesystem.

If your grace period is set to 7 days I can assure you that in an HPC  
environment it's the equivalent of not having quotas effectively. You  
should set it to 2 hours or 4 hours.

In an environment such as ours a runway process can easily generate  
500TB of data or 1 billion inodes in few hours, and choke the file  
system to all users/jobs.

Jaime



Quoting "Buterbaugh, Kevin L" <Kevin.Buterbaugh at Vanderbilt.Edu>:

> Hi Renar,
>
> I?m sure this is the case, but I don?t see anywhere in this thread   
> where this is explicitly stated ? you?re not doing your tests as   
> root, are you?  root, of course, is not bound by any quotas.
>
> Kevin
>
> On Jul 31, 2017, at 2:04 PM, Grunenberg, Renar   
> <Renar.Grunenberg at huk-coburg.de<mailto:Renar.Grunenberg at huk-coburg.de>>   
> wrote:
>
>
> Hallo J. Eric, hallo Jaime,
> Ok after we hit the softlimit we see that the graceperiod are go to   
> 7 days. I think that?s the default. But was does it mean.
> After we reach the ?hard?-limit. we see additionaly the gbytes  in_doubt.
> My interpretation now we can write many gb to the nospace-left event  
>  in the filesystem.
> But our intention is to restricted some application to write only to  
>  the hardlimit in the fileset. Any hints to accomplish this?
>
>
>
> Renar Grunenberg
> Abteilung Informatik ? Betrieb
>
> HUK-COBURG
> Bahnhofsplatz
> 96444 Coburg
> Telefon:        09561 96-44110
> Telefax:        09561 96-44104
> E-Mail: Renar.Grunenberg at huk-coburg.de<mailto:Renar.Grunenberg at huk-coburg.de>
> Internet:       www.huk.de<http://www.huk.de/>
>
> ________________________________
> HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter  
> Deutschlands a. G. in Coburg
> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
> Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr.  
> Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel  
> Thomas (stv.).
> ________________________________
> Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte  
> Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht  
> irrtümlich erhalten haben,
> informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser   
> Nachricht ist nicht gestattet.
>
> This information may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this   
> information in error) please notify the
> sender immediately and destroy this information.
> Any unauthorized copying, disclosure or distribution of the material  
>  in this information is strictly forbidden.
> ________________________________
>
>
>
> Von:   
> gpfsug-discuss-bounces at spectrumscale.org<mailto:gpfsug-discuss-bounces at spectrumscale.org> [mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von J. Eric   
> Wonderley
> Gesendet: Montag, 31. Juli 2017 19:55
> An: gpfsug main discussion list   
> <gpfsug-discuss at spectrumscale.org<mailto:gpfsug-discuss at spectrumscale.org>>
> Betreff: Re: [gpfsug-discuss] Quota and hardlimit enforcement
>
> Hi Renar:
> What does 'mmlsquota -j fileset filesystem' report?
> I did not think you would get a grace period of none unless the   
> hardlimit=softlimit.
>
> On Mon, Jul 31, 2017 at 1:44 PM, Grunenberg, Renar   
> <Renar.Grunenberg at huk-coburg.de<mailto:Renar.Grunenberg at huk-coburg.de>>   
> wrote:
> Hallo All,
> we are on Version 4.2.3.2 and see some missunderstandig in the   
> enforcement of hardlimit definitions on a flieset quota. What we see  
>  is we put some 200 GB files on following quota definitions:  quota   
> 150 GB Limit 250 GB Grace none.
> After the creating of one 200 GB we hit the softquota limit, thats   
> ok. But After the the second file was created!! we expect an io   
> error but it don?t happen. We define all well know Parameters   
> (-Q,..) on the filesystem . Is this a bug or a Feature? mmcheckquota  
>  are already running at first.
> Regards Renar.
>
>
> Renar Grunenberg
> Abteilung Informatik ? Betrieb
>
> HUK-COBURG
> Bahnhofsplatz
> 96444 Coburg
>
> Telefon:
>
> 09561 96-44110
>
> Telefax:
>
> 09561 96-44104
>
> E-Mail:
>
> Renar.Grunenberg at huk-coburg.de<mailto:Renar.Grunenberg at huk-coburg.de>
>
> Internet:
>
> www.huk.de<http://www.huk.de/>
>
> ________________________________
> HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter  
> Deutschlands a. G. in Coburg
> Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
> Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
> Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
> Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr.  
> Hans Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel  
> Thomas (stv.).
> ________________________________
> Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte  
> Informationen.
> Wenn Sie nicht der richtige Adressat sind oder diese Nachricht  
> irrtümlich erhalten haben,
> informieren Sie bitte sofort den Absender und vernichten Sie diese Nachricht.
> Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser   
> Nachricht ist nicht gestattet.
>
> This information may contain confidential and/or privileged information.
> If you are not the intended recipient (or have received this   
> information in error) please notify the
> sender immediately and destroy this information.
> Any unauthorized copying, disclosure or distribution of the material  
>  in this information is strictly forbidden.
> ________________________________
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org<http://spectrumscale.org/>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>






          ************************************
           TELL US ABOUT YOUR SUCCESS STORIES
          http://www.scinethpc.ca/testimonials
          ************************************
---
Jaime Pinto
SciNet HPC Consortium - Compute/Calcul Canada
www.scinet.utoronto.ca - www.computecanada.ca
University of Toronto
661 University Ave. (MaRS), Suite 1140
Toronto, ON, M5G1M1
P: 416-978-2755
C: 416-505-1477

----------------------------------------------------------------
This message was sent using IMP at SciNet Consortium, University of Toronto.




More information about the gpfsug-discuss mailing list