[gpfsug-discuss] pagepool shrink doesn't release all memory

Aaron Knister aaron.s.knister at nasa.gov
Fri Feb 23 03:24:00 GMT 2018


This is also interesting (although I don't know what it really means). 
Looking at pmap run against mmfsd I can see what happens after each step:

# baseline
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020000000000 1048576K 1048576K 1048576K 1048576K      0K rwxp [anon]
Total:           1613580K 1191020K 1189650K 1171836K      0K

# tschpool 64G
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020000000000 67108864K 67108864K 67108864K 67108864K      0K rwxp [anon]
Total:           67706636K 67284108K 67282625K 67264920K      0K

# tschpool 1G
00007fffe4639000  59164K      0K      0K      0K      0K ---p [anon]
00007fffd837e000  61960K      0K      0K      0K      0K ---p [anon]
0000020001400000 139264K 139264K 139264K 139264K      0K rwxp [anon]
0000020fc9400000 897024K 897024K 897024K 897024K      0K rwxp [anon]
0000020009c00000 66052096K      0K      0K      0K      0K rwxp [anon]
Total:           67706636K 1223820K 1222451K 1204632K      0K

Even though mmfsd has that 64G chunk allocated there's none of it 
*used*. I wonder why Linux seems to be accounting it as allocated.

-Aaron

On 2/22/18 10:17 PM, Aaron Knister wrote:
> I've been exploring the idea for a while of writing a SLURM SPANK plugin 
> to allow users to dynamically change the pagepool size on a node. Every 
> now and then we have some users who would benefit significantly from a 
> much larger pagepool on compute nodes but by default keep it on the 
> smaller side to make as much physmem available as possible to batch work.
> 
> In testing, though, it seems as though reducing the pagepool doesn't 
> quite release all of the memory. I don't really understand it because 
> I've never before seen memory that was previously resident become 
> un-resident but still maintain the virtual memory allocation.
> 
> Here's what I mean. Let's take a node with 128G and a 1G pagepool.
> 
> If I do the following to simulate what might happen as various jobs 
> tweak the pagepool:
> 
> - tschpool 64G
> - tschpool 1G
> - tschpool 32G
> - tschpool 1G
> - tschpool 32G
> 
> I end up with this:
> 
> mmfsd thinks there's 32G resident but 64G virt
> # ps -o vsz,rss,comm -p 24397
>     VSZ   RSS COMMAND
> 67589400 33723236 mmfsd
> 
> however, linux thinks there's ~100G used
> 
> # free -g
>               total       used       free     shared    buffers     cached
> Mem:           125        100         25          0          0          0
> -/+ buffers/cache:         98         26
> Swap:            7          0          7
> 
> I can jump back and forth between 1G and 32G *after* allocating 64G 
> pagepool and the overall amount of memory in use doesn't balloon but I 
> can't seem to shed that original 64G.
> 
> I don't understand what's going on... :) Any ideas? This is with Scale 
> 4.2.3.6.
> 
> -Aaron
> 

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776



More information about the gpfsug-discuss mailing list