[gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy

[Konstantin Arnold]

Hi Tejas,

we have observed this behavior too, typically when automounter or open 
file handles are involved. We tried to close the file handles (kill the 
process that has them open), next tried to unmount the automouter path 
and then mmumount filesystem again. In cases where we could not find and 
close the open file handles (e.g. process cannot be killed anymore) or 
you're stuck with unmounting the autoumounter path, we had to stop GPFS 
and verify the kernel module gets unloaded. In case the kernel module 
stays loaded, our only option was to reboot the node.

Best
Konstantin



On 06/29/2016 05:58 PM, Tejas Rao wrote:
> "umount -l" returns with "not mounted" message. See below.
>
>
> [root@ ~]# umount -l /gpfs01
> umount: /gpfs01: not mounted
>
> [root@ ~]# umount -l /dev/gpfs01
> umount: /dev/gpfs01: not mounted
> [root@ ~]#
>
>
> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01
> Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ...
> mount: /dev/gpfs01 already mounted or /gpfs01 busy
> mmmount: Command failed. Examine previous error messages to determine cause.
>
> Thanks.
>
>
> On 6/29/2016 11:46, Buterbaugh, Kevin L wrote:
>> Hi Tejas,
>>
>> One thing that I have had occasional success with is a lazy unmount -
>> i.e. “unmount -l /gpfs01”.  Doesn’t help more of the time than it does
>> help in my experience, but it’s one thing I try before bouncing a
>> node.  HTHAL…
>>
>> Kevin
>>
>>> On Jun 29, 2016, at 10:43 AM, Tejas Rao
>>> <<mailto:raot at bnl.gov>raot at bnl.gov> wrote:
>>>
>>> Hi all,
>>>
>>> I cannot get the filesystem mounted again on few nodes.
>>>
>>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01
>>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ...
>>> mount: /dev/gpfs01 already mounted or /gpfs01 busy
>>> mmmount: Command failed. Examine previous error messages to determine
>>> cause.
>>>
>>>
>>>
>>> Things I have tried and checked.
>>>
>>>
>>>
>>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both
>>> /dev/gpfs01 and /gpfs01 are not seen anywhere.
>>>
>>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules.
>>> See below.
>>>
>>> 3. mmumount gpfs01 -f
>>>
>>>
>>> A reboot of the client would probably fix it but as the client is
>>> used in production, this is not easy to do.
>>>
>>>
>>> Are there any tricks/hacks to remount the filesystem again without
>>> rebooting the client.
>>>
>>>
>>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown
>>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of
>>> GPFS file systems
>>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons
>>> Shutting down!
>>> 'shutdown' command about to kill process 11431
>>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra
>>> mmfsenv: Module mmfslinux is still in use.
>>> Unmount all GPFS file systems and issue the command:
>>>  mmfsadm cleanup
>>> Unloading module mmfs26
>>> ERROR: Module mmfs26 is in use
>>> mmfsenv: Error unloading module mmfs26.
>>> Unloading module mmfslinux
>>> ERROR: Module mmfslinux is in use by mmfs26
>>> mmfsenv: Error unloading module mmfslinux.
>>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished
>>> [root@~]#
>>>
>>> Thanks.
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>> —
>> Kevin Buterbaugh - Senior System Administrator
>> Vanderbilt University - Advanced Computing Center for Research and
>> Education
>> Kevin.Buterbaugh at vanderbilt.edu
>> <mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633
>>
>>
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 

-------------------------------------------------------------------------------------------
Konstantin Arnold | University of Basel & SIB
Klingelbergstrasse 50/70 | CH-4056 Basel | Phone: +41 61 267 15 82
Email: konstantin.arnold at unibas.ch