[gpfsug-discuss] mount: /dev/gpfs01 already mounted or /gpfs01 busy

[Tejas Rao]

Yes, open file handles were probably the issue, although I did not see 
them in lsof. I restarted my application and now I can mount /gpfs01/ 
again.

Thanks,

Tejas.


On 6/29/2016 12:05, Konstantin Arnold wrote:
> Hi Tejas,
>
> we have observed this behavior too, typically when automounter or open 
> file handles are involved. We tried to close the file handles (kill 
> the process that has them open), next tried to unmount the automouter 
> path and then mmumount filesystem again. In cases where we could not 
> find and close the open file handles (e.g. process cannot be killed 
> anymore) or you're stuck with unmounting the autoumounter path, we had 
> to stop GPFS and verify the kernel module gets unloaded. In case the 
> kernel module stays loaded, our only option was to reboot the node.
>
> Best
> Konstantin
>
>
>
> On 06/29/2016 05:58 PM, Tejas Rao wrote:
>> "umount -l" returns with "not mounted" message. See below.
>>
>>
>> [root@ ~]# umount -l /gpfs01
>> umount: /gpfs01: not mounted
>>
>> [root@ ~]# umount -l /dev/gpfs01
>> umount: /dev/gpfs01: not mounted
>> [root@ ~]#
>>
>>
>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01
>> Wed Jun 29 11:56:58 EDT 2016: mmmount: Mounting file systems ...
>> mount: /dev/gpfs01 already mounted or /gpfs01 busy
>> mmmount: Command failed. Examine previous error messages to determine 
>> cause.
>>
>> Thanks.
>>
>>
>> On 6/29/2016 11:46, Buterbaugh, Kevin L wrote:
>>> Hi Tejas,
>>>
>>> One thing that I have had occasional success with is a lazy unmount -
>>> i.e. “unmount -l /gpfs01”.  Doesn’t help more of the time than it does
>>> help in my experience, but it’s one thing I try before bouncing a
>>> node.  HTHAL…
>>>
>>> Kevin
>>>
>>>> On Jun 29, 2016, at 10:43 AM, Tejas Rao
>>>> <<mailto:raot at bnl.gov>raot at bnl.gov> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I cannot get the filesystem mounted again on few nodes.
>>>>
>>>> [root@ ~]# /usr/lpp/mmfs/bin/mmmount gpfs01
>>>> Wed Jun 29 11:29:50 EDT 2016: mmmount: Mounting file systems ...
>>>> mount: /dev/gpfs01 already mounted or /gpfs01 busy
>>>> mmmount: Command failed. Examine previous error messages to determine
>>>> cause.
>>>>
>>>>
>>>>
>>>> Things I have tried and checked.
>>>>
>>>>
>>>>
>>>> 1. Checked /proc/mounts and /etc/mtab and lsof /dev/gpfs01 and both
>>>> /dev/gpfs01 and /gpfs01 are not seen anywhere.
>>>>
>>>> 2. Tried mmshutdown/mmstartup and it fails to unload mmfs modules.
>>>> See below.
>>>>
>>>> 3. mmumount gpfs01 -f
>>>>
>>>>
>>>> A reboot of the client would probably fix it but as the client is
>>>> used in production, this is not easy to do.
>>>>
>>>>
>>>> Are there any tricks/hacks to remount the filesystem again without
>>>> rebooting the client.
>>>>
>>>>
>>>> [root@~]# /usr/lpp/mmfs/bin/mmshutdown
>>>> Wed Jun 29 11:10:52 EDT 2016: mmshutdown: Starting force unmount of
>>>> GPFS file systems
>>>> Wed Jun 29 11:10:57 EDT 2016: mmshutdown: Shutting down GPFS daemons
>>>> Shutting down!
>>>> 'shutdown' command about to kill process 11431
>>>> Unloading modules from /lib/modules/2.6.32-504.8.1.el6.x86_64/extra
>>>> mmfsenv: Module mmfslinux is still in use.
>>>> Unmount all GPFS file systems and issue the command:
>>>>  mmfsadm cleanup
>>>> Unloading module mmfs26
>>>> ERROR: Module mmfs26 is in use
>>>> mmfsenv: Error unloading module mmfs26.
>>>> Unloading module mmfslinux
>>>> ERROR: Module mmfslinux is in use by mmfs26
>>>> mmfsenv: Error unloading module mmfslinux.
>>>> Wed Jun 29 11:11:03 EDT 2016: mmshutdown: Finished
>>>> [root@~]#
>>>>
>>>> Thanks.
>>>> _______________________________________________
>>>> gpfsug-discuss mailing list
>>>> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>>
>>> —
>>> Kevin Buterbaugh - Senior System Administrator
>>> Vanderbilt University - Advanced Computing Center for Research and
>>> Education
>>> Kevin.Buterbaugh at vanderbilt.edu
>>> <mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> gpfsug-discuss mailing list
>>> gpfsug-discuss at spectrumscale.org
>>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>>
>>
>> _______________________________________________
>> gpfsug-discuss mailing list
>> gpfsug-discuss at spectrumscale.org
>> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>>
>