[gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2
Felipe Knop
knop at us.ibm.com
Thu Jun 13 20:25:16 BST 2019
Kiran,
If SELinux is disabled (SELinux mode set to 'disabled') then the crash
should not happen, and it should be OK to upgrade to (say) 3.10.0-957.21.2
or stay at that level.
Felipe
----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
From: KG <spectrumscale at kiranghag.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 06/13/2019 12:56 PM
Subject: [EXTERNAL] Re: [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6
kernel 3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Hi
As per the flash -
https://www-01.ibm.com/support/docview.wss?uid=ibm10887213&myns=s033&mynp=OCSTXKQY&mync=E&cm_sp=s033-_-OCSTXKQY-_-E
this bug doesnt appear if SELinux is disabled.
If customer is willing to disable SELinux, will it be ok to upgrade (or
stay on upgraded level and avoid downgrade)?
On Tue, Jun 11, 2019 at 9:24 PM Felipe Knop <knop at us.ibm.com> wrote:
Renar,
With the change below, which is a retrofit of a change deployed in newer
kernels, an inconsistency has taken place between the GPFS kernel
portability layer and the kernel proper. A known result of that
inconsistency is a kernel crash. One known sequence leading to the crash
involves the mkdir() call.
We are working on an official notification on the issue.
Felipe
----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
Inactive hide details for "Grunenberg, Renar" ---06/11/2019 08:28:07
AM---Hallo Felipe, can you explain is this a generic Probl"Grunenberg,
Renar" ---06/11/2019 08:28:07 AM---Hallo Felipe, can you explain is this
a generic Problem in rhel or only a scale related. Are there a
From: "Grunenberg, Renar" <Renar.Grunenberg at huk-coburg.de>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 06/11/2019 08:28 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6
kernel 3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Hallo Felipe,
can you explain is this a generic Problem in rhel or only a scale
related. Are there any cicumstance already available? We ask redhat but
have no points that this are know to them?
Regards Renar
Renar Grunenberg
Abteilung Informatik - Betrieb
HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon: 09561 96-44110
Telefax: 09561 96-44104
E-Mail: Renar.Grunenberg at huk-coburg.de
Internet: www.huk.de
HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter
Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans
Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.
Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese
Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
ist nicht gestattet.
This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information
in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in
this information is strictly forbidden.
Von: gpfsug-discuss-bounces at spectrumscale.org <
gpfsug-discuss-bounces at spectrumscale.org> Im Auftrag von Felipe Knop
Gesendet: Montag, 10. Juni 2019 15:43
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: Re: [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel
3.10.0-957.21.2
Renar,
Thanks. Of the changes below, it appears that
* security: double-free attempted in security_inode_init_security()
(BZ#1702286)
was the one that ended up triggering the problem. Our investigations now
show that RHEL kernels >= 3.10.0-957.19.1 are impacted.
Felipe
----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
Inactive hide details for "Grunenberg, Renar" ---06/10/2019 08:43:27
AM---Hallo Felipe, here are the change list:"Grunenberg, Renar"
---06/10/2019 08:43:27 AM---Hallo Felipe, here are the change list:
From: "Grunenberg, Renar" <Renar.Grunenberg at huk-coburg.de>
To: "'gpfsug-discuss at spectrumscale.org'" <
gpfsug-discuss at spectrumscale.org>
Date: 06/10/2019 08:43 AM
Subject: [EXTERNAL] [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6
kernel 3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Hallo Felipe,
here are the change list:
RHBA-2019:1337 kernel bug fix update
Summary:
Updated kernel packages that fix various bugs are now available for Red
Hat Enterprise Linux 7.
The kernel packages contain the Linux kernel, the core of any Linux
operating system.
This update fixes the following bugs:
* Mellanox CX-5 MAC learning with OVS H/W offload not working
(BZ#1686292)
* RHEL7.4 NFS4.1 client and server repeated SEQUENCE / TEST_STATEIDs with
SEQUENCE Reply has SEQ4_STATUS_RECALLABLE_STATE_REVOKED set - NFS server
should return NFS4ERR_DELEG_REVOKED or NFS4ERR_BAD_STATEID for revoked
delegations (BZ#1689811)
* PANIC: "BUG: unable to handle kernel paging request" in the mtip32xx
mtip_init_cmd_header routine (BZ#1689929)
* The nvme cli delete-ns command hangs indefinitely. (BZ#1690519)
* drm/nouveau: nv50 - Graphics become sluggish or frozen for nvidia
Pascal cards (Regression from 1584963) - Need to flush fb writes when
rewinding push buffer (BZ#1690761)
* [CEE/SD] Ceph+NFS server crashed and rebooted due to CephFS kernel
client issue (BZ#1692266)
* [Mellanox OVS offload] tc fails to calculate the checksum in case vlan
trunk and header rewrite (BZ#1693110)
* aio O_DIRECT writes to non-page-aligned file locations on ext4 can
result in the overlapped portion of the page containing zeros
(BZ#1693561)
* [HP WS 7.6 bug] Audio driver does not recognize multi function audio
jack microphone input (BZ#1693562)
* XFS returns ENOSPC when using extent size hint with space still
available (BZ#1693796)
* OVN requires IPv6 to be enabled (BZ#1694981)
* breaks DMA API for non-GPL drivers (BZ#1695511)
* ovl_create can return positive retval and crash the host (BZ#1696292)
* ceph: append mode is broken for sync/direct write (BZ#1696595)
* Problem building module due to -EXPORT_SYMBOL_GPL/-EXPORT_SYMBOL
(BZ#1697241)
* Failed to load kpatch module after install the rpm package occasionally
on ppc64le (BZ#1697867)
* [Hyper-V][RHEL7] Stop suppressing PCID bit (BZ#1697940)
* Resizing an online EXT4 filesystem on a loopback device hangs
(BZ#1698110)
* dm table: propagate BDI_CAP_STABLE_WRITES (BZ#1699722)
* [ESXi][RHEL7.6]After upgrade to kernel-3.10.0-957.el7, system is unable
to discover newly added VMware LSI Logic SAS virtual disks without a
reboot. (BZ#1699723)
* kernel: zcrypt: fix specification exception on z196 at ap probe
(BZ#1700706)
* XFS: Metadata corruption detected at xfs_attr3_leaf_write_verify()
(BZ#1701293)
* stime showed huge values related to wrong calculation of time deltas
(L3:) (BZ#1701743)
* Kernel panic due to NULL pointer dereference at
sysfs_do_create_link_sd.isra.2+0x34 while loading [ipmi_si] module using
hard-coded device (BZ#1701991)
* IPv6 ECMP modulo N hashing inefficient when X^2 rt6i_nsiblings
(BZ#1702282)
* security: double-free attempted in security_inode_init_security()
(BZ#1702286)
* Missing wakeup leaves task stuck waiting in blk_queue_enter()
(BZ#1702921)
* Satellite Capsule sync triggers several XFS corruptions (BZ#1702922)
* BUG: SELinux doesn't handle NFS crossmnt well (BZ#1702923)
* md_clear flag missing from /proc/cpuinfo on late microcode update
(BZ#1712993)
* MDS mitigations are not enabled after double microcode update
(BZ#1712998)
* WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:90 __static_key_slow_dec
+0xa6/0xb0 (BZ#1713004)
Users of kernel are advised to upgrade to these updated packages, which
fix these bugs.
Full details and references:
https://access.redhat.com/errata/RHBA-2019:1337?sc_cid=701600000006NHXAA2
Revision History:
Issue Date: 2019-06-04
Updated: 2019-06-04
Regards Renar
Renar Grunenberg
Abteilung Informatik - Betrieb
HUK-COBURG
Bahnhofsplatz
96444 Coburg
Telefon: 09561 96-44110
Telefax: 09561 96-44104
E-Mail: Renar.Grunenberg at huk-coburg.de
Internet: www.huk.de
HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter
Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans
Olav Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.
Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese
Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht
ist nicht gestattet.
This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information
in error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in
this information is strictly forbidden.
Von: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von Felipe
Knop
Gesendet: Montag, 10. Juni 2019 06:41
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
3.10.0-957.21.2
Hi,
Though we are still learning what workload results in the problem, it
appears that even minimal I/O on the file system may cause the OS to
crash. One pattern that we saw was 'mkdir'. There is a chance that the DR
site was not yet impacted because no I/O workload has been run there. In
that case, rolling back to the prior kernel level (one which has been
tested before) may be advisable.
Felipe
----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
Inactive hide details for KG ---06/09/2019 09:38:55 AM---One of my
customer already upgraded their DR site. Is rollback advisedKG
---06/09/2019 09:38:55 AM---One of my customer already upgraded their DR
site. Is rollback advised? They will be running from DR
From: KG <spectrumscale at kiranghag.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 06/09/2019 09:38 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6
kernel 3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
One of my customer already upgraded their DR site.
Is rollback advised? They will be running from DR site for a day in
another week.
On Sat, Jun 8, 2019, 03:37 Felipe Knop <knop at us.ibm.com> wrote:
Zach,
This appears to be affecting all Scale versions,
including 5.0.2 -- but only when moving to the
new 3.10.0-957.21.2 kernel. (3.10.0-957 is not
impacted)
Felipe
----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
Zachary Mance ---06/07/2019 05:51:37 PM---Which
versions of Spectrum Scale versions are you
referring to? 5.0.2-3?
---------------------------
From: Zachary Mance <zmance at ucar.edu>
To: gpfsug main discussion list <
gpfsug-discuss at spectrumscale.org>
Date: 06/07/2019 05:51 PM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum
Scale with RHEL7.6 kernel 3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org
Which versions of Spectrum Scale versions are you
referring to? 5.0.2-3?
---------------------------------------------------------------------------------------------------------------
Zach Mance zmance at ucar.edu (303) 497-1883
HPC Data Infrastructure Group / CISL / NCAR
---------------------------------------------------------------------------------------------------------------
On Fri, Jun 7, 2019 at 3:45 PM Felipe Knop <
knop at us.ibm.com> wrote:
All,
There
have been reported issues (including kernel crashes) on Spectrum Scale with
the latest RHEL7.6 kernel 3.10.0-957.21.2. Please consider delaying upgrades
to this kernel until further information is provided.
Thanks,
Felipe
----
Felipe
Knop
knop at us.ibm.com
GPFS
Development and Security
IBM
Systems
IBM
Building 008
2455
South Rd, Poughkeepsie, NY 12601
(845)
433-9314 T/L 293-9314
_______________________________________________
gpfsug-discuss
mailing list
gpfsug-discuss
at
spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss
mailing list
gpfsug-discuss
at
spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[attachment "graycol.gif" deleted by Felipe
Knop/Poughkeepsie/IBM]
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=ruNEnNWRM7KKCMlL1L1FqB8Ivd1BJ06q9bTmFf91ers&s=ccj51O58apypgvaYh1EVyKuP6GiWRZRSg-z00jTT0UI&e=
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190613/9edaba2c/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190613/9edaba2c/attachment.gif>
More information about the gpfsug-discuss
mailing list