[gpfsug-discuss] WG: Spectrum Scale with RHEL7.6 kernel 3.10.0-957.21.2

Mon Jun 10 14:43:10 BST 2019

Renar,

Thanks. Of the changes below, it appears that

* security: double-free attempted in security_inode_init_security()
(BZ#1702286)

was the one that ended up triggering the problem.  Our investigations now
show that RHEL kernels >= 3.10.0-957.19.1 are impacted.

  Felipe

----
Felipe Knop                                     knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314  T/L 293-9314

From:	"Grunenberg, Renar" <Renar.Grunenberg at huk-coburg.de>
To:	"'gpfsug-discuss at spectrumscale.org'"
            <gpfsug-discuss at spectrumscale.org>
Date:	06/10/2019 08:43 AM
Subject:	[EXTERNAL] [gpfsug-discuss] WG: Spectrum Scale with RHEL7.6
            kernel	3.10.0-957.21.2
Sent by:	gpfsug-discuss-bounces at spectrumscale.org

Hallo Felipe,

here are the change list:
RHBA-2019:1337 kernel bug fix update

Summary:

Updated kernel packages that fix various bugs are now available for Red Hat
Enterprise Linux 7.

The kernel packages contain the Linux kernel, the core of any Linux
operating system.

This update fixes the following bugs:

* Mellanox CX-5 MAC learning with OVS H/W offload not working (BZ#1686292)

* RHEL7.4 NFS4.1 client and server repeated SEQUENCE / TEST_STATEIDs with
SEQUENCE Reply has SEQ4_STATUS_RECALLABLE_STATE_REVOKED set - NFS server
should return NFS4ERR_DELEG_REVOKED or NFS4ERR_BAD_STATEID for revoked
delegations (BZ#1689811)

* PANIC: "BUG: unable to handle kernel paging request" in the mtip32xx
mtip_init_cmd_header routine (BZ#1689929)

* The nvme cli delete-ns command hangs indefinitely. (BZ#1690519)

* drm/nouveau: nv50 - Graphics become sluggish or frozen for nvidia Pascal
cards (Regression from 1584963) - Need to flush fb writes when rewinding
push buffer (BZ#1690761)

* [CEE/SD] Ceph+NFS server crashed and rebooted due to CephFS kernel client
issue (BZ#1692266)

* [Mellanox OVS offload] tc fails to calculate the checksum in case vlan
trunk and header rewrite (BZ#1693110)

* aio O_DIRECT writes to non-page-aligned file locations on ext4 can result
in the overlapped portion of the page containing zeros (BZ#1693561)

* [HP WS 7.6 bug]  Audio driver does not recognize multi function audio
jack microphone input (BZ#1693562)

* XFS returns ENOSPC when using extent size hint with  space still
available (BZ#1693796)

* OVN requires IPv6 to be enabled (BZ#1694981)

* breaks DMA API for non-GPL drivers (BZ#1695511)

* ovl_create can return positive retval and crash the host (BZ#1696292)

* ceph: append mode is broken for sync/direct write (BZ#1696595)

* Problem building module due to -EXPORT_SYMBOL_GPL/-EXPORT_SYMBOL
(BZ#1697241)

* Failed to load kpatch module after install the rpm package occasionally
on ppc64le (BZ#1697867)

* [Hyper-V][RHEL7] Stop suppressing PCID bit (BZ#1697940)

* Resizing an online EXT4 filesystem on a loopback device hangs
(BZ#1698110)

* dm table: propagate BDI_CAP_STABLE_WRITES (BZ#1699722)

* [ESXi][RHEL7.6]After upgrade to kernel-3.10.0-957.el7, system is unable
to discover newly added VMware LSI Logic SAS virtual disks without a
reboot. (BZ#1699723)

* kernel: zcrypt: fix specification exception on z196 at ap probe
(BZ#1700706)

* XFS: Metadata corruption detected at xfs_attr3_leaf_write_verify()
(BZ#1701293)

* stime showed huge values related to wrong calculation of time deltas
(L3:) (BZ#1701743)

* Kernel panic due to NULL pointer dereference at
sysfs_do_create_link_sd.isra.2+0x34 while loading [ipmi_si] module using
hard-coded device (BZ#1701991)

* IPv6 ECMP modulo N hashing inefficient when X^2 rt6i_nsiblings
(BZ#1702282)

* security: double-free attempted in security_inode_init_security()
(BZ#1702286)

* Missing wakeup leaves task stuck waiting in blk_queue_enter()
(BZ#1702921)

* Satellite Capsule sync triggers several XFS corruptions (BZ#1702922)

* BUG: SELinux doesn't handle NFS crossmnt well (BZ#1702923)

* md_clear flag missing from /proc/cpuinfo on late microcode update
(BZ#1712993)

* MDS mitigations are not enabled after double microcode update
(BZ#1712998)

* WARNING: CPU: 0 PID: 0 at kernel/jump_label.c:90 __static_key_slow_dec
+0xa6/0xb0 (BZ#1713004)

Users of kernel are advised to upgrade to these updated packages, which fix
these bugs.

Full details and references:

https://access.redhat.com/errata/RHBA-2019:1337?sc_cid=701600000006NHXAA2

Revision History:

Issue Date: 2019-06-04
Updated:    2019-06-04

Regards Renar

Renar Grunenberg
Abteilung Informatik - Betrieb

HUK-COBURG
Bahnhofsplatz
96444 Coburg

 Telefon:  09561 96-44110                 

 Telefax:  09561 96-44104                 

 E-Mail:   Renar.Grunenberg at huk-coburg.de 

 Internet: www.huk.de                     

HUK-COBURG Haftpflicht-Unterstützungs-Kasse kraftfahrender Beamter
Deutschlands a. G. in Coburg
Reg.-Gericht Coburg HRB 100; St.-Nr. 9212/101/00021
Sitz der Gesellschaft: Bahnhofsplatz, 96444 Coburg
Vorsitzender des Aufsichtsrats: Prof. Dr. Heinrich R. Schradin.
Vorstand: Klaus-Jürgen Heitmann (Sprecher), Stefan Gronbach, Dr. Hans Olav
Herøy, Dr. Jörg Rheinländer (stv.), Sarah Rössler, Daniel Thomas.
Diese Nachricht enthält vertrauliche und/oder rechtlich geschützte
Informationen.
Wenn Sie nicht der richtige Adressat sind oder diese Nachricht irrtümlich
erhalten haben,
informieren Sie bitte sofort den Absender und vernichten Sie diese
Nachricht.
Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Nachricht ist
nicht gestattet.

This information may contain confidential and/or privileged information.
If you are not the intended recipient (or have received this information in
error) please notify the
sender immediately and destroy this information.
Any unauthorized copying, disclosure or distribution of the material in
this information is strictly forbidden.

Von: gpfsug-discuss-bounces at spectrumscale.org [
mailto:gpfsug-discuss-bounces at spectrumscale.org] Im Auftrag von Felipe Knop
Gesendet: Montag, 10. Juni 2019 06:41
An: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Betreff: Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
3.10.0-957.21.2

Hi,

Though we are still learning what workload results in the problem, it
appears that even minimal I/O on the file system may cause the OS to crash.
One pattern that we saw was 'mkdir'. There is a chance that the DR site was
not yet impacted because no I/O workload has been run there. In that case,
rolling back to the prior kernel level (one which has been tested before)
may be advisable.

Felipe

----
Felipe Knop knop at us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314

Inactive hide details for KG ---06/09/2019 09:38:55 AM---One of my customer
already upgraded their DR site. Is rollback advisedKG ---06/09/2019
09:38:55 AM---One of my customer already upgraded their DR site. Is
rollback advised? They will be running from DR

From: KG <spectrumscale at kiranghag.com>
To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
Date: 06/09/2019 09:38 AM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6 kernel
3.10.0-957.21.2
Sent by: gpfsug-discuss-bounces at spectrumscale.org

One of my customer already upgraded their DR site.

Is rollback advised? They will be running from DR site for a day in another
week.

On Sat, Jun 8, 2019, 03:37 Felipe Knop <knop at us.ibm.com> wrote:
      Zach,

      This appears to be affecting all Scale versions, including 5.0.2 --
      but only when moving to the new 3.10.0-957.21.2 kernel. (3.10.0-957
      is not impacted)

      Felipe

      ----
      Felipe Knop knop at us.ibm.com
      GPFS Development and Security
      IBM Systems
      IBM Building 008
      2455 South Rd, Poughkeepsie, NY 12601
      (845) 433-9314 T/L 293-9314

      Zachary Mance ---06/07/2019 05:51:37 PM---Which versions of Spectrum
      Scale versions are you referring to? 5.0.2-3?
      ---------------------------

      From: Zachary Mance <zmance at ucar.edu>
      To: gpfsug main discussion list <gpfsug-discuss at spectrumscale.org>
      Date: 06/07/2019 05:51 PM
      Subject: [EXTERNAL] Re: [gpfsug-discuss] Spectrum Scale with RHEL7.6
      kernel 3.10.0-957.21.2
      Sent by: gpfsug-discuss-bounces at spectrumscale.org

      Which versions of Spectrum Scale versions are you referring to?
      5.0.2-3?
      ---------------------------------------------------------------------------------------------------------------

      Zach Mance  zmance at ucar.edu  (303) 497-1883

      HPC Data Infrastructure Group / CISL / NCAR
      ---------------------------------------------------------------------------------------------------------------

      On Fri, Jun 7, 2019 at 3:45 PM Felipe Knop <knop at us.ibm.com> wrote:
                  All,

                  There have been reported issues (including kernel
                  crashes) on Spectrum Scale with the latest RHEL7.6 kernel
                  3.10.0-957.21.2. Please consider delaying upgrades to
                  this kernel until further information is provided.

                  Thanks,

                  Felipe

                  ----
                  Felipe Knop knop at us.ibm.com
                  GPFS Development and Security
                  IBM Systems
                  IBM Building 008
                  2455 South Rd, Poughkeepsie, NY 12601
                  (845) 433-9314 T/L 293-9314

                  _______________________________________________
                  gpfsug-discuss mailing list
                  gpfsug-discuss at spectrumscale.org
                  http://gpfsug.org/mailman/listinfo/gpfsug-discuss
                  _______________________________________________
                  gpfsug-discuss mailing list
                  gpfsug-discuss at spectrumscale.org
                  http://gpfsug.org/mailman/listinfo/gpfsug-discuss

      _______________________________________________
      gpfsug-discuss mailing list
      gpfsug-discuss at spectrumscale.org
      http://gpfsug.org/mailman/listinfo/gpfsug-discuss[attachment
      "graycol.gif" deleted by Felipe Knop/Poughkeepsie/IBM]
      _______________________________________________
      gpfsug-discuss mailing list
      gpfsug-discuss at spectrumscale.org
      http://gpfsug.org/mailman/listinfo/gpfsug-discuss
 _______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=oNT2koCZX0xmWlSlLblR9Q&m=yWFAPveNSlMNNB5WT9HWp-2gQFFcYeCEsQdME5UvoGw&s=xZFqiCTjE-2e_6gM6MkzBcALK0hp-3ZquA7bt2GIjt8&e=

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190610/52920dca/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20190610/52920dca/attachment.gif>