[gpfsug-discuss] rmda errors scatter thru gpfs logs

Aaron Knister aaron.s.knister at nasa.gov
Fri Jan 27 22:54:51 GMT 2017


This is rather disconcerting. We just finished upgrading our nsd servers 
from 3.5.0.31 to 4.1.1.10 (All clients were previously migrated from 
3.5.0.31 to 4.1.1.10). After finishing that upgrade I'm now seeing these 
errors with some frequency (a couple every few minutes). Anyone have 
insight?

On 1/18/17 11:58 AM, Brian Marshall wrote:
> As background, we recently upgraded GPFS from 4.2.0 to 4.2.1  and
> updated the Mellanox OFED on our compute cluster to allow it to move
> from CentOS 7.1 to 7.2
>
> We do some transient warnings from the Mellanox switch gear about
> various port counters that we are tracking down with them.
>
> Jobs and filesystem seem stable, but the logs are concerning.
>
> On Wed, Jan 18, 2017 at 10:22 AM, Aaron Knister
> <aaron.s.knister at nasa.gov <mailto:aaron.s.knister at nasa.gov>> wrote:
>
>     I'm curious about this too. We see these messages sometimes when
>     things have gone horribly wrong but also sometimes during recovery
>     events. Here's a recent one:
>
>     loremds20 (manager/nsd node):
>     Mon Jan 16 14:19:02.048 2017: [E] VERBS RDMA rdma read error
>     IBV_WC_REM_ACCESS_ERR to 10.101.11.6 (lorej006) on mlx5_0 port 1
>     fabnum 3 vendor_err 136
>     Mon Jan 16 14:19:02.049 2017: [E] VERBS RDMA closed connection to
>     10.101.11.6 (lorej006) on mlx5_0 port 1 fabnum 3 due to RDMA read
>     error IBV_WC_REM_ACCESS_ERR index 11
>
>     lorej006 (client):
>     Mon Jan 16 14:19:01.990 2017: [N] VERBS RDMA closed connection to
>     10.101.53.18 (loremds18) on mlx5_0 port 1 fabnum 3 index 2
>     Mon Jan 16 14:19:01.995 2017: [N] VERBS RDMA closed connection to
>     10.101.53.19 (loremds19) on mlx5_0 port 1 fabnum 3 index 0
>     Mon Jan 16 14:19:01.997 2017: [I] Recovering nodes: 10.101.53.18
>     10.101.53.19
>     Mon Jan 16 14:19:02.047 2017: [W] VERBS RDMA async event
>     IBV_EVENT_QP_ACCESS_ERR on mlx5_0 qp 0x7fffe550f1c8.
>     Mon Jan 16 14:19:02.051 2017: [E] VERBS RDMA closed connection to
>     10.101.53.20 (loremds20) on mlx5_0 port 1 fabnum 3 error 733 index 1
>     Mon Jan 16 14:19:02.071 2017: [I] Recovered 2 nodes for file system
>     tnb32.
>     Mon Jan 16 14:19:02.140 2017: [I] VERBS RDMA connecting to
>     10.101.53.20 (loremds20) on mlx5_0 port 1 fabnum 3 index 0
>     Mon Jan 16 14:19:02.160 2017: [I] VERBS RDMA connected to
>     10.101.53.20 (loremds20) on mlx5_0 port 1 fabnum 3 sl 0 index 0
>
>     I had just shut down loremds18 and loremds19 so there was certainly
>     recovery taking place and during that time is when the error seems
>     to have occurred.
>
>     I looked up the meaning of IBV_WC_REM_ACCESS_ERR here
>     (http://www.rdmamojo.com/2013/02/15/ibv_poll_cq/
>     <http://www.rdmamojo.com/2013/02/15/ibv_poll_cq/>) and see this:
>
>     IBV_WC_REM_ACCESS_ERR (10) - Remote Access Error: a protection error
>     occurred on a remote data buffer to be read by an RDMA Read, written
>     by an RDMA Write or accessed by an atomic operation. This error is
>     reported only on RDMA operations or atomic operations. Relevant for
>     RC QPs.
>
>     my take on it during recovery it seems like one end of the
>     connection more or less hanging up on the other end (e.g. Connection
>     reset by peer
>     /ECONNRESET).
>
>     But like I said at the start, we also see this when there something
>     has gone awfully wrong.
>
>     -Aaron
>
>     On 1/18/17 3:59 AM, Simon Thompson (Research Computing - IT
>     Services) wrote:
>
>         I'd be inclined to look at something like:
>
>         ibqueryerrors -s
>         PortXmitWait,LinkDownedCounter,PortXmitDiscards,PortRcvRemotePhysicalErrors
>         -c
>
>         And see if you have a high number of symbol errors, might be a cable
>         needs replugging or replacing.
>
>         Simon
>
>         From: <gpfsug-discuss-bounces at spectrumscale.org
>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>
>         <mailto:gpfsug-discuss-bounces at spectrumscale.org
>         <mailto:gpfsug-discuss-bounces at spectrumscale.org>>> on behalf of
>         "J. Eric
>         Wonderley" <eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>
>         <mailto:eric.wonderley at vt.edu <mailto:eric.wonderley at vt.edu>>>
>         Reply-To: "gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>
>         <mailto:gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>>"
>         <gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>
>         <mailto:gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>>>
>         Date: Tuesday, 17 January 2017 at 21:16
>         To: "gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>
>         <mailto:gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>>"
>         <gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>
>         <mailto:gpfsug-discuss at spectrumscale.org
>         <mailto:gpfsug-discuss at spectrumscale.org>>>
>         Subject: [gpfsug-discuss] rmda errors scatter thru gpfs logs
>
>         I have messages like these frequent my logs:
>         Tue Jan 17 11:25:49.731 2017: [E] VERBS RDMA rdma write error
>         IBV_WC_REM_ACCESS_ERR to 10.51.10.5 (cl005) on mlx5_0 port 1
>         fabnum 0
>         vendor_err 136
>         Tue Jan 17 11:25:49.732 2017: [E] VERBS RDMA closed connection to
>         10.51.10.5 (cl005) on mlx5_0 port 1 fabnum 0 due to RDMA write error
>         IBV_WC_REM_ACCESS_ERR index 23
>
>         Any ideas on cause..?
>
>
>
>         _______________________________________________
>         gpfsug-discuss mailing list
>         gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>         http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>         <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>     --
>     Aaron Knister
>     NASA Center for Climate Simulation (Code 606.2)
>     Goddard Space Flight Center
>     (301) 286-2776 <tel:%28301%29%20286-2776>
>     _______________________________________________
>     gpfsug-discuss mailing list
>     gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
>     http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>     <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>

-- 
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776



More information about the gpfsug-discuss mailing list