[gpfsug-discuss] What is this error message telling me?

Thu Sep 27 22:53:23 BST 2018

Hi John,

Thanks for the explanation and the link to your presentation … just what I was needing.

Kevin
—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu> - (615)875-9633

On Sep 27, 2018, at 11:37 AM, John Lewars <jlewars at us.ibm.com<mailto:jlewars at us.ibm.com>> wrote:

Hi Kevin,

The message below indicates that the mmfsd code had a pending message on a socket, and, when it looked at the low level socket statistics, GPFS found indications that the TCP connection was in a 'bad state'.  GPFS determines a connection to be a 'bad state' if:

1) the CA_STATE for the socket is not in 0 (or open) state, which means the state must be disorder, recovery, or loss.  See this paper for more details on CA_STATE:
https://wiki.aalto.fi/download/attachments/69901948/TCP-CongestionControlFinal.pdf<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.aalto.fi%2Fdownload%2Fattachments%2F69901948%2FTCP-CongestionControlFinal.pdf&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=gTy7ZlQruWdS9uOBQZQRqABi8AwFEXjqKQvfm%2FSnsHo%3D&reserved=0>

or

2) the RTO is greater than 10 seconds and there are unacknowledged messages pending on the socket (unacked > 0).

In the example below we see that rto=27008000, which means that the non-fast path TCP retransmission timeout is about 27 seconds, and that probably means the connection has experienced significant packet loss.  If there was no expel following this message, I would suspect there was some transient packet loss that was recovered from.

There are plenty of places in which to find more details on RTO, but you might want to start with wikipedia (https://en.wikipedia.org/wiki/Transmission_Control_Protocol<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FTransmission_Control_Protocol&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=2AodSr9rUeGUtRUU37hDR29ejB%2BUmPxhGbtNGx4%2BSzY%3D&reserved=0>) which states:

In addition, senders employ a retransmission timeout(RTO) that is based on the estimated round-trip time<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FRound-trip_time&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=i2g8cJKLq4TDNg1UNVXZ23kNsiiPNMWHgllAmMfe0tg%3D&reserved=0> (or RTT) between the sender and receiver, as well as the variance in this round trip time. The behavior of this timer is specified in RFC 6298<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc6298&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=%2BeYWIqeeGCSQwaqqVivAeksyaio4C%2BXMr2%2BMGEKvZVM%3D&reserved=0>. There are subtleties in the estimation of RTT. For example, senders must be careful when calculating RTT samples for retransmitted packets; typically they use Karn's Algorithm<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FKarn%2527s_Algorithm&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=TkOxj60PftecM92%2FPpIv2Pznqo6c9LFO3lAQECRUG0c%3D&reserved=0> or TCP timestamps (see RFC 1323<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc1323&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=UovNpxntUSX7xQkhxNnn3jZHKsnSONfmQOUDUf873OE%3D&reserved=0>). These individual RTT samples are then averaged over time to create a Smoothed Round Trip Time (SRTT) using Jacobson<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FVan_Jacobson&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=yxUf%2F7GxtYVQMMmd3WDrlEpOrToiTQmUcCJq7HB6pHU%3D&reserved=0>'s algorithm. This SRTT value is what is finally used as the round-trip time estimate.
[. . .]
Reliability is achieved by the sender detecting lost data and retransmitting it. TCP uses two primary techniques to identify loss. Retransmission timeout (abbreviated as RTO) and duplicate cumulative acknowledgements (DupAcks).

Note that older versions of the Spectrum Scale code had a third criteria in checking for 'bad state', which included checking if unacked was greater than 8, but that check would sometimes call-out a socket that was working fine, so this third check has been removed via the APAR IJ02566.  All Spectrum Scale V5 code has this fix and the 4.2.X code stream picked up this fix in PTF 7 (4.2.3.7 ships APAR IJ02566).

More details on debugging expels using these TCP connection messages are in the presentation you referred to, which I posted here:https://www.ibm.com/developerworks/community/wikis/home?lang=en_us#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/DEBUG%20Expels<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ibm.com%2Fdeveloperworks%2Fcommunity%2Fwikis%2Fhome%3Flang%3Den_us%23!%2Fwiki%2FGeneral%2520Parallel%2520File%2520System%2520(GPFS)%2Fpage%2FDEBUG%2520Expels&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=BgfLGFTA9%2FlqDjx4HMnBQMkvSj0dK0t97yNV%2FHGQpa4%3D&reserved=0>

Regards,
John Lewars
Technical Computing Development, IBM Poughkeepsie

----- Forwarded by Lyle Gayne/Poughkeepsie/IBM on 09/27/2018 11:15 AM -----

Hi All,

2018-09-27_09:48:50.923-0500: [E] The TCP connection to IP address 1.2.3.4 some client <c0n509> (socket 442) state is unexpected: ca_state=1 unacked=3 rto=27008000

Seeing errors like the above and trying to track down the root cause.  I know that at last weeks’ GPFS User Group meeting at ORNL this very error message was discussed, but I don’t recall the details and the slides haven’t been posted to the website yet.  IIRC, the “rto” is significant …

I’ve Googled, but haven’t gotten any hits, nor have I found anything in the GPFS 4.2.2 Problem Determination Guide.

Thanks in advance…

—
Kevin Buterbaugh - Senior System Administrator
Vanderbilt University - Advanced Computing Center for Research and Education
Kevin.Buterbaugh at vanderbilt.edu<mailto:Kevin.Buterbaugh at vanderbilt.edu>- (615)875-9633

_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org<http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fgpfsug.org%2Fmailman%2Flistinfo%2Fgpfsug-discuss&data=02%7C01%7CKevin.Buterbaugh%40Vanderbilt.Edu%7C106784daf7e9407e00ef08d624978b49%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C636736630629232470&sdata=O9Ki0Wnpc4kAZBKhf9crcU%2BruP8mqgnX%2FUtPXkZDDQk%3D&reserved=0>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180927/c876f29e/attachment.htm>