[gpfsug-discuss] What is this error message telling me?

[Jim Doherty]

The data  is also shown in an internaldump as a part of the mmfsadm dump tscomm data,  the RTO & RTT times are listed in microseconds.  So the RTO here in my example is 18.5 seconds (see below).   You  can get the same information from the  Linux networking command   ss -i.    The normal setting for RTO is 200 ms.    Seeing retransmits and backups will drive up the RTO time.    When I look at internaldumps from node expels it is not unusual to see 13 backoffs and retransmits and RTO to have hit 120 seconds   at which point the tcp/ip connection times out.

 10.0.0.31.24/0
    state 1 established snd_wscale 10 rcv_wscale 10 rto 18558000 ato 40000
    retransmits 4 probes 0 backoff 4 options: TSTAMP SACK WSCALE
    rtt 2761650 rttvar 3238039 snd_ssthresh 4 snd_cwnd 5 unacked 0
    snd_mss 1992 rcv_mss 1992 pmtu 2044 advmss 1992 rcv_ssthresh 157708
    sacked 0 lost 0 retrans 0 fackets 0 reordering 3 ca_state 'open'


 Jim

    On Thursday, September 27, 2018, 11:14:43 AM EDT, Buterbaugh, Kevin L <Kevin.Buterbaugh at Vanderbilt.Edu> wrote:  
 
  Hi All,
2018-09-27_09:48:50.923-0500: [E] The TCP connection to IP address 1.2.3.4 some client <c0n509> (socket 442) state is unexpected: ca_state=1 unacked=3 rto=27008000
Seeing errors like the above and trying to track down the root cause.  I know that at last weeks’ GPFS User Group meeting at ORNL this very error message was discussed, but I don’t recall the details and the slides haven’t been posted to the website yet.  IIRC, the “rto” is significant … 
I’ve Googled, but haven’t gotten any hits, nor have I found anything in the GPFS 4.2.2 Problem Determination Guide.
Thanks in advance…
—Kevin Buterbaugh - Senior System AdministratorVanderbilt University - Advanced Computing Center for Research and EducationKevin.Buterbaugh at vanderbilt.edu - (615)875-9633


_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://gpfsug.org/pipermail/gpfsug-discuss_gpfsug.org/attachments/20180927/23e7e029/attachment.htm>