Possible connection leak?

[zlesnr]

I'm seeing issues where connections are not cleaned up after a request is issued. In a case I'm working on, a connection to several hundred devices exhausts my local open file handle limit and causes the program to stop functioning.

while true; do lsof -i -n -P | grep main | awk '{ print $9 }' |awk -F'>' '{print $2}' | sort | uniq -c | sort -n ; sleep 1; echo;done

This yields something like the following:

   5 192.168.10.17:443
   5 192.168.10.24:443
   5 192.168.10.30:443
   5 192.168.10.36:443
   5 192.168.10.38:443
   5 192.168.10.40:443
   5 192.168.10.48:443
   5 192.168.10.50:443
   5 192.168.10.54:443

This indicates that there are 5 connections to each host I'm connected to, which I believe is the result of the HTTPs connection pooling provided by http.Client. Though the decode of the JSON doesn't appear to perform the ReadAll necessary to clean up the client connections to the given host. Note also that for a given run, each send() call creates a new client, which I believe further exacerbates the issue in my conditions here. It's late on a Friday and I'm heading into the weekend, but wanted to make a note.

[cheynearista]

@zlesnr Interesting. Are they never getting cleaned up? transport.go shows that there should be a 90sec Idle connection timeout, and MaxIdleConnsPerHost should be 2. Are these in TIME_WAIT or CLOSE_WAIT?

[zlesnr]

Everything looks to be in ESTABLISHED state. I'm also trying to close out connections when I'm done with the node.

	if err != nil {
		log.Error(err)
	}

	err = handle.Close()
	if err != nil {
		log.Error(err)
	}

I thought the above would be enough to close out the open connections for a host, but I can't tell if all the calls are being made with the same handle object.

[sfunkhouser]

If anyone else looks at this, if the issue was the same as I was having you can now do:

node.GetConnection().SetDisableKeepAlive(true)

And it should clear up the connections.