Solving “Out of Socket Memory”

We some times see “Out of Socket Memory” errors in the messages file in the logs location.

[root@vx132s ~]# tail -f /var/log/messages

Apr 22 15:05:39 ztm-n08 kernel: [12624150.315458] Out of socket memory

At this point there are a couple of cases where this condition falls,

1. There are too many Orphan Sockets.

2. The TCP memory is using high or it is running out of memory

Most of the times, these issues come under the Condition 1.

In order to find out how much memory is configured for the TCP , we can get these details from

[root@vx132s sysconfig]# cat /proc/sys/net/ipv4/tcp_mem

743136 990848 1486272

The above tcp memory contains 3 parts.

min : below this number of pages TCP is not bothered about its memory consumption. 

pressure: when the amount of memory allocated to TCP by the kernel exceeds this threshold, the kernel starts to moderate the memory consumption. This mode is exited when memory consumption falls under min.

max : the max number of pages allowed for queuing by all TCP sockets. When the system goes above this threshold, the kernel will start throwing the "Out of socket memory" error in the logs.

Now lets see how much of the memory does TCP uses , we can get these details using

[root@vx132s sysconfig]# cat /proc/net/sockstat

sockets: used 651

TCP: inuse 6 orphan 2314 tw 0 alloc 12 mem 1893

UDP: inuse 9 mem 0

UDPLITE: inuse 0

RAW: inuse 0

FRAG: inuse 0 memory 0

we can see from the above output that the current memory used by TCP is 1895 pages. This value is way lower than the maximum pages allocated (1486272 ).

From this we can dismiss that TCP Is using high memory. Now our next step is find the orphan sockets. We can get that information from the above output as 2314. We can get the limit of the orphan sockets using

[root@vx132s sysconfig]# cat /proc/sys/net/ipv4/tcp_max_orphans 

65536

The orphan socket count 2314 is very less when compared to 65536. If this number is bigger than the one from tcp_max_orphans then this can be a reason for the "Out of socket memory" Normally an orphan socket is a socket that isn't associated with a file descriptor, usually after the close() call and there is no longer a file descriptor that reference it, but the socket still exists in memory, until TCP is done with it.

Each orphan sockets eats up to 64K of unswappable memory. The file tcp_max_orphans file shows the maximum number of TCP sockets not attached to any user file handle, held by system that the kernel can support. If the number normally exceeds the orphaned connections they are reset and a warning is printed.

We can solve this by increasing tcp_max_orphans value like,

echo 400000 > /proc/sys/net/ipv4/tcp_max_orphans

In some cases the kernel may show more sockets by multiplying the number of orphans by 2x or 4x to artificially increase the score of the bad socket. It is always better to get the number of the orphaned sockets during peak server utilization and multiple by 4 and add that to the max value of orphan sockets value

More to come , happy learning