Could not expand chunk pool for ipnat node
Could not expand chunk pool for ipnat node. No memory available
-Process= "Chunk Manager", ipl= 4, pid= 1 -Traceback= 0x80137D24 0x8028E1FC 0x802937F0 0x802AD8B8 0x802AC97C 0x802AC6F4 0x 8027E0F8 0x80281720 17:37:32: %SYS-2-CHUNKEXPANDFAIL: Could not expand chunk pool for ipnat node. No memory available -Process= "Chunk Manager", ipl= 4, pid= 1 -Traceback= 0x80137D24 0x802AC73C 0x8027E0F8 0x80281720 -Traceback= 0x80137D24 0x802AC73C 0x8027E0F8 0x80281720 17:37:58: %SYS-2-MALLOCFAIL: Memory allocation of 65536 bytes failed from 0x802A D8B4, alignment 8 Pool: Processor Free: 4719952 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool
You will see these messages in your log when your router hasn’t got enough memory to allocate for more Dynamic NAT.
check your NAT statistics
core-router#show ip nat statistics Total active translations: 3646 (25 static, 3621 dynamic; 819 extended) Outside interfaces: Ethernet2, Dialer0, Dialer1, Virtual-Access2 Inside interfaces: Ethernet0 Hits: 16308412 Misses: 188984 CEF Translated packets: 15976311, CEF Punted packets: 653723 Expired translations: 292334 Dynamic mappings: – Inside Source [Id: 1] route-map nonat interface Dialer0 refcount 146 Queued Packets: 0 core-router#
I’m using cisco router that serve almost 10 computers, and these are the statistics of a normal ADSL Cisco 800 Series router, and The Dynamic NAT (in red color) is quite large in number, this happened to me when one of our machine was infected with viruses and it was sending out packets to multiple random IP’s. The machine IP (192.168.0.22) was NATING out with different ports and was almost 100 IP:ports per 5 seconds, this number increased so rapidly that dynamic NAT statics reached to thousands and filled up the memory causing the router to stopped working and the above messages started. After rebooting the router everything seems fine, but the memory filled up again after 5-10 minutes.
This can be viruses and can be extra load on router if its not very powerful, so here is how I found the problem.
I checked the NAT translation
core-router#show ip nat translations tcp 168.33.123.49:2038 192.168.0.22:2038 212.125.66.24:7793 212.125.66.24:7793 tcp 168.33.123.49:2039 192.168.0.22:2039 212.125.66.24:7794 212.125.66.24:7794 tcp 168.33.123.49:2040 192.168.0.22:2040 212.125.66.24:7795 212.125.66.24:7795 tcp 168.33.123.49:2041 192.168.0.22:2041 212.125.66.24:7796 212.125.66.24:7796 tcp 168.33.123.49:2042 192.168.0.22:2042 212.125.66.24:7797 212.125.66.24:7797 tcp 168.33.123.49:2043 192.168.0.22:2043 212.125.66.24:7798 212.125.66.24:7798 tcp 168.33.123.49:2044 192.168.0.22:2044 212.125.66.24:7799 212.125.66.24:7799 tcp 168.33.123.49:2045 192.168.0.22:2045 212.125.66.24:7800 212.125.66.24:7800 tcp 168.33.123.49:2046 192.168.0.22:2046 212.125.66.24:7801 212.125.66.24:7801 tcp 168.33.123.49:2047 192.168.0.22:2047 212.125.66.24:7802 212.125.66.24:7802 tcp 168.33.123.49:2048 192.168.0.22:2048 212.125.66.24:7803 212.125.66.24:7803 tcp 168.33.123.49:2049 192.168.0.22:2049 212.125.66.24:7804 212.125.66.24:7804 tcp 168.33.123.49:2050 192.168.0.22:2050 212.125.66.24:7805 212.125.66.24:7805 tcp 168.33.123.49:2051 192.168.0.22:2051 212.125.66.24:7806 212.125.66.24:7806 tcp 168.33.123.49:2052 192.168.0.22:2052 212.125.66.24:7807 212.125.66.24:7807 tcp 168.33.123.49:2053 192.168.0.22:2053 212.125.66.24:7808 212.125.66.24:7808 tcp 168.33.123.49:2054 192.168.0.22:2054 212.125.66.24:7809 212.125.66.24:7809 tcp 168.33.123.49:2055 192.168.0.22:2055 212.125.66.24:7810 212.125.66.24:7810 tcp 168.33.123.49:2056 192.168.0.22:2056 212.125.66.24:7811 212.125.66.24:7811 tcp 168.33.123.49:2057 192.168.0.22:2057 212.125.66.24:7812 212.125.66.24:7812 tcp 168.33.123.49:2058 192.168.0.22:2058 212.125.66.24:7813 212.125.66.24:7813 tcp 168.33.123.49:2059 192.168.0.22:2059 212.125.66.24:7814 212.125.66.24:7814 tcp 168.33.123.49:2060 192.168.0.22:2060 212.125.66.24:7815 212.125.66.24:7815 tcp 168.33.123.49:2061 192.168.0.22:2061 212.125.66.24:7816 212.125.66.24:7816 tcp 168.33.123.49:2062 192.168.0.22:2062 212.125.66.24:7817 212.125.66.24:7817 tcp 168.33.123.49:2063 192.168.0.22:2063 212.125.66.24:7818 212.125.66.24:7818 tcp 168.33.123.49:2064 192.168.0.22:2064 212.125.66.24:7819 212.125.66.24:7819 tcp 168.33.123.49:2065 192.168.0.22:2065 212.125.66.24:7820 212.125.66.24:7820 –More–
Here the IP 192.168.0.22 is the internal IP of the infected machine, its external IP is 168.33.123.49, which is translating to 168.33.123.49 with dynamic port staring from 2038 to 2039, 2040 and so forth. This translation was so rapid that it could fill up the memory in less than 10 minutes.
You can try to limit the dynamic rate by
core-router#ip nat translation max-entries 500
but this is not the permanent solution, as the only solution (for me) was to clean the machine 192.168.0.22 from viruses and by cleaning it fixed the problem.