We're planting a tree for every job application! Click here to learn more

Troubleshooting Tomcat in a memory-lean environment

Waqqas Dadabhoy

20 Jan 2021

4 min read

 Troubleshooting Tomcat in a memory-lean environment
  • tomcat

onemusicapi.com runs on AWS Elastic Beanstalk with a Tomcat 8.5 platform, using t3.micro instances. t3.micro instances have 1GB of RAM, and Elastic Beanstalk's monitoring puts instances in a Warning state if memory use exceeds 90% of available memory. We found that about a day or two after an EC2 instance was brought online, it would go into Warning state, due to memory use, even if the service wasn't actually affected.

TL;DR: We found Tomcat was using more memory than expected, and the following helped:

  • Reducing -Xmx
  • Limiting the code cache size
  • Reducing the number of threads

Step one: measuring heap

The first step was to reduce the heap size (-Xmx) and monitor memory use. During this step, we realized that we should be able to reproduce the problem by setting -Xms & -Xmx to the same value, so that the heap allocation wouldn't change. However, the memory issue continued the same way it had - a day or two after being created, the instance would go into Warning state.

Step two: beyond the heap

The next step was to check what was using memory on the system. Using the command:

ps -eo size,pid,user,command --sort -size | head

... we noticed that Java process was using several hundred megabytes of memory over what -Xmx had been configured to. We were able to figure out how much of the memory was going to:

  • Thread stacks.
  • JVM overhead (by comparing it to the memory of a freshly started jshell).

However, this still left about 150MB of memory unexplained.

We assumed that this might be due to Tomcat using native memory for its internal buffers, so we tried changing every setting in that might seem like it might affect memory use.

Step three: measuring native memory

The next step was to enable Native Memory Tracking, and get "before & after" snapshots of memory use. This immediately showed us that the code cache was using over 64MB of memory. We then set the initial & maximum code cache size to get more predictable memory use. Here is the Native Memory Tracking output:

Total: reserved=1716451KB, committed=378495KB                                  |  Total: reserved=1753206KB, committed=441570KB
-                 Java Heap (reserved=235520KB, committed=235520KB)               -                 Java Heap (reserved=235520KB, committed=235520KB)
                            (mmap: reserved=235520KB, committed=235520KB)                                     (mmap: reserved=235520KB, committed=235520KB) 
                                                                               |   
-                     Class (reserved=1120844KB, committed=82468KB)            |  -                     Class (reserved=1136217KB, committed=101681KB)
                            (classes# 11852)                                   |                              (classes# 13397)
                            (  instance classes# 11125, array classes# 727)    |                              (  instance classes# 12600, array classes# 797)
                            (malloc=2636KB# 35841)                             |                              (malloc=3673KB# 55415) 
                            (mmap: reserved=1118208KB, committed=79832KB)      |                              (mmap: reserved=1132544KB, committed=98008KB) 
                            (  Metadata:   )                                                                  (  Metadata:   )
                            (    reserved=69632KB, committed=67840KB)          |                              (    reserved=83968KB, committed=83200KB)
                            (    used=66088KB)                                 |                              (    used=80220KB)
                            (    free=1752KB)                                  |                              (    free=2980KB)
                            (    waste=0KB =0.00%)                                                            (    waste=0KB =0.00%)
                            (  Class space:)                                                                  (  Class space:)
                            (    reserved=1048576KB, committed=11992KB)        |                              (    reserved=1048576KB, committed=14808KB)
                            (    used=10314KB)                                 |                              (    used=12109KB)
                            (    free=1678KB)                                  |                              (    free=2699KB)
                            (    waste=0KB =0.00%)                                                            (    waste=0KB =0.00%)
                                                                               |   
-                    Thread (reserved=82615KB, committed=9175KB)               |  -                    Thread (reserved=102238KB, committed=12062KB)
                            (thread# 80)                                       |                              (thread# 99)
                            (stack: reserved=82232KB, committed=8792KB)        |                              (stack: reserved=101764KB, committed=11588KB)
                            (malloc=289KB# 482)                                |                              (malloc=358KB# 596) 
                            (arena=94KB# 158)                                  |                              (arena=116KB# 196)
                                                                               |   
-                      Code (reserved=249146KB, committed=23006KB)             |  -                      Code (reserved=251533KB, committed=64609KB)
                            (malloc=1458KB# 8342)                              |                              (malloc=3845KB# 17739) 
                            (mmap: reserved=247688KB, committed=21548KB)       |                              (mmap: reserved=247688KB, committed=60764KB) 
                                                                               |   
-                        GC (reserved=1211KB, committed=1211KB)                |  -                        GC (reserved=1323KB, committed=1323KB)
                            (malloc=439KB# 3369)                               |                              (malloc=551KB# 4476) 
                            (mmap: reserved=772KB, committed=772KB)                                           (mmap: reserved=772KB, committed=772KB) 
                                                                               |   
-                  Compiler (reserved=424KB, committed=424KB)                  |  -                  Compiler (reserved=991KB, committed=991KB)
                            (malloc=294KB# 1015)                               |                              (malloc=861KB# 2304) 
                            (arena=131KB# 5)                                                                  (arena=131KB# 5)
                                                                               |   
-                  Internal (reserved=938KB, committed=938KB)                  |  -                  Internal (reserved=1295KB, committed=1295KB)
                            (malloc=906KB# 1817)                               |                              (malloc=1263KB# 2206) 
                            (mmap: reserved=32KB, committed=32KB)                                             (mmap: reserved=32KB, committed=32KB) 
                                                                               |   
-                     Other (reserved=134KB, committed=134KB)                  |  -                     Other (reserved=170KB, committed=170KB)
                            (malloc=134KB# 22)                                 |                              (malloc=170KB# 40) 
                                                                               |   
-                    Symbol (reserved=17441KB, committed=17441KB)              |  -                    Symbol (reserved=19006KB, committed=19006KB)
                            (malloc=14684KB# 157347)                           |                              (malloc=16153KB# 175495) 
                            (arena=2757KB# 1)                                  |                              (arena=2853KB# 1)
                                                                               |   
-    Native Memory Tracking (reserved=3356KB, committed=3356KB)                |  -    Native Memory Tracking (reserved=4162KB, committed=4162KB)
                            (malloc=19KB# 243)                                 |                              (malloc=28KB# 372) 
                            (tracking overhead=3338KB)                         |                              (tracking overhead=4135KB)
                                                                               |   
-               Arena Chunk (reserved=4368KB, committed=4368KB)                |  -               Arena Chunk (reserved=180KB, committed=180KB)
                            (malloc=4368KB)                                    |                              (malloc=180KB) 
                                                                               |   
-                   Logging (reserved=4KB, committed=4KB)                         -                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB# 191)                                                                 (malloc=4KB# 191) 
                                                                               |   
-                 Arguments (reserved=19KB, committed=19KB)                       -                 Arguments (reserved=19KB, committed=19KB)
                            (malloc=19KB# 504)                                                                (malloc=19KB# 504) 
                                                                               |   
-                    Module (reserved=132KB, committed=132KB)                  |  -                    Module (reserved=146KB, committed=146KB)
                            (malloc=132KB# 1597)                               |                              (malloc=146KB# 1668) 
                                                                               |   
-              Synchronizer (reserved=290KB, committed=290KB)                  |  -              Synchronizer (reserved=393KB, committed=393KB)
                            (malloc=290KB# 2444)                               |                              (malloc=393KB# 3310) 
                                                                               |   
-                 Safepoint (reserved=8KB, committed=8KB)                         -                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)                |                              (mmap: reserved=8KB, committed=8KB)

Step four: threads cost memory

During the above investigation, we noticed that we had 146 threads running, which was excessive for our use-case. So we removed & combined thread pools where we could, to reduce the number of threads. We also found that our web framework was creating large thread pools, but we did not have control over them. This was one of the learning outcomes of this exercise ?? limiting the number of threads to conserve memory.

One usually doesn't find articles on the Web about running Tomcat in memory-constrained environments - our searches all turned up results about multi-gigabyte heaps, where Tomcat's default settings are a much smaller fraction of the memory use. So it was an interesting experience to troubleshoot a situation where we didn't have any search results to use as a starting point.

Did you like this article?

Waqqas Dadabhoy

See other articles by Waqqas

Related jobs

See all

Title

The company

  • Remote

Title

The company

  • Remote

Title

The company

  • Remote

Title

The company

  • Remote

Related articles

JavaScript Functional Style Made Simple

JavaScript Functional Style Made Simple

Daniel Boros

12 Sep 2021

JavaScript Functional Style Made Simple

JavaScript Functional Style Made Simple

Daniel Boros

12 Sep 2021

WorksHub

CareersCompaniesSitemapFunctional WorksBlockchain WorksJavaScript WorksAI WorksGolang WorksJava WorksPython WorksRemote Works
hello@works-hub.com

Ground Floor, Verse Building, 18 Brunswick Place, London, N1 6DZ

108 E 16th Street, New York, NY 10003

Subscribe to our newsletter

Join over 111,000 others and get access to exclusive content, job opportunities and more!

© 2023 WorksHub

Privacy PolicyDeveloped by WorksHub