CPU Usage JVM Thread Level Monitoring

0 votes
5 views
asked Aug 30, 2017 in Hadoop by admin (4,410 points)
SummaryHow to enable and capture useful information about CPU utilization on the thread level from a JVM. 

Symptoms

You have high CPU utilization from a JVM for argument sake lets say the Hive Metastore Server. You do not readily understand why this is the situation and you want to understand what is happening. 

Applies To

All versions of CDH and all JVMs which is almost all services. 

Cause
Instructions

A link to the document that helps understand Java Flight Recorder is here:

https://docs.oracle.com/javacomponents/jmc-5-4/jfr-runtime-guide/run.htm#JFRUH178 

The following properties represent the JVM configuration that needs to be applied to be able to collect the information you want. These values can be added to the Java Configuration Options safety valve for the service to be monitored, for example the "Java Configuration Options for Hive Metastore Server".

-Dcom.sun.management.jmxremote 
-Dcom.sun.management.jmxremote.port=9016 
-Dcom.sun.management.jmxremote.local.only=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false 
-XX:+UnlockCommercialFeatures 
-XX:+FlightRecorder

To monitor the JVM the following command is used:

sudo -u hive /usr/java/jdk1.7.0_67-cloudera/bin/jcmd 1410 JFR.start duration=2m filename=/tmp/test.jfr compress=true

Here is the explanation of this command:

1. [sudo -u hive] - This will run the command as user hive to have appropriate permissions to monitor the JVM, otherwise the command will fail.   If monitoring another service, make sure the user used is the user running the JVM process.

2. [/usr/java/jdk1.7.0_67-cloudera/bin/jcmd] - This is full path to the jcmd, this could be different on customer cluster. The full path is not necessary if the JAVA_HOME and PATH environment is setup correctly. 

3. [1410] - This is the PID of Hive Metastore Server. You can see this if you use "ps -ef | grep HiveMeta" or if you use "jcmd | grep HiveMeta".

4. [JFR.start] - This tells jcmd to run the flight recorder. To check the status you can use a command like this:

sudo -u hive /usr/java/jdk1.7.0_67-cloudera/bin/jcmd 1410 JFR.check

5. [duration=2m] - This is how long the Flight recorder will run. We have tested the modifier "s,m,h" meaning "seconds,minutes,hours".

6. [filename=/tmp/test.jfr] - This is the file that will store the Flight recording. 

7. [compress=true] - This will compress the flight recording. 

Also, one can use the Java Mission Control which is GUI based. The instructions are below to start a Flight recording. 

https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr004.html

Please log in or register to answer this question.

...