Navigator Metadata server API is not returning values

0 votes
2 views
asked Aug 28, 2017 in Hadoop by admin (4,410 points)
Summary

Applies To
  • Navigator Metadata Server
  • NAVMS cursorMark API
Symptoms

Navigator Metadata Server crashed whenever the Navigator metadata API curl command ran. The example of the curl command looked like:

https://navigator-metadata-host:7187/api/v9/entities/?query=sourceType%3Ahive%20AND%20type%3Afield&limit=2147483647

Cause

The Navigator Metadata API curl command comes back with lots of data because there are more than 600 databases with thousands of tables and thousands of fields in the hive metastore. This requires a very high java heap size for the Navigator Metadata Server. The Java heap size for the Navigator Metadata Server is too low for the requirement.
 

Instructions

Bump up the Navigator Metadata Server java heap size to much bigger will allow the curl command finishing successfully.
Also, for scalability in the future, we suggest to modify the custom code to use: cursorMark.
With cursorMark api, you can page through the results rather than loading everything into memory. Please take a look at the documentation on Cloudera Navigator API. From your own Navigator API console, you can read more details about cursorMark at:

http://<nav-server>:7187/api-console/index.html#!/entities/getEntities_0_1


Below is an example screenshot:

Cloudera Navigator API Page

For example, you can use this way:

https://<nav-server>:7187/api/v9/entities/paging?query=sourceType%3Ahive%20AND%20type%3Afield&limit=100

and then based on cursorMark value returns from above, run the next command with it:

https://<nav-server>:7187/api/v10/entities/paging?query=sourceType%3Ahive%20AND%20type

Please log in or register to answer this question.

...