Unable to create or drop Table in KUDU

0 votes
8 views
asked Aug 20, 2017 in Hadoop by admin (4,410 points)
Summary
Symptoms
1. Client exhibits below behavior:

DROP table "yyyy.xxxxx" in Impala-shell reports below error
AnalysisException: Table does not exist: yyyy.xxxxx

CREATE table "yyyy .xxxxx" in Impala-shell reports below error
ImpalaRuntimeException: Error creating Kudu table 'yyyy.xxxxx' CAUSED BY: ImpalaRuntimeException: Table ''yyyy.xxxxx' already exists in Kudu.

2. Kudu Master logs reports time out of the RPC tablet replica creation request to Tablet Servers 
catalog_manager.cc:3142] Tablet 048027ceeabc4d63a0698750ccff8baa (table yyyy.xxxxx [id=bfa772493eee49ae9743f1dc7b4af02e]) was not created within the allowed timeout. Replacing with a new tablet 4b1bc39811a147a3a31d330219a4cc03

3. Tablet Servers may fail the tablet replica creation due to insufficient disk space to preallocate for tablet WAL files.
consensus_peers.cc:329] T 048027ceeabc4d63a0698750ccff8baa P 3137a7256689453cab2950f7bace8ffd -> Peer e908c7180f6340c38604648f1753048c (xxxxxxxxxx:7050): Couldn't send request to peer e908c7180f6340c38604648f1753048c for tablet 048027ceeabc4d63a0698750ccff8baa. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: FAILED: IO error: Failed to open new log: Insufficient disk space to allocate 1073741824 bytes under path /data01/kudu/wal/wals/048027ceeabc4d63a0698750ccff8baa/.kudutmp.newsegmentQpSCZM (756072448 bytes free vs 0 bytes reserved) (error 28). Retrying in the next heartbeat period. Already tried 211659 times.
Applies To

CDH 5.10.x
CDH 5.11.x
Kudu
Kudu-Impala client

Cause

Tablet replica may be failing during creation or taking more time (than the configured Kudu master RPC timeout value) to complete in the Tablet Servers due to increased (abnormally high) WAL file segment size configuration (--log_segment_size_mb). Basically  "log_segment_size_mb" determines the size of WAL segments, which are, by default, preallocated at tablet creation time.

At times, Tablet replica creation failure may be associated with Tablet Servers complain about insufficient disk space to allocate/accomodate WAL segment size in the configured Kudu Tablet Server WAL Directory

Instructions

Check whether the default value of "log_segment_size_mb" (which has a default value of 64 MB in CDH 5.10.0) is overwritten with higher value in Tserver Safety value in CM. If so, follow below steps to revert it back

CM > Kudu > Configuration > Tablet Server Advanced Configuration Snippet (Safety Valve) for gflagfile > Remove the flag "log_segment_size_mb" > Save and Restart Kudu service

Please log in or register to answer this question.

...