Hive Incremental Data Load

Hive Incremental Data Load

This tutorials will explain how to incrementally load data in Hive tables. Incremental load in Hive can be achieved using transient table and partition overwrite.


Initial Data Load

LOAD DATA local inpath '/path_to_file/data_files/order_incr_1.txt' OVERWRITE INTO TABLE orders_temp;

set hive.exec.dynamic.partition.mode=nonstrict;
INSERT OVERWRITE TABLE orders_incr PARTITION(order_date) SELECT order_id,cust_id,order_status,order_date from orders_temp;


Approach 1 : Incremental Data Load into Partitioned Table using transient table when size of target table is small.


Approach 2 : Incremental Data Load into Partitioned Table using transient table

If the above command fail with below error then set dynamic partition mode to non-strict.
FAILED: SemanticException [Error 10096]: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict