Algorithms, Design, Code and more: Hive Plan Generation Hooks

Sunday, May 18, 2014

Hive Abstract Semantic Analyzer Hook

Hive allows Pre & Post Analyzer hooks to be added to the normal hive plan query generation flow via the AbstractSemanticAnalyzerHook class.

A custom hook needs to extend AbstractSemanticAnalyzerHook & override the preAnalyze or postAnalyze method as necessary.

Simple Sematic Analyzer Hook:
A sematic analyzer hook that logs a message each in the preAnalyze & postAnalyze methods, is shown below:

package org.apache.hadoop.hive.ql.parse;

public class SimpleSemanticPreAnalyzerHook extends AbstractSemanticAnalyzerHook{
static final private Log LOG = LogFactory.getLog(SimpleSemanticPreAnalyzerHook.class.getName());
static final private LogHelper console = new LogHelper(LOG);

@Override
public ASTNode preAnalyze(HiveSemanticAnalyzerHookContext context,
ASTNode ast) throws SemanticException {
console.printInfo("!! SimpleSemanticPreAnalyzerHook preAnalyze called !!");
return super.preAnalyze(context, ast);
}

@Override
public void postAnalyze(HiveSemanticAnalyzerHookContext context,
List<Task<? extends Serializable>> rootTasks)
throws SemanticException {
console.printInfo("!! SimpleSemanticPreAnalyzerHook postAnalyze called !!");
super.postAnalyze(context, rootTasks);
}
}

Configurations for Simple Sematic Analyzer Hook:

Thursday, April 10, 2014

Hive Query Plan Generation

Hive query is passed through several built-in modules for the final plan to be generated.

The stages/ modules are:

Query
=> (1) Parser
=> (2) Semantic Analyzer
=> (3) Logical Plan Generation
=> (4) Optimizer
=> (5) Physical Plan Generation
=> Executor to run on Hadoop