Map-Reduce Logs on Hive-Tez

Question

I want to get the interpretation of Map-Reduce logs after running a query on Hive-Tez ? What the lines after INFO: conveys ? Here I have attached a sample

INFO  : Session is already open
INFO  : Dag name: SELECT a.Model...)
INFO  : Tez session was closed. Reopening...
INFO  : Session re-established.
INFO  : 
INFO  : Status: Running (Executing on YARN cluster with App id application_14708112341234_1234)

INFO  : Map 1: -/-  Map 3: -/-  Map 4: -/-  Map 7: -/-  Reducer 2: 0/15     Reducer 5: 0/26 Reducer 6: 0/13 
INFO  : Map 1: -/-  Map 3: 0/118    Map 4: 0/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26  Reducer 6: 0/13
INFO  : Map 1: 0/118    Map 3: 0/118    Map 4: 0/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13 
INFO  : Map 1: 0/118    Map 3: 0/118    Map 4: 0(+5)/118    Map 7: 0/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13 
INFO  : Map 1: 0/118    Map 3: 0(+5)/118    Map 4: 0(+7)/118    Map 7: 0(+1)/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13 
INFO  : Map 1: 0/118    Map 3: 0(+15)/118   Map 4: 0(+18)/118   Map 7: 0(+1)/1  Reducer 2: 0/15 Reducer 5: 0/26 Reducer 6: 0/13 

Beall

Answer 1

The log you posted is DAG execution log. The DAG consists of Map 1,Map 3,Map 4,Map 7 mappers vertices and reducers: Reducer 2,Reducer 5,Reducer 6

Map 1: -/- - this means that the vertex is not initialized, the number of mappers are not calculated yet.

Map 4: 0(+7)/118 - this means that totally there are 118 mappers, 7 of them are running in parallel, 0 completed yet, 118-7=111 are pending.

Reducer 2: 0/15 - this means that totally there are 15 reducers, 0 of them are running, 0 of them are completed (15 reducers pending).

Negative figures (there are no such in your example) = number of failed or killed mappers or reducers

Qubole has explanation about Tez log: https://docs.qubole.com/en/latest/user-guide/hive/using-hive-on-tez/hive-tez-tuning.html#understanding-log-pane

Clever

Just what I was looking for! Do you have a source for this? – Tried

I have no sources for this. But you can easily check this looking at the job tracker during tez execution. – Clever

@Tried Have found this on Qubole: docs.qubole.com/en/latest/user-guide/hive/using-hive-on-tez/… – Clever

If I saw failed/killed mappers/reducers, in other words, negative figures, will the failed ones restart on their own and complete? – Jethro

@MemphisMeng Yes, they will restart 3 times and if not successful then fail, all other containers will be killed and entire job failed – Clever

@MemphisMeng If the same container restarted 3 times and finally failed, all job will fail. Negative figures can be also kills for example when speculative execution enabled, slower task gets killed after running some time – Clever