Container killed by the ApplicationMaster Exit code is 143

Question

I've been getting the following error in several cases:

2017-03-23 11:55:10,794 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1490079327128_0048_r_000003_0: Container killed by the ApplicationMaster.

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

I noticed it happens one large sorts but when i change the "Sort Allocation Memory" it does not help.

I tried changing other memory properties but yet, the solution eludes me. Is there a good explanation to how Mapreduce works and what's the interaction between the different components? What should I change? where do I locate the Java error leading to this?

Illuviation

hive jobs throwing this error right? – Golden

Exactly, and I cant find the JAVA exception – Illuviation

I am asking about, is it hive jobs or your own map reduce jobs? – Golden

There hive Jobs. – Illuviation

The table having partition? – Golden

I cant use partitioning or bucketing in this case – Illuviation

could you share your schema and what you are trying to do? – Golden

Hello @Illuviation Did you solve the problem? If so do you remember it? – Fane

Answer 1

Exit code 143 is related to Memory/GC issues. Your default Mapper/reducer memory setting may not be sufficient to run the large data set. Thus, try setting up higher AM, MAP and REDUCER memory when a large yarn job is invoked.

Please check this link out: https://community.hortonworks.com/questions/96183/help-troubleshoot-container-killed-by-the-applicat.html

Please look into: https://www.slideshare.net/SparkSummit/top-5-mistakes-when-writing-spark-applications-63071421

Excellent source to optimize your code.

Quitrent

Answer 2

I found out I mixed up two separate things. The 143 exit code is from the metrics collector which is down. The Jobs are killed, as far as I understand, due to no memory issues. The problem is with large window functions that cant reduce the data till the last one which contains all the data.

Although, the place in the logs where it gives the reason why the job was killed, still eludes me.

Illuviation

Answer 3

This issue is related to memory issue. you can resolve it by increasing driver memory, executor memory and executor cores. If you are working on spark shell then while opening spark shell you can provide these configs to increase memory according to your data size.

Bik