How do I suppress the bloat of useless information when using the DUMP command while using grunt via 'pig -x local'?

Question

I'm working with PigLatin, using grunt, and every time I 'dump' stuffs, my console gets clobbered with blah blah, blah non-info, is there a way to surpress all that?

grunt> A = LOAD 'testingData' USING PigStorage(':'); dump A; 

2013-05-06 19:42:04,146 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2013-05-06 19:42:04,147 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false ...
...
--- another like 50 lines of useless context clobbering junk here... till --- ...
...
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

now my like 4 lines of info looking for:

(daemon,*,1,1,System Services,/var/root,/usr/bin/false)
(uucp,*,,,/var/spool/uucp,/usr/sbin/uucico)
(taskgated,*,13,13,Task Gate Daemon,/var/empty,/usr/bin/false)
(networkd,*,24,24,Network Services,/var/empty,/usr/bin/false)
(installassistant,*,25,25,/usr/bin/false)

grunt>                                                                                       

---> obviously if it errors, fine lotsa info helpful, but not when it basically works great.

Viscous

I tried the above , it doesn't work... However this link works #16628248 – Highup

Answer 1

You need to set the log4j properties. For example:

$PIG_HOME/conf/pig.properties :
enable:
# log4jconf=./conf/log4j.properties

rename: log4j.properties.template -> log4j.properties

log4j.properties :
set info to error:
log4j.logger.org.apache.pig=info, A

You may also set the Hadoop related logging level as well:

log4j.logger.org.apache.hadoop = error, A

Landwaiter

There seems to be a bug in Pig 0.12.0 which causes this to fail. – Elnaelnar

@Xaero182 I think that 0.12.0 uses logback instead of log4j.Try adding this JAVA OPT : -Dlogback.configurationFile=/path/to/your/logback.xml file. – Sandstorm

Answer 2

An easy way to do this seems to be to redirect standard error as below.

But it will suppress all errors.

pig -x local 2> /dev/null

Also found that if you remove or rename your hadoop install directory to basically make it inaccessible to pig then all those INFO messages go away.
Changing logging levels in hadoop didn't help, just so that you know.

Blamed

This, frankly, is the only one that answers the question directly and provides a simple solution. Creating and editing a log4j.properties file doesn't really cut it for me. – Fortyniner

It is the best way for me – Memoried

Answer 3

When you start pig, pass it a log4j.properties file with pig -4 <filename>.

In my case there was a log4j.properties in the conf directory and setting the level of the logger named org.apache.pig to ERROR is sufficient to make the logger less verbose.

log4j.logger.org.apache.pig=ERROR, A

Lumbard

Answer 4

pig has debug log level one need to set that in pig.properties file,

# Logging level. debug=OFF|ERROR|WARN|INFO|DEBUG (default: INFO)
#
# debug=INFO

The reason one get large logs on console, e.g. change it to ERROR

Tangential

Do we need to restart Unix after this change? it is not working for me. I updated it but it is not working. – Synopsize