how to hide "py4j.java_gateway:Received command c on object id p0"?

Question

Once logging is started in INFO level I keep getting bunch of py4j.java_gateway:Received command c on object id p0 on your logs. How can I hide it?

Sparteine

Answer 1

using the logging module run:

logging.getLogger("py4j").setLevel(logging.ERROR)

Sparteine

happy to help, stackoverflow rocks – Sparteine

Answer 2

None of these answers have worked for me. Even after using the above solutions, I was still getting the logging errors.

After a long search I came across the following that solved my issue.

import logging
logger = spark._jvm.org.apache.log4j
logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

I found this solution in the Databricks Knowledge base article

Bastia

why do we need this line logger = spark._jvm.org.apache.log4j? it do not change or update anything – Punch

Answer 3

The best way to control pyspark and py4j logging is by setting the following snippet:

import logging
logging.getLogger("py4j").setLevel(<pyspark-level>)
logging.getLogger('pyspark').setLevel(<py4j-level>)
logger = logging.getLogger('pyspark')

For your case you should write:

import logging
logging.getLogger("py4j").setLevel(logging.INFO)
logging.getLogger('pyspark').setLevel(logging.ERROR)
logger = logging.getLogger('pyspark')

Sentimentality

Answer 4

For me it was nosetests overwriting all these levels. The only solution that worked was passing --logging-level=INFO as argument to nose. Ref: http://nose.readthedocs.io/en/latest/plugins/logcapture.html#cmdoption-logging-level

Inexact