Cloudera Data Science Workbench allows you to update Spark’s internal logging configuration on a per-project basis. Spark 2 uses Apache Log4j, which can be configured through a properties file. By default, a log4j.properties file found in the root of your project will be appended to the existing Spark logging properties for every session and job.
Note: To specify a custom location, set the environmental variable LOG4J_CONFIG to the file location relative to your project.
- Create a new file log4j.properties in the root of your project
- Add the one of the following configurations to set the logging level for Spark jobs.
PySpark logging levels should be set as follows:
And Scala logging levels should be set as:
- Start a new session and reproduce the issue