What I am trying to achieve is to build and install Oozie 4.2.0 that will enable me to submit Spark jobs to a YARN cluster.
I build the distro by executing: oozie-4.2.0/bin/mkdistro.sh -Puber -Phadoop-2 -DskipTests
. That created oozie-4.2.0-distro.tar.gz
package and inside I can find oozie-4.2.0-sharelib.tar.gz
. However, many tutorials online state that I should use oozie-4.2.0-sharelib-yarn.tar.gz
in order to use YARN. Such a file is not contained in the distro package. How can I make the build process output the YARN version of sharelibs?
I tried to continue with the non-YARN version, but when submitting the example Spark job (and adjusting the HDFS and YARN addresses in job.properties along with master property from local[*]
to yarn
) I got an error:
Error: Could not load YARN classes. This copy of Spark may not have been compiled with YARN support.