Skip to content

yzhang921/learning-spark

 
 

Repository files navigation

buildstatus Examples for Learning Spark

Examples for the Learning Spark book. These examples require a number of libraries and as such have long build files. We have also added a stand alone example with minimal dependcies and a small build file in the mini-complete-example directory.

Requirements

  • JDK 1.7 or higher
  • Scala 2.10.3
  • scala-lang.org
  • Spark 1.0 sanp shot
  • Protobuf compiler
  • On debian you can install with sudo apt-get install protobuf-compiler
  • R & the CRAN package Imap are required for the ChapterSixExample
  • The Python examples require urllib3

Python examples

From spark just run ./bin/pyspark ./src/python/[example]

Spark Submit

You can also create an assembly jar with all of the dependcies for running either the java or scala versions of the code and run the job with the spark-submit script

./sbt/sbt assembly OR mvn package cd $SPARK_HOME; ./bin/spark-submit --class com.oreilly.learningsparkexamples.[lang].[example] ../learning-spark-examples/target/scala-2.10/learning-spark-examples-assembly-0.0.1.jar

Learning Spark

About

Example code from Learning Spark book

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 50.5%
  • Scala 29.8%
  • Python 12.4%
  • Shell 6.9%
  • Other 0.4%