Lets make some simple examples with scala

We'll make a very simple scala object compile it and use it in the python process



In [1]:

    
!mkdir helloscala









    



mkdir: cannot create directory `helloscala': File exists



In [2]:

    
%%file helloscala/hw.scala

object HelloScala
{ 
    def sayHi(): String = "Hi! from scala"
    def sum(x: Int, y: Int) = x + y
}









    



Overwriting helloscala/hw.scala

We'll compile things with sbt (the scala build tool)

This makes us a jar that we can load with spark.



In [3]:

    
%%bash
cd helloscala
sbt package









    



Loading /mnt/data/analytics/mniekerk/scala/sbt/bin/sbt-launch-lib.bash
[info] Set current project to helloscala (in build file:/mnt/data/analytics/mniekerk/notebooks/spylon/helloscala/)
[info] Compiling 1 Scala source to /mnt/data/analytics/mniekerk/notebooks/spylon/helloscala/target/scala-2.10/classes...
[info] Packaging /mnt/data/analytics/mniekerk/notebooks/spylon/helloscala/target/scala-2.10/helloscala_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 3 s, completed Sep 12, 2016 11:54:09 AM



In [4]:

    
import spylon
import spylon.spark as sp
c = sp.SparkConfiguration()

c._spark_home = "/path/to/spark-1.6.2-bin-hadoop2.6"
c.master = ["local[4]"]

Add the jar we built previously so that we can import our scala stuff



In [5]:

    
c.jars = ["./helloscala/target/scala-2.10/helloscala_2.10-0.1-SNAPSHOT.jar"]



In [7]:

    
(sc, sqlContext) = c.sql_context("MyApplicationName")

Lets load our helpers and import the Scala class we just wrote



In [8]:

    
from spylon.spark.utils import SparkJVMHelpers
helpers = SparkJVMHelpers(sc)



In [9]:

    
Hi = helpers.import_scala_object("HelloScala")



In [13]:

    
print Hi.__doc__









    



Help on class HelloScala$ in package HelloScala$:

HelloScala$ {
|  
|  Methods defined here:
|  
|  sayHi() : String
|  
|  sum(int, int) : int
|  
|  ------------------------------------------------------------
|  Fields defined here:
|  
|  MODULE$ : HelloScala$
|  
|  ------------------------------------------------------------
|  Internal classes defined here:
|  
}



In [10]:

    
Hi.sayHi()









    Out[10]:





u'Hi! from scala'



In [11]:

    
Hi.sum(4, 6)









    Out[11]:





10