Scala example using Spark SQL over Cloudant as a source

This sample notebook is written in Scala and expects the Scala 2.10 runtime. Make sure the kernel is started and we are connected when executing this notebook.

The data source for this example can be found at: http://examples.cloudant.com/crimes/

Replicate the database into your own Cloudant account before you execute this script.

1. Work with the Spark Context

A Spark Context handle sc is available with every notebook create in the Spark Service. Use it to understand the Spark version used, the environment settings, and create a Spark SQL Context object off of it.



In [1]:

    
sc.version









    Out[1]:





1.4.1



In [2]:

    
val sqlCtx = new org.apache.spark.sql.SQLContext(sc)

2. Work with a Cloudant database

A Dataframe object can be created directly from a Cloudant database. To configure the database as source, pass these options:

1 - package name that provides the classes (like CloudantDataSource) implemented in the connector to extend BaseRelation. For the Cloudant Spark connector this will be com.cloudant.spark

2 - cloudant.host parameter to pass the Cloudant account name

3 - cloudant.user parameter to pass the Cloudant user name

4 - cloudant.password parameter to pass the Cloudant account password



In [3]:

    
val df = sqlCtx.read.format("com.cloudant.spark").option("cloudant.host","examples.cloudant.com").option("cloudant.username", "examples").option("cloudant.password","xxxx").load("crimes")









    



Use dbName=crimes, indexName=null, jsonstore.rdd.partitions=5, jsonstore.rdd.maxInPartition=-1, jsonstore.rdd.minInPartition=10, jsonstore.rdd.requestTimeout=100000,jsonstore.rdd.concurrentSave=-1,jsonstore.rdd.bulkSize=1

3. Work with a Dataframe

At this point all transformations and functions should behave as specified with Spark SQL. (http://spark.apache.org/sql/)

There are, however, a number of things the Cloudant Spark connector does not support yet, or things that are simply not working. For that reason we call this connector a BETA release and are only gradually improving it towards GA.

Please direct your any change requests at support@cloudant.com



In [4]:

    
df.printSchema()









    



root
 |-- _id: string (nullable = true)
 |-- _rev: string (nullable = true)
 |-- geometry: struct (nullable = true)
 |    |-- coordinates: array (nullable = true)
 |    |    |-- element: double (containsNull = true)
 |    |-- type: string (nullable = true)
 |-- properties: struct (nullable = true)
 |    |-- compnos: string (nullable = true)
 |    |-- domestic: boolean (nullable = true)
 |    |-- fromdate: long (nullable = true)
 |    |-- main_crimecode: string (nullable = true)
 |    |-- naturecode: string (nullable = true)
 |    |-- reptdistrict: string (nullable = true)
 |    |-- shooting: boolean (nullable = true)
 |    |-- source: string (nullable = true)
 |-- type: string (nullable = true)



In [5]:

    
df.count()









    Out[5]:





271



In [6]:

    
df.select("properties.naturecode").show()









    



+----------+
|naturecode|
+----------+
|    DISTRB|
|       EDP|
|    ARREST|
|        AB|
|      CD14|
|    UNKEMS|
|      REQP|
|       EDP|
|       MVA|
|     IVPER|
|      NIDV|
|        AB|
|    IVPREM|
|     IVPER|
|     IVPER|
|       MVA|
|      CD11|
|    LARCEN|
|       MVA|
|    ARREST|
+----------+



In [8]:

    
df.filter(df.col("properties.naturecode").startsWith("DISTRB")).show()









    



+--------------------+--------------------+--------------------+--------------------+-------+
|                 _id|                _rev|            geometry|          properties|   type|
+--------------------+--------------------+--------------------+--------------------+-------+
|79f14b64c57461584...|1-d9518df5c255e4b...|[List(-71.0598744...|[142035012,true,1...|Feature|
|79f14b64c57461584...|1-798ef404b141dfb...|[List(-71.0554045...|[142035053,false,...|Feature|
|79f14b64c57461584...|1-08cd46894f9c579...|[List(-71.1317963...|[142035113,false,...|Feature|
|79f14b64c57461584...|1-be4512491803441...|[List(-71.1331290...|[142035116,false,...|Feature|
|79f14b64c57461584...|1-2e3e1fe35278b5d...|[List(-71.0784045...|[142035162,false,...|Feature|
|79f14b64c57461584...|1-e03133da93c2644...|[List(-71.0824845...|[142035211,false,...|Feature|
|79f14b64c57461584...|1-4c21d07bfb9f45a...|[List(-71.1013666...|[142035316,false,...|Feature|
+--------------------+--------------------+--------------------+--------------------+-------+