Toree Magics

Magics are special "functions" which enable features or execute some special code. Magics can receive input arguments when they are invoked. There are two types of magics: cell magics and line magics. Magics invocations are not case sensitive.

Table of Contents

  1. Line Magics
    1. LsMagic
    2. Truncation
    3. ShowTypes
    4. AddJar
    5. AddDeps
  2. Cell Magics
    1. DataFrame
    2. Html
    3. JavaScript
    4. PySpark
    5. SparkR
    6. SparkSQL

Line MagicsTop

Line magics are run on a single line and can have other code and line magics within the same cell. Line magics use the following syntax:

%magicname [args]

%LsMagicTop

The LsMagic is a magic to list all the available magics.


In [1]:
%LsMagic


Available line magics:
%addjar %lsmagic %showtypes %adddeps %truncation

Available cell magics:
%%sql %%html %%javascript %%rdd %%scala %%sparkr %%pyspark

Type %<magic_name> for usage info.
         

%TruncationTop

Toree will, by default, truncate results from statements. This can be managed through the %Truncation magic. To see the current state of the truncation setting you can invoke the magic.


In [2]:
// invoke the truncation magic to see if truncation is on or off
%Truncation


Truncation is currently on 

In [3]:
// return a value to see the truncation
(1 to 200)


Out[3]:
Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170...

In [4]:
%Truncation off
(1 to 200)


Output will NOT be truncated
Out[4]:
Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200)

In [5]:
%Truncation on
(1 to 200)


Output WILL be truncated.
Out[5]:
Range(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170...

%ShowTypesTop

The type information for a result is hidden by default. This behavior can be changed by using the %ShowTypes magic. You can view the current state of %ShowTypes by invoking it with no arguments.


In [6]:
%ShowTypes


ShowTypes is currently off 

In [7]:
"Hello types!"


Out[7]:
Hello types!

In [8]:
%ShowTypes on
"Hello types!"


Types will be printed.
Out[8]:
String = Hello types!

In [9]:
(1, "Hello types!")


Out[9]:
(Int, String) = (1,Hello types!)

In [10]:
%ShowTypes off
"Hello types!"


Types will not be printed
Out[10]:
Hello types!

%AddJarTop

AddJar is a magic that allows the addition of jars to Torree's environment. You can see the arguments for AddJar by invoking it with no arguments.


In [11]:
%AddJar


Usage: %AddJar <jar_url>

Option   Description                        
------   -----------                        
-f       forces re-download of specified jar
--magic  loads jar as a magic extension     

In [12]:
%AddJar https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.0.0b/lwjgl-3.0.0b.jar


Starting download from https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.0.0b/lwjgl-3.0.0b.jar
Finished download of lwjgl-3.0.0b.jar

In [13]:
org.lwjgl.Version.getVersion()


Out[13]:
3.0.0b SNAPSHOT

%AddDepsTop

AddDeps is a magic to add dependencies from a maven repository. You can see the arguments for AddDeps by invoking it with no arguments.


In [14]:
%AddDeps


Usage: %AddDeps my.company artifact-id version

Option                        Description                          
------                        -----------                          
--abort-on-resolution-errors  Abort (no downloads) when resolution 
                                fails                              
--repository                  Adds an additional repository to     
                                available list                     
--trace                       Prints out trace of download progress
--transitive                  Retrieve dependencies recursively    
--verbose                     Prints out additional information    

Note, that by default the AddDeps magic will only retrieve the specified dependency. If you want the transitive dependencies provide the --transitive flag.


In [15]:
%AddDeps org.joda joda-money 0.11 --transitive --trace --verbose


Marking org.joda:joda-money:0.11 for download
Preparing to fetch from:
-> file:/tmp/toree_add_deps5662724810625125387/
-> https://repo1.maven.org/maven2
=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom.sha1
=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.pom
===> 1 (joda-money-0.11.pom.sha1): Is 40 total bytes
===> 1 (joda-money-0.11.pom.sha1): Downloaded 40 bytes (100.00%)
=> 1 (joda-money-0.11.pom.sha1): Finished downloading
===> 1 (joda-money-0.11.pom): Is 22792 total bytes
===> 1 (joda-money-0.11.pom): Downloaded 15727 bytes (69.00%)
===> 1 (joda-money-0.11.pom): Downloaded 22792 bytes (100.00%)
=> 1 (joda-money-0.11.pom): Finished downloading
=> 1 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar.sha1
=> 2 (): Downloading https://repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar
===> 1 (joda-money-0.11.jar.sha1): Is 40 total bytes
===> 1 (joda-money-0.11.jar.sha1): Downloaded 40 bytes (100.00%)
=> 1 (joda-money-0.11.jar.sha1): Finished downloading
===> 2 (joda-money-0.11.jar): Is 63725 total bytes
===> 2 (joda-money-0.11.jar): Downloaded 15713 bytes (24.66%)
===> 2 (joda-money-0.11.jar): Downloaded 32097 bytes (50.37%)
===> 2 (joda-money-0.11.jar): Downloaded 48481 bytes (76.08%)
===> 2 (joda-money-0.11.jar): Downloaded 63725 bytes (100.00%)
=> 2 (joda-money-0.11.jar): Finished downloading
-> New file at /tmp/toree_add_deps5662724810625125387/https/repo1.maven.org/maven2/org/joda/joda-money/0.11/joda-money-0.11.jar

In [16]:
org.joda.money.CurrencyUnit.AUD


Out[16]:
AUD

Cell MagicsTop

Cell magics are magics which take the whole cell as their argument. They take the following form:

%%magicname
line1
line2
...

%%DataFrameTop

The %%DataFrame magic is used to convert a Spark SQL DataFrame into various formats. Currently, json, html, and csv are supported. The magic takes an expression, which evauluates to a dataframe, to perform the conversion. So, we first need to create a DataFrame object for reference.


In [1]:
case class DFRecord(key: String, value: Int)
val sqlc = sqlContext
import sqlc.implicits._
val df = sc.parallelize(1 to 10).map(x => DFRecord(x.toString, x)).toDF()

The default output is html


In [1]:
%%dataframe


Out[1]:
%%dataframe [arguments]
DATAFRAME_CODE

DATAGRAME_CODE can be any numbered lines of code, as long as the
last line is a reference to a variable which is a DataFrame.
    Option    Description                           
------    -----------                           
--help    Displays the help and usage text for  
            this magic.                         
--limit   The type of the output: html          
            (default), csv, json (default: 10)  
--output  The type of the output: html          
            (default), csv, json (default: html)

In [2]:
%%dataframe
df


Out[2]:
keyvalue
11
22
33
44
55
66
77
88
99
1010

You can specify the --output argument to change the output type.


In [3]:
%%dataframe --output=csv
df


Out[3]:
key,value
1,1
2,2
3,3
4,4
5,5
6,6
7,7
8,8
9,9
10,10

There is also an option to limit the number of records returned.


In [4]:
%%dataframe --limit=3
df


Out[4]:
keyvalue
11
22
33

%%HtmlTop

The %%HTML magic allows you to return HTML.


In [17]:
%%html
<p>
Hello, <strong>Magics</strong>!
</p>


Out[17]:

Hello, Magics!

%%JavaScriptTop

The %%JavaScript magic allows to return JavaScript. The JavaScript code will run in the notebook.


In [18]:
%%JavaScript
alert("Hello, Magics!")


Out[18]:

%%PySparkTop

The %%PySpark exposes an environment with and a python interpreter and a shared SparkContext.


In [19]:
%%PySpark
from operator import add
print(sc.parallelize(range(1, 100)).reduce(add))


Out[19]:
4950

%%SparkRTop

The %%SparkR exposes an environment with and an R interpreter and a shared SparkContext.


In [20]:
%%SparkR
df <- createDataFrame(sqlContext, faithful)
head(df)


* installing *binary* package ‘SparkR’ ...
* DONE (SparkR)
Loading required package: methods

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    filter, na.omit

The following objects are masked from ‘package:base’:

    intersect, rbind, sample, subset, summary, table, transform

NULL
[1] "Received Id 680be9df-a384-455d-b548-405560772cc1 Code df <- createDataFrame(sqlContext, faithful)\nhead(df)"
[1] "Code expr df <- createDataFrame(sqlContext, faithful)"
[2] "Code expr head(df)"                                   
[1] "Result type character 7"
[1] "Success 680be9df-a384-455d-b548-405560772cc1   eruptions waiting"
[2] "Success 680be9df-a384-455d-b548-405560772cc1 1     3.600      79"
[3] "Success 680be9df-a384-455d-b548-405560772cc1 2     1.800      54"
[4] "Success 680be9df-a384-455d-b548-405560772cc1 3     3.333      74"
[5] "Success 680be9df-a384-455d-b548-405560772cc1 4     2.283      62"
[6] "Success 680be9df-a384-455d-b548-405560772cc1 5     4.533      85"
[7] "Success 680be9df-a384-455d-b548-405560772cc1 6     2.883      55"
[1] "Marking success with output: eruptions waiting\n1     3.600      79\n2     1.800      54\n3     3.333      74\n4     2.283      62\n5     4.533      85\n6     2.883      55"
Out[20]:
eruptions waiting
1     3.600      79
2     1.800      54
3     3.333      74
4     2.283      62
5     4.533      85
6     2.883      55

%%SparkSQLTop

The %%SparkSQL magic allows for SQL queries to be performed against tables saved in spark.


In [21]:
val sqlc = sqlContext
import sqlc.implicits._
case class Record(key: String, value: Int)
val df = sc.parallelize(1 to 10).map(x => Record(x.toString, x)).toDF()
df.registerTempTable("MYTABLE")

In [22]:
%%SQL
SELECT * FROM MYTABLE WHERE value >= 6


Out[22]:
+---+-----+
|key|value|
+---+-----+
|  6|    6|
|  7|    7|
|  8|    8|
|  9|    9|
| 10|   10|
+---+-----+


In [23]:
%%SQL
SELECT * FROM MYTABLE WHERE value >= 4


Out[23]:
+---+-----+
|key|value|
+---+-----+
|  4|    4|
|  5|    5|
|  6|    6|
|  7|    7|
|  8|    8|
|  9|    9|
| 10|   10|
+---+-----+