Line magics are run on a single line and can have other code and line magics within the same cell. Line magics use the following syntax:
%magicname [args]The LsMagic is a magic to list all the available magics.
In [1]:
%LsMagic
Toree will, by default, truncate results from statements. This can be managed through the %Truncation magic. To see the current state of the truncation setting you can invoke the magic.
In [2]:
// invoke the truncation magic to see if truncation is on or off
%Truncation
In [3]:
// return a value to see the truncation
(1 to 200)
Out[3]:
In [4]:
%Truncation off
(1 to 200)
Out[4]:
In [5]:
%Truncation on
(1 to 200)
Out[5]:
The type information for a result is hidden by default. This behavior can be changed by using the %ShowTypes magic. You can view the current state of %ShowTypes by invoking it with no arguments.
In [6]:
%ShowTypes
In [7]:
"Hello types!"
Out[7]:
In [8]:
%ShowTypes on
"Hello types!"
Out[8]:
In [9]:
(1, "Hello types!")
Out[9]:
In [10]:
%ShowTypes off
"Hello types!"
Out[10]:
AddJar is a magic that allows the addition of jars to Torree's environment. You can see the arguments for AddJar by invoking it with no arguments.
In [11]:
%AddJar
In [12]:
%AddJar https://repo1.maven.org/maven2/org/lwjgl/lwjgl/3.0.0b/lwjgl-3.0.0b.jar
In [13]:
org.lwjgl.Version.getVersion()
Out[13]:
AddDeps is a magic to add dependencies from a maven repository. You can see the arguments for AddDeps by invoking it with no arguments.
In [14]:
%AddDeps
Note, that by default the AddDeps magic will only retrieve the specified dependency. If you want the transitive dependencies provide the --transitive flag.
In [15]:
%AddDeps org.joda joda-money 0.11 --transitive --trace --verbose
In [16]:
org.joda.money.CurrencyUnit.AUD
Out[16]:
Cell magics are magics which take the whole cell as their argument. They take the following form:
%%magicname
line1
line2
...The %%DataFrame magic is used to convert a Spark SQL DataFrame into various formats. Currently, json, html, and csv are supported. The magic takes an expression, which evauluates to a dataframe, to perform the conversion. So, we first need to create a DataFrame object for reference.
In [1]:
case class DFRecord(key: String, value: Int)
val sqlc = sqlContext
import sqlc.implicits._
val df = sc.parallelize(1 to 10).map(x => DFRecord(x.toString, x)).toDF()
The default output is html
In [1]:
%%dataframe
Out[1]:
In [2]:
%%dataframe
df
Out[2]:
You can specify the --output argument to change the output type.
In [3]:
%%dataframe --output=csv
df
Out[3]:
There is also an option to limit the number of records returned.
In [4]:
%%dataframe --limit=3
df
Out[4]:
The %%HTML magic allows you to return HTML.
In [17]:
%%html
<p>
Hello, <strong>Magics</strong>!
</p>
Out[17]:
The %%JavaScript magic allows to return JavaScript. The JavaScript code will run in the notebook.
In [18]:
%%JavaScript
alert("Hello, Magics!")
Out[18]:
The %%PySpark exposes an environment with and a python interpreter and a shared SparkContext.
In [19]:
%%PySpark
from operator import add
print(sc.parallelize(range(1, 100)).reduce(add))
Out[19]:
The %%SparkR exposes an environment with and an R interpreter and a shared SparkContext.
In [20]:
%%SparkR
df <- createDataFrame(sqlContext, faithful)
head(df)
Out[20]:
The %%SparkSQL magic allows for SQL queries to be performed against tables saved in spark.
In [21]:
val sqlc = sqlContext
import sqlc.implicits._
case class Record(key: String, value: Int)
val df = sc.parallelize(1 to 10).map(x => Record(x.toString, x)).toDF()
df.registerTempTable("MYTABLE")
In [22]:
%%SQL
SELECT * FROM MYTABLE WHERE value >= 6
Out[22]:
In [23]:
%%SQL
SELECT * FROM MYTABLE WHERE value >= 4
Out[23]: