Error messages

Very important
Have lots of information in them
Take the time to read them.
If you get multiple error messages, the first one is the most important, then the last one, then the others in the middle.

An error message has 4 parts:

The stack trace
The error itself and a small description (the most important) (only 1 line)
Sometimes: extra details
Sometimes: hint to help you get more information

Here is an example of code with a user error. Don't try to find the error by code inspection, check only the error:



In [1]:

    
import numpy as np
import theano
import theano.tensor as T
x = T.vector()
y = T.vector()
z = x + x
z = z * y
f = theano.function([x, y], z)
f(np.ones((2,)), np.ones((3,)))









    



---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-1c52c188c23d> in <module>()
      7 z = z * y
      8 f = theano.function([x, y], z)
----> 9 f(np.ones((2,)), np.ones((3,)))

/u/bastienf/repos/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    616                         self.fn.nodes[self.fn.position_of_error],
    617                         self.fn.thunks[self.fn.position_of_error],
--> 618                         storage_map=self.fn.storage_map)
    619                 else:
    620                     # For the c linker We don't have access from

/u/bastienf/repos/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map)
    295     exc_value = exc_type(str(exc_value) + detailed_err_msg +
    296                          '\n' + '\n'.join(hints))
--> 297     reraise(exc_type, exc_value, exc_trace)
    298 
    299 

/u/bastienf/repos/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    605         t0_fn = time.time()
    606         try:
--> 607             outputs = self.fn()
    608         except Exception:
    609             if hasattr(self.fn, 'position_of_error'):

ValueError: Input dimension mis-match. (input[0].shape[0] = 2, input[1].shape[0] = 3)
Apply node that caused the error: Elemwise{Composite{((i0 + i0) * i1)}}(<TensorType(float64, vector)>, <TensorType(float64, vector)>)
Toposort index: 0
Inputs types: [TensorType(float64, vector), TensorType(float64, vector)]
Inputs shapes: [(2,), (3,)]
Inputs strides: [(8,), (8,)]
Inputs values: [array([ 1.,  1.]), array([ 1.,  1.,  1.])]
Outputs clients: [['output']]

Debugprint of the apply node: 
Elemwise{Composite{((i0 + i0) * i1)}} [@A] <TensorType(float64, vector)> ''   
 |<TensorType(float64, vector)> [@B] <TensorType(float64, vector)>
 |<TensorType(float64, vector)> [@C] <TensorType(float64, vector)>

Storage map footprint:
 - <TensorType(float64, vector)>, Input, Shape: (3,), ElemSize: 8 Byte(s), TotalSize: 24 Byte(s)
 - <TensorType(float64, vector)>, Input, Shape: (2,), ElemSize: 8 Byte(s), TotalSize: 16 Byte(s)
 TotalSize: 40 Byte(s) 0.000 GB
 TotalSize inputs: 40 Byte(s) 0.000 BG

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.

Where in your code does this error come from?

The HINT tells you how to get a better error message. In Python, you can pass the parameter mode=theano.Mode(optimizer=TODO) to theano.function, instead of changing the Theano flag.

This allows you to see in your code, which line caused the problem!



In [ ]:

    
# TODO: finish to define the mode below
mode=...

import numpy as np
import theano
import theano.tensor as T
x = T.vector()
y = T.vector()
z = x + x
z.name = "z1"
z = z * y
z.name = "z2"
f = theano.function([x, y], z, mode=mode)
f(np.ones((2,)), np.ones((3,)))

Stack trace

The stack trace can be very useful. You don't need to understand the part in Theano (where the file is inside theano/), just check the part coming from your code files.

DebugMode

Checks and double-checks everything, extremely slow.

Compares Python, C and GPU implementations.
Compares values before and after each optimization.
By default, raises an error on NaN.
Sensitive: so frequently reported errors are OK.

Use the Theano flag mode=DebugMode or the the parameter mode=theano.compile.DebugMode() to theano.function().

Python debugger (PDB)

PDB is similar to GDB

n: next
c: continue
l: list
p var: print
s: step,
bt: print the stack trace
...

To get into PDB:

import pdb; pdb.set_trace()
python -m pdb script.py

To get into PDB with Theano:

Some Theano flags:
- mode=FAST_COMPILE disables most of optimizations, and C code generation.
- linker=py disables C code generation.
- warn_float64=pdb
- ...
PdbBreakpoint: breakpoint during execution.

Breakpoint during execution



In [ ]:

    
import theano
import theano.tensor as T
from theano.tests.breakpoint import PdbBreakpoint
input, target = T.fvectors(['x', 'y'])

mse = (input - target) ** 2

# Conditional breakpoint to be activated if the total
# MSE > 100. The breakpoint will monitor the inputs,
# targets as well as the individual error values
breakpointOp = PdbBreakpoint("MSE too high")
condition = T.gt(mse.sum(), 100)
mse, monitored_input, monitored_target = breakpointOp(
    condition, mse, input, target)

# Compile the theano function
fct = theano.function([input, target], mse)

# Use the function
print fct([10, 0], [10, 5]) # Will NOT activate the breakpoint
print fct([0, 0], [10, 5]) # Will activate the breakpoint

Printing during execution



In [ ]:

    
import theano

x = theano.tensor.vector()
o = theano.printing.Print("a message")(x)
f = theano.function([x], o)
d = f([3, 4])

Printing attributes of a variable



In [ ]:

    
o = theano.printing.Print("Attributes of x:", attrs=('min', 'mean', 'max'))(x)
f = theano.function([x], o)
d = f([3, 1, 4, 9])

Most Frequent NaN Causes

Hyperparameters (ex: learning rate)
Initialization of parameters
Numerical Stability
Algorithm Related

Run in NanGuardMode, DebugMode, or MonitorMode.

NanGuardMode

Can check for:

Nan
Inf
Big values (greater than 1e10)



In [ ]:

    
import numpy

import theano
import theano.compile.nanguardmode
from theano import tensor as T

x = T.matrix()
w = theano.shared(numpy.random.randn(5, 7).astype(theano.config.floatX))
y = T.dot(x, w)
mode=theano.compile.nanguardmode.NanGuardMode(nan_is_error=True,
                  inf_is_error=True,
                  big_is_error=True)
fun = theano.function(
    [x], y, mode=mode)
infa = numpy.tile(
    (numpy.asarray(100.) ** 1000000), (3, 5))
fun(infa)

Test Value

Give sample values to symbolic variables and have the graph execute as it is being built. This allows to get some type of error like shape errors when you build the graph instead of during the execution.



In [ ]:

    
# Can also be 'off', 'ignore', 'raise', 'pdb'
theano.config.compute_test_value = 'warn'

# input which will be of shape (5, 10)
x, y  = T.matrices('xy')

# provide Theano with a default test-value
x.tag.test_value = numpy.random.rand(5, 10)
y.tag.test_value = numpy.random.rand(4, 10)

x + y # warn about the shape error

Others

theano.tensor.opt.AssertOp
- Assertion during execution
theano.printing.{debugprint,pydotprint}
- Text and graphic display of Theano graphs
Flag: profile=True
- Profiles the execution time of Theano function
Flag: exception_verbosity=True
- Extra verbose errors, ex: for MemoryError
Flag: warn_float64='pdb'
- Allows to know where float64 are build in the graph

Advanced

Flag: on_opt_error=True
- Makes Theano stop on optimization error instead of skipping the optimization in question
Flag: optimizer_verbose=True
- Makes Theano print each optimization it applies