Forward Differentiation from scratch in Julia

In one implementation of forward-mode automatic differentiation (autodiff), we use "dual numbers" to carry forward partial derivatives along with our calculation. A dual number is similar to a complex number in that it has a real valued component, and an extra component. In the case of dual numbers the extra component is an "infinitesimal" component $\epsilon$. Whereas in complex numbers, $i$ is defined by $i^2 = -1$, for dual numbers, $\epsilon$ is defined by $\epsilon^2 = 0$.



In [1]:

    
# create a dual number type
immutable Dual
    value::Float64
    eps::Float64
end

We'll attempt to differentiate the function

$$f(x) = x^2 + x \sin(x)$$

which seems moderately interesting.



In [2]:

    
# define a test function:
f(x) = x^2 + x * sin(x)









    Out[2]:





f (generic function with 1 method)



In [3]:

    
f(1.)









    Out[3]:





1.8414709848078965

So we'll need to know how to multiply, add and take the sine of our Dual type.



In [4]:

    
import Base: +, *, sin



In [5]:

    
*(x::Dual, y::Dual) = Dual(x.value * y.value, x.value * y.eps + y.value * x.eps)









    Out[5]:





* (generic function with 152 methods)



In [6]:

    
+(x::Dual, y::Dual) = Dual(x.value + y.value, x.eps + y.eps)









    Out[6]:





+ (generic function with 164 methods)



In [7]:

    
sin(x::Dual) = Dual(sin(x.value), cos(x.value) * x.eps)









    Out[7]:





sin (generic function with 11 methods)

Sweet! Now we'll try to run it!



In [8]:

    
f(3.)









    Out[8]:





9.423360024179601



In [9]:

    
x = Dual(3., 1.)
f(x)









    Out[9]:





Dual(9.423360024179601,3.171142518258531)

Check it:



In [10]:

    
fprime(x) = 2 * x + sin(x) + x * cos(x)









    Out[10]:





fprime (generic function with 1 method)



In [11]:

    
fprime(3.)









    Out[11]:





3.171142518258531