Avoiding numerical pitfalls

The harmonic series is convergent in floating point arithmetics

\begin{align*} \sum_{n=1}^{\infty} \; \frac{1}{n} \quad = \quad 34.1220356680478715816207113675773143768310546875 \end{align*}

(usually it is famously known to diverge to $\infty$)

References

Proof of divergence: http://www.free-education-resources.com/www.mathematik.net/harmonische-reihen/hr1s20.htm

Proof of convergence for floating point: http://www.maths.tcd.ie/pub/ims/bull71/recipnote.pdf

Resolution: http://fredrik-j.blogspot.de/2009/02/how-not-to-compute-harmonic-numbers.html

Basics

In floating point arithmetic, subtraction is rather inaccurate. Observe that 2-1.8 is not 0.2. However, Python catches this well known phenomenon in its str-method and converts the output to a convenient number. The following two lines illustrate not only numeric subtaction inaccuracy, but also the difference between repr and str. repr is designed to represent the value accurately, while str is intended for a convenient output format.


In [7]:
print repr(2-1.8)
print str(2-1.8)


0.19999999999999996
0.2

Just to mention for completeness: Don't use exact equals-operator on floats:


In [20]:
print (2-1.8 == 0.2)
#Python-hack that actually works surprisingly well:
print (str(2-1.8) == str(0.2))

#Recommended method with control over matching-precision:
threshold = 0.000000001
print (((2-1.8) - 0.2)**2 < threshold)


False
True
True

Let's solve a quadratic equation.

Naive solving becomes bad if low and large coefficients occur. Consider the equation $3 x^2 + 10^5 x + 5 = 0$


In [201]:
import numpy as np
import scipy.optimize as opt

a = 3.0
b = 10e5
c = 5.0
pol = np.polynomial.Polynomial((c, b, a))

def f(x):
    return a*x**2+b*x+c
    #return (a*x+b)*x+c

def f1(x):
    return 2*a*x+b

def f2(x):
    return 2*a

def solve_pq():
    p = b/a
    q = c/a
    D = (p/2.0)**2 - q
    r1 = -p/2.0+D**0.5
    r2 = -p/2.0-D**0.5
    return (r1, r2)

def solve_pq2():
    p = b/a
    q = c/a
    D = (p/2.0)**2 - q
    r1 = -2.0*q/(p+2.0*D**0.5)
    r2 = -p/2.0-D**0.5
    return (r1, r2)

def solve_companion():
    p = b/a
    q = c/a
    C = np.array([[0.0, -q], [1.0, -p]])
    return np.linalg.eigvals(C)

def solve_newton(r):
    return opt.newton(f, r, tol=1.48e-10)#, fprime=None, args=(), tol=1.48e-08, maxiter=50, fprime2=None

def solve_newton2(r):
    return opt.newton(f, r, tol=1.48e-10, fprime=f1)#, args=(), tol=1.48e-08, maxiter=50, fprime2=None

def solve_newton3(r):
    return opt.newton(f, r, tol=1.48e-10, fprime=f1, fprime2=f2)

result = solve_pq()
print "pq"
print repr(result)
print repr(f(result[0]))
print repr(f(result[1]))

result = solve_pq2()
print "pq2"
print repr(result)
print repr(f(result[0]))
print repr(f(result[1]))

result = solve_companion()
print "companion"
print repr(result)
print repr(f(result[0]))
print repr(f(result[1]))

result[0] = solve_newton(result[0])
result[1] = solve_newton(result[1])
print "newton"
print repr(result)
print repr(f(result[0]))
print repr(f(result[1]))

result = np.polynomial.polynomial.polyroots((c, b, a))
print "numpy"
print repr(result)
print repr(f(result[1]))
print repr(f(result[0]))


pq
(-5.0000089686363935e-06, -333333.33332833333)
-8.968561393096763e-06
6.103515625e-05
pq2
(-5.000000000075e-06, -333333.33332833333)
-8.881784197001252e-16
6.103515625e-05
companion
array([ -4.99997986e-06,  -3.33333333e+05])
2.0135269062748762e-05
6.103515625e-05
newton
array([ -5.00000000e-06,  -3.33333333e+05])
0.0
0.0
numpy
array([ -3.33333333e+05,  -4.99997986e-06])
2.0135269062748762e-05
6.103515625e-05

Logarithms can avoid floating point overflow

Especially probabilities often involve large factorials. These can become astronomically large.


In [202]:
import math

n = 645645665476.43e160
m = 125624536575.76e150

#print repr(n**4/m**4)
print repr((n/m)**4)
print repr(math.exp(4*math.log(n)-4*math.log(m)))


6.977166201106374e+42
6.977166201106792e+42

Don't invert that matrix

References

http://www.johndcook.com/blog/2010/01/19/dont-invert-that-matrix/

Summary:

  • There's hardly ever a good reason to invert a matrix
  • Solve linear equation systems directly
  • Apply a QR-decomposition to solve multiple systems with the same matrix (but different right side)
  • Even if the inverse is given for free, direct solving is still more accurate
  • Inverses of sparse matrices are in general dense

In [318]:
import numpy as np
import scipy

#m = np.array([[0.5e90, 0.00008, -0.1, 46786767], [-0.5, 0.2, -0.00001, 0.000008653], [1200000000000000.00002, -600.8, -0.5, 0.0], [-12000, 600.8, -0.698065, 650.0]])
m = np.array([[0.5, 0.00008, -0.1, 4667], [-0.5, 0.2, -0.00001, 0.000008653], [1200.00002, -600.8, -0.5, 0.0], [-12000, 600.8, -0.698065, 650.0]])


#print m
#mI = m**(-1)
mI = np.linalg.inv(m)
#print mI
#print m.dot(mI)

ev = [1.0e-12, 2.0, 88.8, -0.005]
A = m.dot(np.diag(ev)).dot(mI)
print A
print ""
AI = np.linalg.inv(A)
print AI
print ""
print A.dot(AI)

b = np.array([1.0, 2.0, 3.0, 4.0])

# Required is x solving Ax = b
def err(x1):
    v = np.dot(A, x1)-b
    return np.sqrt(np.inner(v, v))

x = np.dot(AI, b)
print err(x)

x2 = scipy.linalg.solve(A, b)
print err(x2)

# A = QR
Q, R = np.linalg.qr(A)
Qb = np.dot(Q.T, b)
x3 = scipy.linalg.solve_triangular(R, Qb)
print err(x3)


[[  6.44990502e-02   5.36839163e+04   1.73710929e+01  -4.99717837e-01]
 [  2.05431541e-05   7.90998814e+00   1.81976767e-03  -1.47605213e-04]
 [  3.06640151e-01   2.60768591e+05   8.66019730e+01  -2.20514772e+00]
 [  5.25260458e-01   3.82363798e+05   1.21503426e+02  -3.78146023e+00]]

[[  6.87262213e+06   3.31201536e+11   6.09022361e+07  -4.93512719e+07]
 [ -6.87282197e+06  -3.31201415e+11  -6.09021971e+07   4.93512708e+07]
 [  1.64947730e+10   7.94883409e+14   1.46165275e+11  -1.18443052e+11]
 [ -1.64947727e+11  -7.94883395e+15  -1.46165273e+12   1.18443050e+12]]

[[  1.00000000e+00   2.50000000e+00   0.00000000e+00  -3.66210938e-04]
 [ -3.72529030e-09   1.00000000e+00  -2.98023224e-08  -5.96046448e-08]
 [ -1.22070312e-04   2.00000000e+00   1.00097656e+00  -4.88281250e-04]
 [  1.22070312e-04   4.00000000e+00   0.00000000e+00   9.99023438e-01]]
31.0161248539
13.9283883114
13.0758905086

Polynomes

  • Horner's method for numerically stable evaluation
  • NumPy's Polynome-class has this built-in

For expansion:

  • sometimes we use monomials to expand input data
  • a subsequent algorithm is supposed to figure out the factors and assemble polynomials
  • these will be unstable in a naive approach (if high degrees are involved, degrees above 60 can already cause issues)
  • instead of monomials use a numerically stable base like legendre polynomials
  • you might need to adjust them for your value-set

In [ ]: