# Examples and Exercises from Think Stats, 2nd Edition

http://thinkstats2.com



In [1]:

%matplotlib inline

import numpy as np
import pandas as pd

import thinkstats2
import thinkplot



## Multiple regression

Let's load up the NSFG data again.



In [2]:

import first

live, firsts, others = first.MakeFrames()



Here's birth weight as a function of mother's age (which we saw in the previous chapter).



In [3]:

import statsmodels.formula.api as smf

formula = 'totalwgt_lb ~ agepreg'
model = smf.ols(formula, data=live)
results = model.fit()
results.summary()




Out[3]:

OLS Regression Results

Dep. Variable:       totalwgt_lb     R-squared:             0.005

Method:             Least Squares    F-statistic:           43.02

Date:             Thu, 28 Feb 2019   Prob (F-statistic): 5.72e-11

Time:                 09:58:53       Log-Likelihood:      -15897.

No. Observations:        9038        AIC:                3.180e+04

Df Residuals:            9036        BIC:                3.181e+04

Df Model:                   1

Covariance Type:      nonrobust

coef     std err      t      P>|t|  [0.025    0.975]

Intercept     6.8304     0.068   100.470  0.000     6.697     6.964

agepreg       0.0175     0.003     6.559  0.000     0.012     0.023

Omnibus:       1024.052   Durbin-Watson:         1.618

Prob(Omnibus):   0.000    Jarque-Bera (JB):   3081.833

Skew:           -0.601    Prob(JB):               0.00

Kurtosis:        5.596    Cond. No.               118.

Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



We can extract the parameters.



In [4]:

inter = results.params['Intercept']
slope = results.params['agepreg']
inter, slope




Out[4]:

(6.830396973311047, 0.017453851471802877)



And the p-value of the slope estimate.



In [5]:

slope_pvalue = results.pvalues['agepreg']
slope_pvalue




Out[5]:

5.722947107312786e-11



And the coefficient of determination.



In [6]:

results.rsquared




Out[6]:

0.004738115474710369



The difference in birth weight between first babies and others.



In [7]:

diff_weight = firsts.totalwgt_lb.mean() - others.totalwgt_lb.mean()
diff_weight




Out[7]:

-0.12476118453549034



The difference in age between mothers of first babies and others.



In [8]:

diff_age = firsts.agepreg.mean() - others.agepreg.mean()
diff_age




Out[8]:

-3.5864347661500275



The age difference plausibly explains about half of the difference in weight.



In [9]:

slope * diff_age




Out[9]:

-0.06259709972169267



Running a single regression with a categorical variable, isfirst:



In [10]:

live['isfirst'] = live.birthord == 1
formula = 'totalwgt_lb ~ isfirst'
results = smf.ols(formula, data=live).fit()
results.summary()




Out[10]:

OLS Regression Results

Dep. Variable:       totalwgt_lb     R-squared:             0.002

Method:             Least Squares    F-statistic:           17.74

Date:             Thu, 28 Feb 2019   Prob (F-statistic): 2.55e-05

Time:                 09:58:53       Log-Likelihood:      -15909.

No. Observations:        9038        AIC:                3.182e+04

Df Residuals:            9036        BIC:                3.184e+04

Df Model:                   1

Covariance Type:      nonrobust

coef     std err      t      P>|t|  [0.025    0.975]

Intercept           7.3259     0.021   356.007  0.000     7.286     7.366

isfirst[T.True]    -0.1248     0.030    -4.212  0.000    -0.183    -0.067

Omnibus:       988.919   Durbin-Watson:         1.613

Prob(Omnibus):  0.000    Jarque-Bera (JB):   2897.107

Skew:          -0.589    Prob(JB):               0.00

Kurtosis:       5.511    Cond. No.               2.58

Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



Now finally running a multiple regression:



In [11]:

formula = 'totalwgt_lb ~ isfirst + agepreg'
results = smf.ols(formula, data=live).fit()
results.summary()




Out[11]:

OLS Regression Results

Dep. Variable:       totalwgt_lb     R-squared:             0.005

Method:             Least Squares    F-statistic:           24.02

Date:             Thu, 28 Feb 2019   Prob (F-statistic): 3.95e-11

Time:                 09:58:53       Log-Likelihood:      -15894.

No. Observations:        9038        AIC:                3.179e+04

Df Residuals:            9035        BIC:                3.182e+04

Df Model:                   2

Covariance Type:      nonrobust

coef     std err      t      P>|t|  [0.025    0.975]

Intercept           6.9142     0.078    89.073  0.000     6.762     7.066

isfirst[T.True]    -0.0698     0.031    -2.236  0.025    -0.131    -0.009

agepreg             0.0154     0.003     5.499  0.000     0.010     0.021

Omnibus:       1019.945   Durbin-Watson:         1.618

Prob(Omnibus):   0.000    Jarque-Bera (JB):   3063.682

Skew:           -0.599    Prob(JB):               0.00

Kurtosis:        5.588    Cond. No.               137.

Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.



As expected, when we control for mother's age, the apparent difference due to isfirst is cut in half.

If we add age squared, we can control for a quadratic relationship between age and weight.



In [12]:

live['agepreg2'] = live.agepreg**2
formula = 'totalwgt_lb ~ isfirst + agepreg + agepreg2'
results = smf.ols(formula, data=live).fit()
results.summary()




Out[12]:

OLS Regression Results

Dep. Variable:       totalwgt_lb     R-squared:             0.007

Method:             Least Squares    F-statistic:           22.64

Date:             Thu, 28 Feb 2019   Prob (F-statistic): 1.35e-14

Time:                 09:58:54       Log-Likelihood:      -15884.

No. Observations:        9038        AIC:                3.178e+04

Df Residuals:            9034        BIC:                3.181e+04

Df Model:                   3

Covariance Type:      nonrobust

coef     std err      t      P>|t|  [0.025    0.975]

Intercept           5.6923     0.286    19.937  0.000     5.133     6.252

isfirst[T.True]    -0.0504     0.031    -1.602  0.109    -0.112     0.011

agepreg             0.1124     0.022     5.113  0.000     0.069     0.155

agepreg2           -0.0018     0.000    -4.447  0.000    -0.003    -0.001

Omnibus:       1007.149   Durbin-Watson:         1.616

Prob(Omnibus):   0.000    Jarque-Bera (JB):   3003.343

Skew:           -0.594    Prob(JB):               0.00

Kurtosis:        5.562    Cond. No.           1.39e+04

Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.[2] The condition number is large, 1.39e+04. This might indicate that there arestrong multicollinearity or other numerical problems.



When we do that, the apparent effect of isfirst gets even smaller, and is no longer statistically significant.

These results suggest that the apparent difference in weight between first babies and others might be explained by difference in mothers' ages, at least in part.

## Data Mining

We can use join to combine variables from the preganancy and respondent tables.



In [13]:

import nsfg

live = live[live.prglngth>30]
resp.index = resp.caseid
join = live.join(resp, on='caseid', rsuffix='_r')



And we can search for variables with explanatory power.

Because we don't clean most of the variables, we are probably missing some good ones.



In [14]:

import patsy

def GoMining(df):
"""Searches for variables that predict birth weight.

df: DataFrame of pregnancy records

returns: list of (rsquared, variable name) pairs
"""
variables = []
for name in df.columns:
try:
if df[name].var() < 1e-7:
continue

formula = 'totalwgt_lb ~ agepreg + ' + name
formula = formula.encode('ascii')

model = smf.ols(formula, data=df)
if model.nobs < len(df)/2:
continue

results = model.fit()
except (ValueError, TypeError, patsy.PatsyError):
continue

variables.append((results.rsquared, name))

return variables




In [15]:

variables = GoMining(join)



The following functions report the variables with the highest values of $R^2$.



In [16]:

import re

"""Reads Stata dictionary files for NSFG data.

returns: DataFrame that maps variables names to descriptions
"""

all_vars = vars1.append(vars2)
all_vars.index = all_vars.name
return all_vars

def MiningReport(variables, n=30):
"""Prints variables with the highest R^2.

t: list of (R^2, variable name) pairs
n: number of pairs to print
"""

variables.sort(reverse=True)
for r2, name in variables[:n]:
key = re.sub('_r$', '', name) try: desc = all_vars.loc[key].desc if isinstance(desc, pd.Series): desc = desc[0] print(name, r2, desc) except (KeyError, IndexError): print(name, r2)  Some of the variables that do well are not useful for prediction because they are not known ahead of time.  In [17]: MiningReport(variables)  Combining the variables that seem to have the most explanatory power.  In [18]: formula = ('totalwgt_lb ~ agepreg + C(race) + babysex==1 + ' 'nbrnaliv>1 + paydu==1 + totincr') results = smf.ols(formula, data=join).fit() results.summary()   Out[18]: OLS Regression Results Dep. Variable: totalwgt_lb R-squared: 0.060 Model: OLS Adj. R-squared: 0.059 Method: Least Squares F-statistic: 79.98 Date: Thu, 28 Feb 2019 Prob (F-statistic): 4.86e-113 Time: 09:59:13 Log-Likelihood: -14295. No. Observations: 8781 AIC: 2.861e+04 Df Residuals: 8773 BIC: 2.866e+04 Df Model: 7 Covariance Type: nonrobust coef std err t P>|t| [0.025 0.975] Intercept 6.6303 0.065 102.223 0.000 6.503 6.757 C(race)[T.2] 0.3570 0.032 11.215 0.000 0.295 0.419 C(race)[T.3] 0.2665 0.051 5.175 0.000 0.166 0.367 babysex == 1[T.True] 0.2952 0.026 11.216 0.000 0.244 0.347 nbrnaliv > 1[T.True] -1.3783 0.108 -12.771 0.000 -1.590 -1.167 paydu == 1[T.True] 0.1196 0.031 3.861 0.000 0.059 0.180 agepreg 0.0074 0.003 2.921 0.004 0.002 0.012 totincr 0.0122 0.004 3.110 0.002 0.005 0.020 Omnibus: 398.813 Durbin-Watson: 1.604 Prob(Omnibus): 0.000 Jarque-Bera (JB): 1388.362 Skew: -0.037 Prob(JB): 3.32e-302 Kurtosis: 4.947 Cond. No. 221. Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.  ## Logistic regression Example: suppose we are trying to predict y using explanatory variables x1 and x2.  In [19]: y = np.array([0, 1, 0, 1]) x1 = np.array([0, 0, 0, 1]) x2 = np.array([0, 1, 1, 1])  According to the logit model the log odds for the$i$th element of$y$is$\log o = \beta_0 + \beta_1 x_1 + \beta_2 x_2 $So let's start with an arbitrary guess about the elements of$\beta$:  In [20]: beta = [-1.5, 2.8, 1.1]  Plugging in the model, we get log odds.  In [21]: log_o = beta[0] + beta[1] * x1 + beta[2] * x2 log_o   Out[21]: array([-1.5, -0.4, -0.4, 2.4])  Which we can convert to odds.  In [22]: o = np.exp(log_o) o   Out[22]: array([ 0.22313016, 0.67032005, 0.67032005, 11.02317638])  And then convert to probabilities.  In [23]: p = o / (o+1) p   Out[23]: array([0.18242552, 0.40131234, 0.40131234, 0.9168273 ])  The likelihoods of the actual outcomes are$p$where$y$is 1 and$1-p$where$y$is 0.  In [24]: likes = np.where(y, p, 1-p) likes   Out[24]: array([0.81757448, 0.40131234, 0.59868766, 0.9168273 ])  The likelihood of$y$given$\beta$is the product of likes:  In [25]: like = np.prod(likes) like   Out[25]: 0.1800933529673034  Logistic regression works by searching for the values in$\beta$that maximize like. Here's an example using variables in the NSFG respondent file to predict whether a baby will be a boy or a girl.  In [26]: import first live, firsts, others = first.MakeFrames() live = live[live.prglngth>30] live['boy'] = (live.babysex==1).astype(int)  The mother's age seems to have a small effect.  In [27]: model = smf.logit('boy ~ agepreg', data=live) results = model.fit() results.summary()   Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Out[27]: Logit Regression Results Dep. Variable: boy No. Observations: 8884 Model: Logit Df Residuals: 8882 Method: MLE Df Model: 1 Date: Thu, 28 Feb 2019 Pseudo R-squ.: 6.144e-06 Time: 09:59:16 Log-Likelihood: -6156.7 converged: True LL-Null: -6156.8 LLR p-value: 0.7833 coef std err z P>|z| [0.025 0.975] Intercept 0.0058 0.098 0.059 0.953 -0.185 0.197 agepreg 0.0010 0.004 0.275 0.783 -0.006 0.009  Here are the variables that seemed most promising.  In [28]: formula = 'boy ~ agepreg + hpagelb + birthord + C(race)' model = smf.logit(formula, data=live) results = model.fit() results.summary()   Optimization terminated successfully. Current function value: 0.692944 Iterations 3 Out[28]: Logit Regression Results Dep. Variable: boy No. Observations: 8782 Model: Logit Df Residuals: 8776 Method: MLE Df Model: 5 Date: Thu, 28 Feb 2019 Pseudo R-squ.: 0.0001440 Time: 09:59:16 Log-Likelihood: -6085.4 converged: True LL-Null: -6086.3 LLR p-value: 0.8822 coef std err z P>|z| [0.025 0.975] Intercept -0.0301 0.104 -0.290 0.772 -0.234 0.173 C(race)[T.2] -0.0224 0.051 -0.439 0.660 -0.122 0.077 C(race)[T.3] -0.0005 0.083 -0.005 0.996 -0.163 0.162 agepreg -0.0027 0.006 -0.484 0.629 -0.014 0.008 hpagelb 0.0047 0.004 1.112 0.266 -0.004 0.013 birthord 0.0050 0.022 0.227 0.821 -0.038 0.048  To make a prediction, we have to extract the exogenous and endogenous variables.  In [29]: endog = pd.DataFrame(model.endog, columns=[model.endog_names]) exog = pd.DataFrame(model.exog, columns=model.exog_names)  The baseline prediction strategy is to guess "boy". In that case, we're right almost 51% of the time.  In [30]: actual = endog['boy'] baseline = actual.mean() baseline   Out[30]: 0.507173764518333  If we use the previous model, we can compute the number of predictions we get right.  In [31]: predict = (results.predict() >= 0.5) true_pos = predict * actual true_neg = (1 - predict) * (1 - actual) sum(true_pos), sum(true_neg)   Out[31]: (3944.0, 548.0)  And the accuracy, which is slightly higher than the baseline.  In [32]: acc = (sum(true_pos) + sum(true_neg)) / len(actual) acc   Out[32]: 0.5115007970849464  To make a prediction for an individual, we have to get their information into a DataFrame.  In [33]: columns = ['agepreg', 'hpagelb', 'birthord', 'race'] new = pd.DataFrame([[35, 39, 3, 2]], columns=columns) y = results.predict(new) y   Out[33]: 0 0.513091 dtype: float64  This person has a 51% chance of having a boy (according to the model). ## Exercises Exercise: Suppose one of your co-workers is expecting a baby and you are participating in an office pool to predict the date of birth. Assuming that bets are placed during the 30th week of pregnancy, what variables could you use to make the best prediction? You should limit yourself to variables that are known before the birth, and likely to be available to the people in the pool.  In [34]: import first live, firsts, others = first.MakeFrames() live = live[live.prglngth>30]  The following are the only variables I found that have a statistically significant effect on pregnancy length.  In [35]: import statsmodels.formula.api as smf model = smf.ols('prglngth ~ birthord==1 + race==2 + nbrnaliv>1', data=live) results = model.fit() results.summary()   Out[35]: OLS Regression Results Dep. Variable: prglngth R-squared: 0.011 Model: OLS Adj. R-squared: 0.011 Method: Least Squares F-statistic: 34.28 Date: Thu, 28 Feb 2019 Prob (F-statistic): 5.09e-22 Time: 09:59:18 Log-Likelihood: -18247. No. Observations: 8884 AIC: 3.650e+04 Df Residuals: 8880 BIC: 3.653e+04 Df Model: 3 Covariance Type: nonrobust coef std err t P>|t| [0.025 0.975] Intercept 38.7617 0.039 1006.410 0.000 38.686 38.837 birthord == 1[T.True] 0.1015 0.040 2.528 0.011 0.023 0.180 race == 2[T.True] 0.1390 0.042 3.311 0.001 0.057 0.221 nbrnaliv > 1[T.True] -1.4944 0.164 -9.086 0.000 -1.817 -1.172 Omnibus: 1587.470 Durbin-Watson: 1.619 Prob(Omnibus): 0.000 Jarque-Bera (JB): 6160.751 Skew: -0.852 Prob(JB): 0.00 Kurtosis: 6.707 Cond. No. 10.9 Warnings:[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.  Exercise: The Trivers-Willard hypothesis suggests that for many mammals the sex ratio depends on “maternal condition”; that is, factors like the mother’s age, size, health, and social status. See https://en.wikipedia.org/wiki/Trivers-Willard_hypothesis Some studies have shown this effect among humans, but results are mixed. In this chapter we tested some variables related to these factors, but didn’t find any with a statistically significant effect on sex ratio. As an exercise, use a data mining approach to test the other variables in the pregnancy and respondent files. Can you find any factors with a substantial effect?  In [36]: # Solution def GoMining(df): """Searches for variables that predict birth weight. df: DataFrame of pregnancy records returns: list of (rsquared, variable name) pairs """ df['boy'] = (df.babysex==1).astype(int) variables = [] for name in df.columns: try: if df[name].var() < 1e-7: continue formula='boy ~ agepreg + ' + name model = smf.logit(formula, data=df) nobs = len(model.endog) if nobs < len(df)/2: continue results = model.fit() except: continue variables.append((results.prsquared, name)) return variables variables = GoMining(join)   Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692961 Iterations 3 Optimization terminated successfully. Current function value: 0.692849 Iterations 3 Optimization terminated successfully. Current function value: 0.692996 Iterations 3 Optimization terminated successfully. Current function value: 0.692903 Iterations 4 Optimization terminated successfully. Current function value: 0.692724 Iterations 6 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 5 Optimization terminated successfully. Current function value: 0.692577 Iterations 6 Optimization terminated successfully. Current function value: 0.686599 Iterations 4 Optimization terminated successfully. Current function value: 0.693022 Iterations 3 Optimization terminated successfully. Current function value: 0.692903 Iterations 4 Optimization terminated successfully. Current function value: 0.693031 Iterations 3 Optimization terminated successfully. Current function value: 0.692963 Iterations 3 Optimization terminated successfully. Current function value: 0.692964 Iterations 3 Optimization terminated successfully. Current function value: 0.692905 Iterations 3 Optimization terminated successfully. Current function value: 0.693092 Iterations 3 Optimization terminated successfully. Current function value: 0.693107 Iterations 3 Optimization terminated successfully. Current function value: 0.692589 Iterations 6 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692774 Iterations 4 Optimization terminated successfully. Current function value: 0.692843 Iterations 5 Optimization terminated successfully. Current function value: 0.692612 Iterations 3 Optimization terminated successfully. Current function value: 0.692947 Iterations 3 Optimization terminated successfully. Current function value: 0.692817 Iterations 3 Optimization terminated successfully. Current function value: 0.692947 Iterations 3 Optimization terminated successfully. Current function value: 0.693023 Iterations 3 Optimization terminated successfully. Current function value: 0.692967 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.692645 Iterations 3 Optimization terminated successfully. Current function value: 0.692555 Iterations 3 Optimization terminated successfully. Current function value: 0.693021 Iterations 3 Optimization terminated successfully. Current function value: 0.693066 Iterations 3 Optimization terminated successfully. Current function value: 0.692911 Iterations 3 Optimization terminated successfully. Current function value: 0.692917 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692861 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692859 Iterations 3 Optimization terminated successfully. Current function value: 0.692814 Iterations 3 Optimization terminated successfully. Current function value: 0.692389 Iterations 3 Optimization terminated successfully. Current function value: 0.692517 Iterations 3 Optimization terminated successfully. Current function value: 0.692452 Iterations 3 Optimization terminated successfully. Current function value: 0.692701 Iterations 3 Optimization terminated successfully. Current function value: 0.692290 Iterations 4 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692926 Iterations 3 Optimization terminated successfully. Current function value: 0.693004 Iterations 3 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.692861 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692806 Iterations 3 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.692987 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.692871 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692942 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692922 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.692573 Iterations 3 Optimization terminated successfully. Current function value: 0.692988 Iterations 3 Optimization terminated successfully. Current function value: 0.692817 Iterations 5 Optimization terminated successfully. Current function value: 0.692588 Iterations 5 Optimization terminated successfully. Current function value: 0.692877 Iterations 5 Optimization terminated successfully. Current function value: 0.692652 Iterations 5 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692918 Iterations 4 Optimization terminated successfully. Current function value: 0.692984 Iterations 3 Optimization terminated successfully. Current function value: 0.692952 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692893 Iterations 5 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692776 Iterations: 35 Optimization terminated successfully. Current function value: 0.692638 Iterations 6 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.692838 Iterations 5 Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 4 Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 4 Optimization terminated successfully. Current function value: 0.692810 Iterations 4 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.692979 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.686305 Iterations 4 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692959 Iterations 3 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692984 Iterations 3 Optimization terminated successfully. Current function value: 0.692928 Iterations 5 Optimization terminated successfully. Current function value: 0.692910 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692881 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692861 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.692806 Iterations 3 Optimization terminated successfully. Current function value: 0.692808 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.693004 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693039 Iterations 4 Optimization terminated successfully. Current function value: 0.692886 Iterations 3 Optimization terminated successfully. Current function value: 0.692957 Iterations 3 Optimization terminated successfully. Current function value: 0.692888 Iterations 4 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692921 Iterations 3 Optimization terminated successfully. Current function value: 0.692996 Iterations 3 Optimization terminated successfully. Current function value: 0.692907 Iterations 3 Optimization terminated successfully. Current function value: 0.692439 Iterations 3 Optimization terminated successfully. Current function value: 0.692916 Iterations 4 Optimization terminated successfully. Current function value: 0.692951 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692950 Iterations 3 Optimization terminated successfully. Current function value: 0.693037 Iterations 3 Optimization terminated successfully. Current function value: 0.693021 Iterations 3 Optimization terminated successfully. Current function value: 0.693018 Iterations 3 Optimization terminated successfully. Current function value: 0.692850 Iterations 3 Optimization terminated successfully. Current function value: 0.693056 Iterations 3 Optimization terminated successfully. Current function value: 0.692885 Iterations 4 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692963 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692959 Iterations 4 Optimization terminated successfully. Current function value: 0.692319 Iterations 6 Optimization terminated successfully. Current function value: 0.692984 Iterations 5 Optimization terminated successfully. Current function value: 0.692841 Iterations 4 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692879 Iterations 4 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692871 Iterations 3 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.692997 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692956 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692780 Iterations 3 Optimization terminated successfully. Current function value: 0.693055 Iterations 4 Optimization terminated successfully. Current function value: 0.693104 Iterations 4 Optimization terminated successfully. Current function value: 0.693022 Iterations 3 Optimization terminated successfully. Current function value: 0.693069 Iterations 3 Optimization terminated successfully. Current function value: 0.693052 Iterations 3 Optimization terminated successfully. Current function value: 0.692988 Iterations 3 Optimization terminated successfully. Current function value: 0.693037 Iterations 3 Optimization terminated successfully. Current function value: 0.693063 Iterations 3 Optimization terminated successfully. Current function value: 0.693052 Iterations 3 Optimization terminated successfully. Current function value: 0.693078 Iterations 3 Optimization terminated successfully. Current function value: 0.693078 Iterations 3 Optimization terminated successfully. Current function value: 0.692964 Iterations 4 Optimization terminated successfully. Current function value: 0.692801 Iterations 4 Optimization terminated successfully. Current function value: 0.693074 Iterations 3 Optimization terminated successfully. Current function value: 0.692959 Iterations 4 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693004 Iterations 3 Optimization terminated successfully. Current function value: 0.692911 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692833 Iterations 3 Optimization terminated successfully. Current function value: 0.693025 Iterations 3 Optimization terminated successfully. Current function value: 0.692823 Iterations 3 Optimization terminated successfully. Current function value: 0.692970 Iterations 4 Optimization terminated successfully. Current function value: 0.692945 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693055 Iterations 3 Optimization terminated successfully. Current function value: 0.692908 Iterations 4 Optimization terminated successfully. Current function value: 0.692703 Iterations 4 Optimization terminated successfully. Current function value: 0.692853 Iterations 5 Optimization terminated successfully. Current function value: 0.692981 Iterations 3 Optimization terminated successfully. Current function value: 0.693053 Iterations 3 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692962 Iterations 3 Optimization terminated successfully. Current function value: 0.692928 Iterations 4 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.692943 Iterations 3 Optimization terminated successfully. Current function value: 0.693049 Iterations 4 Optimization terminated successfully. Current function value: 0.692901 Iterations 3 Optimization terminated successfully. Current function value: 0.692951 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692843 Iterations 4 Optimization terminated successfully. Current function value: 0.692834 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692783 Iterations 3 Optimization terminated successfully. Current function value: 0.692799 Iterations 3 Optimization terminated successfully. Current function value: 0.692850 Iterations 3 Optimization terminated successfully. Current function value: 0.692841 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.693118 Iterations 3 Optimization terminated successfully. Current function value: 0.693128 Iterations 3 Optimization terminated successfully. Current function value: 0.692958 Iterations 3 Optimization terminated successfully. Current function value: 0.692898 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.692829 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.692972 Iterations 3 Optimization terminated successfully. Current function value: 0.692901 Iterations 4 Optimization terminated successfully. Current function value: 0.692929 Iterations 3 Optimization terminated successfully. Current function value: 0.692894 Iterations 3 Optimization terminated successfully. Current function value: 0.692820 Iterations 4 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692960 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.692968 Iterations 4 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.693028 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692733 Iterations 4 Optimization terminated successfully. Current function value: 0.692645 Iterations 3 Optimization terminated successfully. Current function value: 0.693029 Iterations 3 Optimization terminated successfully. Current function value: 0.692842 Iterations 3 Optimization terminated successfully. Current function value: 0.692821 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692721 Iterations 4 Optimization terminated successfully. Current function value: 0.693054 Iterations 3 Optimization terminated successfully. Current function value: 0.692742 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692958 Iterations 3 Optimization terminated successfully. Current function value: 0.692975 Iterations 3 Optimization terminated successfully. Current function value: 0.692848 Iterations 3 Optimization terminated successfully. Current function value: 0.692942 Iterations 3 Optimization terminated successfully. Current function value: 0.692953 Iterations 3 Optimization terminated successfully. Current function value: 0.692915 Iterations 3 Optimization terminated successfully. Current function value: 0.692931 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692940 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693000 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.692979 Iterations 3 Optimization terminated successfully. Current function value: 0.693000 Iterations 3 Optimization terminated successfully. Current function value: 0.692984 Iterations 3 Optimization terminated successfully. Current function value: 0.692978 Iterations 3 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.692829 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693000 Iterations 3 Optimization terminated successfully. Current function value: 0.693000 Iterations 3 Optimization terminated successfully. Current function value: 0.692979 Iterations 3 Optimization terminated successfully. Current function value: 0.692996 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692990 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692950 Iterations 3 Optimization terminated successfully. Current function value: 0.693037 Iterations 3 Optimization terminated successfully. Current function value: 0.693039 Iterations 3 Optimization terminated successfully. Current function value: 0.693091 Iterations 3 Optimization terminated successfully. Current function value: 0.693065 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692987 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692840 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692984 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693023 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693016 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692970 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692990 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.692965 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692818 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692680 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692674 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692884 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692865 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692883 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692941 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692948 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692917 Iterations 3 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.692968 Iterations 3 Optimization terminated successfully. Current function value: 0.692850 Iterations 3 Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.692844 Iterations 3 Optimization terminated successfully. Current function value: 0.692963 Iterations 4 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692984 Iterations 3 Optimization terminated successfully. Current function value: 0.692937 Iterations 3 Optimization terminated successfully. Current function value: 0.692960 Iterations 4 Optimization terminated successfully. Current function value: 0.692970 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692911 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.692929 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.692845 Iterations 3 Optimization terminated successfully. Current function value: 0.692981 Iterations 3 Optimization terminated successfully. Current function value: 0.692718 Iterations 4 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692575 Iterations 4 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.692673 Iterations 4 Optimization terminated successfully. Current function value: 0.692988 Iterations 3 Optimization terminated successfully. Current function value: 0.692778 Iterations 3 Optimization terminated successfully. Current function value: 0.693018 Iterations 3 Optimization terminated successfully. Current function value: 0.692804 Iterations 3 Optimization terminated successfully. Current function value: 0.693023 Iterations 3 Optimization terminated successfully. Current function value: 0.692752 Iterations 4 Optimization terminated successfully. Current function value: 0.693023 Iterations 3 Optimization terminated successfully. Current function value: 0.692671 Iterations 4 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.692709 Iterations 4 Optimization terminated successfully. Current function value: 0.693046 Iterations 3 Optimization terminated successfully. Current function value: 0.692650 Iterations 4 Optimization terminated successfully. Current function value: 0.693063 Iterations 3 Optimization terminated successfully. Current function value: 0.692875 Iterations 3 Optimization terminated successfully. Current function value: 0.693098 Iterations 3 Optimization terminated successfully. Current function value: 0.692806 Iterations 4 Optimization terminated successfully. Current function value: 0.693074 Iterations 3 Optimization terminated successfully. Current function value: 0.693058 Iterations 3 Optimization terminated successfully. Current function value: 0.693024 Iterations 3 Optimization terminated successfully. Current function value: 0.692892 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692959 Iterations 3 Optimization terminated successfully. Current function value: 0.692853 Iterations 3 Optimization terminated successfully. Current function value: 0.692615 Iterations 3 Optimization terminated successfully. Current function value: 0.692979 Iterations 3 Optimization terminated successfully. Current function value: 0.693046 Iterations 3 Optimization terminated successfully. Current function value: 0.692917 Iterations 3 Optimization terminated successfully. Current function value: 0.692977 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693004 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.692932 Iterations 3 Optimization terminated successfully. Current function value: 0.692619 Iterations 4 Optimization terminated successfully. Current function value: 0.692779 Iterations 3 Optimization terminated successfully. Current function value: 0.692886 Iterations 3 Optimization terminated successfully. Current function value: 0.692739 Iterations 3 Optimization terminated successfully. Current function value: 0.692662 Iterations 3 Optimization terminated successfully. Current function value: 0.692621 Iterations 3 Optimization terminated successfully. Current function value: 0.692792 Iterations 3 Optimization terminated successfully. Current function value: 0.692926 Iterations 3 Optimization terminated successfully. Current function value: 0.692879 Iterations 3 Optimization terminated successfully. Current function value: 0.692805 Iterations 4 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.692673 Iterations 4 Optimization terminated successfully. Current function value: 0.692940 Iterations 3 Optimization terminated successfully. Current function value: 0.692903 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.692941 Iterations 4 Optimization terminated successfully. Current function value: 0.692972 Iterations 3 Optimization terminated successfully. Current function value: 0.692952 Iterations 3 Optimization terminated successfully. Current function value: 0.692977 Iterations 3 Optimization terminated successfully. Current function value: 0.692920 Iterations 3 Optimization terminated successfully. Current function value: 0.692856 Iterations 3 Optimization terminated successfully. Current function value: 0.692955 Iterations 4 Optimization terminated successfully. Current function value: 0.692852 Iterations 3 Optimization terminated successfully. Current function value: 0.692573 Iterations 3 Optimization terminated successfully. Current function value: 0.692976 Iterations 3 Optimization terminated successfully. Current function value: 0.692879 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693069 Iterations 3 Optimization terminated successfully. Current function value: 0.692955 Iterations 3 Optimization terminated successfully. Current function value: 0.692868 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.693018 Iterations 3 Optimization terminated successfully. Current function value: 0.693027 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692949 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692945 Iterations 3 Optimization terminated successfully. Current function value: 0.692962 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692971 Iterations 4 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692579 Iterations 4 Optimization terminated successfully. Current function value: 0.692949 Iterations 3 Optimization terminated successfully. Current function value: 0.692801 Iterations 4 Optimization terminated successfully. Current function value: 0.692987 Iterations 4 Optimization terminated successfully. Current function value: 0.693094 Iterations 3 Optimization terminated successfully. Current function value: 0.693036 Iterations 3 Optimization terminated successfully. Current function value: 0.692917 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692824 Iterations 3 Optimization terminated successfully. Current function value: 0.692862 Iterations 3 Optimization terminated successfully. Current function value: 0.692862 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692976 Iterations 3 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.692965 Iterations 3 Optimization terminated successfully. Current function value: 0.692967 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692916 Iterations 4 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.692951 Iterations 3 Optimization terminated successfully. Current function value: 0.692962 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.692952 Iterations 3 Optimization terminated successfully. Current function value: 0.692946 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692974 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692806 Iterations 3 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.692987 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692610 Iterations 4 Optimization terminated successfully. Current function value: 0.692904 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692859 Iterations 5 Optimization terminated successfully. Current function value: 0.692712 Iterations 4 Optimization terminated successfully. Current function value: 0.692948 Iterations 4 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.692871 Iterations 3 Optimization terminated successfully. Current function value: 0.692883 Iterations 3 Optimization terminated successfully. Current function value: 0.692801 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692994 Iterations 3 Optimization terminated successfully. Current function value: 0.692722 Iterations 3 Optimization terminated successfully. Current function value: 0.692909 Iterations 3 Optimization terminated successfully. Current function value: 0.692561 Iterations 3 Optimization terminated successfully. Current function value: 0.692803 Iterations 3 Optimization terminated successfully. Current function value: 0.692870 Iterations 3 Optimization terminated successfully. Current function value: 0.692755 Iterations 3 Optimization terminated successfully. Current function value: 0.692702 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692963 Iterations 3 Optimization terminated successfully. Current function value: 0.692879 Iterations 3 Optimization terminated successfully. Current function value: 0.692855 Iterations 3 Optimization terminated successfully. Current function value: 0.692837 Iterations 3 Optimization terminated successfully. Current function value: 0.692729 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692990 Iterations 3 Optimization terminated successfully. Current function value: 0.692875 Iterations 3 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.692813 Iterations 3 Optimization terminated successfully. Current function value: 0.692475 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.692833 Iterations 3 Optimization terminated successfully. Current function value: 0.692529 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692898 Iterations 3 Optimization terminated successfully. Current function value: 0.692640 Iterations 3 Optimization terminated successfully. Current function value: 0.692784 Iterations 3 Optimization terminated successfully. Current function value: 0.692875 Iterations 3 Optimization terminated successfully. Current function value: 0.692782 Iterations 3 Optimization terminated successfully. Current function value: 0.692938 Iterations 3 Optimization terminated successfully. Current function value: 0.692871 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692939 Iterations 18 Optimization terminated successfully. Current function value: 0.693009 Iterations 4 Optimization terminated successfully. Current function value: 0.693005 Iterations 4 Optimization terminated successfully. Current function value: 0.692976 Iterations 4 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 4 Optimization terminated successfully. Current function value: 0.692958 Iterations 4 Optimization terminated successfully. Current function value: 0.693002 Iterations 4 Optimization terminated successfully. Current function value: 0.692983 Iterations 4 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692955 Iterations 4 Optimization terminated successfully. Current function value: 0.692967 Iterations 4 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.692859 Iterations 4 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692925 Iterations 4 Optimization terminated successfully. Current function value: 0.692969 Iterations 4 Optimization terminated successfully. Current function value: 0.692983 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692894 Iterations 4 Optimization terminated successfully. Current function value: 0.692874 Iterations 4 Optimization terminated successfully. Current function value: 0.692859 Iterations 4 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692924 Iterations 4 Optimization terminated successfully. Current function value: 0.692982 Iterations 4 Optimization terminated successfully. Current function value: 0.692914 Iterations 4 Optimization terminated successfully. Current function value: 0.692969 Iterations 4 Optimization terminated successfully. Current function value: 0.692983 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692695 Iterations 4 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692955 Iterations 4 Optimization terminated successfully. Current function value: 0.692946 Iterations 4 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.692859 Iterations 4 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692925 Iterations 4 Optimization terminated successfully. Current function value: 0.692969 Iterations 4 Optimization terminated successfully. Current function value: 0.692983 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692955 Iterations 4 Optimization terminated successfully. Current function value: 0.692889 Iterations 4 Optimization terminated successfully. Current function value: 0.692823 Iterations 4 Optimization terminated successfully. Current function value: 0.692968 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692859 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692925 Iterations 4 Optimization terminated successfully. Current function value: 0.692969 Iterations 4 Optimization terminated successfully. Current function value: 0.692983 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692989 Iterations 3 Optimization terminated successfully. Current function value: 0.692997 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692923 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692990 Iterations 4 Optimization terminated successfully. Current function value: 0.692986 Iterations 4 Optimization terminated successfully. Current function value: 0.692929 Iterations 5 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692939 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692990 Iterations 4 Optimization terminated successfully. Current function value: 0.692986 Iterations 4 Optimization terminated successfully. Current function value: 0.692929 Iterations 5 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692966 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692941 Iterations 4 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692978 Iterations 4 Optimization terminated successfully. Current function value: 0.692986 Iterations 4 Optimization terminated successfully. Current function value: 0.692929 Iterations 5 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692974 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 3 Optimization terminated successfully. Current function value: 0.692930 Iterations 3 Optimization terminated successfully. Current function value: 0.692929 Iterations 3 Optimization terminated successfully. Current function value: 0.692933 Iterations 3 Optimization terminated successfully. Current function value: 0.693045 Iterations 3 Optimization terminated successfully. Current function value: 0.693078 Iterations 3 Optimization terminated successfully. Current function value: 0.692739 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692997 Iterations 3 Optimization terminated successfully. Current function value: 0.692750 Iterations 3 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692873 Iterations 3 Optimization terminated successfully. Current function value: 0.692830 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693082 Iterations 3 Optimization terminated successfully. Current function value: 0.692636 Iterations 3 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692709 Iterations: 35 Optimization terminated successfully. Current function value: 0.692863 Iterations 3 Optimization terminated successfully. Current function value: 0.692926 Iterations 3 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.692726 Iterations 3 Optimization terminated successfully. Current function value: 0.692774 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 4 Optimization terminated successfully. Current function value: 0.692861 Iterations 3 Optimization terminated successfully. Current function value: 0.692705 Iterations 4 Optimization terminated successfully. Current function value: 0.692723 Iterations 4 Optimization terminated successfully. Current function value: 0.692803 Iterations 4 Optimization terminated successfully. Current function value: 0.692956 Iterations 3 Optimization terminated successfully. Current function value: 0.692786 Iterations 3 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692989 Iterations 4 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692952 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692970 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692862 Iterations: 35 Optimization terminated successfully. Current function value: 0.692861 Iterations 4 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692658 Iterations 4 Optimization terminated successfully. Current function value: 0.692789 Iterations 4 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692855 Iterations 4 Optimization terminated successfully. Current function value: 0.692855 Iterations 4 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692749 Iterations 4 Optimization terminated successfully. Current function value: 0.692862 Iterations 4 Optimization terminated successfully. Current function value: 0.692862 Iterations 3 Optimization terminated successfully. Current function value: 0.692821 Iterations 3 Optimization terminated successfully. Current function value: 0.692772 Iterations 3 Optimization terminated successfully. Current function value: 0.692687 Iterations 4 Optimization terminated successfully. Current function value: 0.692947 Iterations 4 Optimization terminated successfully. Current function value: 0.692771 Iterations 4 Optimization terminated successfully. Current function value: 0.692755 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693006 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Optimization terminated successfully. Current function value: 0.692949 Iterations 3 Optimization terminated successfully. Current function value: 0.692912 Iterations 3 Optimization terminated successfully. Current function value: 0.692879 Iterations 3 Optimization terminated successfully. Current function value: 0.692950 Iterations 3 Optimization terminated successfully. Current function value: 0.692991 Iterations 3 Optimization terminated successfully. Current function value: 0.692929 Iterations 4 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692960 Iterations: 35 Optimization terminated successfully. Current function value: 0.693007 Iterations 3 Optimization terminated successfully. Current function value: 0.692981 Iterations 3 Optimization terminated successfully. Current function value: 0.692975 Iterations 3 Optimization terminated successfully. Current function value: 0.692961 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.692957 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692983 Iterations 3 Optimization terminated successfully. Current function value: 0.692258 Iterations 5 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692696 Iterations: 35 Optimization terminated successfully. Current function value: 0.692993 Iterations 3 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692853 Iterations 3 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.692971 Iterations 3 Optimization terminated successfully. Current function value: 0.692639 Iterations 3 Optimization terminated successfully. Current function value: 0.692917 Iterations 3 Optimization terminated successfully. Current function value: 0.692760 Iterations 3 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692832 Iterations 3 Optimization terminated successfully. Current function value: 0.693028 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692888 Iterations 3 Optimization terminated successfully. Current function value: 0.692770 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.692976 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.692992 Iterations 3 Optimization terminated successfully. Current function value: 0.692976 Iterations 3 Optimization terminated successfully. Current function value: 0.692866 Iterations 3 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.692896 Iterations 3 Optimization terminated successfully. Current function value: 0.692758 Iterations 3 Optimization terminated successfully. Current function value: 0.693081 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 4 Optimization terminated successfully. Current function value: 0.692927 Iterations 4 Optimization terminated successfully. Current function value: 0.692827 Iterations 4 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692890 Iterations 4 Optimization terminated successfully. Current function value: 0.692941 Iterations 4 Optimization terminated successfully. Current function value: 0.692920 Iterations 4 Optimization terminated successfully. Current function value: 0.692920 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692982 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693001 Iterations 4 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692997 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.693008 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693005 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692956 Iterations 3 Optimization terminated successfully. Current function value: 0.692931 Iterations 4 Optimization terminated successfully. Current function value: 0.692996 Iterations 4 Optimization terminated successfully. Current function value: 0.692996 Iterations 4 Optimization terminated successfully. Current function value: 0.692730 Iterations 5 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692898 Iterations 4 Optimization terminated successfully. Current function value: 0.692879 Iterations 5 Optimization terminated successfully. Current function value: 0.692887 Iterations 4 Optimization terminated successfully. Current function value: 0.692987 Iterations 4 Optimization terminated successfully. Current function value: 0.692986 Iterations 4 Optimization terminated successfully. Current function value: 0.692995 Iterations 4 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692849 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692974 Iterations 4 Optimization terminated successfully. Current function value: 0.692987 Iterations 4 Optimization terminated successfully. Current function value: 0.692957 Iterations 4 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 4 Optimization terminated successfully. Current function value: 0.693013 Iterations 3 Optimization terminated successfully. Current function value: 0.692980 Iterations 3 Optimization terminated successfully. Current function value: 0.692849 Iterations 4 Optimization terminated successfully. Current function value: 0.692998 Iterations 4 Optimization terminated successfully. Current function value: 0.692974 Iterations 4 Optimization terminated successfully. Current function value: 0.692987 Iterations 4 Optimization terminated successfully. Current function value: 0.692957 Iterations 4 Optimization terminated successfully. Current function value: 0.693009 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 4 Optimization terminated successfully. Current function value: 0.692969 Iterations 4 Optimization terminated successfully. Current function value: 0.693005 Iterations 4 Optimization terminated successfully. Current function value: 0.692946 Iterations 4 Optimization terminated successfully. Current function value: 0.692862 Iterations 4 Optimization terminated successfully. Current function value: 0.692905 Iterations 4 Optimization terminated successfully. Current function value: 0.692962 Iterations 4 Optimization terminated successfully. Current function value: 0.692977 Iterations 4 Optimization terminated successfully. Current function value: 0.693010 Iterations 3 Optimization terminated successfully. Current function value: 0.693000 Iterations 3 Optimization terminated successfully. Current function value: 0.692998 Iterations 3 Optimization terminated successfully. Current function value: 0.693012 Iterations 3 Warning: Maximum number of iterations has been exceeded. Current function value: 0.692939 Iterations: 35 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692831 Iterations 3 Optimization terminated successfully. Current function value: 0.692999 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692997 Iterations 3 Optimization terminated successfully. Current function value: 0.692795 Iterations 4 /home/downey/anaconda3/envs/ThinkStats2/lib/python3.7/site-packages/statsmodels/base/model.py:508: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals "Check mle_retvals", ConvergenceWarning) Optimization terminated successfully. Current function value: 0.692693 Iterations 4 Optimization terminated successfully. Current function value: 0.692457 Iterations 4 Optimization terminated successfully. Current function value: 0.692815 Iterations 4 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.692989 Iterations 4 Optimization terminated successfully. Current function value: 0.693008 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 4 Optimization terminated successfully. Current function value: 0.692673 Iterations 5 Optimization terminated successfully. Current function value: 0.692982 Iterations 3 Optimization terminated successfully. Current function value: 0.692985 Iterations 3 Optimization terminated successfully. Current function value: 0.692986 Iterations 3 Optimization terminated successfully. Current function value: 0.692922 Iterations 3 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.692973 Iterations 4 Optimization terminated successfully. Current function value: 0.693011 Iterations 3 Optimization terminated successfully. Current function value: 0.693015 Iterations 3 Optimization terminated successfully. Current function value: 0.692942 Iterations 3 Optimization terminated successfully. Current function value: 0.692921 Iterations 3 Optimization terminated successfully. Current function value: 0.693002 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.692810 Iterations 4 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692995 Iterations 3 Optimization terminated successfully. Current function value: 0.692979 Iterations 3 Optimization terminated successfully. Current function value: 0.693014 Iterations 3 Optimization terminated successfully. Current function value: 0.693003 Iterations 3 Optimization terminated successfully. Current function value: 0.692996 Iterations 4 Optimization terminated successfully. Current function value: 0.692996 Iterations 4 Optimization terminated successfully. Current function value: 0.692971 Iterations 3   In [37]: # Solution #Here are the 30 variables that yield the highest pseudo-R^2 values. MiningReport(variables)   totalwgt_lb 0.009696855253233383 birthwgt_lb 0.009274460080281988 BD-3 BIRTHWEIGHT IN POUNDS - 1ST BABY FROM THIS PREGNANCY constat3 0.0010985419170438382 3RD PRIORITY CODE FOR CURRENT CONTRACEPTIVE STATUS lbw1 0.0010519527860076705 LOW BIRTHWEIGHT - BABY 1 nplaced 0.001010368752280555 # OF R'S BIO CHILDREN SHE PLACED FOR ADOPTION (BASED ON BPA) fmarout5 0.0009096579032891183 FORMAL MARITAL STATUS AT PREGNANCY OUTCOME rmarout6 0.000818252143711895 INFORMAL MARITAL STATUS AT PREGNANCY OUTCOME - 6 CATEGORIES infever 0.0008115919859909004 EVER USED INFERTILITY SERVICES OF ANY KIND frsteatd 0.0007675331422082321 AGE (IN MOS) WHEN 1ST SUPPLEMENTED - 1ST FROM THIS PREG splstwk1 0.0007334122339932581 IF-1 H/P DOING WHAT LAST WEEK (EMPLOYMENT STATUS) 1ST MENTION pmarpreg 0.0007245809157658822 WHETHER PREGNANCY ENDED BEFORE R'S 1ST MARRIAGE (PREMARITALLY) usefstp 0.0007122387685902787 EF-3 USE METHOD AT FIRST SEX WITH 1ST PARTNER IN PAST 12 MONTHS? outcom02 0.0007015744602576479 OUTCOME OF PREGNANCY - 2ND nummult34 0.0006606172426639745 NUMBER OF METHODS REPORTED IN (OCT 2001) coh1dur 0.0006550110146368304 DURATION (IN MONTHS) OF R'S FIRST COHABITATION brnout_r 0.0006438582787924307 brnout 0.0006438582787924307 bpa_bdscheck1 0.0006384992730465999 WHETHER 1ST LIVEBORN BABY FROM THIS PREGNANCY WAS BPA OR BDS nummult41 0.000624852395994635 NUMBER OF METHODS REPORTED IN (MAY 2002) agepreg_i 0.0006221097878729154 AGEPREG IMPUTATION FLAG educmom 0.0005903653895802385 MOTHER'S (OR MOTHER-FIGURE'S) EDUCATION marout03 0.0005883792792801268 FORMAL MARITAL STATUS WHEN PREGNANCY ENDED - 3RD abort12 0.0005779076671296179 FA-3B RECEIVED ABORTION LAST 12 MONTHS p1yhsage 0.0005631662500977797 CI-6 PARTNER'S AGE AT 1ST SEX-1ST REPORTED PARTNER IN LAST 12 MOS numfirsm1 0.0005601782001567468 TOTAL NUMBER OF RESPONSES IN EB-1 FIRSMETH - PRESCRIPTION METHODS lbw1_i 0.0005499451546340239 LBW1 IMPUTATION FLAG agecon_i 0.0005303754891523571 AGECON IMPUTATION FLAG mar1con1_i 0.0005217496825273837 MAR1CON1 IMPUTATION FLAG nummult35 0.0005188253090530059 NUMBER OF METHODS REPORTED IN (NOV 2001) nummult39 0.0005146121683266003 NUMBER OF METHODS REPORTED IN (MAR 2002)   In [38]: # Solution # Eliminating variables that are not known during pregnancy and # others that are fishy for various reasons, here's the best model I could find: formula='boy ~ agepreg + fmarout5==5 + infever==1' model = smf.logit(formula, data=join) results = model.fit() results.summary()   Optimization terminated successfully. Current function value: 0.691874 Iterations 4 Out[38]: Logit Regression Results Dep. Variable: boy No. Observations: 8884 Model: Logit Df Residuals: 8880 Method: MLE Df Model: 3 Date: Thu, 28 Feb 2019 Pseudo R-squ.: 0.001653 Time: 09:59:57 Log-Likelihood: -6146.6 converged: True LL-Null: -6156.8 LLR p-value: 0.0001432 coef std err z P>|z| [0.025 0.975] Intercept -0.1805 0.118 -1.534 0.125 -0.411 0.050 fmarout5 == 5[T.True] 0.1582 0.049 3.217 0.001 0.062 0.255 infever == 1[T.True] 0.2194 0.065 3.374 0.001 0.092 0.347 agepreg 0.0050 0.004 1.172 0.241 -0.003 0.013  Exercise: If the quantity you want to predict is a count, you can use Poisson regression, which is implemented in StatsModels with a function called poisson. It works the same way as ols and logit. As an exercise, let’s use it to predict how many children a woman has born; in the NSFG dataset, this variable is called numbabes. Suppose you meet a woman who is 35 years old, black, and a college graduate whose annual household income exceeds$75,000. How many children would you predict she has born?



In [39]:

# Solution

# I used a nonlinear model of age.

join.numbabes.replace([97], np.nan, inplace=True)
join['age2'] = join.age_r**2




In [40]:

# Solution

formula='numbabes ~ age_r + age2 + age3 + C(race) + totincr + educat'
formula='numbabes ~ age_r + age2 + C(race) + totincr + educat'
model = smf.poisson(formula, data=join)
results = model.fit()
results.summary()




Optimization terminated successfully.
Current function value: 1.677002
Iterations 7

Out[40]:

Poisson Regression Results

Dep. Variable:     numbabes       No. Observations:       8884

Model:              Poisson       Df Residuals:           8877

Method:               MLE         Df Model:                  6

Date:          Thu, 28 Feb 2019   Pseudo R-squ.:        0.03686

Time:              09:59:57       Log-Likelihood:       -14898.

converged:           True         LL-Null:              -15469.

LLR p-value:        3.681e-243

coef     std err      z      P>|z|  [0.025    0.975]

Intercept       -1.0324     0.169    -6.098  0.000    -1.364    -0.701

C(race)[T.2]    -0.1401     0.015    -9.479  0.000    -0.169    -0.111

C(race)[T.3]    -0.0991     0.025    -4.029  0.000    -0.147    -0.051

age_r            0.1556     0.010    15.006  0.000     0.135     0.176

age2            -0.0020     0.000   -13.102  0.000    -0.002    -0.002

totincr         -0.0187     0.002    -9.830  0.000    -0.022    -0.015

educat          -0.0471     0.003   -16.076  0.000    -0.053    -0.041



Now we can predict the number of children for a woman who is 35 years old, black, and a college graduate whose annual household income exceeds $75,000  In [41]: # Solution columns = ['age_r', 'age2', 'age3', 'race', 'totincr', 'educat'] new = pd.DataFrame([[35, 35**2, 35**3, 1, 14, 16]], columns=columns) results.predict(new)   Out[41]: 0 2.496802 dtype: float64  Exercise: If the quantity you want to predict is categorical, you can use multinomial logistic regression, which is implemented in StatsModels with a function called mnlogit. As an exercise, let’s use it to guess whether a woman is married, cohabitating, widowed, divorced, separated, or never married; in the NSFG dataset, marital status is encoded in a variable called rmarital. Suppose you meet a woman who is 25 years old, white, and a high school graduate whose annual household income is about$45,000. What is the probability that she is married, cohabitating, etc?



In [42]:

# Solution

# Here's the best model I could find.

formula='rmarital ~ age_r + age2 + C(race) + totincr + educat'
model = smf.mnlogit(formula, data=join)
results = model.fit()
results.summary()




Optimization terminated successfully.
Current function value: 1.084053
Iterations 8

Out[42]:

MNLogit Regression Results

Dep. Variable:     rmarital       No. Observations:      8884

Model:              MNLogit       Df Residuals:          8849

Method:               MLE         Df Model:                30

Date:          Thu, 28 Feb 2019   Pseudo R-squ.:       0.1682

Time:              09:59:58       Log-Likelihood:      -9630.7

converged:           True         LL-Null:             -11579.

LLR p-value:          0.000

rmarital=2     coef     std err      z      P>|z|  [0.025    0.975]

Intercept        9.0156     0.805    11.199  0.000     7.438    10.593

C(race)[T.2]    -0.9237     0.089   -10.418  0.000    -1.097    -0.750

C(race)[T.3]    -0.6179     0.136    -4.536  0.000    -0.885    -0.351

age_r           -0.3635     0.051    -7.150  0.000    -0.463    -0.264

age2             0.0048     0.001     6.103  0.000     0.003     0.006

totincr         -0.1310     0.012   -11.337  0.000    -0.154    -0.108

educat          -0.1953     0.019   -10.424  0.000    -0.232    -0.159

rmarital=3     coef     std err      z      P>|z|  [0.025    0.975]

Intercept        2.9570     3.020     0.979  0.328    -2.963     8.877

C(race)[T.2]    -0.4411     0.237    -1.863  0.062    -0.905     0.023

C(race)[T.3]     0.0591     0.336     0.176  0.860    -0.600     0.718

age_r           -0.3177     0.177    -1.798  0.072    -0.664     0.029

age2             0.0064     0.003     2.528  0.011     0.001     0.011

totincr         -0.3258     0.032   -10.175  0.000    -0.389    -0.263

educat          -0.0991     0.048    -2.050  0.040    -0.194    -0.004

rmarital=4     coef     std err      z      P>|z|  [0.025    0.975]

Intercept       -3.5238     1.205    -2.924  0.003    -5.886    -1.162

C(race)[T.2]    -0.3213     0.093    -3.445  0.001    -0.504    -0.139

C(race)[T.3]    -0.7706     0.171    -4.509  0.000    -1.106    -0.436

age_r            0.1155     0.071     1.626  0.104    -0.024     0.255

age2            -0.0007     0.001    -0.701  0.483    -0.003     0.001

totincr         -0.2276     0.012   -19.621  0.000    -0.250    -0.205

educat           0.0667     0.017     3.995  0.000     0.034     0.099

rmarital=5     coef     std err      z      P>|z|  [0.025    0.975]

Intercept       -2.8963     1.305    -2.220  0.026    -5.453    -0.339

C(race)[T.2]    -1.0407     0.104   -10.038  0.000    -1.244    -0.837

C(race)[T.3]    -0.5661     0.156    -3.635  0.000    -0.871    -0.261

age_r            0.2411     0.079     3.038  0.002     0.086     0.397

age2            -0.0035     0.001    -2.977  0.003    -0.006    -0.001

totincr         -0.2932     0.015   -20.159  0.000    -0.322    -0.265

educat          -0.0174     0.021    -0.813  0.416    -0.059     0.025

rmarital=6     coef     std err      z      P>|z|  [0.025    0.975]

Intercept        8.0533     0.814     9.890  0.000     6.457     9.649

C(race)[T.2]    -2.1871     0.080   -27.211  0.000    -2.345    -2.030

C(race)[T.3]    -1.9611     0.138   -14.188  0.000    -2.232    -1.690

age_r           -0.2127     0.052    -4.122  0.000    -0.314    -0.112

age2             0.0019     0.001     2.321  0.020     0.000     0.003

totincr         -0.2945     0.012   -25.320  0.000    -0.317    -0.272

educat          -0.0742     0.018    -4.169  0.000    -0.109    -0.039



Make a prediction for a woman who is 25 years old, white, and a high school graduate whose annual household income is about \$45,000.



In [43]:

# Solution

# This person has a 75% chance of being currently married,
# a 13% chance of being "not married but living with opposite
# sex partner", etc.

columns = ['age_r', 'age2', 'race', 'totincr', 'educat']
new = pd.DataFrame([[25, 25**2, 2, 11, 12]], columns=columns)
results.predict(new)




Out[43]:

.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}

.dataframe tbody tr th {
vertical-align: top;
}

text-align: right;
}

0
1
2
3
4
5

0
0.750028
0.126397
0.001564
0.033403
0.021485
0.067122




In [ ]: