In [1]:

    

from theano.sandbox import cuda
cuda.use('gpu2')


    

    




Using gpu device 2: GeForce GTX TITAN X (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)

In [2]:

    

%matplotlib inline
import utils; reload(utils)
from utils import *
from __future__ import division, print_function


    

    




Using Theano backend.

In [144]:

    

from keras.layers import TimeDistributed, Activation
from numpy.random import choice


    

In [107]:

    

path = get_file('nietzsche.txt', origin="https://s3.amazonaws.com/text-datasets/nietzsche.txt")
text = open(path).read().lower()
print('corpus length:', len(text))


    

    




corpus length: 600901

In [272]:

    

!tail {path} -n25


    

    




are thinkers who believe in the saints.


144

It stands to reason that this sketch of the saint, made upon the model
of the whole species, can be confronted with many opposing sketches that
would create a more agreeable impression. There are certain exceptions
among the species who distinguish themselves either by especial
gentleness or especial humanity, and perhaps by the strength of their
own personality. Others are in the highest degree fascinating because
certain of their delusions shed a particular glow over their whole
being, as is the case with the founder of christianity who took himself
for the only begotten son of God and hence felt himself sinless; so that
through his imagination--that should not be too harshly judged since the
whole of antiquity swarmed with sons of god--he attained the same goal,
the sense of complete sinlessness, complete irresponsibility, that can
now be attained by every individual through science.--In the same manner
I have viewed the saints of India who occupy an intermediate station
between the christian saints and the Greek philosophers and hence are
not to be regarded as a pure type. Knowledge and science--as far as they
existed--and superiority to the rest of mankind by logical discipline
and training of the intellectual powers were insisted upon by the
Buddhists as essential to sanctity, just as they were denounced by the
christian world as the indications of sinfulness.

In [101]:

    

#path = 'data/wiki/'
#text = open(path+'small.txt').read().lower()
#print('corpus length:', len(text))

#text = text[0:1000000]


    

    




corpus length: 137587200

In [124]:

    

chars = sorted(list(set(text)))
vocab_size = len(chars)+1
print('total chars:', vocab_size)


    

    




total chars: 60

In [134]:

    

chars.insert(0, "\0")


    

In [270]:

    

''.join(chars[1:-6])


    

    
Out[270]:





'\n !"\'(),-.0123456789:;=?[]_abcdefghijklmnopqrstuvwxyz'

In [135]:

    

char_indices = dict((c, i) for i, c in enumerate(chars))
indices_char = dict((i, c) for i, c in enumerate(chars))


    

In [136]:

    

idx = [char_indices[c] for c in text]


    

In [276]:

    

idx[:10]


    

    
Out[276]:





[43, 45, 32, 33, 28, 30, 32, 1, 1, 1]

In [274]:

    

''.join(indices_char[i] for i in idx[:70])


    

    
Out[274]:





'preface\n\n\nsupposing that truth is a woman--what then? is there not gro'

In [139]:

    

maxlen = 40
sentences = []
next_chars = []
for i in range(0, len(idx) - maxlen+1):
    sentences.append(idx[i: i + maxlen])
    next_chars.append(idx[i+1: i+maxlen+1])
print('nb sequences:', len(sentences))


    

    




0
nb sequences: 600862

In [ ]:

    

sentences = np.concatenate([[np.array(o)] for o in sentences[:-2]])
next_chars = np.concatenate([[np.array(o)] for o in next_chars[:-2]])


    

In [277]:

    

sentences.shape, next_chars.shape


    

    
Out[277]:





((600860, 40), (600860, 40))

In [213]:

    

n_fac = 24


    

In [232]:

    

model=Sequential([
        Embedding(vocab_size, n_fac, input_length=maxlen),
        LSTM(512, input_dim=n_fac,return_sequences=True, dropout_U=0.2, dropout_W=0.2,
             consume_less='gpu'),
        Dropout(0.2),
        LSTM(512, return_sequences=True, dropout_U=0.2, dropout_W=0.2,
             consume_less='gpu'),
        Dropout(0.2),
        TimeDistributed(Dense(vocab_size)),
        Activation('softmax')
    ])


    

In [233]:

    

model.compile(loss='sparse_categorical_crossentropy', optimizer=Adam())


    

In [219]:

    

def print_example():
    seed_string="ethics is a basic foundation of all that"
    for i in range(320):
        x=np.array([char_indices[c] for c in seed_string[-40:]])[np.newaxis,:]
        preds = model.predict(x, verbose=0)[0][-1]
        preds = preds/np.sum(preds)
        next_char = choice(chars, p=preds)
        seed_string = seed_string + next_char
    print(seed_string)


    

In [236]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 640s - loss: 1.5152   






    
Out[236]:





<keras.callbacks.History at 0x7f8cf2398810>

In [220]:

    

print_example()


    

    




ethics is a basic foundation of all thatscrriets sdi  ,s lrrbmh
fceelsora tec
 n yiefma
cnostencnrut -  o
pen.htt" oaiosovo  stialpts es rb b
ea ie
ohatnmauyielueysiutlmo,es  etfrne oh
ohnio iis e.eosme o rdorfdbteirnse ohdnotafi enicron e eietnyn sytt e ptsrdrede httmi ah
oo, tdye es r,igyct toehitu abrh ei isiem-r natra lnspamlltefae a
cni vuui
twgt fatieh

In [236]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 640s - loss: 1.5152   






    
Out[236]:





<keras.callbacks.History at 0x7f8cf2398810>

In [222]:

    

print_example()


    

    




ethics is a basic foundation of all that he maluces indofely and is; pticrast', and re onerog je ivesantamale as whered
and ror and kytinf? on chaninn nurdeln--ans prory. heke the pepadinar; anf bom,
puntely"" ones to bucf, alcherstol the qisleves: the the wite dadong the gur is prang not galcaula rewinl
more by than sic appads not pepow o mee, a more
bins c

In [235]:

    

model.optimizer.lr=0.001


    

In [236]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 640s - loss: 1.5152   






    
Out[236]:





<keras.callbacks.History at 0x7f8cf2398810>

In [237]:

    

print_example()


    

    




ethics is a basic foundation of all that schools pedhaps a new prisons of the ashamed in which
a coverbine estimates of the assumption that one avoid; he will curse about pain:
     people, he-equally present to
the lalier,
nature. that he has
rendered and henceforth distrain and impulses to perceive that each other
former and dangerous, and cannot at
the pu

In [250]:

    

model.optimizer.lr=0.0001


    

In [239]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 639s - loss: 1.2892   






    
Out[239]:





<keras.callbacks.History at 0x7f8cf167b410>

In [240]:

    

print_example()


    

    




ethics is a basic foundation of all that account has its granitify them.

131. the new "dilence," out of the
same light,
interpretation thereof: under the "thinking"
there, to counter-arguments in the monality, so many language:
though
all nobilitys of higher impulses, man and hence to everything of seldom man.



chapter i. woman decides according the injur

In [242]:

    

model.save_weights('data/char_rnn.h5')


    

In [257]:

    

model.optimizer.lr=0.00001


    

In [243]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 640s - loss: 1.2544   






    
Out[243]:





<keras.callbacks.History at 0x7f8cf2398f50>

In [249]:

    

print_example()


    

    




ethics is a basic foundation of all that is military
contemplation of distance itself is in physician!

249.. in every
to strick in the man of disguise and in the
will to wind at any progress, the
religious estimates of vehapance has a powerful and religious nature of manner, who had the problem of
decided expression of his equality, which, sometimes power? 

In [258]:

    

model.fit(sentences, np.expand_dims(next_chars,-1), batch_size=64, nb_epoch=1)


    

    




Epoch 1/1
600860/600860 [==============================] - 640s - loss: 1.2218   






    
Out[258]:





<keras.callbacks.History at 0x7f8cf2a21590>

In [264]:

    

print_example()


    

    




ethics is a basic foundation of all that
the belief in the importance. the employs concerning
seriousness and
materialism, it is circles which alone is already attained, that he sees
also the day after thinking
of mankind, brightness, resistance--and after the value of "nature" in order to nevertheless
have taken a system of liberal fatalists are willing him

In [283]:

    

print_example()


    

    




ethics is a basic foundation of all that were beought by the temptation of truth--for the rest of a sublime medely and take part of life, which lacks himself the
credibility about this, in short, and raise such a
gods; and on the
other hand, the explanation of
the case, as the most ingredient, and insight, and approach as to the
peculiarly prolonged "distrus

In [282]:

    

model.save_weights('data/char_rnn.h5')


    

In [ ]: