In [2]:
import string
In [15]:
from vanilla_neural_nets.recurrent_neural_network.network import VanillaRecurrentNeuralNetwork
from vanilla_neural_nets.recurrent_neural_network.optimization_algorithm import RNNGradientDescent
from vanilla_neural_nets.recurrent_neural_network.backpropagate_through_time import RNNBackPropagateThroughTime
from vanilla_neural_nets.recurrent_neural_network.parameter_initialization import OneOverRootNWeightInitializer
from vanilla_neural_nets.recurrent_neural_network.training_data import WordLevelRNNTrainingDataBuilder
In [4]:
VOCABULARY_SIZE = 1000
In [5]:
CORPUS = """
I spent seven-plus weeks this summer in the Balkans and Eastern Europe. For a week-plus, I necessarily reconnected with an old love interest to tighten loose ends. For another, I discovered Kosovo, falling hard for the youth, optimism and bullishness of its decidedly ugly capital Pristina, and watching films alongside castles, technologists, dashing Danish women and qebapa at Prizen’s DokuFest. In a third week, I went hiking in the Albanian Alps as originally planned. In a fourth, I rebuilt this website and constructed syllabi for two data science courses I was to later teach. For the remaining ~3 weeks, I wondered: what the fuck am I really doing here?
My two-plus-year trip around the world was perhaps the most meaningful experience of my life. Still, almost three years after finish, I have not fully processed what it truly meant to me. I was on fire, back then. I felt concrete purpose in my steps. I carried unwavering bewilderment in my eyes and an unbeatable smile on my face. The people I met were characters in a beautiful, awesome story. Arriving in a new city, backpack aback and a half-charged iPod in my front pocket was a moment of romance. Parting ways with new friends with a “have fun out there” was my full-hearted salute to a fellow soldier on the battlefield of travel. There was nothing else I wanted to do with my time.
Upon return, I picked up a new hobby: data science. I moved to New York City where my hobby was my job. I started speaking differently – discussing technologies, companies and investment – using words I previously found alien, and ugly. Relationships became interesting: all the depth, beauty and rhythm I once found in constant change was right there in my peers. In my boss, Alex. In my neighbor, Olga. None of these things are bad. Simply, I grew as a person into someone new.
Until this summer, traveling to new places seemed like panacea. More than anything, this was supported by my own personal experiences on the road – by the immense joy and creative energy it had always brought. This summer, getting lost in new places and eating new foods was not the experience it always was. My relationship with travel has started to change.
Travel was enthralling because it was a challenge. The theme of my trip ’round the world was thus: to relentlessly flog myself mentally, physically and emotionally, and see what came out the other end. And I did it. I became fluent in a new language. I traveled by land from Sweden to Côte d’Ivoire. I pedaled 10,000 kilometers on a bicycle. I fell in love. I learned how to really solve problems. And when it was over, landing in Philadelphia International, I cried, cried and cried. I hadn’t the slightest regret in the world. I did, almost meticulously, exactly what I set out to do.
This summer brought one challenge: revisiting a previous love. That week, in all frankness, was exhilarating. That week – while we did nothing more than camp, eat, laugh and walk – was one of the sickest adventures in recent years. But that week aside – that challenge aside – there was nothing more than unknown cities I knew how to navigate; unknown, 48-hour strangers I knew how to befriend; unknown languages I felt I could speak. I was not in awe. This summer was one of hiking, delicious produce, an education in Balkan history and my tent, all baked into a lukewarm soup. It was not like my trip before; it was not that which I know travel can bring.
Moving forward, I’m conscious of the following: it’s not about travel; it’s about doing things that scare you. It’s about “breaking off” something big – “I’m going to cycle from Cairo to Cape Town;” “I’m going to learn the viola in the next 60 days;” “I’m going to teach myself product management and get hired as a Product Manager” – and knocking it down. Then, once more, every second does become exhilarating in proportion to your goal. The people you meet do become pointed characters in your story. Life becomes a circuitous and enriching journey to a tenable end.
Travel is an evolutionary process. At the start, the novelty of diverse company, foreign currency and what’s for breakfast suffices to keep our cheeks pinned wide and our spirit on edge. Thereafter, this decays: the uncomfortable becomes comfortable with time and experience, and we must fight harder and harder to get the same high.
So, what’s the solution? Where do we go from here?
First and foremost, we must realize that we, humans, are always growing, and fundamentally, our path to happiness is growing in suit. This is a path marked not by the things we do, but the context in which we do them. Our first trip, kiss and job are memorable not because of the concrete action, necessarily, but more so because of the novelty of the experience and the emotional reward that brings.
Next, it follows that our once-stalwart road to ecstasy might not be our next. Sadly, I will never be able to “do” my trip again; a two-year trip around the world is not my next high. This took considerable time, effort and humility to realize.
Lastly, the potential for explosive inspiration and liquid-like passion for life itself is always there. To find it, we must be honest with ourselves: honest about what we truly want, honest about what scares us, honest about our tolerance for risk. Then, we take these answers and throw ourselves into something foreign, something enriching, something that demands we find something in and of ourselves that we haven’t found before.
Moving forward, I’d like to travel and work. Increasingly (and really, I started at 0…) I see work as beautiful thing, and fortunately, my hobby is my job. I recently finished two weeks of work in Bogotá – creating data science courses for Platzi – the first time I’d really combined my passion for travel with the professional world. All of a sudden, arriving to a hotel, meeting new people, going for runs and taking photos became enthralling – rimmed with playful novelty by virtue of being somewhere new – and brimming with adrenaline, as I wanted to do a great job. Those two weeks were memorable because I faced a new challenge. And for what it’s worth, if someone said “here’s two weeks in Bogotá – the capital of your favorite country in the world – now go, just, walk around,” I likely would have said no (preferring to study data science things in my bedroom, or something).
I currently write from my new apartment in Casablanca, Morocco. Here, I’m pursuing a self-directed “Masters” in machine learning and speaking some French. (Incidentally, few people understand what I’m doing, why I’m doing it and how I’ll ever be hired again, which only adds to the challenge.)
I can’t wait to not move. I can’t wait to memorize the smells of my street and find the best baked goods within a 3-block radius. I can’t wait to learn more about the people in my co-working space. In fact, I can’t wait for tomorrow.
"""
In [6]:
training_data = WordLevelRNNTrainingDataBuilder.build(
corpus=CORPUS,
vocabulary_size=VOCABULARY_SIZE
)
In [28]:
HIDDEN_LAYER_SIZE = 50
BACKPROP_THROUGH_TIME_STEPS = 4
LEARNING_RATE = 0.05
N_EPOCHS = 25
RANDOM_STATE = 12345
In [29]:
rnn = VanillaRecurrentNeuralNetwork(
vocabulary_size=VOCABULARY_SIZE,
hidden_layer_size=HIDDEN_LAYER_SIZE,
backprop_through_time_steps=BACKPROP_THROUGH_TIME_STEPS,
backprop_through_time_class=RNNBackPropagateThroughTime,
optimization_algorithm_class=RNNGradientDescent,
weight_initializer_class=OneOverRootNWeightInitializer,
learning_rate=LEARNING_RATE,
n_epochs=N_EPOCHS,
random_state=RANDOM_STATE,
log_training_loss=True
)
In [30]:
rnn.fit(training_data.X_train, training_data.y_train)
In [31]:
import numpy as np
def generate_new_sentence(network):
sentence = [sentence_start_index]
while not sentence[-1] == sentence_end_index:
next_word_index_predictions = network.predict(sentence)
next_word_index = np.argmax( np.random.multinomial(1, next_word_index_predictions[-1]) )
if next_word_index in training_data.index_to_token_lookup.keys():
sentence.append(next_word_index)
return [training_data.index_to_token_lookup[index] for index in sentence[1:-1]]
def conditional_sentence_join(corpus):
corpus = [('' if char in string.punctuation else ' ') + char for char in corpus]
return ''.join(corpus).strip()
def generate_new_corpus(network):
new_corpus = []
for _ in range(NUMBER_OF_SENTENCES):
sentence = []
while len(sentence) < MIN_SENTENCE_LENGTH or len(sentence) > MAX_SENTENCE_LENGTH:
sentence = generate_new_sentence(network)
new_corpus += sentence
return new_corpus
In [32]:
NUMBER_OF_SENTENCES = 10
MIN_SENTENCE_LENGTH = 10
MAX_SENTENCE_LENGTH = 25
sentence_start_index = training_data.token_to_index_lookup['SENTENCE_START']
sentence_end_index = training_data.token_to_index_lookup['SENTENCE_END']
In [33]:
new_corpus = generate_new_corpus(rnn)
print( conditional_sentence_join(new_corpus) )
In [34]:
from vanilla_neural_nets.recurrent_neural_network.network import VanillaLSTM
from vanilla_neural_nets.recurrent_neural_network.optimization_algorithm import LSTMGradientDescent
from vanilla_neural_nets.recurrent_neural_network.backpropagate_through_time import LSTMBackpropagateThroughTime
In [35]:
lstm = VanillaLSTM(
vocabulary_size=VOCABULARY_SIZE,
hidden_layer_size=HIDDEN_LAYER_SIZE,
backprop_through_time_steps=BACKPROP_THROUGH_TIME_STEPS,
backprop_through_time_class=LSTMBackpropagateThroughTime,
optimization_algorithm_class=LSTMGradientDescent,
weight_initializer_class=OneOverRootNWeightInitializer,
learning_rate=LEARNING_RATE,
n_epochs=N_EPOCHS,
random_state=RANDOM_STATE,
log_training_loss=True
)
In [36]:
lstm.fit(training_data.X_train, training_data.y_train)
In [43]:
new_corpus = generate_new_corpus(lstm)
print( conditional_sentence_join(new_corpus) )