Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_200_512_dr.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_200_512_dr_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> (3) -> output]
  (1): nn.LookupTable
  (2): nn.LSTM(200 -> 512)
  (3): nn.Dropout(0.400000)
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 7.8905910143412)
('Valid Perplexity: ', 23.700125891221)
('Test Perplexity: ', 22.736589678571)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  tex - mex waffles <begin_ingredients> 1 1 / 2 cups all - purpose flour <ing> 1 / 2 cup yellow cornmeal <ing> 1 1 / 3 cups longhorn cheese or 1 1 / 3 cups sharp cheddar cheese , grated <ing> 1 tablespoon baking powder <ing> 2 teaspoons sugar <ing> 1 / 2 teaspoon salt <ing> 1 teaspoon mild chili powder <ing> 1 2 / 3 cups milk <ing> 2 eggs , lightly beaten <ing> 1 / 3 cup vegetable oil <ing> 2 tablespoons finely chopped green chilies <end>

Gend:  <beg> preheat oven to 350 degrees f ( 175 degrees c ) . grease a 9x13 inch baking dish . <step> in a large bowl , combine the cream cheese , cream cheese , vanilla , sugar , and vanilla bean mixture . stir in the eggs , milk , and vanilla . pour the wet

True:  preheat the waffle maker . <step> in a large bowl , combine the flour , cornmeal , cheese , baking powder , sugar , salt and chili powder . add the milk , eggs , vegetable oil and chiles , and mix just until moistened . mixture may be a little lumpy . <step> bake in preheated waffle maker according to manufacturer 's instructions . makes 4 to 5 large waffles . <step> serve with your favorite salsa , sour cream and additional grated cheese . <end>



Input:  pasta olive olio <begin_ingredients> 1 lb pasta <ing> 1 / 2 red onion , sliced <ing> 1 / 4 red & yellow pepper , sliced <ing> 1 bag fresh spinach <ing> 2 – 3 tbsp olive oil <ing> salt , pepper & oregano <ing> salt , pepper & oregano <ing> salt , pepper & oregano <end>

Gend:  <beg> in a large pot , heat olive oil over medium heat . add onion and garlic and cook for 2 minutes . add garlic and cook for another minute . add garlic and cook for another minute . add the tomatoes and cook for another minute . add the chicken broth and bring to a

True:  in a pot with of boiling salted water , add the pasta and cook until just al dente . the pasta will get tossed again with the vegetables and cook a little more . drain the pasta but reserve some of the cooking liquid . meanwhile , in a sauté pan on medium heat , add the oil and get good and warm . once , warm , add the onions and peppers . also add the salt , pepper and oregano . cook until they just begin to soften about 3 minutes or so . <end>



Input:  baked brie and bacon jam phyllo cups <begin_ingredients> 1 pack mini phyllo cups ( i used athens ) <ing> 4 oz brie cheese ( i used alouette creme de brie ) <ing> 8 slices bacon <ing> 1 / 3 cup apricot jam ( i used polaner ) <ing> 2 tbsps apple cider vinegar <ing> thyme for serving <ing> crisco to grease <end>

Gend:  <beg> 1 . preheat oven to 350 degrees . <step> 2 . in a large bowl , combine the flour , baking powder , salt , and sugar . <step> 2 . add the milk and vanilla extract and mix well . <step> 3 . add the flour and mix until just combined . <step> 4

True:  1 . in microwave cook bacon until crisp , then allow to cool before chopping into chunks . set aside . <step> 2 . in small skillet heat apricot jam and vinegar over medium heat until well combined . next , stir in bacon and bring ingredients to a rolling boil . once this is reached , reduce flame to low and allow to simmer for 5 - 6 minutes . <step> 3 . using a teaspoon , spoon brie into each phyllo cup before placing on lightly greased baking sheet . when all cups are prepared , top each with a dollop of bacon jam , then bake in the oven at 350f for 10 minutes until brie is just beginning to bubble . immediately plate and serve , pairing with fresh thyme if desired . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  chocolate pancakes <begin_ingredients> 1 1 / 3 cups all purpose flour <ing> 1 / 3 cup plus 1 tablespoon cocoa powder <ing> 6 tablespoons granulated sugar <ing> 1 1 / 2 teaspoons baking powder <ing> 3 / 4 teaspoon salt <ing> 1 / 4 teaspoon baking soda <ing> 2 large eggs <ing> 3 tablespoons melted unsalted butter , plus extra for greasing pan and buttering pancakes <ing> 3 / 4 cup whole milk <ing> 1 / 2 cup whole milk <ing> 5 ounces semi - sweet chocolate , finely chopped <ing> 1 teaspoon pure vanilla extract <end>

Gend:  <beg> preheat the oven to 350 degrees . grease a 9 - inch cake pan with cooking spray . <step> in a large bowl , combine the flour , sugar , baking powder , baking soda , salt , and cinnamon . stir in the melted butter , and mix until combined . <step> in a

True:  1 in large bowl , sift together flour , cocoa , sugar , baking powder , salt , and baking soda . in medium bowl , whisk eggs , melted butter , and milk . pour liquid ingredients into dry and gently mix until batter is just moistened ( there will be some lumps ) . 2 melt small piece of butter in large non - stick skillet over medium heat . drop 2 heaping tablespoons batter in hot skillet and spread into 2 1 / 2 inch disc . space discs 2 inches apart . cook until bubbles form over entire top , about 30 seconds . using heatproof , non - stick spatula , carefully flip pancake and continue to cook until done , 15 to 30 seconds more . 3 in small saucepan , heat milk to just simmering . add chocolate and whisk until smooth . whisk in vanilla . serve over pancakes . <end>



Input:  slow cooker balsamic beef <begin_ingredients> 1 large yellow onion <ing> 2 - 3 garlic cloves <ing> 2 - 3 pound beef chuck , roast , or stew meat <ing> 1 teaspoon kosher salt <ing> 1 bay leaf <ing> 5 - 8 sprigs fresh thyme <ing> ½ cup balsamic vinegar <ing> ½ cup beef stock <ing> 1 yellow onion <ing> 2 red bell peppers <ing> 8 buns , rolls , or other bread <ing> ½ cup unsalted butter , softened <ing> 1 tablespoon garlic powder <ing> 2 tablespoons grated parmesan cheese <ing> 1 recipe slow cooker balsamic beef ( recipe above ) <ing> 1 recipe one - pot stove - top mac and cheese <end>

Gend:  <beg> in a large skillet , heat the olive oil over medium heat . add the onion and garlic and cook , stirring occasionally , until the onion is translucent , about 5 minutes . add the garlic and cook for another minute . add the garlic and cook for another minute . add the tomatoes

True:  slice the onions into rings and the garlic into discs and place them in the bottom of the slow cooker . <step> place the beef on top of the onions and garlic along with all the remaining ingredients . <step> set the slow cooker to high and cook for 4 hours , or cook on low for 6 hours . <step> you 'll know the meat is ready when it easily falls apart when pulled with two forks . <step> for some bonus goodness , strain the cooking liquid into a small saucepan , whisk in some flour , and cook over medium to thicken into a nice gravy for dipping ! <step> thinly slice the onion and peppers , and sauté them in a touch of oil over medium heat until soft , about 7 minutes . <step> while those are sautéing , mix together the butter , garlic powder , and parmesan cheese until thoroughly combined . <step> spread the butter mixture on all the bun or roll halves , and place under the broiler until golden brown , about 3 minutes . watch them so they do n't burn ! <step> set out the garlic bread buns , the sautéed peppers and onions , the balsamic beef and the mac and cheese , and let everyone go to town creating their new favorite sandwich ! <end>



Input:  egg sandwiches with wilted spinach <begin_ingredients> 1 tsp . plus 1 tbs . olive oil <ing> 2 eggs <ing> 2 tbs . low - fat ( 1 % ) milk or water <ing> 2 thin slices manchego cheese , about 1 1 / 2 oz . total <ing> 2 whole wheat english muffins , split and toasted <ing> 1 garlic clove , smashed <ing> 2 generous handfuls baby spinach leaves , about 3 oz . <end>

Gend:  <beg> 1 . preheat oven to 400 ° . <step> 2 . in a large bowl , combine the flour , sugar , baking powder , and salt . in a small bowl , whisk together the eggs , milk , and vanilla . <step> 2 . in a large bowl , whisk together the eggs

True:  preheat an oven to 375f ° . lightly grease two 6 - oz . ramekins with the 1 tsp . olive oil . <step> in a bowl , lightly beat the eggs with the milk . divide the egg mixture between the prepared ramekins . transfer to the oven and bake until the eggs are puffy and set , 15 to 18 minutes . <step> near the end of the baking time , place a cheese slice on the bottom half of each english muffin . place the cheese - topped muffins on a rack in the hot oven until the cheese melts slightly , 2 to 3 minutes . remove from the oven and set aside . <step> meanwhile , in a fry pan over medium heat , warm the 1 tbs . olive oil and add the garlic clove . cook just until the garlic sizzles and is fragrant , about 1 minute . add the spinach and cook , stirring , until just wilted , 30 seconds to 1 minute . remove from the heat and discard the garlic clove . <step> remove the ramekins from the oven and let cool slightly . carefully run a paring knife around the inside edges to release the eggs . turn out each egg portion onto a muffin , on top of the melted cheese . mound equal portions of the spinach over the eggs and then cover with the muffin tops . eat immediately , or wrap loosely in waxed paper and take to go . serves 2 . <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  hummus with hen of the woods mushrooms <begin_ingredients> 1 cup pine nuts <ing> 1 tsp . extra - virgin olive oil , plus more <ing> canola oil , for frying <ing> 1 lb . thinly sliced mushrooms , preferably hen of the woods or oyster <ing> 1 lb . thinly sliced mushrooms , preferably hen of the woods or oyster <ing> kosher salt , to taste <ing> ¼ cup fresh lemon juice <ing> 2 cloves garlic , roughly chopped <ing> 12 oz . tahini ( available at amazon ) <ing> 6 cups canned chickpeas , rinsed <ing> 1 tsp . ground cumin <ing> 1 tbsp . finely chopped parsley <ing> paprika , to garnish <end>

Gend:  <beg> <end> the rice and rice in a large pot . <step> cook the rice according to the package directions . <step> in a large bowl , combine the rice , rice , rice vinegar , soy sauce , sugar , and salt . <step> add the rice and stir to combine . <step> add the

True:  1 . heat oven to 300 ° . toss pine nuts and olive oil in a bowl until evenly coated . transfer to a baking sheet ; bake , stirring occasionally , until golden brown , about 10 minutes . set aside . pour canola oil to a depth of 2″ in a 6 - qt . dutch oven ; heat over medium - high heat until a deep - fry thermometer reads 350 ° . working in batches , add mushrooms , and fry until crisp , about 4 minutes . transfer to paper towels to drain and season with salt . <step> 2 . make the hummus : process lemon juice and garlic in a food processor until smooth . add tahini , chickpeas , cumin , and salt ; purée . spread on bottom of a serving dish , forming a shallow well in center ; fill well with mushrooms and pine nuts . sprinkle with parsley and paprika , and drizzle with olive oil . <end>



Input:  easy paella <begin_ingredients> 2 tablespoons olive oil <ing> 2 chicken legs with skin <ing> salt <ing> pepper <ing> 1 cup chopped onion <ing> 1 / 2 cup chopped green pepper <ing> 1 / 2 cup chopped red pepper <ing> 1 bay leaf <ing> 2 links spicy sausage , sliced <ing> 2 cups chopped canned tomatoes <ing> 1 tablespoon paprika <ing> 2 cups rice <ing> 5 cups chicken stock <ing> 6 clams , scrubbed and soaked in cold water to remove sand <ing> 6 mussels , scrubbed <ing> 6 shrimp , shelled and deveined <ing> 1 cup frozen peas , thawed <end>

Gend:  <beg> heat oil in a large skillet over medium heat . add onion and garlic and cook for 2 minutes . add garlic and cook for another minute . add garlic and cook for another minute . add garlic and cook for another minute . add the garlic and cook for another minute . add the

True:  chicken <step> dinner <step> simmer <end>



Input:  banana chocolate - chunk ice cream <begin_ingredients> 1 / 2 cup plus 2 tablespoons shredded sweetened coconut <ing> 2 pints vanilla ice cream , slightly softened <ing> 2 small bananas , cut into 1 / 2 - inch chunks <end>

Gend:  <beg> combine first 4 ingredients in a medium bowl . stir in milk and vanilla . <step> pour mixture into a greased 9 x 13 - inch baking dish . <step> bake at 350 ° for 40 minutes or until crust is golden brown . <end> cake with a knife . <end> cake . <end> cake

True:  preheat oven to 350 degrees . spread coconut evenly on a rimmed baking sheet ; toast in oven , tossing about halfway through , until pale golden brown , 3 to 5 minutes . let cool completely . <step> mix ice cream , chocolate , banana , and 1 / 2 cup toasted coconut in the bowl of an electric mixer fitted with the paddle attachment until combined , about 1 minute . press mixture into a 1 - quart airtight container ; freeze until firm , about 50 minutes . serve scoops of ice cream sprinkled with remaining 2 tablespoons coconut . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.14 

1-gram Score:  50.2
2-gram Score:  16.2
3-gram Score:  6
4-gram Score:  3.1