Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_200_512_2_full.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_200_512_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> output]
  (1): nn.LookupTable
  (2): nn.LSTM(200 -> 512)
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 6.4472819164084)
('Valid Perplexity: ', 32.517989261361)
('Test Perplexity: ', 30.289007907978)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  blackberry and apricot compote <begin_ingredients> 2 firm - ripe apricots <ing> 1 / 4 cup water <ing> 1 / 4 cup sugar <ing> 1 / 4 teaspoon freshly grated lime zest <ing> 1 cup dry white wine <ing> 1 cup picked - over blackberries <ing> accompaniment if desired : almond macaroons <end>

Gend:  <beg> place ice cream in a blender , and process until smooth . <step> combine all ingredients in a blender , and process until smooth . <end> . <end> , or serve immediately or refrigerate . <end> , if necessary , chill up to 1 hour . <end> and serve . <end> ! <end> ! <end>

True:  halve apricots lengthwise and pit . cut each apricot half into 4 wedges and transfer to a heatproof bowl . <step> in a small saucepan bring water and sugar to a boil over moderate heat , stirring until sugar is dissolved . stir in zest and wine and simmer 5 minutes . pour syrup over apricots and stir in blackberries . chill mixture 20 minutes . serve compote with macaroons . <end>



Input:  beta - carotene bomb salad <begin_ingredients> 2 beta - carotene bomb salad <ing> 1 large beetroot <ing> 50 grams pea shoots <ing> 1⁄2 lime <ing> extra virgin olive oil <end>

Gend:  <beg> add the cream , milk , salt , and pepper to a medium saucepan . add the cream and the cream and stir to combine . cook until the mixture is completely absorbed , about 2 minutes . stir in the lemon juice and serve . <end> with a slotted spoon and a straw .

True:  * grate the carrots and the beetroot . if you have a food processor that can do that , definitely use it as it would be a huge time saver . <step> * mix all together with the pea shoots . <step> * squeeze the juice of half a lime . <step> * add a bit of extra virgin olive oil and enjoy ! <end>



Input:  cabbage omelet <begin_ingredients> 1 small onion , peeled , cut in half and thinly sliced <ing> 1 / 2 head cabbage , shredded <ing> 3 eggs , slightly beaten <ing> 1 / 2 - 1 teaspoon salt ( or to taste ) <ing> 1 / 4 teaspoon black pepper ( or to taste ) <ing> 1 - 2 tablespoon canola oil <end>

Gend:  <beg> heat a large skillet over medium heat . add the oil , and heat to medium - high . add the garlic and ginger and stir - fry for 30 seconds . add the garlic and ginger and cook for 2 minutes . add the garlic and ginger and cook until fragrant , about 1

True:  heat a non - stick skillet on medium heat . <step> add oil to skillet and heat , do not burn . <step> add onions to hot oil and saute until well cooked but not yet browned , about 7 minutes . <step> add cabbage to skillet and saute , stirring constantly until cooked and takes on a translucent appearance . onions should be starting to brown when cabbage is cooked . about 5 to 7 minutes . <step> stir in salt and pepper . <step> pour beaten eggs over cabbage / onion mixture and mix into vegetables , cook until eggs are cooked . about 3 minutes . <step> serve as a side dish with a main meat course or as a main dish with steamed jasmine white rice . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  egg sandwiches with wilted spinach <begin_ingredients> 1 tsp . plus 1 tbs . olive oil <ing> 2 eggs <ing> 2 tbs . low - fat ( 1 % ) milk or water <ing> 2 thin slices manchego cheese , about 1 1 / 2 oz . total <ing> 2 whole wheat english muffins , split and toasted <ing> 1 garlic clove , smashed <ing> 2 generous handfuls baby spinach leaves , about 3 oz . <end>

Gend:  <beg> 1 . heat the olive oil in a large skillet over medium - high heat . add the onion and garlic and cook , stirring occasionally , until softened , about 3 minutes . add the garlic and cook , stirring , until fragrant , about 1 minute . add the garlic and cook for

True:  preheat an oven to 375f ° . lightly grease two 6 - oz . ramekins with the 1 tsp . olive oil . <step> in a bowl , lightly beat the eggs with the milk . divide the egg mixture between the prepared ramekins . transfer to the oven and bake until the eggs are puffy and set , 15 to 18 minutes . <step> near the end of the baking time , place a cheese slice on the bottom half of each english muffin . place the cheese - topped muffins on a rack in the hot oven until the cheese melts slightly , 2 to 3 minutes . remove from the oven and set aside . <step> meanwhile , in a fry pan over medium heat , warm the 1 tbs . olive oil and add the garlic clove . cook just until the garlic sizzles and is fragrant , about 1 minute . add the spinach and cook , stirring , until just wilted , 30 seconds to 1 minute . remove from the heat and discard the garlic clove . <step> remove the ramekins from the oven and let cool slightly . carefully run a paring knife around the inside edges to release the eggs . turn out each egg portion onto a muffin , on top of the melted cheese . mound equal portions of the spinach over the eggs and then cover with the muffin tops . eat immediately , or wrap loosely in waxed paper and take to go . serves 2 . <end>



Input:  coffee cake in a mug with cinnamon oatmeal struesel topping <begin_ingredients> 1 tablespoon butter , softened <ing> 4 teaspoons sugar <ing> 2 tablespoons vanilla yogurt <ing> ? cup flour <ing> ? teaspoon baking powder <ing> pinch of salt <ing> 1 tablespoon butter , softened <ing> 2 tablespoons quick oatmeal <ing> 1 scant tablespoon brown sugar <ing> ? teaspoon cinnamon <end>

Gend:  <beg> 1 . in a medium bowl , mix together flour , sugar , cinnamon , and salt . <step> 2 . in a separate bowl , mix together flour , baking powder , and salt . <step> 3 . in a large bowl , combine the wet ingredients and mix well . <step> 3 .

True:  place 1 tablespoon of room temperature butter in mug . if cold , place in microwave for about 10 seconds . do not melt . <step> add sugar and mix well . <step> add yogurt and dry ingredients , stirring until just combined . <step> mixture should be thick . smooth across the bottom of the mug . <step> in small bowl , combine streusel topping ingredients using a fork until well mixed . sprinkle on top of coffee cake in mug . <step> cook at full power in microwave for 1 minute . top should be shiny , look slightly underdone and firm to touch . if additional cooking time is needed , do only 10 seconds at a time to avoid overcooking . <step> eat while warm ! ! ! ! <end>



Input:  grits spanakopita <begin_ingredients> 1 ( 10 - oz . ) package frozen chopped spinach <ing> 1 cup uncooked regular grits <ing> 2 ( 4 - oz . ) packages feta cheese , crumbled <ing> 1 / 2 cup finely chopped green onions <ing> 1 / 4 cup chopped fresh basil <ing> 2 garlic cloves , pressed <ing> 1 tablespoon fresh lemon juice <ing> 3 / 4 teaspoon salt <ing> freshly ground pepper to taste <ing> 2 cups panko ( japanese breadcrumbs ) <ing> 1 / 2 cup grated parmesan cheese <ing> 1 / 4 cup finely chopped fresh parsley <ing> vegetable cooking spray <end>

Gend:  <beg> 1 . bring a large pot of salted water to a boil . add pasta and cook until al dente . drain . <step> 2 . meanwhile , in a large skillet , cook bacon until crisp and brown on both sides , about 3 minutes per side . transfer to a plate ; set

True:  1 . thaw spinach ; drain well , pressing between paper towels . <step> 2 . prepare grits according to package directions . stir feta , next 5 ingredients , and spinach into grits . add freshly ground pepper to taste . <step> 3 . stir together panko and next 2 ingredients ; sprinkle half of mixture into a well greased 13 - x 9 - inch baking dish . pour hot grits mixture into prepared pan . sprinkle remaining panko mixture over grits . cover and chill 8 hours . <step> 4 . invert chilled grits onto a cutting board ; cut into 24 squares . cut each diagonally into 2 triangles . coat both sides well with cooking spray . <step> 5 . cook triangles , in batches , on a hot greased griddle or in a nonstick skillet over medium heat 2 minutes on each side or until golden . place on a wire rack on a baking sheet , and keep warm in a 200 ? oven up to 30 minutes . <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  healthy sweet and sour chicken <begin_ingredients> 3 - 4 boneless skinless chicken breasts , pounded to ½ inch thickness and chopped into 1 inch pieces <ing> ⅓ cup corn starch <ing> 2 tablespoons oil <ing> 1 red bell pepper , chopped <ing> 1 green bell pepper <ing> ½ white or yellow onion , chopped <ing> ¾ cup sugar <ing> ½ cup apple cider vinegar ( may sub white vinegar ) <ing> 2 tablespoons soy sauce <ing> 1 teaspoon garlic powder <ing> ½ teaspoon onion * salt <ing> ¼ cup ketchup <ing> 1 tablespoon cornstarch + 2 tablespoon cold water <ing> 1 tablespoon cornstarch + 2 tablespoon cold water <end>

Gend:  <beg> heat 1 tablespoon oil in a large skillet over medium - high heat . add garlic and ginger and stir - fry until fragrant , about 30 seconds . add garlic and ginger and cook , stirring , until fragrant , about 1 minute . add chicken and cook , stirring , until fragrant ,

True:  first prepare the sauce . add sugar , vinegar , soy sauce , garlic powder , onion salt , and ketchup to a medium sauce pan . stir and bring to a boil . in a small bowl whisk together the 1 tablespoon corn starch and cold water until dissolved . add to sauce pan and stir until thickened , then reduce to low heat . <step> add chicken pieces and corn starch to a large ziplock bag . seal and shake to coat chicken . drizzle a large pan or skillet with oil . add coated chicken . saute over medium heat for about 5 minutes . add peppers and onions . continue to saute over medium heat until chicken is browned and cooked through . <step> add sauce to chicken and peppers . stir to coat well . serve warm with cooked rice if desired . <end>



Input:  fresh salmon - cilantro burgers <begin_ingredients> 1 / 4 cup reduced - fat mayonnaise <ing> 1 tablespoon chopped fresh cilantro <ing> 1 tablespoon fresh lime juice <ing> 1 / 8 teaspoon salt <ing> 1 / 8 teaspoon freshly ground black pepper <ing> 1 ( 1 - pound ) salmon fillet , skinned and cut into 1 - inch pieces <ing> 1 / 4 cup dry breadcrumbs <ing> 2 tablespoons cilantro leaves <ing> 2 tablespoons chopped green onions <ing> 1 tablespoon chopped seeded jalapeño pepper <ing> 2 tablespoons fresh lime juice <ing> 1 / 2 teaspoon salt <ing> 1 / 4 teaspoon freshly ground black pepper <ing> cooking spray <ing> 4 ( 1 1 / 2 - ounce ) hamburger buns with sesame seeds , toasted <ing> 12 ( 1 / 4 - inch - thick ) slices english cucumber <ing> 4 leaf lettuce leaves <end>

Gend:  <beg> 1 . combine first 4 ingredients in a small bowl . rub steaks with 1 / 4 teaspoon salt and pepper . add steak to pan ; grill 2 minutes on each side or until lightly browned . remove from grill ; let stand 5 minutes . <step> grill shrimp , covered with grill lid

True:  1 . combine first 5 ingredients in a small bowl ; cover and chill . <step> 2 . place salmon in a food processor ; pulse until coarsely chopped . add breadcrumbs and next 6 ingredients ( through 1 / 4 teaspoon black pepper ) ; pulse 4 times or until well blended . divide salmon mixture into 4 equal portions , shaping each into a 3 / 4 - inch - thick patty . <step> 3 . heat a grill pan over medium - high heat . coat pan with cooking spray . add patties to pan ; cook 2 minutes . carefully turn patties over ; cook 2 minutes or until done . <step> 4 . spread about 1 tablespoon mayonnaise mixture over bottom half of each hamburger bun . top each serving with 1 salmon patty , 3 cucumber slices , 1 lettuce leaf , and top half of bun . <step> spinach salad : combine 2 tablespoons hoisin sauce , 1 tablespoon rice vinegar , 2 teaspoons canola oil , 2 teaspoons water , and 1 / 4 teaspoon chile paste in a large bowl ; stir well with a whisk . add half of 1 ( 5 - ounce ) package baby spinach , 1 / 2 cup yellow bell pepper strips , and 1 / 4 cup thinly sliced red onion to bowl ; toss to coat . <end>



Input:  pumpkin spice ice cream <begin_ingredients> 2 cups heavy cream <ing> 1 cup whole milk <ing> 1 vanilla bean , split and scraped <ing> 3 / 4 cup dark brown sugar <ing> 8 egg yolks <ing> 1 cup pumpkin purée <ing> 1 teaspoon ground cinnamon <ing> 1 / 2 teaspoon ground cloves <ing> 1 / 4 teaspoon ground nutmeg ( see note above ) <ing> 1 teaspoon bourbon <ing> kosher salt , to taste <end>

Gend:  <beg> in a small bowl , combine the cream and milk . stir in the vanilla and lemon zest . pour the mixture into the prepared pan and top with the berries . <step> bake for 30 minutes , or until the top is golden brown . <end> with a spatula . <end> with a slotted

True:  1 in a medium saucepan over medium heat , bring cream and milk to a simmer . stir in vanilla bean , cover , remove from heat , and let steep for 1 hour . discard vanilla bean or wash and dry for another use . 2 in a second saucepan whisk together sugar , egg yolks , pumpkin purée , and spices until very well combined . slowly poor milk mixture into saucepan , whisking constantly , until fully incorporated . set over medium heat and cook , stirring frequently , until a custard forms on the back of a spoon and finger swiped across it leaves a clean line . 3 pour custard through a fine mesh strainer into an airtight container , then stir in bourbon and salt to taste in 1 / 4 teaspoon increments ( i used 3 / 4 teaspoons for a slightly salty kick ) and chill overnight . 4 the next day , churn according to manufacturer 's instructions . transfer ice cream to airtight container and chill in freezer for at least 4 to 5 hours before serving . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.15 

1-gram Score:  55
2-gram Score:  16.8
3-gram Score:  6
4-gram Score:  3.1