Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_200_512_2.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_200_512_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> output]
  (1): nn.LookupTable
  (2): nn.LSTM(200 -> 512)
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 7.0723899372472)
('Valid Perplexity: ', 29.78677168081)
('Test Perplexity: ', 27.753701742162)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  korean beef and noodles <begin_ingredients> 1 teaspoon cornstarch <ing> 8 ounces beef eye round , trimmed and thinly sliced <ing> 1 tablespoon low sodium soy sauce <ing> 1 1 / 2 teaspoons minced peeled fresh ginger <ing> 2 teaspoons sambal oelek or 2 teaspoons thai chili paste <ing> 1 / 2 teaspoon dark sesame oil <ing> 3 garlic cloves , minced <ing> cooking spray <ing> 1 ( 3 3 / 4 ounce ) package uncooked cellophane noodles <ing> 1 teaspoon dark sesame oil <ing> 1 teaspoon vegetable oil <ing> 1 / 2 teaspoon crushed red pepper flakes <ing> 5 garlic cloves , minced <ing> 3 cups sliced shiitake mushroom caps ( about 6 ounces mushrooms ) <ing> diagonally sliced green onion <ing> julienne - cut carrot <ing> 1 ( 10 ounce ) bag fresh spinach <ing> 1 / 3 cup low sodium soy sauce <ing> 1 tablespoon brown sugar <ing> 1 tablespoon rice vinegar <ing> 1 tablespoon dark sesame oil <ing> 1 tablespoon sesame seeds , toasted <end>

Gend:  <beg> in a large skillet , heat oil over medium - high . add onion and cook , stirring , until softened , about 5 minutes . add garlic and ginger and cook , stirring , until fragrant , about 1

True:  to prepare beef , sprinkle cornstarch over beef ; toss to combine . add 1 tablespoon soy sauce and next 4 ingredients ( 1 tablespoon soy sauce through 3 garlic cloves ) ; toss well to coat . cover and refrigerate 30 minutes to 1 hour . <step> heat a large nonstick skillet or wok coated with cooking spray over medium - high heat . add beef mixture ; stir - fry 3 minutes or until done . remove mixture from pan . cover and keep warm . <step> to prepare noodles , pour boiling water over noodles ; let stand 10 minutes or until tender . drain and rinse with cold water . drain . snip noodles several times with kitchen shears . <step> to prepare vegetables , wipe skillet or wok clean with paper towels . heat 1 teaspoon sesame oil and vegetable oil in pan over medium - high heat . add red pepper and 5 garlic cloves ; stir - fry 30 seconds . add mushrooms , onions , and carrot ; stir - fry 3 minutes . add half of spinach ; stir - fry 2 minutes or until spinach wilts . add remaining spinach ; stir - fry 2 minutes or until spinach wilts . <step> reduce heat to medium - low . add beef mixture and noodles to pan , stirring well to combine . combine 1 / 3 cup soy sauce , brown sugar , vinegar , and 1 tablespoon sesame oil , stirring with a whisk . drizzle over noodle mixture ; stir well to combine . cook over medium - low heat 3 minutes or until thoroughly heated . sprinkle with sesame seeds . <end>



Input:  artichoke filling for ravioli or tortellini <begin_ingredients> 1 small leek , washed well and thinly sliced <ing> 8 small artichoke hearts <ing> 1 1 / 2 tablespoons butter , melted <ing> salt & freshly ground black pepper <ing> salt & freshly ground black pepper <ing> 2 tablespoons olive oil <ing> 2 eggs , plus <ing> 1 egg yolk , beaten together <ing> 3 / 4 cup freshly grated parmesan cheese <end>

Gend:  <beg> in a large skillet , heat the olive oil over medium - high heat . add the onions and garlic and cook until softened , about 5 minutes . add the garlic and cook for another minute . <step> add

True:  cut the artichoke hearts into four wedges ; place them , the leek , butter , salt and pepper in a skillet and saute over moderately low heat for 15 minutes . <step> puree in a blender or food processor until smooth ; stir in olive oil , eggs and cheese . <end>



Input:  summer fruit samosas <begin_ingredients> 2 cups meat of fresh young thai coconuts <ing> about 1 / 4 cup fresh coconut water , more if needed <ing> 1 tablespoon fresh or frozen berries ( optional ) , i used lingonberries <ing> about 2 cups of mixed fruit such as cherries , strawberries , raspberries , blueberries , blackberries , currants , peaches , nectarines , plums - pits removed , sliced when necessary <ing> 4 tablespoons coconut sugar <ing> zest of 1 orange <ing> 1 tablespoon of orange juice <ing> seeds of 1 vanilla bean <end>

Gend:  <beg> combine all ingredients in a blender and blend until smooth . <end> . <end> . <end> , or pour into a chilled glasses . <end> . <end> , or freeze by serving to the ice cream . <end> and chill

True:  blend the coconut meat , berries , and water in a high speed blender , adding the water one tablespoon at a time . you may need more or less water , depending on the coconut meat quality . thicker , more mature meat requires more water than the thin , tender variety . make sure that the mixture is not too runny , but not too thick either . <step> spread your mixture evenly onto teflex - lined dehydrator trays ( i ended up with one and a half 15 - inch tray this time ) . dehydrate at 115f for 2 - 4 hours or more , until coconut sheets are ready to be separated from teflex . flip the sheet over and peel away the teflex . dry for another 30 minutes or so , making sure to not let your wrappers get too dry and crispy . they should be very pliable . <step> gently mix all ingredients together in a bowl . dehydrate at 115f for 2 - 3 hours , mixing occasionally . <step> transfer your coconut sheet onto a dry cutting board . cut the sheet down the middle , and then crosswise to receive 8 even rectangles , about 3 by 7 inches in size . place a generous tablespoon of the fruit filling onto one end of the rectangle . fold one corner over the filling diagonally , to meet the other side and make a triangle . continue folding in this manner , until the whole wrapper has been folded . serve with greek or oat yogurt . you might end up with some leftover fruit , which is great to have for breakfast as a yogurt or porridge topping . <end>



Input:  deviled eggs <begin_ingredients> 4 hard - boiled eggs <ing> 1 / 2 cup fresh basil leaf , roughly chopped <ing> 3 tablespoons mayonnaise <end>

Gend:  <beg> peel and cut each slice into 1 / 2 inch slices . <step> in a bowl , combine the mayonnaise , sour cream , sour cream , and a pinch of salt . mix well . <step> spread a thin

True:  peel hard boiled eggs and cut them lengthwise . <step> carefully remove yolks ; place yolks in a bowl . <step> set egg whites aside . <step> mash yolks with a fork . <step> place basil and mayonnaise in bowl of food processor ; blend about 2 minutes . <step> add mixture to yolks , mix . <step> season with salt and pepper if desired . <step> fill egg whites with filling and serve . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  spinach with chickpeas <begin_ingredients> 2 pounds baby spinach <ing> 3 tablespoons extra - virgin olive oil , divided <ing> 1 medium red onion , finely chopped <ing> 5 cloves garlic , minced <ing> 1 19 - ounce can chickpeas , rinsed <ing> 1 1 / 2 teaspoons dried thyme <ing> 1 1 / 2 teaspoons dried oregano <ing> 1 1 / 2 teaspoons ground cumin <ing> 1 teaspoon kosher salt <ing> 1 / 2 teaspoon hot paprika <ing> 1 / 2 cup golden raisins <ing> 1 / 2 cup reduced - sodium chicken broth , or vegetable broth <end>

Gend:  <beg> 1 . in a large bowl , combine the tomatoes , tomato , onion , garlic , olive oil , salt and pepper . mix well and set aside . <step> 3 . heat a large skillet over medium -

True:  1 . rinse spinach and let drain in a colander . with water still clinging to it , place half the spinach in a dutch oven over medium heat . cook , tossing with tongs and adding the remaining spinach by the handful until all is added and wilted , 6 to 8 minutes . drain in the colander . let cool slightly , then coarsely chop . <step> 2 . carefully wipe out the pan , then heat 1 tablespoon oil over medium heat . add onion and garlic and cook , stirring , until the onion is tender and lightly browned , 8 to 10 minutes . stir in chickpeas , thyme , oregano , cumin , salt and paprika . using a potato masher , mash some of the chickpeas , then cook , stirring , for 3 minutes . stir in raisins and broth , scraping up any browned bits . add the chopped spinach and stir gently to combine . remove from the heat and let stand 10 minutes . drizzle with the remaining 2 tablespoons oil just before serving . <end>



Input:  deviled eggs <begin_ingredients> 6 hard - boiled eggs <ing> 1 / 4 cup mayonnaise <ing> 1 teaspoon white vinegar <ing> 1 teaspoon yellow mustard <ing> 1 / 8 teaspoon salt <ing> 1 dash black pepper <ing> garnish with a sprinkle of paprika and chopped fresh parsley <end>

Gend:  <beg> beat eggs and eggs in a bowl . <step> add eggs , salt and pepper to taste . <step> add eggs and mix well . <step> add the eggs and one egg . <step> beat until well combined . <step>

True:  slice boiled eggs in half , remove yolks into a medium bowl . <step> smash yolks with fork , or beat with mixer until smooth . <step> mix in mayonaise , vinegar , mustard salt and pepper . <step> spoon or pipe into halved egg whites . <step> garnish with a sprinkle of paprika and a sprig of parsley . <end>



Input:  huevos rancheros <begin_ingredients> 8 inch flour tortilla ( traditional recipe uses corn ) <ing> refried beans <ing> salsa <ing> guacamole <ing> 2 eggs <ing> grated cheese ( your preference or whatever you have ) <end>

Gend:  <beg> preheat the oven to 350 degrees . <step> cut the bread into 1 / 2 inch slices . <step> in a large bowl , mix together the ground beef , bread crumbs , egg , and bread crumbs . <step>

True:  heat a tablespoon or two of oil in a large skillet over medium heat . add tortilla and cook on both sides until just lightly browned and crispy . remove to plate . add eggs to skillet and cook to desired doneness . i cook mine until they are just slightly runny in the center but my husband prefers his over well ( solid centers ) . <step> spread a generous amount of refried beans over crispy tortilla . top with a generous spread of salsa and follow that with guacamole . <step> place cooked eggs on top and sprinkle with salt and pepper , then cheese . you can put the plate under the broiler to melt the cheese if you like . garnish with more salsa and guacamole . enjoy ! <end>



Input:  gnocchi alla romana <begin_ingredients> 3 cups milk <ing> 1 teaspoon salt <ing> 1 / 2 cup unsalted butter <ing> 1 cup semolina ( malt - o - meal ) <ing> 1 cup freshly grated parmigiano - reggiano cheese <ing> 4 egg yolks <end>

Gend:  <beg> preheat oven to 350 . <step> in a large bowl , combine the cake mix , pudding mix , oil and egg . beat on medium - high speed until smooth . stir in vanilla . <step> pour mixture into

True:  preheat the oven to 425 defrees . butter a jelly roll pan that has 3 / 4 inches sides . <step> in a 3 to 4 quart sauce pan , heat the milk , salt , and 6 tbs of the butter nearly to the boiling point over medium heat . <step> whisking vigorously , add the semolina in a thin stream and cook for about a minute , switching to a wooden spoon as the batter thickens . <step> remove from the heat and stir 1 / 2 cup of the grated cheese , then beat in the egg yolks . working quickly pour the batter into the prepared pan and spread to a thicknes off 1 / 2 inch . allow to cool and cover with plastic until ready to cook . <step> using a 3 - inch diameter water glass , cut a rows of rounds across the sheet . <step> arrangein a well buttered baking dish . if there is any butter left , mealt it and pour on top of gnocchis . sprinkle with the remaining 1 / 2 cup of grated cheese . place in the oven and cook 15 to 20 minutes , or until the top is light golden brown . remove and serve inmediately . <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  hummus with hen of the woods mushrooms <begin_ingredients> 1 cup pine nuts <ing> 1 tsp . extra - virgin olive oil , plus more <ing> canola oil , for frying <ing> 1 lb . thinly sliced mushrooms , preferably hen of the woods or oyster <ing> 1 lb . thinly sliced mushrooms , preferably hen of the woods or oyster <ing> kosher salt , to taste <ing> ¼ cup fresh lemon juice <ing> 2 cloves garlic , roughly chopped <ing> 12 oz . tahini ( available at amazon ) <ing> 6 cups canned chickpeas , rinsed <ing> 1 tsp . ground cumin <ing> 1 tbsp . finely chopped parsley <ing> paprika , to garnish <end>

Gend:  <beg> 1 . combine the first 7 ingredients in a small bowl . <step> 2 . place a large baking sheet and drizzle with oil and roast to coat . <step> 4 . in a small bowl , combine the soy

True:  1 . heat oven to 300 ° . toss pine nuts and olive oil in a bowl until evenly coated . transfer to a baking sheet ; bake , stirring occasionally , until golden brown , about 10 minutes . set aside . pour canola oil to a depth of 2″ in a 6 - qt . dutch oven ; heat over medium - high heat until a deep - fry thermometer reads 350 ° . working in batches , add mushrooms , and fry until crisp , about 4 minutes . transfer to paper towels to drain and season with salt . <step> 2 . make the hummus : process lemon juice and garlic in a food processor until smooth . add tahini , chickpeas , cumin , and salt ; purée . spread on bottom of a serving dish , forming a shallow well in center ; fill well with mushrooms and pine nuts . sprinkle with parsley and paprika , and drizzle with olive oil . <end>



Input:  no - onions aloo curry – ( potato curry ) <begin_ingredients> 4 large potatoes – boiled , peeled & chopped into medium size <ing> 2 large tomatoes – chopped into medium size <ing> 2 big green chilies – deseeded & chopped into medium size <ing> 1 cup water – ( less 3 tbsp ) <ing> 2 tbsp oil – any flavorless oil for sautéing everything into a curry <ing> 1 tbsp fresh coriander – chopped for garnishing - optional <ing> 1 pinch turmeric powder – ( indian spice name – haldi ) <ing> 2 whole dry red chili - ( indian spice name – sukhi sabut lal mirch ) <ing> 1½ tsp cumin seed - ( indian spice name – sabut jeera ) <ing> 2 tsp coriander powder - ( indian spice name – dhania powder ) <ing> 2 tsp cumin powder - ( indian spice name – jeera powder ) <ing> 2 tsp red chili powder - ( indian spice name – lal mirch powder ) - optional <ing> ½ tsp black pepper powder - ( indian spice name – kali mirch powder ) <end>

Gend:  <beg> cook pasta according to package directions . drain and set aside . <step> 2 . in a large bowl , combine all ingredients and mix well . <step> 5 . serve with the lettuce salad . <end> , or a

True:  1 . heat oil in the non - stick at high , as soon as the oil is heated , add turmeric , whole dry red chili & cumin seed . <step> 2 . when the cumin seed starts to pop , add chopped tomatoes and stir on high for 1 min . <step> 3 . reduce the heat to medium , add coriander powder , cumin powder , red chili powder , black pepper powder and stir for another 1 min . <step> 4 . now add boiled - chopped potatoes , stir gently to mix all the spices together with potatoes and cook for 3 min . <step> 5 . after 3 min , add water and reduce the heat to lowest and cook for another 4 - 5min . <step> 6 . after 4 - 5min , increase the heat to high and bring to a boil and turn off the heat . <step> 7 . decorate with fresh coriander ( optional ) & serve hot . <end>



Input:  slow cooker charro beans <begin_ingredients> 1 / 2 pound bacon <ing> 1 pound dry pinto beans <ing> 4 cups water <ing> 2 cups beef broth <ing> 6 cloves garlic , minced <ing> fresh jalapeno ( seeds & membranes removed ) , diced , optional <ing> 1 tablespoon cumin <ing> 1 teaspoon garlic powder <ing> 1 teaspoon chili powder <ing> 1 ( 15 - ounce ) can rotel ( diced tomatoes & green chiles ) <ing> 1 cup fresh cilantro leaves , chopped <ing> salt & freshly ground black pepper <ing> salt & freshly ground black pepper <end>

Gend:  <beg> 1 . in a large saucepan , cook the rice noodles according to the package directions . drain and set aside . <step> 2 . in a large skillet , heat the oil over medium - high heat . add

True:  place beans in a colander , rinse well , and remove any stones or shriveled beans . <step> cook bacon until just crispy . drain , chop , and set aside . <step> pour beans into slow cooker . cover with water and beef broth . add chopped bacon , garlic , jalapeño ( if using ) , cumin , garlic powder , and chili powder . cover and cook on low for 7 hours or until almost done . stir in rotel and chopped cilantro , cover , and cook for an additional hour or until tender ( total cooking time will probably be between 8 to 10 hours ) . when beans are done , season with salt and freshly ground pepper , to taste ( i add 1 / 2 teaspoon of salt at a time , tasting in between ) . <end>



Input:  korean barbecued short ribs <begin_ingredients> 4 beef short ribs ( about 1 lb . each ) <ing> 1 / 4 cup soy sauce <ing> 1 / 4 cup dry sherry <ing> 1 tablespoon sugar <ing> 1 tablespoon minced fresh ginger <end>

Gend:  <beg> heat grill to medium - high . <step> combine first 6 ingredients in a small bowl . <step> rub steaks with 1 / 4 teaspoon of the salt and pepper . <step> place salmon on grill rack coated with cooking

True:  1 . trim thick outer layer of fat from ribs ; discard . rinse meat and pat dry . <step> 2 . on the meaty side of each rib , 1 / 4 inch in from the edge opposite the bone , make a cut the length of the rib , down into meat but not quite through it . turn the rib over . about halfway between bone and first cut , make another lengthwise cut almost through meat . flip the rib over again and make a third lengthwise cut parallel to and around the bone almost through meat , leaving bone attached . pull out to unfold meat . <step> 3 . in a 1 - gallon heavy plastic food bag , combine soy , sherry , sugar , and ginger ; shake to mix . add short ribs to bag , seal , and turn over to coat meat with marinade . <step> 4 . set bag in a bowl ; chill at least 30 minutes or up to 1 day , turning several times . remove ribs and discard marinade . <step> 5 . lay ribs , pulled open so meat is flat , on a barbecue grill over a solid bed of hot coals or high heat on a gas grill ( you can hold your hand at grill level only 2 to 3 seconds ) . close lid ; open vents for charcoal . brown meat , turning once , 7 to 10 minutes . transfer to a platter . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.02 

1-gram Score:  56.9
2-gram Score:  15.4
3-gram Score:  4.5
4-gram Score:  1.5