Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_200_512_norem.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_200_512_norem_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> output]
  (1): nn.LookupTable
  (2): nn.LSTM(200 -> 512)
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 9.0876709171962)
('Valid Perplexity: ', 25.55261758451)
('Test Perplexity: ', 24.14035993694)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  thai - style chicken coleslaw <begin_ingredients> 2 small shallots , finely chopped <ing> 1 red chili , fresh , seeded and finely chopped <ing> 2 tablespoons honey <ing> 1 / 4 cup rice vinegar <ing> 1 lime , juice of <ing> 3 tablespoons fish sauce <ing> 3 tablespoons soy sauce <ing> 1 tablespoon chives , finely chopped fresh chives <ing> 2 chicken fillets , cooked and shredded <ing> 1 / 2 a medium cabbage , finely shredded <ing> 2 carrots , medium size , cut into very thin strips <ing> 6 mint leaves , finely chopped <ing> 1 / 4 cup coriander , lightly chopped <ing> 1 / 4 cup peanuts , coarsely chopped <ing> 3 - 4 slices pink pickled ginger ( to garnish ) <end>

Gend:  <beg> 1 . in a large bowl , combine the rice , rice , rice vinegar , rice wine , and rice . stir . add the rice and stir . cover and cook on low for 8 to 10 hours

True:  place the chicken in a large serving bowl . <step> combine the shallots , chilli , honey , rice vinegar , fish sauce , lime juice , soy sauce and chives . pour the mixture over the shredded chicken . keep mix in fridge until ready to serve . <step> when ready to serve add the cabbage , carrot , mint leaves , coriander and peanuts to the chicken mix . <step> toss lightly until well combined and finish with a garnish of pink pickled ginger . <step> serve the salad well chilled . <end>



Input:  perfect vanilla bean rice pudding <begin_ingredients> 1 / 2 vanilla bean <ing> 1 / 2 cup ( approximately ) plain , cooked rice <ing> 2 cups milk ( pref . low fat ) <ing> 1 / 2 cup heavy cream <ing> 1 large egg <ing> 1 / 3 cup sugar <end>

Gend:  <beg> 1 . in a large bowl , stir together the flour , baking powder , sugar , salt , and baking soda . add the eggs , milk , vanilla , and oil . beat until smooth . <step> 2

True:  take a medium saucepan . split the vanilla bean in half and scrape out the seeds . combine vanilla bean seeds with rice , milk and cream and place in the saucepan . bring milk mixture to a simmer over low heat , stirring occasionally , and cook until the liquid reduces by at least half , about 30 minutes . the mixture should be roughly 2 / 3 rice and 1 / 3 liquid . <step> in a medium bowl , whisk together the egg and sugar until smooth . pour in a small amount ( approximately 1 / 2 cup ) of the milk from the rice while whisking constantly . stirring with a whisk , stream the sugar mixture back into the large pot . cook over medium - low heat until the pudding bubbles and thickens enough to coat the back of a spoon . remove from heat and transfer to a separate bowl to cool . <step> store , covered , in the refrigerator . <end>



Input:  sweet & spicy chinese chicken <begin_ingredients> 2 bunches green onions <ing> 1 / 4 cup olive oil <ing> 1 tablespoon minced garlic <ing> 12 oz pineapple juice <ing> 1 / 4 cup chili garlic sauce <ing> 1 / 4 cup cider vinegar <ing> 1 / 4 cup brown sugar <ing> 2 tablespoons soy sauce <ing> 3 - 4 pounds boneless , skinless chicken <end>

Gend:  <beg> mix all ingredients together and pour over chicken . <step> cover and cook on low for 8 hours . <step> remove chicken from slow cooker and shred with a fork . <step> place ribs on grill and cook for 5

True:  instructions chop green onion ; set aside . in large bowl , whisk together olive oil , garlic , pineapple juice , chili sauce , vinegar , brown sugar , green onions and soy sauce . divide chicken into two gallon size freezer bags ( be sure to label bags first ! ) divide sauce into bags over top of chicken . freeze until needed . thaw ; pour contents into slow cooker . cook on low for 4 - 5 hours or high for 2 - 3 hours , or place in casserole dish and bake at 400 degrees for 30 - 40 minutes . serve over cooked rice . <end>



Input:  creamy chipotle shrimp <begin_ingredients> 1 lb raw shrimp , peeled & devined <ing> 1 cup diced yellow onion <ing> 1 1 / 2 cups canned corn <ing> 1 / 4 cup fat free cream cheese ( i used philadelphia ) <ing> 2 tbsps chipotle peppers in adobo ( i used la costena ) <ing> 1 / 4 cup whole wheat panko crumbs ( i used ian 's ) <ing> 1 / 3 cup shredded reduced fat cheddar cheese ( i used sargento ) <ing> 1 1 / 2 tsps extra virgin olive oil <ing> 1 / 2 tsp lime juice <ing> 2 tsps cilantro <ing> 1 tsp sea salt & coarse black peppercorn - approx . <end>

Gend:  <beg> 1 . combine all ingredients in a food processor and pulse until well combined . <step> 2 . heat oil in a large skillet over medium heat . add garlic and cook until fragrant , about 1 minute . add

True:  1 . in food processor puree chipotle peppers until smooth . transfer 2 tbsps puree into small bowl , then combine with cream cheese and lime juice , stirring thoroughly . next , add shrimp to large bowl before seasoning with salt and pepper . pour over with sauce , then allow to marinate in refrigerator for at least 1 hour . <step> 2 . before cooking , in small bowl combine panko crumbs , cheddar cheese , cilantro , and 1 / 2 tsp evoo . set aside . <step> 3 . in oven - proof skillet sauté onions in remaining tsp evoo over medium - high heat until soft and translucent . next stir in corn , sautéing for several minutes before adding in shrimp . cook until shrimp just begins to turn pink on one side , then remove from heat . <step> 4 . evenly sprinkle panko mixture atop skillet , then bake at 350f for 12 - 15 minutes until crispy and browned . immediately plate and serve . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  salmon burger with baby spinach <begin_ingredients> 2 hamburger buns <ing> 1 can ( 7.5 ounces ) salmon , drained and flaked <ing> 1 / 2 celery stalk , finely chopped ( 2 tablespoons ) <ing> 1 / 2 shallot , finely chopped ( 1 tablespoon ) <ing> 1 tablespoon mayonnaise , plus more for serving <ing> coarse salt and ground pepper <ing> coarse salt and ground pepper <ing> 2 teaspoons extra - virgin olive oil <ing> baby spinach , for serving <end>

Gend:  <beg> in a large bowl , combine the chicken , onion , garlic , onion , garlic , and red pepper flakes ; season with salt and pepper . <step> in a large skillet , heat the oil over medium -

True:  in a food processor , pulse 1 bun until fine crumbs form ( you should have about 1 / 4 cup ) . transfer breadcrumbs to a large bowl and add salmon , celery , shallot , and mayonnaise . season with salt and pepper ; mix well to combine . form into a 3 / 4 - inch - thick patty , transfer to a plate , and refrigerate 10 minutes . <step> in a medium nonstick skillet , heat oil over medium . add patty and cook , flipping once , until browned and heated through , about 8 minutes . <step> meanwhile , split and toast remaining bun . serve burger on bun with mayonnaise and spinach . <end>



Input:  salmon burger with baby spinach <begin_ingredients> 2 hamburger buns <ing> 1 can ( 7.5 ounces ) salmon , drained and flaked <ing> 1 / 2 celery stalk , finely chopped ( 2 tablespoons ) <ing> 1 / 2 shallot , finely chopped ( 1 tablespoon ) <ing> 1 tablespoon mayonnaise , plus more for serving <ing> coarse salt and ground pepper <ing> coarse salt and ground pepper <ing> 2 teaspoons extra - virgin olive oil <ing> baby spinach , for serving <end>

Gend:  <beg> in a large bowl , combine the chicken , onion , garlic , onion , garlic , and red pepper flakes ; season with salt and pepper . <step> in a large skillet , heat the oil over medium -

True:  in a food processor , pulse 1 bun until fine crumbs form ( you should have about 1 / 4 cup ) . transfer breadcrumbs to a large bowl and add salmon , celery , shallot , and mayonnaise . season with salt and pepper ; mix well to combine . form into a 3 / 4 - inch - thick patty , transfer to a plate , and refrigerate 10 minutes . <step> in a medium nonstick skillet , heat oil over medium . add patty and cook , flipping once , until browned and heated through , about 8 minutes . <step> meanwhile , split and toast remaining bun . serve burger on bun with mayonnaise and spinach . <end>



Input:  huevos rancheros <begin_ingredients> 8 inch flour tortilla ( traditional recipe uses corn ) <ing> refried beans <ing> salsa <ing> guacamole <ing> 2 eggs <ing> grated cheese ( your preference or whatever you have ) <end>

Gend:  <beg> 1 . preheat the oven to 350 degrees . <step> 2 . in a large bowl , mix together the flour , baking powder , salt and baking soda . <step> 3 . in a large bowl , combine the

True:  heat a tablespoon or two of oil in a large skillet over medium heat . add tortilla and cook on both sides until just lightly browned and crispy . remove to plate . add eggs to skillet and cook to desired doneness . i cook mine until they are just slightly runny in the center but my husband prefers his over well ( solid centers ) . <step> spread a generous amount of refried beans over crispy tortilla . top with a generous spread of salsa and follow that with guacamole . <step> place cooked eggs on top and sprinkle with salt and pepper , then cheese . you can put the plate under the broiler to melt the cheese if you like . garnish with more salsa and guacamole . enjoy ! <end>



Input:  stuffed baked mussels <begin_ingredients> 3 pounds prince edward island mussels , about 50 to 60 <ing> 1 / 2 cup dry white wine <ing> 2 - 1 / 2 cups basic tomato sauce <ing> 1 cup fresh basil leaves <ing> 1 cup fresh bread crumbs <ing> 1 / 2 cup extra virgin olive oil <end>

Gend:  <beg> preheat oven to 350 degrees f ( 175 degrees c ) . <step> in a large bowl , combine the chicken , onion , garlic , red pepper , and onion . stir to combine . <step> in a large

True:  mussel <step> baking <step> fall <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  mediterranean sandwich <begin_ingredients> 2 tsps . olive oil <ing> 1 tsp . balsamic vinegar <ing> 1 chicken breast <ing> 1 crusty bread , sliced <ing> 3 sun dried tomatoes , chopped <ing> feta cheese , sliced ( or goat cheese ) <ing> basil ( optional ) <end>

Gend:  <beg> heat the oil in a large skillet over medium heat . add the onion and cook , stirring occasionally , until the onions are soft and translucent , about 5 minutes . add the garlic and cook for 1 minute

True:  mix the olive oil and balsamic vinegar . set aside . <step> cook the chicken breast and cut into strips . <step> warm the bread on both sides . <step> drizzle some of the dressing onto the bottom half of the bread , then add the chicken , tomatoes , cheese , and basil . pour the rest of the dressing on top , cover with the top half of the bread , and serve . <end>



Input:  easy homemade pizza sauce <begin_ingredients> 1 ( 6 ounce ) can tomato paste <ing> 1 / 3 cup water <ing> ¼ teaspoon dried basil <ing> ¼ teaspoon chili powder <ing> ¼ teaspoon cumin <ing> 1 / 8 teaspoon garlic powder <ing> 1 / 8 teaspoon onion powder <ing> ½ teaspoon dried oregano <ing> 1 teaspoon paprika <ing> ½ teaspoon red chile flakes , optional <ing> 1 / 8 teaspoon salt <ing> 1 teaspoon sugar <end>

Gend:  <beg> combine all ingredients in a large bowl . <step> place ribs in a greased 9 x 13 inch baking dish . <step> bake at 350 degrees for 30 minutes . <step> remove from oven and let cool . <step> combine

True:  mix all ingredients together in a medium mixing bowl , and refrigerate at least one hour before using . <end>



Input:  mediterranean sandwich <begin_ingredients> 2 tsps . olive oil <ing> 1 tsp . balsamic vinegar <ing> 1 chicken breast <ing> 1 crusty bread , sliced <ing> 3 sun dried tomatoes , chopped <ing> feta cheese , sliced ( or goat cheese ) <ing> basil ( optional ) <end>

Gend:  <beg> heat the oil in a large skillet over medium heat . add the onion and cook , stirring occasionally , until the onions are soft and translucent , about 5 minutes . add the garlic and cook for 1 minute

True:  mix the olive oil and balsamic vinegar . set aside . <step> cook the chicken breast and cut into strips . <step> warm the bread on both sides . <step> drizzle some of the dressing onto the bottom half of the bread , then add the chicken , tomatoes , cheese , and basil . pour the rest of the dressing on top , cover with the top half of the bread , and serve . <end>



Input:  chipotle ranch taco salad <begin_ingredients> 3 / 4 lb . lean ground beef <ing> 1 can ( 15.5 oz . ) low sodium red kidney beans , rinsed and drained <ing> 1 / 2 cup wish - bone® ranch dressing , divided <ing> 6 corn tortillas , cut into thin strips <ing> 8 cups torn romaine lettuce leaves <ing> 2 medium tomatoes , chopped <ing> 1 / 2 cup shredded low fat cheddar cheese ( about 2 oz . ) <ing> 1 / 2 cup chopped red onion <ing> 1 / 2 tsp . ground chipotle chile powder , divided <end>

Gend:  <beg> 1 . combine first 4 ingredients in a small bowl ; stir in soy sauce . cover and chill 2 hours . <step> 2 . preheat oven to 400 ° . <step> 2 . combine 1 / 4 cup flour

True:  preheat oven to 400 & deg ; . arrange tortilla strips on baking sheet . bake , stirring twice , until crisp , about 10 minutes . reserve handful of tortillas for garnish . <step> meanwhile , brown ground beef with 1 / 4 teaspoon chile powder in 12 - inch nonstick skillet over medium - high heat . remove skillet from heat , then stir in beans and 1 / 4 cup wish - bone & reg ; ranch dressing ; set aside to cool . <step> layer 3 cups lettuce , 1 / 3 of the tomato and onion and 1 / 2 of the beef mixture in trifle dish or deep glass bowl . sprinkle with 1 / 2 of the tortilla strips and 1 / 2 of the cheese ; repeat layers . top with remaining 2 cups lettuce , remaining tomatoes , onion and cheese . sprinkle with reserved tortilla strips . <step> combine remaining 1 / 4 teaspoon chile powder with remaining 1 / 4 cup dressing and serve with salad . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.02 

1-gram Score:  49.6
2-gram Score:  15.8
3-gram Score:  6.2
4-gram Score:  2.4