Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_250_800.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_250_800_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> output]
  (1): nn.LookupTable
  (2): nn.LSTM(250 -> 800)
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 6.4398729601483)
('Valid Perplexity: ', 29.048134934015)
('Test Perplexity: ', 27.320529810014)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  easy cheesy tuna casserole <begin_ingredients> 1 ( 7 1 / 4 ounce ) package kraft macaroni & cheese dinner mix <ing> 1 / 4 cup butter ( not spread or tub product ) or 1 / 4 cup margarine ( not spread or tub product ) <ing> 1 / 4 cup butter ( not spread or tub product ) or 1 / 4 cup margarine ( not spread or tub product ) <ing> 1 / 4 cup milk <ing> 1 ( 10 3 / 4 ounce ) can condensed cream of celery soup <ing> 1 / 2 cup milk <ing> 1 ( 6 ounce ) canned tuna , drained <ing> 1 / 2 medium red pepper , chopped <ing> 2 green onions , sliced <ing> 1 cup crushed potato chip <end>

Gend:  <beg> preheat oven to 350 degrees f ( 175 degrees c ) . <step> in a large skillet over medium heat , cook ground beef and onion until tender . stir in the ground beef and onion ; cook , stirring occasionally , until browned , about 8 minutes . <step> stir in the tomatoes ,

True:  preheat oven to 350f ° <step> prepare the mac - n - cheese dinner according to package directions . <step> stir in soup , milk , tuna , red pepper and onion . <step> spoon into 2 - quart casserole dish ; top with crushed chips . <step> bake 30 minutes or until heated through . <end>



Input:  carbonara pizza <begin_ingredients> all - purpose flour and cornmeal <ing> all - purpose flour and cornmeal <ing> 1 pound whole - wheat pizza dough , room temperature <ing> 1 tablespoon olive oil <ing> 1 / 8 teaspoon pepper <ing> 1 cup skim milk <ing> 1 / 4 cup finely grated parmesan <ing> 1 / 3 cup frozen peas , thawed <ing> 2 slices prosciutto , chopped or torn <end>

Gend:  <beg> 1 . preheat oven to 350f ° . <step> 2 . combine first 3 ingredients in a large bowl . <step> 2 . stir together flour , baking powder , and salt . <step> 3 . add to the prepared baking dish and bake at 350 ° for 40 minutes or until a toothpick inserted

True:  1 . place an oven rack on lowest position and preheat to 500f ° . <step> 2 . on a lightly floured surface , roll dough into a 14 - inch round . generously sprinkle a baking sheet with cornmeal . place dough on sheet . <step> 3 . bake until dough feels dry , about 8 minutes . <step> 4 . heat oil in a pan on medium - high . whisk in 1 1 / 2 tbsp . flour and pepper . cook until bubbling , about 2 minutes . whisk in milk , bring to a boil , reduce heat and simmer , whisking , until thickened , about 2 minutes . stir in parmesan and peas . <step> 5 . spoon mixture on crust , leaving a 1 / 2 - inch border . bake until crust is crisp and topping is bubbling , about 5 minutes . top with prosciutto . cut into 8 slices . serve . <end>



Input:  lighter lasagna <begin_ingredients> 9 - 12 lasagna noodles , cooked ( or use the kind you do n't have to cook first , just add some extra water to the sauce ) <ing> 1 / 2 lb extra lean ground beef , browned and drained <ing> 1 ( 27 ounce ) jar prepared pasta sauce <ing> 1 ( 14 ounce ) can stewed tomatoes <ing> 8 ounces fat - free ricotta cheese <ing> 8 ounces low - fat firm tofu <ing> 2 cups grated lowfat mozzarella cheese <ing> 1 / 4 cup grated parmesan cheese <ing> 2 minced garlic cloves <ing> 1 tablespoon dried basil <ing> 2 teaspoons dried oregano <ing> 1 / 4 teaspoon black pepper <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large skillet , heat oil over medium heat . add onion and garlic ; cook 3 minutes , stirring frequently . <step> add broth , water , and tomatoes ; bring to a boil . <step> reduce heat and simmer , uncovered , for 15 minutes

True:  preheat oven to 350 degrees . <step> in a large mixing bowl , thoroughly combine tofu , ricotta , and herbs . set aside . <step> in another bowl , combine pasta sauce , stewed tomatoes and ground beef . <step> lightly spray a 13 x 9 inch baking dish with nonstick coating . place 3 - 4 noodles in the bottom . spread with one - third of the cheese mixture and top with one third of the tomato sauce mixture . sprinkle with one - third of the mozarella . repeat , creating two more layers . top with parmesan cheese . <step> bake at 350 degrees for 1 hour . let stand 10 minutes before serving . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  spinach with chickpeas <begin_ingredients> 2 pounds baby spinach <ing> 3 tablespoons extra - virgin olive oil , divided <ing> 1 medium red onion , finely chopped <ing> 5 cloves garlic , minced <ing> 1 19 - ounce can chickpeas , rinsed <ing> 1 1 / 2 teaspoons dried thyme <ing> 1 1 / 2 teaspoons dried oregano <ing> 1 1 / 2 teaspoons ground cumin <ing> 1 teaspoon kosher salt <ing> 1 / 2 teaspoon hot paprika <ing> 1 / 2 cup golden raisins <ing> 1 / 2 cup reduced - sodium chicken broth , or vegetable broth <end>

Gend:  <beg> to make the sauce : in a small bowl , combine the soy sauce , garlic , brown sugar , and chili powder . <step> rub the mixture over the pork chops . <step> heat the oil in a large skillet over medium - high heat . cook the steaks for 5 minutes on each

True:  1 . rinse spinach and let drain in a colander . with water still clinging to it , place half the spinach in a dutch oven over medium heat . cook , tossing with tongs and adding the remaining spinach by the handful until all is added and wilted , 6 to 8 minutes . drain in the colander . let cool slightly , then coarsely chop . <step> 2 . carefully wipe out the pan , then heat 1 tablespoon oil over medium heat . add onion and garlic and cook , stirring , until the onion is tender and lightly browned , 8 to 10 minutes . stir in chickpeas , thyme , oregano , cumin , salt and paprika . using a potato masher , mash some of the chickpeas , then cook , stirring , for 3 minutes . stir in raisins and broth , scraping up any browned bits . add the chopped spinach and stir gently to combine . remove from the heat and let stand 10 minutes . drizzle with the remaining 2 tablespoons oil just before serving . <end>



Input:  hamburger muffins <begin_ingredients> 3 tablespoons butter , softened <ing> 12 slices white bread <ing> 1 1 / 4 pounds ground beef <ing> 1 egg <ing> 1 small onion , chopped <ing> 1 ( 10.75 ounce ) can condensed cream of mushroom soup <ing> salt and pepper to taste <ing> salt and pepper to taste <ing> 3 / 4 cup shredded cheddar cheese <end>

Gend:  <beg> preheat oven to 375 degrees . <step> in a large skillet , cook bacon over medium heat until crisp , about 5 minutes . <step> add garlic and cook for 2 minutes , stirring occasionally . <step> add tomatoes and cook for another minute . <step> add cream , salt and pepper , and simmer

True:  preheat the oven to 350 degrees f ( 175 degrees c ) . butter one side of each slice of bread , and press each slice butter - side down into the cups of a muffin tin . <step> in a medium bowl , mix together the ground beef , egg , onion , cream of mushroom soup , salt and pepper until well blended . fill each bread cup with the mixture . sprinkle shredded cheddar cheese over the tops . <step> bake for 30 minutes in the preheated oven , or until meat is cooked through . <end>



Input:  vietnamese spring roll <begin_ingredients> 1 pound of ground pork <ing> 1 / 2 pound shrimp , peeled and de - veined <ing> 1 / 2 cup dried wood ear mushrooms , soaked in hot water and finely chopped <ing> 1 cup shredded carrots <ing> 1 cup shredded taro root <ing> 1 / 2 small onion , finely diced <ing> 3 - 4 cloves of garlic , minced <ing> 1 shallot , minced <ing> 1 tablespoon sugar <ing> 1 tablespoon fish sauce <ing> 1 teaspoon salt <ing> 2 teaspoons black pepper <ing> 1 package spring roll wrappers ( i like tyj spring roll pastry ) <end>

Gend:  <beg> for the sauce : combine all ingredients in a small bowl and mix well . <step> place a rack over a baking sheet . roast for 20 minutes , or until the fish flakes easily with a fork . <end> the meat is cooking on the stove top . <end> the meat will not be

True:  <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  peanut butter jelly ice cream <begin_ingredients> 250ml ( 1 cup ) almond milk <ing> 4 tbsp smooth peanut butter <ing> 30g ( 1 / 4 cup ) ground almonds <ing> 1 tsp vanilla extract <ing> 3 tbsp maple syrup ( or raw honey if on a raw diet ) <ing> 150g ( 1 cup ) fresh strawberries , halved <ing> 2 tsp chia seeds <ing> 1 tbsp of maple syrup / honey , or more to taste <end>

Gend:  <beg> 1 . in a large bowl , combine the cake mix , oil , and water with the salt and mix well . <step> 2 . add the coconut oil and stir to combine . <step> 3 . pour the mixture into the prepared pan and bake for 20 minutes . <step> 8 . remove

True:  to make the jelly <step> simply blend the strawberries , sweetner and chia seeds together until thick and smooth . leave to the side for later . <step> to make the ice cream <step> mix all the ingredients in a blender until smooth . <step> pour the mixture into your ice cream maker and follow instructions . <step> once the ice cream has churned , transfer a third of the ice cream into a freezer - safe container . <step> drizzle on top a tsp of the strawberry jelly and swirl around . <step> smooth on another layer of the ice cream and repeat until all the ice cream is in the container and you have visible strawberry swirls . <step> freeze for approx 3 hours ( or more ) until the ice cream has reached desired consistency . <step> serve and enjoy ! <end>



Input:  oven baked fancy wild rice pilaf <begin_ingredients> 4 tablespoons butter , melted <ing> 1 ( 10 3 / 4 ounce ) can condensed french onion soup <ing> 1 ( 14 ounce ) can beef broth <ing> 1 ( 7 ounce ) can mushroom stems and pieces , drained <ing> 1 ( 6 ounce ) can diced water chestnuts <ing> 1 cup brown rice <ing> 1 / 2 cup wild rice , rinsed <ing> garlic salt <end>

Gend:  <beg> heat oven to 350 degrees . <step> in a large bowl , combine the chicken , soup , sour cream , and taco seasoning . <step> mix in the chicken breast with the sauce . <step> bake in the preheated oven for 20 minutes . <step> remove from oven and let cool . <step> combine

True:  1 . preheat oven to 375 degrees . <step> 2 . in a large mixing bowl , combine all ingredients . <step> transfer mixture to a 2 quart casserole dish . <step> cover and bake for about 1 hour , until all liquid has been absorbed and rice is tender . very easty to double as i do it all the time . <end>



Input:  seashore chowder <begin_ingredients> 6 slices bacon , cut into 1 - inch pieces <ing> 2 cups red potatoes , 1 / 2in cubes <ing> 1 / 2 cup onion , chopped <ing> 1 / 2 cup celery , chopped <ing> 3 tablespoons butter or 3 tablespoons margarine <ing> 3 tablespoons flour <ing> 3 cups milk <ing> 1 ( 8 ounce ) can oysters , drained , rinsed <ing> 1 ( 6 1 / 2 ounce ) can clams , drained , rinsed <ing> 2 teaspoons basil leaves <ing> 1 pinch cayenne pepper <ing> 1 / 4 cup fresh parsley , chopped <end>

Gend:  <beg> in a large pot of boiling salted water cook pasta according to package instructions ; drain and set aside . <step> in a large skillet , heat the oil over medium heat . add the onion and cook , stirring occasionally , until the onion is translucent , about 5 minutes . <step> add the

True:  in 3 - qt saucepan cook bacon over medium heat for 5 minutes , add potatoes , onions and celery , continue cooking stirring often until potatoes are tender about 15 to 20 minutes . remove from pan and set aside . <step> in same pan melt butter , stir in flour until smooth and bubbly . <step> add potato mixture and remaining ingredients except parsley . <step> cook over medium heat , stirring often for 10 to 15 minutes . <step> stir in parsley and serve . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.14 

1-gram Score:  43.8
2-gram Score:  14.7
3-gram Score:  6.8
4-gram Score:  3.6