Encoder-Decoder Analysis

In [1]:

    

report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_bow_200_512_dr.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_bow_200_512_dr_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


    

    




Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> (3) -> (4) -> output]
  (1): nn.LookupTable
  (2): nn.Mean
  (3): nn.Linear(200 -> 512)
  (4): nn.Replicate
}
Decoder: 

nn.gModule

In [2]:

    

print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


    

    




('Train Perplexity: ', 7.9244035678905)
('Valid Perplexity: ', 26.120246947783)
('Test Perplexity: ', 24.310379639844)

In [3]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


    

In [4]:

    

%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


    

In [5]:

    

def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')


    

In [6]:

    

for sample in report['train_samples']:
    print_sample(sample)


    

    




Input:  caramelized onion and gorgonzola pizza <begin_ingredients> 1 / 8 cup butter <ing> 2 large vidalia onions , thinly sliced <ing> 2 teaspoons sugar <ing> 1 ( 10 ounce ) package refrigerated pizza dough <ing> 1 pound gorgonzola cheese , crumbled <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  in a large saute pan , melt butter over medium heat . saute onions in butter until the onions are soft and dark brown , approximately 25 minutes . stir in sugar , and continue cooking for 1 or 2 more minutes . <step> preheat oven to 425 degrees f ( 220 degrees c ) . <step> grease a pizza pan or cookie sheet , and press out the dough to desired thickness . spread onions evenly over the dough , and top with crumbled gorgonzola . <step> bake for 10 to 12 minutes , or until done . <end>



Input:  nicole’s walnut - tarragon chicken salad <begin_ingredients> 3 to 4 cups cooked chicken , shredded ( about a 4 pound roast chicken ) <ing> ½ cup low - fat greek yogurt <ing> 1 / 3 cup low - fat mayonnaise <ing> 1 tablespoon finely chopped tarragon <ing> ½ cup roughly chopped walnuts <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  combine all of the ingredients together in a bowl and serve over raisin bread , lettuce or on a nice crispy baguette ! <end>



Input:  coffee and molasses - brined pork chops <begin_ingredients> 2 cups water <ing> 1 1 / 2 cups chilled strong brewed coffee <ing> 1 / 4 cup kosher salt ( such as diamond crystal ) <ing> 3 tablespoons dark brown sugar <ing> 2 tablespoons molasses <ing> 1 tablespoon worcestershire sauce <ing> 1 cup ice cubes <ing> 4 ( 6 - ounce ) bone - in center - cut pork chops ( about 1 / 2 inch thick ) <ing> 1 cup hickory wood chips <ing> 1 / 8 teaspoon kosher salt <ing> 8 teaspoons pepper - garlic spice rub <ing> cooking spray <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  combine first 6 ingredients in a large bowl , stirring until salt and sugar dissolve . pour salt mixture into a large zip - top plastic bag . add ice and pork ; seal . refrigerate for 3 hours , turning the bag occasionally . <step> soak wood chips in water 1 hour . drain well . <step> prepare grill for indirect grilling , heating one side to medium - high and leaving one side with no heat . <step> place wood chips on hot coals . remove pork from bag ; discard brine . pat pork dry with paper towels . sprinkle 1 / 8 teaspoon salt over pork . rub 8 teaspoons pepper - garlic spice rub evenly over both sides of pork . place pork on grill rack coated with cooking spray over medium - high heat ; grill 2 minutes on each side . place pork on unheated side of grill ; grill 3 minutes on each side or until done . remove from heat . let stand 5 minutes before serving . <end>



Input:  quilt pizza <begin_ingredients> 1 pound pizza dough , brought to room temperature <ing> 1 / 2 cup pizza sauce <ing> 2 cups mozzarella cheese , shredded <ing> 1 small red onion , thinly sliced <ing> 1 cup broccoli florets , steamed <ing> 1 / 2 cup pineapple , chopped <ing> 1 / 2 cup cherry tomatoes , halved <ing> 1 red bell pepper , sliced <ing> 1 / 4 cup olives , sliced <ing> 1 / 2 cup mushrooms , sliced <ing> 4 strips bacon , cooked <ing> 1 / 2 cup sun dried tomatoes <ing> 1 / 2 cup spinach leaves <ing> 1 / 2 cup green onions , chopped <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  1 . preheat oven to 425 f. 2 . lightly grease the backside of a half sheet pan with olive oil and stretch the dough to fit the size of the pan ( about 16″ x 11″ ) . 3 . spread 1 / 2 cup of sauce on the dough and sprinkle 2 cups of cheese on top . 4 . “decorate” the pizza using the topping ingredients . divide into 8 - 12 squares to create a quilt . 5 . brush the edges with olive oil and bake for 18 - 20 minutes . 6 . serve . <end>

In [7]:

    

for sample in report['valid_samples']:
    print_sample(sample)


    

    




Input:  easy crockpot lasagna <begin_ingredients> 1 pound ground beef or turkey <ing> 1 ( 26 - ounce ) jar pasta sauce ( keep jar ! do n't rinse and throw out ! ) <ing> 1 ( 10 - ounce ) box of regular lasagna noodles ( i use brown rice noodles ) <ing> 1 ( 16 - ounce ) container ricotta cheese <ing> 8 ounces sliced mozzarella cheese <ing> 4 cups shredded italian - blend cheese <ing> 1 / 4 cup water <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  brown meat with chosen spices in large skillet <step> pour in entire jar of pasta sauce ( save jar , do n't rinse or throw away ! ) <step> simmer on low for a few minutes <step> put a ladle - full of sauce mixture into the bottom of your crockpot . <step> top with a layer of dried lasagna noodles . you 'll have to break them to fit . <step> smear ricotta cheese on top of lasagna noodles <step> layer veggies , spinach , or other fillers on top of the ricotta cheese <step> add sliced mozzarella cheese <step> add a handful of the shredded cheese <step> ladle on some more of the meat sauce <step> top with more dry noodles and repeat layers until crock is to desired capacity . <step> put the 1 / 4 cup of water into your empty pasta jar and shake . pour tomatoey juice over the whole pile of food . <step> cover and cook on low for 8 hours , high for 4 . <step> check noodles an hour before serving , and push the top ones down into the liquid if they are getting too hard or are curling up . <end>



Input:  petite macaroon cups <begin_ingredients> 1 cup butter or 1 cup margarine <ing> 2 ( 3 ounce ) packages cream cheese , neuf . is fine <ing> 2 cups all - purpose flour <ing> 1 ( 14 ounce ) can eagle brand sweetened condensed milk <ing> 2 eggs , beaten <ing> 1 1 / 2 teaspoons vanilla extract <ing> 1 / 2 teaspoon almond extract <ing> 1 1 / 3 cups flaked coconut <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  in a large bowl , beat butter and cream cheese until fluffy ; stir in flour . cover ; chill 1 hour . ( i did n't count the chill time into the prep time ) <step> preheat oven to 375 degrees f. divide dough into quarters . on floured surface shape 1 quarter into smooth ball . divide that ball into 12 balls . place each ball in a 1 3 / 4 inches miniature muffin cup . press evenly on bottom and up side of each cup . repeat with remaining dough . <step> in a mediium bowl , combine eagle brand , eggs , vanilla , and almond extract : mix well . stif in coconut . fill muffin cups three - fourths full . bake 16 to 18 minutes or until slightly browned . cool in pans . remove from pan using small metal spatula or knife . store leftovers loosely covered at room temperature . <end>



Input:  seafood dean martinis <begin_ingredients> 6 ounces lump crabmeat , picked over for pieces of shell <ing> 3 / 4 pound small , cooked shrimp - - deveined , tails removed <ing> 2 shots premium vodka , chilled <ing> grated peel and juice of 1 lemon <ing> salt <ing> 2 ribs celery from the heart , 1 finely chopped with leafy greens , 1 cut into 8 chunks <ing> 2 ribs celery from the heart , 1 finely chopped with leafy greens , 1 cut into 8 chunks <ing> 2 teaspoons prepared horseradish <ing> 2 teaspoons hot sauce ( eyeball it ) <ing> 1 cup chili sauce <ing> freshly ground pepper <ing> 8 jumbo olives with pimientos , for serving <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  in a bowl , combine the crabmeat and shrimp with the vodka , lemon peel and juice ; season with salt . refrigerate for 10 minutes . in another bowl , mix the chopped celery , horseradish , hot sauce , chili sauce and pepper . add the chilled seafood and toss ; season with salt to taste . <step> divide the seafood among 4 chilled martini glasses . skewer the olives and celery chunks on toothpicks and set in the glasses ; set a cocktail fork next to each glass . <end>



Input:  amazing watermelon greek salad with feta <begin_ingredients> 3 / 4 cup red onion , thinly sliced <ing> 1 tablespoon fresh lime juice <ing> 6 cups seedless watermelon , cubed <ing> 3 / 4 cup feta cheese , cubed ( or crumbled ) <ing> 1 / 2 cup black olives , pitted and halved ( or use sliced ripe olives ) <ing> 2 tablespoons extra virgin olive oil <ing> balsamic vinegar , to taste <ing> fresh parsley , chopped , to garnish <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  in a small bowl , cover the onion slices with the lime juice . the acid in the lime juice will help to mellow the onion a bit . <step> in a large salad bowl , combine the watermelon , feta , olives , and onions ( without the lime juice , but save the juice for the vinegrette ) set aside . <step> wisk together the reserved lime juice , olive oil and as much balsamic vinegar to suit your taste and preferences . keep in mind there is quite a bit of flavour in the salad from the feta and olives so just 1 or 2 tbsp usually does the trick . <step> pour vinegrette over melon mixture and toss . garnish with freshly chopped parsley . <end>

In [8]:

    

for sample in report['test_samples']:
    print_sample(sample)


    

    




Input:  healthy sweet and sour chicken <begin_ingredients> 3 - 4 boneless skinless chicken breasts , pounded to ½ inch thickness and chopped into 1 inch pieces <ing> ⅓ cup corn starch <ing> 2 tablespoons oil <ing> 1 red bell pepper , chopped <ing> 1 green bell pepper <ing> ½ white or yellow onion , chopped <ing> ¾ cup sugar <ing> ½ cup apple cider vinegar ( may sub white vinegar ) <ing> 2 tablespoons soy sauce <ing> 1 teaspoon garlic powder <ing> ½ teaspoon onion * salt <ing> ¼ cup ketchup <ing> 1 tablespoon cornstarch + 2 tablespoon cold water <ing> 1 tablespoon cornstarch + 2 tablespoon cold water <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  first prepare the sauce . add sugar , vinegar , soy sauce , garlic powder , onion salt , and ketchup to a medium sauce pan . stir and bring to a boil . in a small bowl whisk together the 1 tablespoon corn starch and cold water until dissolved . add to sauce pan and stir until thickened , then reduce to low heat . <step> add chicken pieces and corn starch to a large ziplock bag . seal and shake to coat chicken . drizzle a large pan or skillet with oil . add coated chicken . saute over medium heat for about 5 minutes . add peppers and onions . continue to saute over medium heat until chicken is browned and cooked through . <step> add sauce to chicken and peppers . stir to coat well . serve warm with cooked rice if desired . <end>



Input:  detox kale salad <begin_ingredients> 1 bunch of kale ( young leaves are best ) <ing> 1 / 2 cup chopped dried cherries <ing> 1 / 4 cup pine nuts <ing> 1 / 4 cup good quality olive oil <ing> 1 / 2 teaspoon of salt <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  wash the kale thoroughly , remove the ribs and tear the leaves into bite sized pieces . add the cherries , pine nuts , olive oil and salt . toss well with tongs to make sure all the leaves are coated with the oil / salt mixture . let this marinate at room temperature for 4 - 6 hours before serving . this is necessary to soften the kale leaves as well as the cherries which release some juice into the oil which creates a sweet and tangy dressing . <step> you can make this a day ahead , but take it out of the refrigerator a few hours before serving . test seasoning right before serving , and add a pinch more salt if necessary . experiment with other dried fruits and nuts if you like–this is my favorite but i also like cranberries and walnuts . five simple ingredients that transform into something so amazing you won’t believe it’s good for you ! <end>



Input:  kinilaw na talaba…oyster in vinegar chili dip , my style <begin_ingredients> fresh or frozen oysters <ing> 1 cup vinegar <ing> 1 medium shallot <ing> 2 red hot chili <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  clean the oysters , you can do it by running hot water <step> over oysters on a strainer or what i did was put it in a <step> plate then add water , drain then repeat the process <step> till the water runs clear . <step> boil the vinegar on the stove or microwave oven <step> put the oysters , shallot and chili on the boiling vinegar . <step> let it cool , then put in the fridge for 1 - 2 hours . <step> then serve kilawin cold . enjoy with hot steamed rice . <end>



Input:  moroccan pot roast with potatoes and carrots <begin_ingredients> nonstick cooking spray <ing> 2 tablespoons tomato paste <ing> 1 / 2 cup low - sodium beef broth <ing> 3 tablespoons all - purpose flour <ing> 1 teaspoon cumin <ing> 1 teaspoon salt <ing> 1 cup dried plums <ing> 3 / 4 cup dried apricots <ing> 1 1 / 2 pounds baby yukon gold potatoes , scrubbed and quartered <ing> 3 carrots , cut into 1 - inch pieces <ing> 1 / 4 teaspoon black pepper <ing> 1 / 4 teaspoon ground cardamom <ing> 1 / 2 teaspoon cinnamon <ing> 1 2 pound top round roast <ing> 2 teaspoons canola oil <ing> 2 / 3 cup red wine <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , sugar , baking powder , salt and pepper . <step> in a separate bowl , whisk together the egg yolks , milk , butter

True:  1 . coat the inside of a slow cooker with cooking spray . combine tomato paste , broth , flour , cumin and 1 / 2 teaspoon salt in cooker . stir in plums , apricots , potatoes and carrots . 2 . combine pepper , cardamom , cinnamon and remaining salt and sprinkle over entire surface of meat . heat oil in large nonstick skillet over medium - high heat ; add meat and brown on all sides , about 8 minutes . transfer meat to slow cooker , placing it on top of vegetables and fruit . add wine to skillet and boil 1 minute ; pour over meat . cook on low 8 hours . 3 . remove meat from pot and let stand 10 minutes before slicing . <end>

In [9]:

    

print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


    

    




Overall Score:  0.02 

1-gram Score:  46.3
2-gram Score:  13.1
3-gram Score:  4.7
4-gram Score:  2