Encoder-Decoder Analysis

Model Architecture


In [1]:
report_file = '/Users/bking/IdeaProjects/LanguageModelRNN/reports/encdec_bow_200_512_nolin_dr.json'
log_file = '/Users/bking/IdeaProjects/LanguageModelRNN/logs/encdec_bow_200_512_nolin_dr_logs.json'

import json
import matplotlib.pyplot as plt
with open(report_file) as f:
    report = json.loads(f.read())
with open(log_file) as f:
    logs = json.loads(f.read())
print'Encoder: \n\n', report['architecture']['encoder']
print'Decoder: \n\n', report['architecture']['decoder']


Encoder: 

nn.Sequential {
  [input -> (1) -> (2) -> (3) -> output]
  (1): nn.LookupTable
  (2): nn.Mean
  (3): nn.Replicate
}
Decoder: 

nn.gModule

Perplexity on Each Dataset


In [2]:
print('Train Perplexity: ', report['train_perplexity'])
print('Valid Perplexity: ', report['valid_perplexity'])
print('Test Perplexity: ', report['test_perplexity'])


('Train Perplexity: ', 15.320760396434)
('Valid Perplexity: ', 27.390526104579)
('Test Perplexity: ', 26.200217102922)

Loss vs. Epoch


In [3]:
%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][1], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][2], label=str(k) + ' (valid)')
plt.title('Loss v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.show()


Perplexity vs. Epoch


In [4]:
%matplotlib inline
for k in logs.keys():
    plt.plot(logs[k][0], logs[k][3], label=str(k) + ' (train)')
    plt.plot(logs[k][0], logs[k][4], label=str(k) + ' (valid)')
plt.title('Perplexity v. Epoch')
plt.xlabel('Epoch')
plt.ylabel('Perplexity')
plt.legend()
plt.show()


Generations


In [5]:
def print_sample(sample):
    enc_input = ' '.join([w for w in sample['encoder_input'].split(' ') if w != '<pad>'])
    gold = ' '.join([w for w in sample['gold'].split(' ') if w != '<mask>'])
    print('Input: '+ enc_input + '\n')
    print('Gend: ' + sample['generated'] + '\n')
    print('True: ' + gold + '\n')
    print('\n')

In [6]:
for sample in report['train_samples']:
    print_sample(sample)


Input:  chocolate dipped coconut macaroons <begin_ingredients> 3 egg whites <ing> 3 tablespoons caster sugar <ing> 3 cups shredded unsweetened coconut ( i like bob’s red mill ) <ing> 1 1 / 2 teaspoons lemon zest <ing> 5 ounces dark chocolate block ( preferably 72 % dark ) , roughly chopped <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  preheat oven to 325 degrees f <step> in a stand mixer with the whisk attachment on high speed , mix egg whites for about 3 minutes until very frothy , slowly add the sugar along the edges of the mixer during the whisking process . after all the sugar has been added , continue to whisk for another 2 minutes ( egg whites will look like marshmallow fluff ) , add the lemon zest . with a wooden spoon or a rubber spatula gently fold in the lemon zest until just combined . add shredded coconut , gently fold in . <step> using a spoon ( an ice cream scooper will deflate the meringue ) , pick up about a tablespoon of the meringue and pack it lightly ( kind of like a meatball ) just so it holds a shape . place macaroons about an inch apart . bake until golden , 25 - 30 minutes . let them cool for 10 - 15 minutes , or until cool to the touch before dipping . <step> set a saucepan over medium heat , filled halfway with water . place a glass , or heatproof bowl on the saucepan . the bowl should rest directly on the saucepan , though it should not touch the water in the saucepan . when the water is simmering add the chocolate to the bowl , stirring occasionally , until melted . dip the bottoms of the macaroons , then place on cooled lined ( to prevent sticking ) baking sheets ( not a cooling rack ) . refrigerate for about an hour before serving . serve chilled . will refrigerate well for up to 3 days in an airtight container . <end>



Input:  beef chili bake <begin_ingredients> 2 teaspoons vegetable oil <ing> 3 cloves garlic , minced <ing> 8 ounces ground beef <ing> 1 medium onion , chopped <ing> 2 tablespoons chili powder <ing> 1 teaspoon cumin <ing> 1 / 8 teaspoon cayenne , optional <ing> 1 teaspoon salt <ing> 1 tablespoon all - purpose flour <ing> 1 14.5 - oz . can diced tomatoes <ing> 1 15.5 - oz . can red kidney beans <ing> 1 8.5 - oz . box corn muffin mix <ing> 2 scallions , white and light green parts , thinly sliced <end>

Gend:  <beg> 1 . preheat oven to 350 degrees . <step> 2 . in a large bowl , combine the flour , baking powder , salt , and salt . <step> in a large bowl , whisk together the flour , baking

True:  1 . preheat oven to 400 ? f . warm oil in a skillet over medium - high heat . saut ? garlic for 1 minute . add beef ; cook , stirring and breaking it up , until no longer pink , about 4 minutes . add onion ; saut ? until softened , about 5 minutes . <step> 2 . stir chili powder , cumin , cayenne , if desired , and salt into beef mixture ; saut ? for 1 minute . add flour ; cook , stirring , for 1 minute . stir in tomatoes with liquid and 1 / 4 cup water ; stir in beans with liquid . cook , stirring , until thickened , about 2 minutes . transfer mixture to an 8 - inch baking dish . <step> 3 . prepare muffin mix as box label directs . fold in scallions . spread muffin batter over beef mixture . bake until top is golden , 20 to 25 minutes . <end>



Input:  blue ribbon caramel apple pie <begin_ingredients> 2 prepared pie crusts <ing> 1 / 2 teaspoon nutmeg <ing> 1 / 2 teaspoon cinnamon <ing> 1 / 2 teaspoon allspice <ing> 12 kraft® caramel squares <ing> 1 teaspoon milk <ing> 1 / 2 cup chopped pecans <ing> 2 ( 21 ounce ) cans lucky leaf® premium apple pie filling <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  place one pie crust in a baking dish . combine spices with lucky leaf apple pie filling . pour apple mixture into crust . <step> melt caramels and milk in microwave - safe bowl for about a minute , stirring twice . pour melted caramels over the lucky leaf apple pie filling . <step> place pecans on top and cover with the second crust . vent crust for baking . brush top of pie crust with the beaten egg wash and sprinkle with 1 teaspoon sugar . <step> bake at 425 degrees for 45 minutes . <end>




In [7]:
for sample in report['valid_samples']:
    print_sample(sample)


Input:  pacific rim glazed flank steak <begin_ingredients> 1 cup prepared teriyaki marinade <ing> ½ cup chopped onion <ing> ⅓ cup honey <ing> ⅓ cup fresh orange juice <ing> 1 tbsp . dark sesame oil <ing> 1 large clove garlic , crushed and peeled <ing> pepper <ing> 1 2 - lb . beef flank steak <ing> orange slices <ing> rosemary sprigs <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  1 . in medium , shallow dish , combine teriyaki marinade , onion , honey , orange juice , oil , garlic , and pepper to taste ; whisk until blended . remove and reserve ¾ cup of marinade for basting . <step> 2 . with sharp knife , lightly score both sides of flank steak in a crisscross pattern . place steak in marinade in dish , turning to coat . cover with plastic wrap and marinate in refrigerator 30 minutes , turning once . <step> 3 . remove steak from marinade ; discard marinade . place steak on grid over medium , ash - covered coals . grill , uncovered , 17–21 minutes for medium rare to medium , basting occasionally with reserved marinade and turning once . <step> 4 . place remaining basting marinade in small saucepan ; place on grid of grill and bring to a boil . meanwhile , carve steak diagonally across the grain into thin slices ; arrange on platter . spoon hot marinade over beef , as desired . garnish with orange slices and rosemary sprigs . <end>



Input:  yakitori 焼き鳥 <begin_ingredients> 10 - 12 5 - inch bamboo skewers <ing> 1 lb . boneless , skinless chicken thighs , at room temperature <ing> 9 scallions <ing> vegetable oil <ing> ½ cup soy sauce <ing> ½ cup mirin <ing> ¼ cup sake <ing> ¼ cup water <ing> 2 tsp . packed brown sugar <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  soak bamboo skewers in water for 30 minutes . <step> in a small saucepan , add the mirin , soy sauce , sake , water , brown sugar , and the green part of 1 scallion , and bring it to a boil over high heat . once boiling , decrease the heat to low and simmer , uncovered , until the liquid is reduced by half . it will take about 30 minutes . the sauce will be thicker and glossy . let it cool to room temperature before using . reserve ⅓ of the sauce in a small bowl for final coating . <step> cut the white and light green part of scallions into 1 inch pieces . <step> cut chicken into 1 inch cubes . <step> on a flat work surface , fold each slice of chicken in half , insert into chicken at 45 degrees angle , and press down on the skewer to pierce thorough the center . <step> alternate each chicken slice with a piece of scallion lined up perpendicular to the skewer . each skewer will hold about 4 chicken slices and 3 scallion pieces . <step> grease the grate of the broiler / wire rack ( or oven - safe cooling rack ) to avoid the chicken sticking on the grate . place the skewers on top . <step> set the broiler to high and wait until the heating elements are hot . broil for 6 minutes . <step> after 6 minutes , brush the sauce on the meat on both sides and continue to broil for 3 - 4 minutes to caramelize the sauce . <step> transfer the skewers to a serving plate and brush the chicken on top with the reserved sauce ( with a clean brush ) . <end>



Input:  escarole salad with melons and crispy prosciutto <begin_ingredients> 4 thin slices prosciutto ( about 1.5 ounces ) , coarsely chopped <ing> 3 tablespoons minced shallots <ing> 2 tablespoons balsamic vinegar <ing> 1 tablespoon red wine vinegar <ing> 1 1 / 2 teaspoons extra virgin olive oil <ing> 1 / 4 teaspoon salt <ing> 1 / 4 teaspoon freshly ground black pepper <ing> 12 cups torn escarole ( about 1 1 / 4 pounds ) <ing> 2 cups torn radicchio ( about 4 ounces ) <ing> 2 cups cubed peeled honeydew melon <ing> 2 cups cubed peeled cantaloupe <ing> 2 tablespoons sliced almonds , toasted <end>

Gend:  <beg> 1 . preheat oven to 350 degrees . <step> 2 . in a large bowl , combine the flour , baking powder , salt , and salt . <step> in a large bowl , whisk together the flour , baking

True:  preheat oven to 400 ? . <step> arrange prosciutto in a single layer on a baking sheet . bake at 400 ? for 6 minutes or until crisp . <step> combine shallots and next 5 ingredients ( shallots through pepper ) in a large bowl , stirring with a whisk . add escarole and radicchio , and toss to coat . add honeydew and cantaloupe , and toss to combine . place 2 cups salad on each of 6 plates , and top each serving with about 2 teaspoons prosciutto . sprinkle each serving with 1 teaspoon almonds . <end>




In [8]:
for sample in report['test_samples']:
    print_sample(sample)


Input:  chocolate mint rice krispies treats <begin_ingredients> 1 / 4 cup butter <ing> 1 10 oz package of marshmallows <ing> 1 / 2 tsp peppermint extract <ing> 5 1 / 2 cups cocoa krispies <ing> 10 oz white chocolate / almond bark <ing> 1 / 4 cup sugar <ing> green food coloring <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  spray a 9 x 13 baking pan with cooking spray . <step> melt butter in a large saucepan over low heat . add marshmallows and mint extract . stir until completely melted . remove from heat . <step> add the cereal into the marshmallow mixture and stir until well coated . <step> use a buttered spatula to evenly press mixture into the 9 x 13 baking pan . <step> while the krispy treats are cooling , mix 3 - 4 drops of food coloring into the sugar . stir well until color is evenly distributed . break up any clumps . <step> once treats have cooled , turn out onto a cutting board and cut into rectangles ? however big you would like them . <step> melt chocolate according to package directions . <step> dip one end of each treat into chocolate and immediately sprinkle with colored sugar . <step> place on wax paper to cool . <end>



Input:  chocolate mint rice krispies treats <begin_ingredients> 1 / 4 cup butter <ing> 1 10 oz package of marshmallows <ing> 1 / 2 tsp peppermint extract <ing> 5 1 / 2 cups cocoa krispies <ing> 10 oz white chocolate / almond bark <ing> 1 / 4 cup sugar <ing> green food coloring <end>

Gend:  <beg> preheat oven to 350 degrees . <step> in a large bowl , combine the flour , baking powder , salt , and salt . <step> add the flour , baking powder , baking soda , salt , and salt .

True:  spray a 9 x 13 baking pan with cooking spray . <step> melt butter in a large saucepan over low heat . add marshmallows and mint extract . stir until completely melted . remove from heat . <step> add the cereal into the marshmallow mixture and stir until well coated . <step> use a buttered spatula to evenly press mixture into the 9 x 13 baking pan . <step> while the krispy treats are cooling , mix 3 - 4 drops of food coloring into the sugar . stir well until color is evenly distributed . break up any clumps . <step> once treats have cooled , turn out onto a cutting board and cut into rectangles ? however big you would like them . <step> melt chocolate according to package directions . <step> dip one end of each treat into chocolate and immediately sprinkle with colored sugar . <step> place on wax paper to cool . <end>



Input:  a creamy potato gnocchi salad <begin_ingredients> boiling water <ing> 1 pound of potato gnocchi <ing> a couple handsful of frozen petit pois , thawed and drained <ing> 1 small bag of radishes , trimmed and diced ( about 1 / 2 cup ) <ing> 1 small red onion , chopped <ing> 1 large carrot , peeled and chopped <ing> 2 stalks celery , trimmed and chopped <ing> 2 hard boiled eggs , peeled and coarsely chopped <ing> 2 / 3 cup mayonnaise ( 146g ) <ing> 1 / 3 cup salad cream ( 73g ) <ing> the juice of two lemons <ing> 1 tsp sugar <ing> 1 tsp salt <ing> 1 tsp freshly ground black pepper <ing> a small handful of chopped fresh parsley <end>

Gend:  <beg> 1 . preheat oven to 350 degrees . <step> 2 . in a large bowl , combine the flour , baking powder , salt , and salt . <step> in a large bowl , whisk together the flour , baking

True:  bring a large pot of lightly salted water to the boil . add the gnocchi and cook according to package directions . drain well . rinse in cold water . drain again and then tip into a bowl . add all of the vegetables , along with the chopped hard boiled eggs , and stir together to combine . whisk together all of the dressing ingredients . taste and adjust seasoning as desired . fold this dressing into the gnocchi and vegetable mixture . smooth over and cover . chill for about 3 hours before serving . stir again just before serving . if desired sprinkle with some more parsley . <end>



BLEU Analysis


In [9]:
print 'Overall Score: ', report['bleu']['score'], '\n'
print '1-gram Score: ', report['bleu']['components']['1']
print '2-gram Score: ', report['bleu']['components']['2']
print '3-gram Score: ', report['bleu']['components']['3']
print '4-gram Score: ', report['bleu']['components']['4']


Overall Score:  0.02 

1-gram Score:  51.5
2-gram Score:  16.7
3-gram Score:  5.7
4-gram Score:  2.3

In [ ]: