In [13]:
import pandas as pd;
import numpy as np;

In [14]:
regular_tweets = pd.DataFrame(open('dev_test_tweets/nonsarcastic.csv').readlines())
sarcastic_tweets = pd.DataFrame(open('dev_test_tweets/sarcastic.csv').readlines())

In [15]:
# ALL CAPS COUNT
sarcastic_tweets['AllCapsCount'] = [sum([k.isupper() for k in n.split(" ")]) for n in sarcastic_tweets[0]]
regular_tweets['AllCapsCount'] = [sum([k.isupper() for k in n.split(" ")]) for n in regular_tweets[0]]

In [16]:
sarcastic_tweets


Out[16]:
0 AllCapsCount
0 Interesting, I seem to remember one @BarackOba... 2
1 You all should show the #deplorables, #tacobow... 0
2 I thought Trump was talking about the 50's, wi... 2
3 I woke up today in #whitetrashnation. You had ... 3
4 First the #deplorables came for #Chinese busin... 2
5 @jojoh888 They worked hard for their money bu... 0
6 Figuring Out If When His #AltRight and #Deplor... 0
7 The funniest thing about these #deplorables is... 0
8 @JackieTartamell @antcav94 @threelilbirdsss ya... 0
9 Good luck to everyone in America #deplorables ... 0
10 Is this too much? @3WV @tawnehayes #deplorables\n 1
11 This is a dark day for America. Congratulation... 0
12 Ironic that the so called #deplorables aren't ... 0
13 I never thought losing could be so much fun!! ... 1
14 The deplorable faceless trolls are out in forc... 0
15 Thanks Colin for not noticing that I took your... 2
16 Just finished rewatching 5 hours worth of O Ch... 1
17 I'm surprised liberals haven't started crashin... 0
18 Remember when the #TrumpTrain was putting all ... 0
19 This person doesn't want your support; Let him... 0
20 I only voted for this cool sticker #jk #TrumpT... 1
21 Oh look I can't push a button correctly #voter... 1
22 Could Anonymous and Wikileaks be combating the... 0
23 How can you be a president when you don't even... 1
24 I hope it's a landslide. Red across America #T... 1
25 You mean the debates where @HillaryClinton was... 0
26 Like really on top of the stress of pharmacy s... 1
27 Don't you love it when Canadians tell you what... 0
28 I'm wondering if Grahams name is on the Lolita... 0
29 You let the hate win, the racism this is who y... 0
... ... ...
70 #HillaryClinton could still become #POTUS ~ #... 2
71 I guess the airlines will be busy transporting... 4
72 It's NOT a #concession speech! #HillaryClinton... 4
73 All of a sudden, I am supposed to believe that... 2
74 Maybe the rich will be very competent and spur... 1
75 Ah great. The fascism is starting. Loss of cit... 1
76 Remember pics like this Donald doesn't want sh... 1
77 Now that we've addressed our daily #PEOTUS tro... 2
78 #PEOTUS, you won the electoral college vote fa... 1
79 When I see #PEOTUS I read fetus, appropriate a... 3
80 Its freezing and snowing in New York we need ... 1
81 Jackie O and Tacky Hoe #ImWithHer \n 1
82 Dear @ladygaga I'm sure u were waiting to give... 0
83 This is the person you've voted for ! Congratu... 0
84 just gonna let this sit here. congrats america... 1
85 When my coworker greeted me today with "Mornin... 1
86 oh you have no idea what has america come too,... 0
87 I really thought the United States was united ... 1
88 The playlist on @radioeins is really soothing ... 1
89 We had one job, America #LOVETRUMPSHATE\n 1
90 @enews well, if @realDonaldTrump won, isn't th... 0
91 I guess today I'll eat all my feelings. I'm st... 1
92 47% of Americans didn't vote and we got this. ... 0
93 3 more followers lost who next? #Notmypresident\n 0
94 This is awesome! Glad he didn't get caught. He... 0
95 @DRUDGE_REPORT @politico Awesome! I was just w... 3
96 Awesome let's maintain that relationship as we... 0
97 Really? #ObamaDay? What's next #JamesBuchananD... 1
98 @HillaryClinton A promoted tweet this late Hil... 1
99 @JohnKasich... great call. Your write-in won @... 1

100 rows × 2 columns


In [17]:
# '\\ud83d\\ude1c' in set(sarcastic_tweets['0'][11].split(" "))

In [18]:
# master = sarcastic_tweets.append(regular_tweets)
# # master.to_csv('test_tweets_df.csv')

In [39]:
# ToUser
sarcastic_tweets["ToUser"] = sarcastic_tweets[0].map(lambda x: 1 if "@" in x else 0);
regular_tweets["ToUser"] = regular_tweets[0].map(lambda x: 1 if "@" in x else 0);

# Hashtag count
sarcastic_tweets["Hashtags"] = sarcastic_tweets[0].map(lambda x: 1 if "#" in x else 0);
regular_tweets["Hashtags"] = regular_tweets[0].map(lambda x: 1 if "#" in x else 0);

# Labels
sarcastic_tweets['label'] = 'sarcastic'
regular_tweets['label'] = 'genuine'

In [ ]:
master = sarcastic_tweets.append(regular_tweets)
master.to_csv('test_tweets_df.csv')