In [ ]:
from SEARCH_BING_MODULE import bingURL
from SEARCH_YAHOO_MODULE import yahooURL
from SVO_SENT_MODULE_spacy import SVOSENT
import time
In [ ]:
Subject='North Korean'
CAMEO= 'threaten unconventional attack'
Object=''
sentence=Subject+' '+CAMEO+' '+Object # subject+cameo code
In [3]:
bing=bingURL()
t0=time.time()
urls=bing.search_urls(sentence,3)
t1=time.time()
print('time cost:',t1-t0)
print('#urls found:',len(urls))
time cost: 21.602851629257202
#urls found: 55
In [4]:
import SCRAPER
articles=SCRAPER.scrape_from_urls(urls)
The article: As North Korean threat grows, Hawaii prepares for nuclear attack is downloaded and parsed
The article: Sony hack: North Korea threatens US as row deepens is downloaded and parsed
The article: What Next for North Korea, Guam and Trump? Experts Weigh In is downloaded and parsed
The article: is downloaded and parsed
The article: The Guardian is downloaded and parsed
The article: North Korea threatens Guam, Hawaii, rest of U.S. is downloaded and parsed
The article: Mattis: threat of North Korea nuclear attack 'is accelerating' is downloaded and parsed
The article: North Korea to US: if you attack us, we’ll respond with nukes is downloaded and parsed
The article: Breaking National and World News is downloaded and parsed
The article: North Korea video shows rocket attack on Seoul's presidential building is downloaded and parsed
The article: North Korea again threatens nuclear strikes on U.S., South Korea is downloaded and parsed
The article: World news and comment from the Guardian is downloaded and parsed
The article: North Korea's Plenty Scary Without an Overhyped EMP Threat is downloaded and parsed
The article: North Korea Threatens an Electromagnetic Pulse Attack is downloaded and parsed
The article: First Fire, Then Fury, Now North Korea Threatens Missile Attack on U.S. Military Base in Guam is downloaded and parsed
The article: Op-Ed: How to neutralize North Korea's nuclear threat without starting a world war is downloaded and parsed
The article: North Korea threatens ‘merciless strike’ on US is downloaded and parsed
The article: US News and World Report is downloaded and parsed
The article: North Korea threatens nuclear strike, U.N. expands sanctions is downloaded and parsed
The article: War with North Korea inches closer is downloaded and parsed
The article: U.S. Says North Korea Rhetoric a Prelude to Attack is downloaded and parsed
The article: Trump Threatens ‘Fire and Fury’ Against North Korea if It Endangers U.S. is downloaded and parsed
The article: North Korea LIVE: N Korea says 4.7 million volunteers ready to join military is downloaded and parsed
The article: North Korea’s new ICBMs change little about threat to the US is downloaded and parsed
The article: North Korea Threatens US Base With Nukes: Kim Jong Un Regime Warns Of 'Uncontrollable' Nuclear War is downloaded and parsed
The article: North Korea video shows rocket attack on Seoul's presidential building is downloaded and parsed
The article: Mattis: North Korea nuclear threat accelerating is downloaded and parsed
The article: We Can Stop North Korea From Attacking Us. All We Have to Do Is Not Attack Them. is downloaded and parsed
The article: North Korea's Sea-Based Nuclear Capabilities is downloaded and parsed
The article: Hypegram : McCain: When North Korea can threaten the US with a nuke, we should strike is downloaded and parsed
The article: U.S. Strategy to Hobble North Korea Was Hidden in Plain Sight is downloaded and parsed
The article: Avoiding a second Korean War is downloaded and parsed
The article: World is downloaded and parsed
The article: Tillerson sees no 'imminent threat' of North Korea attack; defends Trump's blunt talk is downloaded and parsed
The article: North Korea has threatened a U.S. attack for years. Why aren’t you scared? is downloaded and parsed
The article: North Korea Is Threatening to Attack Guam. Why Guam? is downloaded and parsed
The article: War with North Korea inches closer is downloaded and parsed
The article: How potent are North Korea's threats? is downloaded and parsed
The article: Did North Korea Threaten to ‘Nuke’ U.S. ‘If Even a Single Bullet Is Fired’? is downloaded and parsed
The article: North Korea Threatens Hydrogen Bomb Attack on Manhattan is downloaded and parsed
The article: Charlie Dent: Kim Jong Un won't take actions that threaten his regime is downloaded and parsed
The article: North Korea threatens strike on Guam is downloaded and parsed
The article: Will Donald Trump Start Second Korean War? Wild Gamble Would Risk Lives And Safety Of Millions is downloaded and parsed
The article: North Korea Threatens To Target “Liberal-Governed” Cities In Nuclear Attack is downloaded and parsed
The article: North Korean Military Threatens to Attack Guam is downloaded and parsed
The article: North Korea threatens nuclear attack after claiming Trump CIA director 'shakes little fists' at regime is downloaded and parsed
The article: North Korea threatens to attack Guam is downloaded and parsed
The article: North Korea Threatens To Attack U.S. If Obama Retaliates Over Sony Hacking is downloaded and parsed
The article: Did North Korea Threaten to ‘Nuke’ U.S. ‘If Even a Single Bullet Is Fired’? is downloaded and parsed
The article: North Korea Threatens To Target “Liberal-Governed” Cities In Nuclear Attack is downloaded and parsed
The article: North Korea threatens US as Pyongyang warns it is ready to use nuclear weapons is downloaded and parsed
The article: North Korea Threatens Nuclear Attack as U.S. and South Korea Begin Drills is downloaded and parsed
The article: North Korea Attack on US Power Grid Threatened is downloaded and parsed
The article: Ransomware Attack: North Korea Hackers Capable is downloaded and parsed
The article: North Korea threatens to attack U.S. is downloaded and parsed
cost time: 54.82132577896118
In [5]:
svo_sent = SVOSENT()
articles_list=svo_sent.split_and_clean(articles)
result=svo_sent.batchProcessArticles(articles_list)
#svo_sent.WriteCSV(results,'corpus4_full_dataset')
1 th/ 56 article is done
2 th/ 56 article is done
3 th/ 56 article is done
4 th/ 56 article is done
5 th/ 56 article is done
6 th/ 56 article is done
7 th/ 56 article is done
8 th/ 56 article is done
9 th/ 56 article is done
10 th/ 56 article is done
11 th/ 56 article is done
12 th/ 56 article is done
13 th/ 56 article is done
14 th/ 56 article is done
15 th/ 56 article is done
16 th/ 56 article is done
17 th/ 56 article is done
18 th/ 56 article is done
19 th/ 56 article is done
20 th/ 56 article is done
21 th/ 56 article is done
22 th/ 56 article is done
23 th/ 56 article is done
23 th article has error: Out of bounds nanosecond timestamp: 500-05-10 00:00:00
25 th/ 56 article is done
26 th/ 56 article is done
27 th/ 56 article is done
28 th/ 56 article is done
29 th/ 56 article is done
30 th/ 56 article is done
31 th/ 56 article is done
32 th/ 56 article is done
33 th/ 56 article is done
34 th/ 56 article is done
35 th/ 56 article is done
36 th/ 56 article is done
37 th/ 56 article is done
38 th/ 56 article is done
39 th/ 56 article is done
40 th/ 56 article is done
41 th/ 56 article is done
42 th/ 56 article is done
43 th/ 56 article is done
44 th/ 56 article is done
45 th/ 56 article is done
46 th/ 56 article is done
47 th/ 56 article is done
48 th/ 56 article is done
49 th/ 56 article is done
50 th/ 56 article is done
51 th/ 56 article is done
52 th/ 56 article is done
53 th/ 56 article is done
54 th/ 56 article is done
55 th/ 56 article is done
55 th article has error: 'Sentence'
time cost:11.653934955596924
In [6]:
result
Out[6]:
Sentence
Names
Persons
Organizations
Facilities
Locations
Subjects
Predicates
Objects
Sentiment
Event_date
Article_date
Article_title
0
As North Korean threat grows, Hawaii prepares ...
[Hawaii, U.S.]
[]
[]
[]
[Hawaii, U.S.]
[Hawaii]
[grow, prepare]
[]
-0.8402
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
1
Public service commercials drilled the “duck a...
[]
[]
[]
[]
[]
[]
[]
[]
-0.4767
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
2
But after the Cold War, most places abandoned ...
[]
[]
[]
[]
[]
[]
[]
[]
-0.8008
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
3
Fears of terrorism grew more urgent and, for m...
[]
[]
[]
[]
[]
[]
[grow, become]
[]
-0.8809
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
4
That’s no longer the case in Hawaii.
[Hawaii]
[]
[]
[]
[Hawaii]
[]
[]
[Hawaii]
-0.2960
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
5
Amid increasing North Korean threats against t...
[U.S., Hawaii]
[]
[]
[]
[U.S., Hawaii]
[Hawaii]
[increase, launch, prepare]
[U.S.]
-0.7425
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
6
TV commercials warn the state’s 1.4 million re...
[]
[]
[]
[]
[]
[]
[warn, get, stay, drop]
[]
-0.5574
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
7
State officials are holding online forums and ...
[]
[]
[]
[]
[]
[]
[hold, fly]
[]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
8
On Dec. 1, the nuclear attack warning siren wi...
[]
[]
[]
[]
[]
[]
[warn, hear]
[]
-0.6705
2017-12-10 00:00:00
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
9
A North Korean bomb is “a major, major concern...
[Vern Miyagi, the Hawaii Emergency Management ...
[Vern Miyagi]
[the Hawaii Emergency Management Agency, ’s De...
[]
[Honolulu]
[Vern Miyagi, the Hawaii Emergency Management ...
[say, hold]
[’s Department of Defense, Honolulu]
-0.6486
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
10
He painted a stark picture of what emergency o...
[Oahu]
[]
[Oahu]
[]
[]
[]
[paint, expect, reach]
[Oahu]
-0.3612
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
11
“We are talking about 50,000 to 120,000 trauma...
[Miyagi]
[Miyagi]
[]
[]
[]
[Miyagi]
[talk, explain]
[]
-0.8074
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
12
The expected target: Pearl Harbor.
[Pearl Harbor]
[]
[]
[]
[Pearl Harbor]
[]
[]
[]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
13
More accustomed to educating residents about h...
[Miyagi, Honolulu]
[Miyagi]
[]
[]
[Honolulu]
[Miyagi]
[]
[Honolulu]
-0.4939
0100-11-10 00:00:00
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
14
The explosion would hit an area about eight mi...
[]
[]
[]
[]
[]
[]
[hit, say]
[]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
15
Ninety percent of people would survive the dir...
[]
[]
[]
[]
[]
[]
[survive, hit]
[]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
16
“We anticipate severe damage to Daniel K. Inou...
[Daniel K. Inouye, Hickam Air Force Base, Hono...
[Daniel K. Inouye, Honolulu Harbor]
[Hickam Air Force Base]
[]
[Pearl Harbor]
[Honolulu Harbor, Pearl Harbor]
[]
[]
-0.7003
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
17
There will be widespread structural fires and ...
[]
[]
[]
[]
[]
[]
[]
[]
-0.2960
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
18
There will be damage to hospitals and governme...
[]
[]
[]
[]
[]
[]
[warn]
[]
-0.6486
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
19
He left open the possibility that other island...
[]
[]
[]
[]
[]
[]
[leave, hit]
[]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
20
Hawaii has largely avoided North Korea’s cross...
[Hawaii, North Korea’s]
[]
[]
[]
[Hawaii, North Korea’s]
[Hawaii]
[avoid, make]
[]
-0.6369
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
21
Concerns grew in 2009, when the Obama administ...
[Obama, Kim Jong Il, Hawaii, U.S., Honolulu]
[Kim Jong Il]
[]
[]
[Obama, Hawaii, U.S., Honolulu]
[Kim Jong Il]
[grow, say, send, increase]
[Hawaii, Honolulu]
0.2732
2009-11-10 00:00:00
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
22
But fears have further increased amid a tense ...
[Trump, Kim Jong]
[Trump, Kim Jong]
[]
[]
[]
[]
[increase]
[Trump]
-0.4767
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
23
The U.S. military has recently focused on Guam...
[U.S., Guam, U.S., North Korea, Kim]
[Kim]
[]
[]
[U.S., Guam, U.S., North Korea]
[Kim]
[threaten]
[Guam, North Korea]
-0.1027
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
24
The hurdles for North Korea are high.
[North Korea]
[]
[]
[]
[North Korea]
[]
[]
[North Korea]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
25
It’s not clear that the reclusive country has ...
[Hawaii]
[]
[]
[]
[Hawaii]
[]
[get]
[Hawaii]
-0.6580
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
26
With around 4,000 miles between Pyongyang and ...
[Pyongyang, Honolulu]
[]
[]
[]
[Pyongyang, Honolulu]
[Honolulu]
[]
[Pyongyang]
-0.3612
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
27
In addition, Pacific Command, the U.S. militar...
[Pacific Command, U.S., ’s]
[]
[]
[]
[Pacific Command, U.S., ’s]
[]
[run, say]
[]
0.3612
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
28
“Any attack against us is suicide,” Miyagi sai...
[Miyagi, U.S., North Korea]
[Miyagi]
[]
[]
[U.S., North Korea]
[Miyagi]
[say]
[North Korea]
-0.1531
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
29
Still, preparations are in full swing in Honol...
[Honolulu]
[]
[]
[]
[Honolulu]
[]
[]
[Honolulu]
0.0000
None
2017-11-10 00:00:00
As North Korean threat grows, Hawaii prepares ...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
101
When in 2014 South Korean investigators traced...
[Korea Hydro & Nuclear Power, Shenyang]
[]
[Korea Hydro & Nuclear Power]
[]
[Shenyang]
[Korea Hydro & Nuclear Power]
[refuse, permit, cooperate]
[Shenyang]
-0.2960
2014-11-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
102
And experts agree that should relations betwee...
[Beijing, Washington]
[]
[]
[]
[Beijing, Washington]
[Washington]
[agree, should, gather]
[Beijing]
0.6808
2017-05-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
103
“That is a likely scenario,” says the defector...
[Jang]
[Jang]
[]
[]
[]
[Jang]
[say]
[]
-0.4404
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
104
Potential targets are legion.
[]
[]
[]
[]
[]
[]
[]
[]
0.0000
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
105
The U.S. and South Korea are among most advanc...
[U.S., South Korea]
[]
[]
[]
[U.S., South Korea]
[U.S., South Korea]
[make]
[]
-0.1548
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
106
The U.S. is arguably the most vulnerable, owin...
[U.S.]
[]
[]
[]
[U.S.]
[U.S.]
[intend]
[]
-0.5849
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
107
In the event of all-out inter-Korean war, Nort...
[North Korea, U.S.]
[]
[]
[]
[North Korea, U.S.]
[North Korea]
[launch]
[]
-0.7184
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
108
Pyongyang strategists posit that a delay of a ...
[Pyongyang, Seoul, Washington]
[]
[]
[]
[Pyongyang, Seoul, Washington]
[Pyongyang]
[occupy, negotiate]
[Seoul, Washington]
0.2960
2017-05-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
109
Read More: North Korea’s Nuclear Weapons Are N...
[North Korea’s Nuclear, Reason Enough, North K...
[Reason Enough]
[]
[]
[North Korea’s Nuclear, North Korea, U.S.]
[Reason Enough, North Korea’s Nuclear]
[start, should, develop, hit, say, take]
[]
-0.7096
2017-05-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
110
The presence of countervailing nuclear deterre...
[]
[]
[]
[]
[]
[]
[lessen]
[]
-0.5994
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
111
U.S. policymakers do not have a pre-establishe...
[U.S.]
[]
[]
[]
[U.S.]
[]
[do, establish]
[]
-0.6486
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
112
In a public talk in February 2015, NSA Directo...
[NSA, Admiral Michael Rogers]
[Admiral Michael Rogers]
[NSA]
[]
[]
[Admiral Michael Rogers]
[say, get, acknowledge, get, get, talk, go, do...
[]
0.2263
2015-02-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
113
The brazen Sony hack, accompanied by threats a...
[Sony, Kim Jong]
[Kim Jong]
[Sony]
[]
[]
[]
[accompany]
[]
-0.7269
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
114
In North Korea’s stifling autocracy, the Kim c...
[North Korea, Kim]
[Kim]
[]
[]
[North Korea]
[]
[stifle, treat]
[]
0.0000
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
115
The scheduled release of The Interview also co...
[The Interview, U.N., the Commission of Inquir...
[Kim]
[The Interview, U.N., the Commission of Inquiry]
[]
[North Korea]
[The Interview]
[implicate]
[the Commission of Inquiry, North Korea, Kim]
-0.5574
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
116
This likely contributed to the scale of the re...
[]
[]
[]
[]
[]
[]
[contribute]
[]
0.2263
2017-05-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
117
President Trump is not a man to mince words.
[Trump]
[Trump]
[]
[]
[]
[Trump]
[]
[]
0.0000
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
118
During his presidential campaign, he called Ki...
[Kim, U.S.]
[Kim]
[]
[]
[U.S.]
[]
[call, follow]
[Kim, U.S.]
0.0000
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
119
Naval Strike Group to the Korean Peninsula and...
[Naval Strike Group, the Korean Peninsula, Nor...
[Kim]
[Naval Strike Group, the Korean Peninsula]
[]
[North Korea]
[Naval Strike Group, Kim]
[warn, refuse]
[the Korean Peninsula, North Korea]
-0.7269
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
120
Invective and perceived provocations from the ...
[the Oval Office]
[]
[the Oval Office]
[]
[]
[]
[see, settle]
[the Oval Office]
-0.2960
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
121
For today we are all at the mercy of hotheads ...
[]
[]
[]
[]
[]
[]
[]
[]
0.3612
2017-05-10 00:00:00
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
122
—With reporting by Stephen Kim / SeoulPublicat...
[Stephen Kim]
[Stephen Kim]
[]
[]
[]
[]
[]
[]
0.0000
None
2017-05-10 00:00:00
Ransomware Attack: North Korea Hackers Capable
0
North Korea threatens to attack U.S.(title_end...
[North Korea, State Department, U.S., N. Korea...
[N. Korea]
[State Department, Ulchi Freedom Guardian', CNN]
[]
[North Korea, U.S., U.S., North Korea, U.S.]
[North Korea, State Department, N. Korea, Ulch...
[threaten, attack, say, threaten, conduct, beg...
[U.S., U.S.]
-0.4329
2015-08-10 00:00:00
2015-08-15 00:00:00
North Korea threatens to attack U.S.
1
South Korea conducts the yearly exercises, cal...
[South Korea, Ulchi Freedom Guardian, the Unit...
[]
[Ulchi Freedom Guardian]
[]
[South Korea, the United States, the Korea-U.S.]
[South Korea]
[conduct, call, enhance, protect, maintain]
[the United States, the Korea-U.S.]
0.8910
NaT
2015-08-15 00:00:00
North Korea threatens to attack U.S.
2
Combined Forces Command.
[Combined Forces Command]
[]
[Combined Forces Command]
[]
[]
[]
[combine]
[]
0.0000
NaT
2015-08-15 00:00:00
North Korea threatens to attack U.S.
3
Just as the event itself is annual, so too are...
[]
[]
[]
[]
[]
[]
[]
[]
-0.4215
NaT
2015-08-15 00:00:00
North Korea threatens to attack U.S.
4
"The further Ulchi Freedom Guardian joint mili...
[Ulchi Freedom, North Korea's, National Defens...
[]
[Ulchi Freedom, National Defense Commission]
[]
[North Korea's]
[]
[intensify, take, say]
[North Korea's, National Defense Commission]
0.8225
2015-08-08 00:00:00
2015-08-15 00:00:00
North Korea threatens to attack U.S.
5
A State Department official told CNN on Saturd...
[State Department, CNN]
[]
[State Department, CNN]
[]
[]
[]
[tell]
[CNN]
-0.4215
2015-08-08 00:00:00
2015-08-15 00:00:00
North Korea threatens to attack U.S.
6
The exercises are transparent, defense-minded ...
[South Korea, the State Department]
[]
[the State Department]
[]
[South Korea]
[]
[increase, say]
[South Korea]
0.5106
NaT
2015-08-15 00:00:00
North Korea threatens to attack U.S.
7
Read MorePublication date - 2015-08-15 00:00:0
[]
[]
[]
[]
[]
[]
[]
[]
0.0000
2015-08-15 00:00:00
2015-08-15 00:00:00
North Korea threatens to attack U.S.
1527 rows × 13 columns
In [ ]:
Content source: rsteed11/GAT
Similar notebooks: