In [114]:
import math
from sklearn.feature_extraction.text import CountVectorizer
In [119]:
train_set = (
"the sky is blue.",
"the sun is bright.",
"the blue bright ocean, blue sky",
"the car is blue and yellow",
"the sun and ocean and yellow")
test_set = ("The sun in the sky is bright.",
"We can see the shining sun, the bright sun.")
In [107]:
vectorizer = CountVectorizer()
vectorizer
Out[107]:
In [108]:
vectorizer.fit_transform(train_set)
print vectorizer.vocabulary_
In [120]:
smatrix = vectorizer.transform(train_set).todense()
smatrix
Out[120]:
In [121]:
float(smatrix.shape[0]) / (smatrix[:,4].sum() + 1)
Out[121]:
In [122]:
def tf(docIndex, term):
row = docIndex
column = vectorizer.vocabulary_[term]
tf = float(smatrix[row, column])
tf = tf / smatrix[row,:].sum()
return tf
def idf(term):
column = vectorizer.vocabulary_[term]
return math.log(
float(smatrix.shape[0]) / (smatrix[:,column].sum() + 1)
)
In [123]:
print idf("the")
print idf("ocean")
print idf("car")
In [124]:
print tf(3, "car")
print tf(3, "the")
In [128]:
print tf(3, "car") * idf("car")
print tf(3, "the") * idf("the")
In [148]:
a = []
a.append("""elma armut elma""")
a.append("""elma karpuz elma karpuz elma domates""")
In [149]:
TCDictionary = {}
for docID, d in enumerate(a):
for t in d.lower().split(" "):
tData = TCDictionary.get(t, {"d"})
tData[docID] = tData.get(docID, 0) + 1
TCDictionary[t] = tData
print TCDictionary
In [302]:
a = ("""
Hooker, to his credit, knew his planting scheme would push out the endemic ferns. What he perhaps didn't realise was just how much havoc it would cause.
"I don't think we'll ever get to the point we could call Green Mountain a fully functioning ecosystem, at least not in the short term - that would take thousands of years," says Weber.
"At the moment it's a completely unmanaged mess of invasive species - one after another rises to dominance, others die back, those species crowd the footpaths, reducing the value of the mountain for walkers."
Things are too far gone to attempt restoration of the island's degraded environment
And the BBC didn't help. Engineers arrived in the mid-60s to construct transmitters to beam World Service programming to Africa and South America. They built a new village, a few miles inland from the sleepy capital, Georgetown - and planted a type of mesquite, known as Mexican thorn, to bind the dry soil.
Today the dry, stiff-thorned mesquite runs riot, dominating whole swathes of Ascension's terrain.
"A conservative estimate is that there's now about 38,000 of these bushes," says Weber.
"It's very difficult to control physically - it puts down tap roots that can be 20-30m deep - so we're looking at biological methods of control, bringing in pest species that are specific to this shrub from its native range."
This, alongside Weber's work on Green Mountain preserving introduced ficus trees as makeshift habitats for the struggling native ferns, pushes Ascension's oddity into new areas.
Things are too far gone to attempt restoration of the island's degraded environment. Weber's plan now is to use invasive species as part of a broader strategy, aimed at clawing back over the next century or two some of the mayhem sparked by Hooker and Darwin.
Turtle - Ascension Island
""",
"""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""","""
"As you can see by the vegetation surrounding us, this plan was spectacularly successful," says biologist Dr Sam Weber, standing wreathed in fog amid a mini cloud forest of dripping ficus trees, bamboo, ginger and guava that envelops Green Mountain today. We are a few minutes' drive from the sweltering lava plains, but up here it's mild and breezy.
"Whether he was right is another question. Judged by today's standards, many scientists would argue this is a disaster. On a superficial level it looks like a tropical paradise - it's humid, there's lots of plants - but if you scratch the surface it really doesn't go a lot beyond that. There's none of the complex interrelationships you'd expect in a real tropical cloud forest - and all of the species that were here are vanishing."
They comprise the few fragile grasses that grew on the island before Homo sapiens landed, including the tiny parsley fern, half the length of your little finger. Long thought extinct, it was rediscovered in 2009 clinging to an isolated crag on Green Mountain and after careful propagation at Kew is now being planted out into the wild again.
""",
"""
He would plant trees all over the 859m (2817ft) summit of Green Mountain, Ascension's highest point. Foliage would trap moisture from the warm southeasterly winds that sweep continuously over, letting it drip down to ground level to assure a water supply for the troops. He would introduce grasses to create pasture for livestock, and soil to plant vegetables.
""",
"""
Darwin discussed how to make Ascension more habitable for humans with his friend Joseph Hooker, later director of the Royal Botanic Gardens at Kew, who visited in 1843. Hooker devised a plan.
""",
"""
They turned out to belong to Leendert Hasenbosch, a Dutch mariner who had been set ashore on Ascension the year before as punishment for homosexuality. The diary records Hasenbosch's increasingly desperate search for water and supplies, as he resorted to drinking the blood of turtles and seabirds and, eventually, his own urine.
The poor man's final fate is unknown - no skeleton was ever discovered.
""",
"""
Almost 80 years later, when the British garrisoned Ascension to discourage French attempts at rescuing Napoleon - who had been exiled to St Helena, the nearest point of land, 700 miles southeast - they too could find virtually no fresh water.
"Near the coast, nothing grows," wrote Charles Darwin.
""",
"""
"The island is entirely destitute of trees."
Darwin discussed how to make Ascension more habitable for humans with his friend Joseph Hooker, later director of the Royal Botanic Gardens at Kew, who visited in 1843. Hooker devised a plan.
""",
"""
He would plant trees all over the 859m (2817ft) summit of Green Mountain, Ascension's highest point. Foliage would trap moisture from the warm southeasterly winds that sweep continuously over, letting it drip down to ground level to assure a water supply for the troops. He would introduce grasses to create pasture for livestock, and soil to plant vegetables.
""",
"""
Mr Juncker was addressing the Council of Europe, the main European human rights watchdog, in the French city of Strasbourg. The 47-nation body is not an EU institution, although all 28 EU countries are in it.
Shortly after he took over as president in 2014, the Commission, which drafts EU laws, scrapped 80 proposals, in a drive to reduce red tape.
""",
"""
The former Luxembourg prime minister said the EU had become involved in "too many domains where member states... are better placed to take action".
""",
"""
The "criteria (on visa liberalisation) will not be watered down in the case of Turkey", he said, commenting on the EU's controversial deal with Turkey aimed at tackling Europe's migrant crisis.
""",
"""
Wolff started working with Williams in 2012, after racing for seven years in the German Touring Car Championship, but retired in November.
She spent three years involved in test and development work and took part in four practice sessions over the 2014 and 2015 seasons - becoming the first woman to take part in a grand prix weekend for more than 20 years.
In 2013 she said one the main issues behind a lack of females in the sport was young girls "have no-one to aspire to growing up".
But the Scot dismissed claims women lack the strength to cope with the demands of the sport, adding: "We have 30% less muscle so we have to work hard, but there's no reason why females can't get strong enough to race an F1 car."
The 33-year-old has since launched an initiative to increase the number of women in motorsport and said there were lots of opportunities in F1 besides driving.
""",
"""
A proposal by a US cinema chain to let customers use their phones during some film screenings unleashed such a powerful blast of outrage that the company quickly reversed it. But is allowing texting in cinemas really such terrible idea?
Adam Aron, head of AMC Entertainment, only had to float the idea of making some cinemas more "texting friendly", and film fans all over the world reacted furiously.
"Worst idea ever," was the conclusion of many people on social media. "You're basically pandering to rude people just to make money," said one person on Twitter. "I will not pay to have people texting all around me."
Maybe it would be better - it would quarantine the texters
Ann Hornaday, Film critic
Just two days after Aron's interview was published, the company was forced to withdraw the idea.
"There will be no texting allowed in any of the auditoriums at AMC Theatres," it said in a statement. "Not today, not tomorrow and not in the foreseeable future."
Hollywood film blogger Alex Billington is one of many who rejoiced at AMC's rapid backpedalling."Texting in the theatre completely ruins the movie," he says. "This is something we have to fight against on a social and cultural level."
In 2014, an argument between two men over texting in a cinema in Florida ended in a fatal shooting. While that was an extreme case, most people would agree that glowing phone screens and fingers tapping can be infuriating.
"The only senses you have in the cinema are vision and hearing," says Ann Hornaday, chief film critic at the Washington Post. "Anything that gets in the way of that sensory experience is rightfully considered an inappropriate distraction.
"When we go to the cinema, we all enter into a contract - so we do feel a little betrayed when someone behaves badly."
At the same time, she does understand the challenge cinemas face to attract younger viewers.
""",
"""
Ann Hornaday says making particular areas or auditoriums "texting friendly" could benefit other audience members who did not want to sit next to someone with their phone on. "Maybe it would be better for people like me - it would quarantine the texters," she says.
But Dana Stevens, a film critic for Slate, is not convinced. "I guess if all the rude texters don't mind being cordoned off into a single, awful screening room full of flashing, beeping phones that could work," she says. "But I don't think dividing audiences into two camps… is the way to keep theatrical movie-going alive."
Much of the frustration of film fans comes from the apparent reluctance of cinema staff to enforce cinema rules, enabling some chains to use their strict no-texting policies as a marketing tool. In 2011 Alamo Drafthouse, a cinema chain based in Austin, Texas, posted a YouTube video of a voicemail message left by a seething customer after she was kicked out for using her phone. The video went viral and Alamo Drafthouse went on to use it as a public service announcement before screenings.
The dilemma remains for other chains still struggling to attract new audiences while keeping core customers happy. The latest outcry over texting in cinemas will deter them from considering this option, says Chuck Tyron.
"The entire movie Twittersphere exploded," he says. "I think the cinema chains have been sent a message. I do not see any change happening any time soon."
Subscribe to the BBC News Magazine's email newsletter to get articles sent to your inbox.
Related Topics
""",
"""
The London-based American writer and broadcaster Michael Goldfarb is frequently asked on air why this year's US election has turned out to be so unusual, and whether insurgent Republican candidate Donald Trump can really win. He has to give a short answer. The long answer, he argues here, involves going back 40 years.
Bruce Springsteen is coming to London with the River tour. At £170 for the cheapest pair, I can't afford to see the Boss any more, even if my body could handle standing on Wembley Stadium's pitch for three-and-a-half-hours in an early June drizzle.
It's interesting that Springsteen is re-exploring The River album again. Whenever the anger that simmers in America erupts and reminds the rest of the world that the country is troubled, he seems to be the cultural figure whose work offers an explanation.
In late 1986, midway through Ronald Reagan's second term of office, with the twin scourges of Aids and crack racing through American cities and New Deal ideas of economic and social fairness consumed by the Bonfire of the Vanities taking place on Wall Street, Britain's Guardian newspaper ran an editorial that said, "for good or ill, [America] is becoming a much more foreign land".
I had just celebrated my first anniversary as an expat in London and wrote an essay trying to explain what America was like away from the places Guardian readers knew. I described the massive population dislocations that followed the long recession that had begun in the mid-70s. I referenced Springsteen. The piece ran under the headline "Torn in the USA".
Bruce SpringsteenImage copyrightAP
Now America is going through even worse ructions. But there is nothing fundamentally new. What we are seeing is the continuation of a disintegration that began 40 years ago around the time Springsteen was writing the title song of the album.
The River, which came out in 1980, was very much about guys trying to kick back at father time and stave off the inevitable arrival of life's responsibilities - wife, kids, job, mortgage - and the equally probable onset of life's disappointments in wife, kids, job, mortgage, and in oneself.
The title track is a long, mournful story about that process and the narrator's desire to reconnect to the person he was when younger and full of hope.
"I come from down in the valley / Where mister, when you're young / They bring you up to do/like your daddy done…"
The key point is being brought up to be like your father. Work the same job, carry yourself in the same way, do the right thing. In the song, this tie that binds is seen as restricting the choices you can make in life. Your daddy worked in a steel mill, you will work in a steel mill, or on the line at River Rouge, or down a mine.
Follansbee West Virginia -children play baseball near the Mountain State Carbon Coke WorksImage copyrightGetty Images
Today, what wouldn't many of us give for the economic and social stability that gave resonance to Springsteen's lyrics? A union job, 30 years of work, a pension. Sounds sweet.
The narrator of the song goes on to tell us, "I got a job working construction at the Johnstown company / but lately there ain't been much work on account of the economy."
Springsteen based the song on the struggle of his brother-in-law to stay employed during the bleak days after the Oil Shock of 1973: a half-decade of inflation and economic stagnation. At the time this stagflation was seen as a cyclical event; the economy would rebound soon. It would be boom time for all.
The economy did rebound, but then went into recession in 1982, and rebounded and went into recession at regular intervals, until the near-death experience of 2007/2008.
But very few people understood that an epochal change had taken place in the American economy. GDP would grow. Income wouldn't. Median salaried workers' wages stagnated. Those working low-wage jobs saw their incomes decline. As for job security, a perfect storm of automation, declining union power, and free-trade agreements put an end to that.
People who had been brought up to do as their parents had done wouldn't be able to do those things, they wouldn't even be able to live in the valley or towns surrounding the factories because there was very little work. A great migration south-east and south-west got underway, although "great diffusion" would be just as accurate a phrase.
Mingo Junction, OhioImage copyrightGetty Images
How much does it hurt to leave the certainty in which you were raised, the community of family and church and friends who have known you forever? You go 500 or 1,000 miles south and live in an exurb built at the junction of an interstate in the middle of nowhere and have to work very hard to create that same sense of community.
You find a job, but unlike your daddy's job it isn't unionised, you work for less and you have very little security. When the economy slows, you know your job is at risk.
America grows richer for some but not for you. That hurts.
This is what I wrote about in 1986 for the Guardian.
In the autumn of 1992, when I had lived in the UK for seven years, the BBC World Service sent me back to the US to do a little reporting on the country in advance of the election. In a cafe in rural Maine, completely by chance, I ran into people whose lives were still upended two decades after the oil shock.
There were people in their forties who could not live as their parents had done be-cause the Great Inflation of the 1970s had permanently eroded the value of their salaries. There were people in their late sixties who found that their retirement savings had disappeared on day three of cancer treatment. They wanted someone to bear witness to their calamities.
More from the Magazine
Many people are not only angry, they are angrier than they were a year ago - particularly Republicans (61%) and white people (54%) but also 42% of Democrats, 43% of Latinos and 33% of African Americans.
Candidates have sensed the mood and are adopting the rhetoric. Donald Trump, who has arguably tapped into voters' frustration better than any other candidate, says he is "very, very angry" and will "gladly accept the mantle of anger" while rival Republican Ben Carson says he has encountered "many Americans who are discouraged and angry as they watch the American dream slipping away".
Why are Americans so angry?
The following year, in 1993, I drove around the Midwest for the BBC to see if the election of Bill Clinton had changed things. It hadn't. The hurt was more on the surface if anything. In Cape Girardeau, Missouri, I met a high school principal, who was conservative and evangelical and hurting because a ballot initiative that would have raised property taxes a fraction of a cent to raise money to rebuild his dilapidated school and buy new text books had been voted down. He thought he was part of a community and had been shocked by the selfishness that vote demonstrated. He had recently retired.
There had been epic floods that year along the Mississippi. We walked down by the river and he pointed across to the Illinois shore where the flood waters had yet to recede. The principal told me, "Sometimes, I can't see where the river is going."
That was almost a quarter of a century ago. A lot of people still can't see which way the river is flowing: they want someone to do, whatever it takes to make the hurt go away.
In the last decade more people have joined the hurt club, and now, increasingly, it's the professional classes.
We know the pain of looking at our children and saying, what I had when I was your age I can't give you.
Donald TrumpImage copyrightAP
When I am asked to explain the Trump phenomenon on BBC radio and television, I know I don't have to teach this history. I have to stay focused on the day's events and explain a bit about the vagaries of the primary process. I have to bullet-point data: the polls, the employment figures, the wage stagnation. I have to throw the story forward because incredulous presenters end each interview by asking, "But can Trump win?"
And I think to myself there is no way to explain in a brief interview the power of the 40-year hurt in shaping American society. There is no number or data set that can measure its pain.
At the end of the River, Bruce writes, "Is a dream a lie if it don't come true/or is it something worse?"
Forty years of hurt have driven some people to answer Springsteen's question, that the American dream is something worse than a lie, and from that bleak answer they are looking for a political leader who echoes their anger.
So my answer to presenters' questions whether Trump can win, is now and was before the primaries started: "Yes."
Subscribe to the BBC News Magazine's email newsletter to get articles sent to your inbox.
""",
"""
For 11 years in the 1970s and 1980s skateboarding was banned in Norway. A few hardcore devotees flouted the law and skated in secret, but the country's skateboarders say they are still a long way behind and have a lot of catching up to do, reports Chris Stokel-Walker.
Henning Braaten was nine years old and wanted only one thing from his parents - a skateboard. As he sat in his house in late 1988, he could see his next-door neighbour whizzing around a skate park he had made in the back garden, complete with a quarter pipe propped against the wall of the house.
But Braaten had a problem. For the previous decade skateboarding had been banned in Norway, and widely condemned as a public menace.
Not long after skateboards first went on sale in Norway, officials noted that American children were getting hurt after colliding with traffic - 28 had died in 1977 and 100,000 had been injured. So a prohibition was announced, which came into force on 15 September 1978.
"The Environment Ministry said protecting children is more important than letting big business make money," reported United Press International's Oslo bureau.
"Importing skateboards, selling them or advertising the American sidewalk sport also are banned under the new law."
Braaten had no access to the underground smuggling network that supplied skaters with their boards, so for a while he had to go without.
Joakim Henrik Wang was luckier.
Skater Nico Henriksen on a skating rampImage copyrightJoakim Wang
Image caption
Skateboarding in the summer of 1987
When he was a teenager he went to a second-hand fair organised by his school in 1984. Wandering between the tables, Wang came across a cheap plastic board known by skateboarders as a banana board - long and very basic - on sale for 10 or 20 krone (in the region of £1 or $1.50 in today's money).
"It was one of these old 1970s boards with no grip tape," he says. "I didn't know it was banned when I bought this first board."
The wheels were basic - taken from roller-skates and attached to the board - but Wang quickly became an enthusiast.
"We obviously knew it was illegal," he says, recalling some difficult conversations with his father. "I think my father never looked at it like it was a sport." At the time, many adults viewed it as not only illegal, but anti-social.
Fortunately, Wang had relatives in Germany. There, skating wasn't banned. There were many skateparks, and even more skate shops. In Oslo, there was just one place, in an area frequented by punk-rockers, where you could buy a board - but no wheels.
Improvised skater park with a skateboarder skating inside a rampImage copyrightAnders Wittusen
Image caption
Skateboarders had to improvise during the prohibition years
"I remember the tension of stepping off the boat in Norway with two boards strapped to your back or tucked away in your suitcase, hoping you wouldn't get pulled to one side by customs," Wang remembers.
"I had endless discussions with my father about smuggling the stuff."
He still feels exasperated, 30 years later, with the politicians who introduced the ban.
"We thought it was bullshit that it was illegal," he says.
"I think it's partly if you don't understand it, and you can't control it, you want to ban it. It's a really basic human reaction."
The prohibition was not absolute, however, and allowed the possibility of exemptions. "Exemptions will only be granted to a limited extent and for special purposes," it stated.
A single skatepark in Oslo, Norway's capital, was allowed. It was a place skaters could visit in peace, without fear of arrest - though they might get questioned by police on the way there.
Frognerparken - the only legal skatepark in Norway during the years of the banImage copyrightNORB
Image caption
Frognerparken in Oslo the only legal skatepark in Norway during the years of the ban
"You had to walk around with your membership card for the Oslo club. I have to have this piece of paper in my back pocket to prove I was part of the club which had the only legal ramp? Not even North Korea has banned skateboarding."
For those outside Oslo, subterfuge was required.
Makeshift wooden skateboarding rampImage copyrightAnders Wittusen
In cities across Norway, skaters set up their own parks and pipes made of plywood and planks. Some were small and hastily-built, while others were well-crafted. Most were tucked into copses or deep in the undergrowth, off the beaten path.
Wang was arrested twice. The first time was in his hometown, when he was seen skating near the local train station. "My dad had to come and bail me out," he recounts. "You don't have to do that if your kid plays football, do you?"
The second time was in Oslo, when Wang was walking with a friend in the city centre. The police didn't believe the pair had been skating at the legal ramp in the city; they were taken into custody, and spent the night in a cell.
As the end of the 1980s approached, enforcement of the ban gradually relaxed. "Around 1987 it kind of changed," says Wang. "They mellowed out. They realised the ban was coming to an end and didn't crack down as bad."
When it eventually was lifted in 1989, skateboards flooded the market. Skateparks sprang up across the country in the early 1990s. Brands got in on the craze, tying their products to skateboarding. One of the country's major magazine publishers, better known for celebrity gossip publications, even started printing a skateboarding magazine.
Oslo skater Erik Dybeli at a skatepark in SandvikaImage copyrightJoakin Wang
Image caption
Large numbers of skateparks were built in the 1990s after the ban was lifted
"It was almost an overdose of things going on," recalls Wang.
So what did Norway's decade-long ban achieve? Not a lot of positives, according to those who skate.
"I think a lot of countries are 10 years ahead of us now," says Per Olav Hetland, president of the Norsk Organisasjon For Rullebrett (NORB), a lobbying group and body for skateboarders in Norway.
Per Olav on a skateboardImage copyrightNORB
Image caption
Per Olav, president of NORB
He himself started skating only after the ban, as a teenager in 1995.
"In every other country, the people who started skating in the 1980s continued and developed a culture before us," he says. "In Norway, they simply aren't there. Only a few people continued."
It was not just the ban on skateparks that had a lasting effect, but the lack of magazines, videos and organised classes.
"We didn't really know much about skateboarding apart from the flying skateboard in Back to the Future," says Henning Braaten, now 36, a professional skateboarder, and four-time Norwegian champion.
"Suddenly when the ban was lifted and importers starting getting skateboards, they also got videos from the United States and there was a quick learning curve when you see what's possible."
Braaten now spends some of his time lobbying for the creation of new skateparks, and finds that attitudes among officials are still coloured by the campaign of the 1980s.
Longyearbyen, Norway - skateboarding park under snowImage copyrightGetty Images
Image caption
Snow-covered skatepark in Longyearbyen, Norway
"They question the whole culture and ask when are you going to grow out of this," says Braaten.
"I don't think they can see how big it is and how good it can be to have a skatepark that will take care of the youth.
"A lot of my work has been going to communes and talking to the mayor and trying to convince them that no, we're not rebels any more. We're just people trying to do our sport. I still get the same questions all the time. There's quite a long way to go, still."
Until 2013, funding from the state lottery was disbursed to skateparks at a much lower level than other sporting facilities like football stadiums or ice rinks. That has now changed, but Norway still lags behind some other countries.
Basically every big city in Sweden has a skatepark," says Per Olav Hetland. "But not every big city in Norway does."
But Norway's skateboarders are catching up with the rest of the world. In late February, Oslo held a version of the X Games, a global extreme sport competition. Alongside skiing and snowboarding competitions, a street skating contest was held, and Norway's Hermann Stene finished 10th out of 12 competitors. It was a momentous occasion.
Bronze medallist Vanessa Torres from the US in action during the X Games Oslo 2016Image copyrightVegard Wivestad Grott
Image caption
Norway hosted the X Games, a global extreme sport competition, in February
For Mats Hatlem a 15-year-old skateboarder born after the turn of the millennium, skateboarding the era of the ban already seems like another world.
"To me, this sounds crazy," he says.
"Driving a car is just as dangerous, if not way more. Why skateboarding was banned is something I find quite weird."
""",
"a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a",
"a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a",
"a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a",
"a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a","a",
"""
When Hollywood actress Angelina Jolie found out she carried a faulty variant of the BRCA1 gene, her doctors told her she had an 87% chance of developing breast cancer.
Armed with this knowledge, she chose to undergo a double mastectomy in 2013 to reduce the risk to around 5%.
This kind of genetic testing can now be done much faster and at lower cost, giving clinicians the ability to target treatments more effectively.
And combining this technological breakthrough with cloud computing and artificial intelligence is giving pharmaceutical companies the tools to develop drugs faster and with greater chance of success.
One beneficiary of this new approach is Eric Dishman, founder of tech giant Intel's first health research and innovation laboratory in 1999 and a founding member of its digital health group in 2005.
Eric Dishman, IntelImage copyrightCraig Mitchelldyer
When he was 19 years old he was diagnosed with a rare form of kidney cancer. He then had to endure 23 years of trial-and-error treatment, and was about to go on dialysis, when a tech company offered to sequence his genome for him.
This helped identify the faulty gene causing the cancer and enabled doctors to work out which drugs were likely to be the most effective.
Now 47 and cancer-free, Mr Dishman is determined that other cancer patients should have access to the same treatment he finally did.
"My doctors had never done anything with the genome sequence before. It turns out that 92% of the drugs I had been on would never have worked, but they didn't know that," Mr Dishman says.
It took three months for doctors to analyse his genome sequence, then another four months to compare his genomic results with similar patients in hospitals across the United States.
Mr Dishman wanted to speed this process up and so helped to found the Collaborative Cancer Cloud (CCC), an initiative launched last year by Intel and Oregon Health and Science University (OHSU).
Woman looking at computer screenImage copyrightOregon Health and Science University
Image caption
OHSU and Intel hope cloud collaboration could herald a new era of 'precision medicine'
The CCC enables hospitals and research institutions to share patient genomic, imaging, and clinical data securely for potentially lifesaving discoveries.
"It's using the cloud in a very different way," Mr Dishman says. "If you look around the world, the top cancer centres pool their data and make a little cloud where people come together and do research. That's a great thing to do. But the problem is that only about 4% of the data is available to look at."
The aim of the CCC is to bring the other 96% of the data into the equation, and allow hospitals and research centres to share information more effectively so that they can find the ideal treatment plan for cancer patients based on how individuals with the same genomic patterns have reacted to certain treatments.
More Cloud Computing features from Technology of Business
Cloud computing graphicImage copyrightThinkstock
The wearable tech giving sports teams winning ways
The tech allows users to share anonymised data securely, with every specific query and answer being wrapped in an encrypted shell before being sent.
"All of those that are using a centralised system can adopt a federalised approach and connect with each other," Mr Dishman says.
Intel and Oregon Health and Science University are now working with the two other cancer research institutions to see if the idea will work with other researchers and clinicians.
Cancer images on computer screenImage copyrightOHSU
Image caption
Analysing the DNA sequences of individual cancers takes huge computing power
Another 100 institutions will join this year, and the plan is to extend the cloud platform to other countries as well. It could also be used for researching other "faulty gene" illnesses, such as muscular dystrophy.
"We're still on this journey," Mr Dishman says. "One of the things that we saw is that the oncologists [cancer specialists] that were interested in viewing genomic medicine were so swamped trying to make the tech work that they were not seeing patients.
"One of the ways we measure our success is whether the oncologists can go back to being oncologists."
Virtual drugs
Intel's CCC aims to show exactly what drugs should be used with which patients, but there are still drugs yet to be discovered. And cloud computing is helping with this process, too.
So-called "in silico" research involves using computers to model and test potential molecules and compounds before they ever make into a real world lab.
For instance, North Carolina-based Cloud Pharmaceuticals has developed a platform that combines artificial intelligence and cloud computing to search virtual chemical space for new - as yet unmade - molecular compounds that show the most promise of being developed into effective drugs.
Cloud Pharmaceuticals websiteImage copyrightCloud Pharmaceuticals
Image caption
Cloud Pharmaceuticals thinks cloud collaboration could help create new drugs more cheaply
This hi-tech pre-selection process increases the chances of success, says Cloud Pharmaceuticals chief executive Ed Addison. It also means these candidate molecules can be tested in fewer animals and the drug development cycle speeded up.
"Collaborative drug discovery and development can be securely managed among multiple partners via the cloud, which is what Cloud Pharmaceuticals does," says Mr Addison.
Computer says yes
Meanwhile, Japanese tech company Fujitsu has been working with the University of Tokyo to use IT-based drug discovery to develop pharmacologically active compounds with the potential to become anti-cancer drugs.
Of the four main stages before new drugs arrive on the market, the second stage is compound design and synthesis evaluation - this is the area Fujitsu is working on.
"In silico drug discovery... allows for the design of various compounds that people wouldn't usually come up with, then after narrowing them down, carries out a compound evaluation," said Nozomu Kamiya, director in the healthcare systems unit at Fujitsu.
Graphic of drug moleculeImage copyrightCloud Pharmaceuticals
Image caption
In silico - or computer-aided - design of new molecules could speed up drug discovery
This allows the company to assess how effective a new compound might be "before it is actually physically made", he says.
"Together with the development of cloud computing, I think we will witness in silico drug discovery bringing innovation to new drug-related R&D [research and development] in the near future."
Takudzwa Musiyarira, health specialist at research company Frost & Sullivan, says cloud computing has "enormous potential" to improve healthcare services.
"Importantly, using genome research means that those at risk of developing cancer may be identified and remedial action taken before the onset of the disease," he tells the BBC.
"However, security continues to be a major concern for healthcare IT, especially with cyber threats on the increase, so these projects need to ensure that patient privacy is prioritised."
"I think a lot of countries are 10 years ahead of us now," says Per Olav Hetland, president of the Norsk Organisasjon For Rullebrett (NORB), a lobbying group and body for skateboarders in Norway.
""",
"""
"We were wrong to over-regulate and interfere too much," he said. For decades he has been an influential figure at the heart of EU integration moves.
"It is true we're not very popular when we advocate for Europe. We're no longer respected in our countries when we emphasise the need to give priority to the European Union," Mr Juncker said.
""")
In [303]:
vectorizer.fit_transform(a)
open("/tmp/x.txt", "w+").write(str(vectorizer.vocabulary_))
In [305]:
smatrix = vectorizer.transform(a).todense()
smatrix.tofile("/tmp/x.txt")
In [306]:
# a = []
# a.append("""elma armut elma uzum""")
# a.append("""elma karpuz elma karpuz elma domates elma""")
TCDictionary = {}
for docID, d in enumerate(list(a)):
N = len(d.lower().split(" "))
for t in d.lower().split(" "):
tData = TCDictionary.get(t, {"tf": {} })
w = tData["tf"].get(docID, 0)
tData["tf"][docID] = float(w * N + 1) / N
TCDictionary[t] = tData
for t in TCDictionary:
df = len(TCDictionary[t]["tf"])
idf = len(list(a)) / float(1 + df)
TCDictionary[t]["idf"] = idf
from pprint import pprint as pp
open("/tmp/x.txt", "w").write(str(TCDictionary))
#pp(TCDictionary)
In [259]:
list(a)[22]
Out[259]:
In [253]:
smatrix[:,vectorizer.vocabulary_["learning"]].sum()
Out[253]:
In [ ]: