This aims to get relevant data from:
https://www.autismparentingmagazine.com/applied-behavior-analysis-aba/
In [1]:
import requests # to make GET request
from bs4 import BeautifulSoup # to parse the HTML response
import time # to pause between calls
import pandas as pd # to see CSV
import os
In [2]:
def parse_page(response):
"""This parses pages from the Autism Parent Magazine web site using beautifulsoup"""
global df
global source
# parse the response into an HTML tree soup object
# read the content of the server’s response as a string
page_source = response.text
soup = BeautifulSoup(page_source, 'html5lib')
# take a look
#print(soup.prettify()[:1000])
# Find all "<div class="entry-row" appearances
links=soup.select("div.entry-row")
for link in links:
article=link.select("article")
# Get category (for validation)
for element in article[0]['class']:
if ( 'category' in element ):
category=element
break
# Get text
#article[0].find_all('div')[1].get_text()
text=article[0].find('div', class_='entry-summary entry-content').get_text()
# Get href. for further reading
href=article[0].find('a')['href']
# Get title
title=article[0].find('a')['title']
newrow={"title":title,"source":source,"category":category,"text":text,"href":href}
# Add entry into database
df.loc[len(df.values)]=newrow
In [3]:
os.chdir("../data")
# Create an empty database to keep the history of purchases
columns=["title","source","category","text","href"]
df=pd.DataFrame(columns=columns)
In [4]:
source="https://www.autismparentingmagazine.com/"
# Get list of pages to parse:
f = open("AutismParentMagazine-urls.txt","r")
urls = f.readlines()
f.close()
for ii in range(len(urls)):
url = urls[ii]
urls[ii]=url.strip()
In [5]:
#loop over urls:
for url in urls:
# make a GET request
response = requests.get(url)
parse_page(response)
time.sleep(5)
In [7]:
# Convert to file
df.to_csv("AutismParentMagazine-posts.csv")
In [8]:
df
Out[8]:
title
source
category
text
href
0
Autism, Head Banging and other Self Harming Be...
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
For children with autism spectrum disorder (AS...
https://www.autismparentingmagazine.com/autism...
1
High Quality ABA Treatment: What Every Parent...
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
Dr. Stephen Shore once said “If you’ve met one...
https://www.autismparentingmagazine.com/high-q...
2
Help: I Don’t Know How to Choose an Applied Be...
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
Help! I am going to be starting Applied Behav...
https://www.autismparentingmagazine.com/choosi...
3
HELP: My Autistic Child is Absolutely Terrifie...
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
How do you handle high anxiety of a child on t...
https://www.autismparentingmagazine.com/help-a...
4
HELP: I Need Communication Advice for Autistic...
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
A grandfather from Singapore asks… My eldest g...
https://www.autismparentingmagazine.com/help-i...
5
Help: My Aspie Lacks Motivation
https://www.autismparentingmagazine.com/
category-applied-behavior-analysis-aba
Hi, I am the parent of a 13-year-old boy with ...
https://www.autismparentingmagazine.com/help-m...
6
Exciting Campaign Launched to Tackle the Autis...
https://www.autismparentingmagazine.com/
category-autism-advocacy
It will be some time before we see a positive ...
https://www.autismparentingmagazine.com/autism...
7
How One Restaurateur Makes Eating Out Autism-F...
https://www.autismparentingmagazine.com/
category-autism-advocacy
Restaurant owner Andrew Iredale could hardly b...
https://www.autismparentingmagazine.com/restau...
8
Lloyd Claycomb’s Views on Autism
https://www.autismparentingmagazine.com/
category-autism-advocacy
Autism can be one of the most difficult diagno...
https://www.autismparentingmagazine.com/lloyd-...
9
Erasing the Prejudice – Are People on the Spec...
https://www.autismparentingmagazine.com/
category-autism-advocacy
An Interview with Dr. Tony Attwood, PhD, AS/AS...
https://www.autismparentingmagazine.com/erasin...
10
The Never Ending Battle for Autism Rights – A ...
https://www.autismparentingmagazine.com/
category-autism-advocacy
A civil rights lawyer and autism mom talks abo...
https://www.autismparentingmagazine.com/the-ne...
11
How One Special Bear Changed the Way People Se...
https://www.autismparentingmagazine.com/
category-autism-advocacy
In two short years, Tikko has captured the hea...
https://www.autismparentingmagazine.com/how-on...
12
Winning the Fight for Autism
https://www.autismparentingmagazine.com/
category-autism-advocacy
Every time the professional fighters meet thei...
https://www.autismparentingmagazine.com/winnin...
13
The Right for Autism Services for U.S. Militar...
https://www.autismparentingmagazine.com/
category-autism-advocacy
Autism Services for Military Families by Lesli...
https://www.autismparentingmagazine.com/autism...
14
Bridge the Gap Gala Raises over $100,000
https://www.autismparentingmagazine.com/
category-autism-advocacy
Bridge the Gap Gala Raises over $100,000 by Le...
https://www.autismparentingmagazine.com/bridge...
15
Exposing Autism Speaks – Podcast 4
https://www.autismparentingmagazine.com/
category-autism-advocacy
We started this magazine to help parents, so...
https://www.autismparentingmagazine.com/exposi...
16
Top Ways a Gluten Free Diet Can Help Kids wit...
https://www.autismparentingmagazine.com/
category-autism-and-diet
Recent clinical trials looking at gluten-free ...
https://www.autismparentingmagazine.com/gluten...
17
Asperger’s Syndrome and the Power of a Good Diet
https://www.autismparentingmagazine.com/
category-autism-and-diet
I find the suggestion that there needs to be a...
https://www.autismparentingmagazine.com/asperg...
18
Six Steps to a Fun, Friendly, Family Fitness P...
https://www.autismparentingmagazine.com/
category-autism-and-diet
“Wow, how did you get him/her to do that?,” pa...
https://www.autismparentingmagazine.com/six-st...
19
Q&A Section: How does Gluten free casein free ...
https://www.autismparentingmagazine.com/
category-autism-and-diet
This is the question of the month as featured ...
https://www.autismparentingmagazine.com/qa-sec...
20
Mallory’s Marvelous Muffins
https://www.autismparentingmagazine.com/
category-autism-and-food
I had the pleasure of reviewing Mallory’s Marv...
https://www.autismparentingmagazine.com/mallor...
21
Q and A Do sugary foods make children with aut...
https://www.autismparentingmagazine.com/
category-autism-and-food
Sivakumari asks: My daughter loves to eat ice-...
https://www.autismparentingmagazine.com/sugary...
22
Social Stories for Autistic Children
https://www.autismparentingmagazine.com/
category-autism-articles
Children with autism spectrum disorder (ASD) c...
https://www.autismparentingmagazine.com/social...
23
Creative Ways to Help a Child with Autism Enjo...
https://www.autismparentingmagazine.com/
category-autism-articles
How does your child react in the water—with pl...
https://www.autismparentingmagazine.com/ways-h...
24
Autism and Anxiety Disorder – The Double Diagn...
https://www.autismparentingmagazine.com/
category-autism-articles
Over the years, lack of eye contact from a per...
https://www.autismparentingmagazine.com/autism...
25
Mental Health and Anxiety Treatment with Autis...
https://www.autismparentingmagazine.com/
category-autism-articles
If your child has autism spectrum disorder (AS...
https://www.autismparentingmagazine.com/mental...
26
Helping People With Autism Communicate in a Di...
https://www.autismparentingmagazine.com/
category-autism-articles
Lately, the news has been immersed in images o...
https://www.autismparentingmagazine.com/help-a...
27
I See You: Raising Neurotypical and Autistic S...
https://www.autismparentingmagazine.com/
category-autism-articles
When we brought my seven-year-old stepdaughter...
https://www.autismparentingmagazine.com/raisin...
28
The Awesome Rewards of a School Service Dog fo...
https://www.autismparentingmagazine.com/
category-autism-articles
Families navigating through the world of autis...
https://www.autismparentingmagazine.com/servic...
29
Signs of Autism in Infants and Children
https://www.autismparentingmagazine.com/
category-autism-articles
Early signs of autism spectrum disorder (ASD) ...
https://www.autismparentingmagazine.com/signs-...
...
...
...
...
...
...
204
KinderKord: A Safe Way To Stay Connected
https://www.autismparentingmagazine.com/
category-autism-articles
KinderKord by Joan Lunden KinderKord is a ...
https://www.autismparentingmagazine.com/kinder...
205
My Signals: Autistic Reflections of a Man Who ...
https://www.autismparentingmagazine.com/
category-personal-narrative
My Signals: Autistic Reflections by Patrick Ja...
https://www.autismparentingmagazine.com/signal...
206
Go-Go Blanket: The perfect travel blanket for ...
https://www.autismparentingmagazine.com/
category-autism-news
Review of the Go-Go blanket by Go-Go Blanket...
https://www.autismparentingmagazine.com/go-go-...
207
Black and White: A Colorful Look at the Spectrum
https://www.autismparentingmagazine.com/
category-autism-books
Black and White: A Colorful Look at the Spect...
https://www.autismparentingmagazine.com/black-...
208
Sparrow Migrations: A Novel
https://www.autismparentingmagazine.com/
category-autism-books
Sparrow Migrations: A Novel by Cari Noga Neu...
https://www.autismparentingmagazine.com/sparro...
209
Creative Ways to Help a Child with Autism Enjo...
https://www.autismparentingmagazine.com/
category-autism-articles
How does your child react in the water—with pl...
https://www.autismparentingmagazine.com/ways-h...
210
Easy Ways to Create A Sensory Space for Kids w...
https://www.autismparentingmagazine.com/
category-autism-articles
Children with autism spectrum disorder (ASD) m...
https://www.autismparentingmagazine.com/creati...
211
Simple Ways to Make Bath Time a Splash with Se...
https://www.autismparentingmagazine.com/
category-autism-articles
Like so many parents-to-be, when I was pregnan...
https://www.autismparentingmagazine.com/bathin...
212
Simple Ways to Help Kids with Autism Develop H...
https://www.autismparentingmagazine.com/
category-autism-articles
I learned almost 30 years ago that my extremel...
https://www.autismparentingmagazine.com/simple...
213
Simple Ways Sensory Based Intervention Can Cha...
https://www.autismparentingmagazine.com/
category-autism-articles
One in six children experiences sensory diffic...
https://www.autismparentingmagazine.com/sensor...
214
Amazing New Fidget Pillow Calms Busy Hands and...
https://www.autismparentingmagazine.com/
category-autism-articles
My desire to help my child be successful in sc...
https://www.autismparentingmagazine.com/new-fi...
215
5 Top Reasons Qigong May be Right for Your Spe...
https://www.autismparentingmagazine.com/
category-autism-articles
By Sazini Nzula, PhD, Certified QST Therapist ...
https://www.autismparentingmagazine.com/qigong...
216
Brain Balance Takes a Look at the Challenge of...
https://www.autismparentingmagazine.com/
category-autism-articles
“I can’t wear those jeans!” “No, I won’t eat m...
https://www.autismparentingmagazine.com/brain-...
217
Putting the Squeeze On Sensory Processing Diso...
https://www.autismparentingmagazine.com/
category-autism-articles
Industrial Designer Lisa Fraser never imagined...
https://www.autismparentingmagazine.com/puttin...
218
Sensory Processing Difficulties – Dressing For...
https://www.autismparentingmagazine.com/
category-autism-articles
What is Sensory Processing disorder? Many chil...
https://www.autismparentingmagazine.com/sensor...
219
Simple Ways to Help Your Child with ASD Sleep ...
https://www.autismparentingmagazine.com/
category-autism-articles
Insomnia (sleeplessness) is quite common among...
https://www.autismparentingmagazine.com/autism...
220
Sure Fire Ways to Help Your ASD Child Go To Bed
https://www.autismparentingmagazine.com/
category-autism-articles
In order to have a good day, your child must s...
https://www.autismparentingmagazine.com/sure-f...
221
The Calming Effects of Essential Oils
https://www.autismparentingmagazine.com/
category-autism-articles
More and more we are hearing about essential o...
https://www.autismparentingmagazine.com/the-ca...
222
Helping Your Child with Autism Sleep: An Overv...
https://www.autismparentingmagazine.com/
category-autism-articles
Sleep problems are common amongst children. Fo...
https://www.autismparentingmagazine.com/helpin...
223
Simple Ways to Make Social Skill Connections U...
https://www.autismparentingmagazine.com/
category-autism-articles
When you first hear that a computer game or a ...
https://www.autismparentingmagazine.com/social...
224
Special Interactive Program for Kids Encourage...
https://www.autismparentingmagazine.com/
category-autism-articles
“You’re having twins!” Can you imagine the exc...
https://www.autismparentingmagazine.com/specia...
225
4 Simple Steps to Help Your Child PREP for Soc...
https://www.autismparentingmagazine.com/
category-autism-articles
We have been working with families of children...
https://www.autismparentingmagazine.com/prep-f...
226
5 Easy Ways To Help Improve Executive Functioning
https://www.autismparentingmagazine.com/
category-autism-articles
Rick walks into his classroom, and he is angry...
https://www.autismparentingmagazine.com/ways-i...
227
Tips for Teaching the Most Important Social Sk...
https://www.autismparentingmagazine.com/
category-autism-articles
As a child, we had a player piano in my home. ...
https://www.autismparentingmagazine.com/most-i...
228
5 Simple Ways to Help Your Child with ASD Cope...
https://www.autismparentingmagazine.com/
category-autism-articles
Welcome back to Social Skills Corner. For the ...
https://www.autismparentingmagazine.com/cope-w...
229
Top 5 Social Skills Tips for Teens with Specia...
https://www.autismparentingmagazine.com/
category-autism-articles
Social skills are especially difficult for tee...
https://www.autismparentingmagazine.com/social...
230
How to Help Your Child Develop Social Skills T...
https://www.autismparentingmagazine.com/
category-autism-articles
Socializing forms a big part of our everyday l...
https://www.autismparentingmagazine.com/social...
231
5 Things You Need to Know About Social Skills
https://www.autismparentingmagazine.com/
category-autism-articles
Welcome to Social Skills Corner. We have been...
https://www.autismparentingmagazine.com/knowin...
232
10 Valuable Ways to Cope with Social Interaction
https://www.autismparentingmagazine.com/
category-autism-articles
Coping with Asperger’s syndrome can be a day-t...
https://www.autismparentingmagazine.com/10-val...
233
Using Mindfulness to Conquer Parental Stress Now
https://www.autismparentingmagazine.com/
category-autism-articles
As the parent of a child with autism spectrum ...
https://www.autismparentingmagazine.com/using-...
234 rows × 5 columns
In [ ]:
Content source: trangel/Insight-Data-Science
Similar notebooks: