notebook.community

Edit and run

PTT Crawler

爬蟲架構





In [ ]:



    


#上層爬蟲:處理換頁、資料總筆數問題



    











In [ ]:



    


#中層爬蟲:處理單一頁面內每筆資料頁面(URL)連結問題



    











In [ ]:



    


#下層爬蟲:清理資料，放入DIST。



    











In [25]:



    


import requests
from bs4 import BeautifulSoup
from collections import OrderedDict
import json
import traceback
import re



    











In [2]:



    


HOST = 'https://www.ptt.cc'



    











In [23]:



    


list_ptt = []

#def ptt_crawler(no_pages):
def ptt_crawler(board_name, no_pages):
    res = requests.get(HOST + "/bbs/" + board_name + "/index.html")
    soup = BeautifulSoup(res.text, 'lxml')
    total_page = int(soup.select('a.wide')[1]['href'].split('index')[1].split('.html')[0]) + 1

    for page in range(total_page, total_page - no_pages, -1):
        url = HOST + "/bbs/{}/index{}.html".format(board_name,page)
        print(url)
        #交給中層爬蟲
        list_crawler(url)
    print("[INFO] crawl_ptt complete!")
    
    #將爬取資料寫到txt檔案
    crawler_writer(board_name)
    #將爬取資料寫到json檔案
    crawler_json_writer(board_name)
    
#中層爬蟲   
def list_crawler(url):
    res_ls = requests.get(url)
    soup_ls = BeautifulSoup(res_ls.text, 'lxml')
    for tag in soup_ls.select('div.title > a'):
        title = tag.text
        url_ls = HOST + tag['href']
        print(url_ls)
        #交給下層爬蟲
        article_crawler(url_ls)
    print("[INFO] {} complete!".format(url_ls))

#下層爬蟲
def article_crawler(url_ls):    
    res_article = requests.get(url_ls)
    soup_article = BeautifulSoup(res_article.text, 'lxml')
    article_dict = OrderedDict()
    
    meta_values = soup_article.select('span.article-meta-value')
    try:
        article_dict['author'] = meta_values[0].text
        article_dict['board'] = meta_values[1].text
        article_dict['title'] = meta_values[2].text
        article_dict['tm'] = meta_values[3].text
    except IndexError:
        print("[Error] Error while parsing meta_values")

    #push的資料判斷有問題，不確定if...else判斷式哪裡有錯
    #請老師告訴我語法哪裡有問題
    #謝謝!
    pushs = []
    try:
        for push in soup_article.select('div.push'):
            push_dict = {}
            push_spans = push.select('span')
            if push_spans[0].text != "":
                push_dict['push_type'] = push_spans[0].text.strip()
            else:
                push_dict['push_type'] = None
            if push_spans[1].text != "":
                push_dict['push_author'] = push_spans[1].text.strip()
            else:
                push_dict['push_author'] = None
            print(push_spans[2])
            if push_spans[2].text != ":":
                 push_dict['push_content'] = push_spans[2].text.split(': ')[1].strip()
            else:
                push_dict['push_content'] = None 
            if push_spans[3].text != "":
                push_dict['push_tm'] = push_spans[3].text.strip()
            else:
                push_dict['push_tm'] = None
            pushs.append(push_dict)

                 #將推文加入字典檔
            article_dict['pushs'] = pushs

        for tag in soup_article.select('span.article-meta-value'):
            tag.extract()
        for tag in soup_article.select('span.article-meta-tag'):
            tag.extract()
        for tag in soup_article.select('span.f2'):
            tag.extract()
        for tag in soup_article.select('div.push'):
            tag.extract()
        article_dict['content'] = soup_article.select_one("#main-content").text.strip()    
        print("[INFO] {} complete!".format(url_ls))
    except IndexError:
        print("[Error] Error while parsing push")
        print(traceback.format_exc())
        
    list_ptt.append(article_dict)
    
def crawler_writer(board_name):
    with open('./list_ptt_{}.txt'.format(board_name), 'w', encoding='utf-8') as f:
        for posts in list_ptt:
            f.write('\n')
            for keys in posts:
                valus_str = "{} : {}".format(keys, str(posts[keys]))
                f.write(valus_str + '\n')
    print("[INFO] riter complete!")

def crawler_json_writer(board_name):
    with open('./json_ptt_{}.json'.format(board_name), 'a', encoding='utf-8') as fj:
        fj.write(json.dumps(list_ptt, ensure_ascii=False))
    print("[INFO] json_writer complete!")



    











In [24]:



    


ptt_crawler('ObataTakeshi',2)



    


















    






https://www.ptt.cc/bbs/ObataTakeshi/index52.html
https://www.ptt.cc/bbs/ObataTakeshi/M.1456976001.A.3D9.html
<span class="f3 push-content">: 覺得近期老師合作的作品節奏都有點怪…這篇鉑金也是</span>
<span class="f3 push-content">: 怎麼辦覺得不太好看...</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1456976001.A.3D9.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1459527591.A.A71.html
<span class="f3 push-content">: 等好久啦～～～～～～～～～～～</span>
<span class="f3 push-content">: QAQ</span>
<span class="f3 push-content">: thanks!</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1459527591.A.A71.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1462733392.A.44C.html
<span class="f3 push-content">: thanks!</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1462733392.A.44C.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1464874549.A.080.html
<span class="f3 push-content">: 等了一個月　結果有一半都是明日在窩囊！！！！！！！！</span>
<span class="f3 push-content">: 還是覺得重點在主角完全沒有個人魅力...</span>
<span class="f3 push-content">: 更糟糕的是其他推動劇情的角色也沒什麼魅力...</span>
<span class="f3 push-content">: 最有趣的就是娜賽和勒貝爾吧</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1464874549.A.080.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1467769548.A.B0A.html
<span class="f3 push-content">: 啊啊啊啊啊啊阿啊啊啊啊啊啊</span>
<span class="f3 push-content">: 真的好討厭明日XDDDD</span>
<span class="f3 push-content">: 覺得角色個性單一又死板, 有點追不下去</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1467769548.A.B0A.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1470276929.A.84E.html
<span class="f3 push-content">: 啊啊啊啊啊啊啊啊啊啊</span>
<span class="f3 push-content">:</span>
<span class="f3 push-content">: 每次看到標題都以為要被腰斬結局了...</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1470276929.A.84E.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1472831266.A.8C1.html
<span class="f3 push-content">: 等一個月看到明日一直耍廢真的會氣死！！！！！！！！！</span>
<span class="f3 push-content">: 主角真的很窩囊 超受不了的</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1472831266.A.8C1.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1475603693.A.802.html
<span class="f3 push-content">: 好看!</span>
<span class="f3 push-content">: 小咲在樓頂告白時，主角有一幕表情有點陰，不知道和小</span>
<span class="f3 push-content">: 咲突然變得開朗積極有關?</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1475603693.A.802.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1478276984.A.46E.html
<span class="f3 push-content">: 好像開始變好看的感覺　但一個月好久啊啊啊啊！！！！！</span>
<span class="f3 push-content">: 越來越喜歡勒貝爾XDDDDD</span>
<span class="f3 push-content">: 變好看+1</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1478276984.A.46E.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1479980398.A.91B.html
<span class="f3 push-content">: 第二話車上激戰就有露點了啊？</span>
<span class="f3 push-content">: 有點慢，現在都在等繁中翻譯，第二集才到第六話 orz</span>
<span class="f3 push-content">: 好看</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1479980398.A.91B.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1480734517.A.F96.html
<span class="f3 push-content">: thanks</span>
<span class="f3 push-content">: 新人物蠻方尼的XDDDD　難得這話完全沒有明日　清新（？</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1480734517.A.F96.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1165423774.A.05A.html
<span class="f3 push-content">::)</span>
[Error] Error while parsing push
Traceback (most recent call last):
  File "<ipython-input-23-c49f5fda2096>", line 66, in article_crawler
    push_dict['push_content'] = push_spans[2].text.split(': ')[1].strip()
IndexError: list index out of range

[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1165423774.A.05A.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/index51.html
https://www.ptt.cc/bbs/ObataTakeshi/M.1427557532.A.ED4.html
<span class="f3 push-content">: 老師：歡迎加入前科犯一組　　超幽默我愛老師XDDDDD</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1427557532.A.ED4.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1428315113.A.FC7.html
<span class="f3 push-content">: 怎麼好像有點收尾的味道？忽然一堆人的身份都跑出來？</span>
<span class="f3 push-content">: 雖然之前就有埋梗，不過一回內全部拉出來，好急湊喔！</span>
<span class="f3 push-content">: 最近排名很後面,大概被通知腰斬了</span>
<span class="f3 push-content">: 漫畫跟現實畢竟不一樣</span>
<span class="f3 push-content">: 很明顯幾回內就完結了啊</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1428315113.A.FC7.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1428761057.A.EF2.html
<span class="f3 push-content">: 怎麼辦班長好帥~~~~~~~~~~~</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1428761057.A.EF2.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1429364720.A.15C.html
<span class="f3 push-content">: 還不錯的說  可惜應該快斬了: (</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1429364720.A.15C.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1429864142.A.FDE.html
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1429864142.A.FDE.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1431182592.A.05D.html
<span class="f3 push-content">: 囧...最終話但是沒結束</span>
<span class="f3 push-content">: 竟然連最終話都沒ENDING太慘了吧 真的完全是腰斬=_=</span>
<span class="f3 push-content">: 逼人買單行本啊啊啊啊啊!!!!!!!</span>
<span class="f3 push-content">: (補劇情NOT收藏</span>
<span class="f3 push-content">: 其實我也覺得不好看XD 要不是小火田畫的</span>
<span class="f3 push-content">: 我覺得背景設定在小學生 但是劇情案件卻不太恰當</span>
<span class="f3 push-content">: 反而使得案件收尾很詭異 不太精采</span>
<span class="f3 push-content">: 好慘 就這樣結束</span>
<span class="f3 push-content">: 小火田每一部我都有收 但好猶豫要不要收這部啊..</span>
<span class="f3 push-content">: 分鏡很糟啊 原作很不會說故事</span>
<span class="f3 push-content">: 小畑健只是作畫啊...</span>
<span class="f3 push-content">: 果然被斬了 小田還是適合畫青年漫阿!!</span>
<span class="f3 push-content">: 這部是給小學生看的, 在座的各位都太老了</span>
<span class="f3 push-content">: 聽樓上一講才想到 這篇根本就是爆漫王的完美犯罪</span>
<span class="f3 push-content">: 小學生當主角 每次犯案都不是出於惡意 手法也都很完美</span>
<span class="f3 push-content">: 就我看來 小學生上課不上課 只會開法庭 根本荒謬@@</span>
<span class="f3 push-content">: 小學女生很可口，以外沒了</span>
<span class="f3 push-content">: 覺得不好看 有種PCP的感覺</span>
<span class="f3 push-content">: 腰斬正常, PCP可以不代表學糾可以, 其實PCP也沒多吸引人</span>
<span class="f3 push-content">: 神X小偵探如果是走PCP的路子早就腰斬了</span>
<span class="f3 push-content">: 原作的分鏡到作畫手上也是會變動的啊</span>
<span class="f3 push-content">: 爆漫裡面比較想看的就真像教室跟走光格鬥</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1431182592.A.05D.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1433140621.A.C2C.html
<span class="f3 push-content">: 結局超...ZZZZZZZZZZ</span>
<span class="f3 push-content">: 不過最後畢業典禮大家穿西裝蠻帥的</span>
<span class="f3 push-content">: 幸好還有結局XD</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1433140621.A.C2C.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1433153150.A.314.html
<span class="f3 push-content">: 感覺一開始就設定好了 不然這題材超難發揮</span>
<span class="f3 push-content">: 算是有頭有尾的短篇了 看看左近和藍龍才叫真腰斬</span>
<span class="f3 push-content">: 我認為這是小畑最鳥的作品，畫工是一樣優，但劇本太爛</span>
<span class="f3 push-content">: 本以為是講證據的漫畫，結果是最爛的偵探劇，證據都沒</span>
<span class="f3 push-content">: 啥可推理的，根本就原作者說了算</span>
<span class="f3 push-content">: 有誰能舉例一下哪裡有矛盾之處嗎</span>
<span class="f3 push-content">: 看鼠繪有說後面劇情跟前面有矛盾，不曉得是？</span>
<span class="f3 push-content">: 先前的案件 天都還對血之班會事件感到震驚 像是局外人</span>
<span class="f3 push-content">: 然後最終話就變當事人 應該是指這部份</span>
<span class="f3 push-content">: 這題材剛出就不看好 設定太詭異又不吸引人 但想不到斬這</span>
<span class="f3 push-content">: 麼快</span>
<span class="f3 push-content">: 當小品看 不要當推理看</span>
<span class="f3 push-content">: 連續兩部了</span>
<span class="f3 push-content">: All You Need Is Kill不算吧, 應該是一開始就大概這種長度</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1433153150.A.314.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1442545908.A.E2C.html
<span class="f3 push-content">: 謝謝</span>
<span class="f3 push-content">: 這群13歲的也太幼齒了吧!!!!! 是學糾法庭畫太多轉不回嗎</span>
<span class="f3 push-content">: 喔喔喔喔　太棒了</span>
<span class="f3 push-content">: 平丸亂入XDDDDDDD</span>
<span class="f3 push-content">: 高木好萌</span>
<span class="f3 push-content">: 好懷念以前追連載的時光呀...</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1442545908.A.E2C.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1443421226.A.B32.html
<span class="f3 push-content">: 期待!!!   神一般的組合</span>
<span class="f3 push-content">: 終於 小火田還是適合畫青年漫阿!</span>
<span class="f3 push-content">: 推!</span>
<span class="f3 push-content">: 推畑神跟鶇爺啊!!!!!!!!!!!!</span>
<span class="f3 push-content">: 終於!!希望推出比死筆更精彩的內容</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1443421226.A.B32.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1443507459.A.FD3.html
<span class="f3 push-content">: 這集完全就是高木的少女心小劇場XDDDDDD</span>
<span class="f3 push-content">: 期待新連載~</span>
<span class="f3 push-content">: 看完又想再看一次爆漫了  XD</span>
<span class="f3 push-content">: 人物都變好Q喔</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1443507459.A.FD3.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1443770292.A.C2E.html
<span class="f3 push-content">: 越來越期待了</span>
<span class="f3 push-content">: 超級期待~~~!</span>
<span class="f3 push-content">: 一開始就畫自殺，這樣還能刊載蠻強的</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1443770292.A.C2E.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1443896864.A.107.html
<span class="f3 push-content">: movie版的評價不錯說...</span>
<span class="f3 push-content">: 可能切入角度不同吧</span>
<span class="f3 push-content">: 至少我覺得 亜城木夢叶這個筆名 是精神象徵</span>
<span class="f3 push-content">: 看起來真的被改很大</span>
<span class="f3 push-content">: 劇情方面我其實是給高評價 但是人物刻畫方面就差多了</span>
<span class="f3 push-content">: 見吉消失我就不想看了,典型的「那一種拍電影手法」</span>
<span class="f3 push-content">: 以電影來說是好看的，我很少覺得日本電影好看，但這部節奏</span>
<span class="f3 push-content">: 還不錯，不過以原著精神來看，我覺得這部很不「bakuman」，</span>
<span class="f3 push-content">: 更像是另一個故事，能不能接受就看個人了。</span>
<span class="f3 push-content">: 樓上正解 我是先看電影才看原作 劇情完全不同 不過是少</span>
<span class="f3 push-content">: 數真的很好看的日本電影</span>
<span class="f3 push-content">: 看了很多電影版好評決定去看，結果覺得滿失望的 XD</span>
<span class="f3 push-content">: 我覺得我就算沒看過原作也無法給這部電影好評</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1443896864.A.107.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1444552761.A.DAA.html
<span class="f3 push-content">: 很好看 大推!!!!!</span>
<span class="f3 push-content">: 期待！！</span>
<span class="f3 push-content">: 以身高 體型 臉孔 兩位主角互換 不是比較像原作嗎</span>
<span class="f3 push-content">: 難看…</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1444552761.A.DAA.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1446301071.A.0A4.html
<span class="f3 push-content">: 很多人多說這設定很像未來日記</span>
<span class="f3 push-content">: 比較像REVERSI吧?????</span>
<span class="f3 push-content">: 竟然是月刊~~~~~等好久~~~~~~~~</span>
<span class="f3 push-content">: 死筆＋未來日記</span>
<span class="f3 push-content">: 天使跟男主角他媽好像,不知道是不是故意的...</span>
<span class="f3 push-content">: 被射到整個眼睛都變了XDD  整容術嗎XDDDD</span>
<span class="f3 push-content">: 大家都忘記"我的天使"了 <a href="http://imgur.com/s6Px9DU.jpg" rel="nofollow" target="_blank">http://imgur.com/s6Px9DU.jpg</a></span>
<span class="f3 push-content">: 從打工魔王,康斯坦汀可以看出,天使裡頭也是可以有壞傢伙</span>
<span class="f3 push-content">: 服部哲以前對岩瀨建議過「不要用神明的孩子,神明改成神秘</span>
<span class="f3 push-content">: 客,後續會比較有想像空間",他這篇倒是一開始就說是神了</span>
<span class="f3 push-content">: 恩.....</span>
<span class="f3 push-content">: 打鬥故事...希望這部能活久一點</span>
<span class="f3 push-content">: 先猜像L的反派 應該在10話才出現</span>
<span class="f3 push-content">: 不錯看，可惜姑姑很快就死了</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1446301071.A.0A4.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1447753578.A.83A.html
<span class="f3 push-content">: 可惡...已買日版orz</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1447753578.A.83A.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1449110236.A.2A6.html
<span class="f3 push-content">: 這漫畫名字會有種每期都是完結篇的錯覺</span>
<span class="f3 push-content">: 上一話是姑姪  這一話是多P  A＿A</span>
<span class="f3 push-content">: 是說這雙主角的人設越看越像REVERSI</span>
<span class="f3 push-content">: 就看起來上一次嚐試PCP的風格,這次嚐試REVERSI的風格</span>
<span class="f3 push-content">: 而且第二話就殺掉其它人的乾脆做法也像ZombieGun的思路</span>
<span class="f3 push-content">: 感覺天使應該是他媽的化身之類的</span>
<span class="f3 push-content">: 老師第一次畫露點XD</span>
<span class="f3 push-content">: 恩....</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1449110236.A.2A6.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1451486477.A.03C.html
<span class="f3 push-content">: 好煩～～～～月刊好漫長～～～～～～～</span>
<span class="f3 push-content">: thanks!</span>
<span class="f3 push-content">: 我猜主角應該就算被刺了也沒效，因為他早就喜歡她了</span>
<span class="f3 push-content">: 學校變戰場</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1451486477.A.03C.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1454506356.A.01F.html
<span class="f3 push-content">: 推推</span>
<span class="f3 push-content">: thanks!</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1454506356.A.01F.html complete!
https://www.ptt.cc/bbs/ObataTakeshi/M.1456974560.A.7D1.html
<span class="f3 push-content">: 我覺得主角弱很有意思啊！</span>
<span class="f3 push-content">: 主角真的完全沒有魅力的鬥智漫畫, 這樣的設定實在是</span>
<span class="f3 push-content">: 主角弱就跟大雄一樣 對不多啦王</span>
<span class="f3 push-content">: 多啦王!!</span>
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1456974560.A.7D1.html complete!
[INFO] https://www.ptt.cc/bbs/ObataTakeshi/M.1456974560.A.7D1.html complete!
[INFO] crawl_ptt complete!
[INFO] riter complete!
[INFO] json_writer complete!

Content source: nick800608/TeamProject-FoodRecipe

Similar notebooks:

notebook.community | gallery | about