SecLists Index parsing

Example: http://seclists.org/fulldisclosure/2017/Jan

We will do basic parsing of the index file, mainly to extract the thread hierarchy contained within the <ul> tags.

Normally, we have the file already downloaded via seclists_crawler_raw.py, but for this notebook, we'll download a fresh copy.


In [1]:
import requests
from bs4 import BeautifulSoup
from IPython.display import Pretty
import pprint

pp = pprint.PrettyPrinter(indent=4)


url = 'http://seclists.org/fulldisclosure/2017/Jan'
r = requests.get(url)
raw = r.text
Pretty(raw)


Out[1]:
<!-- SecLists-Message-Count: 99 -->
<!-- MHonArc v2.6.19 -->
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
                      "http://www.w3.org/TR/REC-html40/loose.dtd">
<HTML>
<HEAD>
<link rel="alternate" type="application/rss+xml" title="RSS" href="http://seclists.org/rss/fulldisclosure.rss">
<meta property="og:image" content="http://seclists.org/images/fulldisclosure-img.png" />
<link rel="image_src" href="http://seclists.org/images/fulldisclosure-img.png" />
<title>Full Disclosure: by thread</title>
<link REL="SHORTCUT ICON" HREF="/shared/images/tiny-eyeicon.png" TYPE="image/png">
<META NAME="ROBOTS" CONTENT="NOARCHIVE">
<meta name="theme-color" content="#2A0D45">
<link rel="stylesheet" href="/shared/css/insecdb.css" type="text/css">
<!--Google Analytics Code-->
<script type="text/javascript">
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');

  ga('create', 'UA-11009417-1', 'auto');
  ga('send', 'pageview');

</script>
<!--END Google Analytics Code-->

<!--Google Custom Site Search boilerplate Javascript-->
<script type="text/javascript">
  (function() {
    var cx = 'partner-pub-0078565546631069:bx60rb-fytx';
    var gcse = document.createElement('script'); gcse.type = 'text/javascript'; gcse.async = true;
    gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') +
        '//www.google.com/cse/cse.js?cx=' + cx;
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(gcse, s);
  })();
</script>
<!--End Google Custom Site Search boilerplate Javascript-->

</HEAD>
<BODY BGCOLOR="#2A0D45" TEXT="#000000">

<TABLE CELLPADDING="0" WIDTH="100%" CELLSPACING="0">
<TR><TD ALIGN="left"><A HREF="/"><IMG BORDER=0 ALT="Home page logo"
SRC="/images/sitelogo.png" HEIGHT=90 WIDTH=168></A></TD>
<TD VALIGN="bottom" ALIGN="right">
<!-- Begin TopBanner Code -->
<!-- AdSpeed.com Serving Code 7.9.6 for [Zone] TopBanner [Any Dimension] -->
<script type="text/javascript" src="//g.adspeed.net/ad.php?do=js&amp;zid=14678&amp;wd=-1&amp;ht=-1&amp;target=_blank"></script>
<!-- AdSpeed.com End -->
<!-- End Banner Code -->

</TD></TR></TABLE>
<TABLE WIDTH="100%" CELLPADDING="0" CELLSPACING="0"><TR>
<TD ALIGN="left" WIDTH="130" VALIGN="top" class="sidebar">

<!-- SECWIKI PORTAL INSERT -->

<ul>
<li><a href="https://nmap.org/">Nmap Security Scanner</a>
<ul>
<li><a href="https://nmap.org/">Intro</a></li>
<li><a href="https://nmap.org/book/man.html">Ref Guide</a></li>
<li><a href="https://nmap.org/book/install.html">Install Guide</a></li>
<li><a href="https://nmap.org/download.html">Download</a></li>
<li><a href="https://nmap.org/changelog.html">Changelog</a></li>
<li><a href="https://nmap.org/book/">Book</a></li>
<li><a href="https://nmap.org/docs.html">Docs</a></li>
</ul>
<li><a href="http://seclists.org/">Security Lists</a>
<ul>
<li><a href="http://seclists.org/nmap-announce/">Nmap Announce</a></li>
<li><a href="http://seclists.org/nmap-dev/">Nmap Dev</a></li>
<li><a href="http://seclists.org/bugtraq/">Bugtraq</a></li>
<li><a href="http://seclists.org/fulldisclosure/">Full Disclosure</a></li>
<li><a href="http://seclists.org/pen-test/">Pen Test</a></li>
<li><a href="http://seclists.org/basics/">Basics</a></li>
<li><a href="http://seclists.org/">More</a></li>
</ul>
<li><a href="http://sectools.org">Security Tools</a>
<ul>
<li><a href="http://sectools.org/tag/pass-audit/">Password audit</a></li>
<li><a href="http://sectools.org/tag/sniffers/">Sniffers</a></li>
<li><a href="http://sectools.org/tag/vuln-scanners/">Vuln scanners</a></li>
<li><a href="http://sectools.org/tag/web-scanners/">Web scanners</a></li>
<li><a href="http://sectools.org/tag/wireless/">Wireless</a></li>
<li><a href="http://sectools.org/tag/sploits/">Exploitation</a></li>
<li><a href="http://sectools.org/tag/packet-crafters/">Packet crafters</a></li>
<li><a href="http://sectools.org/">More</a></li>
</ul>
<li><a href="http://insecure.org/">Site News</a></li>
<li><a href="http://insecure.org/advertising.html">Advertising</a></li>
<li><a href="http://insecure.org/fyodor/">About/Contact</a></li>
<li>
<!-- SiteSearch Google -->
<form action="https://nmap.org/search.html" id="cse-search-box-sidebar">
  <div>
    <input type="hidden" name="cx" value="partner-pub-0078565546631069:bx60rb-fytx">
    <input type="hidden" name="cof" value="FORID:9">
    <input type="hidden" name="ie" value="ISO-8859-1">
    <input type="text" name="q" size="16">
    <input type="submit" name="sa" value="Site Search">
  </div>
</form>
<!-- End SiteSearch Google -->
</li>
<!-- These can come back if I ever update them ...
<li><a href="http://insecure.org/links.html">Exceptional Links</a></li>
<li><a href="http://insecure.org/reading.html">Good Reading</a></li>
<li><a href="http://insecure.org/sploits.html">Exploit World</a></li>
-->
<li><a href="http://insecure.org/advertising.html">Sponsors:</a>
  <br><br>
  
<!-- Begin Sidebar Banner Code -->
<A HREF="http://www.acunetix.com/web-vulnerability-manager/?utm_source=insecure&utm_medium=box&utm_term=chess&utm_campaign=insecure "><IMG SRC="/shared/images/Acunetix/acx_Chess-WB.gif" BORDER=0 ALT="Acunetix"></A>
<!-- End Sidebar Banner Code -->

<BR><BR>

<!-- Begin Bottom (Google) Sidebar Banner Code -->
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- SidebarSkyScraper -->
<ins class="adsbygoogle"
     style="display:inline-block;width:120px;height:600px"
     data-ad-client="ca-pub-0078565546631069"
     data-ad-slot="9829251079"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>
<!-- End Bottom (Google) Sidebar Banner Code -->

</li>
</ul>

</TD>
<TD BGCOLOR="#FFFFFF" VALIGN="top" ALIGN="left"><IMG
SRC="/shared/images/topleftcurve.gif" alt="/"><TABLE CELLPADDING="4" WIDTH="100%" style="table-layout: fixed;"><TR><TD BGCOLOR="#FFFFFF">

<div style="text-align: center">
<A HREF="/fulldisclosure/"><img src="/images/fulldisclosure-logo.png" border="0" width="80" style="vertical-align: middle" alt="fulldisclosure logo"></A>
<FONT SIZE="+2"><B><a href="http://seclists.org/fulldisclosure/">Full Disclosure</a>: by thread</B></FONT><BR>
<ul class="inline"><li class="first"><A HREF="/rss/fulldisclosure.rss"><img src="/images/feed-icon-16x16.png" border=0 width=16 height=16 alt="RSS icon">RSS Feed</A></li>
<li>&nbsp;<A HREF="https://nmap.org/mailman/listinfo/fulldisclosure"><img src="/images/about-icon-16x16.png" border=0 width=16 height=16 alt="About icon">About List</A></li>
<li>&nbsp;<A HREF="/"><img src="/images/up-icon-16x16.png" border=0 width=16 height=16 alt="Up icon">All Lists</A></li>
</ul>

</div>

<form action="http://insecure.org/search.html" id="top-search-box">
<table border="0" width="100%">
<tr>
<td align="left" valign="middle"><a href="/cgi-bin/nav.cgi?from=fulldisclosure%2F2017%2FJan&goto=prev&view=thread"><img src="/images/left-icon-16x16.png" border=0 width=16 height=16>Previous period</a></td>
<td align="center">
<!-- Google Custom SiteSearch -->
    <input type="hidden" name="cx" value="partner-pub-0078565546631069:bx60rb-fytx" />
    <input type="hidden" name="cof" value="FORID:9" />
    <input type="hidden" name="ie" value="ISO-8859-1" />
    <input type="text" name="q" size="24" />
    <input type="submit" name="sa" value="Search" />
</td>
<!-- End Google Custom SiteSearch -->
<td align="right" valign="middle"><a href="/cgi-bin/nav.cgi?from=fulldisclosure%2F2017%2FJan&goto=next&view=thread">Next period<img src="/images/right-icon-16x16.png" border=0 width=16 height=16></a></td>
</tr>
</table>
</form>
<script type="text/javascript" src="http://www.google.com/coop/cse/brand?form=top-search-box&amp;lang=en"></script>
<p>
99 messages
<a href="#begin">starting</a> Jan 03 17 and
<a href="#end">ending</a> Jan 31 17
<br>
<a href="date.html">Date index</a> |
Thread index |
<a href="author.html">Author index</a>
</p>
<a name="begin">
<ul>
<li><a name="0" href="0">Zend Framework / zend-mail &lt; 2.4.11 Remote Code Execution	(CVE-2016-10034)</a> <em>Dawid Golunski (Jan 03)</em></li>
<li><a name="1" href="1">CINtruder v0.3 released...</a> <em>psy (Jan 03)</em></li>
<li><a name="2" href="2">Advisories Unsafe Dll in Audacity, telegram and Akamai</a> <em>filipe (Jan 03)</em></li>
<li><a name="3" href="3">Persisted Cross-Site Scripting (XSS) in Confluence Jira	Software</a> <em>jlss (Jan 03)</em>
<ul>
<li><a name="9" href="9">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>Moritz Naumann (Jan 04)</em>
<ul>
<li><a name="12" href="12">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>jlss (Jan 06)</em>
</li>
</ul>
</li>
<li>&lt;Possible follow-ups&gt;</li>
<li><a name="11" href="11">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira	Software</a> <em>David Black (Jan 06)</em>
</li>
 </ul>
</li>
<li><a name="4" href="4">0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
<ul>
<li><a name="5" href="5">Re: 0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
<ul>
<li><a name="6" href="6">Re: 0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
</li>
</ul>
</li>
</ul>
</li>
<li><a name="7" href="7">Executable installers are vulnerable^WEVIL (case 43):	SoftMaker&apos;s Office service pack installers allow escalation	of privilege</a> <em>Stefan Kanthak (Jan 03)</em></li>
<li><a name="8" href="8">Executable installers are vulnerable^WEVIL (case 45):	ReadPDF&apos;s installers allow escalation of privilege</a> <em>Stefan Kanthak (Jan 03)</em></li>
<li><a name="10" href="10">Stop User Enumeration does not stop user enumeration	(WordPress plugin)</a> <em>dxw Security (Jan 04)</em></li>
<li><a name="13" href="13">CarolinaCon-13 - May 2017 - Call for Papers/Presenters and Attendees</a> <em>Vic Vandal (Jan 06)</em></li>
<li><a name="14" href="14">Trango Altum AC600 Default root Login</a> <em>Ian Ling (Jan 06)</em></li>
<li><a name="15" href="15">YSTS 11th Edition - CFP</a> <em>Luiz Eduardo (Jan 06)</em></li>
<li><a name="16" href="16">BSides Las Vegas 2017 CFP is open.</a> <em>Daemon Tamer (Jan 09)</em></li>
<li><a name="17" href="17">Hotlinking Vulnerability in PHProxy 0.5b2</a> <em>Celso Bento (Jan 09)</em></li>
<li><a name="18" href="18">enigma2-plugin-extensions-webadmin Remote Code Execution (IoT)</a> <em>Fabian Fingerle (Jan 09)</em></li>
<li><a name="19" href="19">pev 0.80 released</a> <em>Fernando Mercês (Jan 09)</em></li>
<li><a name="20" href="20">CSRF/XSS in Responsive Poll allows unauthenticated attackers	to do almost anything an admin can (WordPress plugin)</a> <em>dxw Security (Jan 10)</em></li>
<li><a name="21" href="21">Docker 1.12.6 - Security Advisory</a> <em>Nathan McCauley (Jan 10)</em>
<ul>
<li><a name="22" href="22">Re: [oss-security] Docker 1.12.6 - Security Advisory</a> <em>Kurt Seifried (Jan 10)</em>
<ul>
<li><a name="29" href="29">Re: [oss-security] Docker 1.12.6 - Security Advisory</a> <em>Andreas Stieger (Jan 11)</em>
</li>
</ul>
</li>
</ul>
</li>
<li><a name="23" href="23">Blackboard LMS 9.1 SP14 - (Profile) Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="24" href="24">BlackBoard LMS 9.1 SP14 - (Title) Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="25" href="25">Bit Defender #39 - Auth Token Bypass Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="26" href="26">Huawei Flybox B660 - (POST Reboot) CSRF Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="27" href="27">Boxoft Wav v1.1.0.0 - Buffer Overflow Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="28" href="28">Cobi Tools v1.0.8 iOS - Persistent Web Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>
<li><a name="30" href="30">[CVE-2016-3403] [Zimbra] Multiple CSRF in Administration interface - all versions</a> <em>Sysdream Labs (Jan 12)</em></li>
<li><a name="31" href="31">Multiple vulnerabilities in cPanel  &lt;= 60.0.34</a> <em>Open Security (Jan 12)</em></li>
<li><a name="32" href="32">ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)</a> <em>Fernando Gont (Jan 12)</em></li>
<li><a name="33" href="33">Nginx (Debian-based + Gentoo distros) - Root Privilege Escalation [CVE-2016-1247 UPDATE]</a> <em>Dawid Golunski (Jan 13)</em></li>
<li><a name="34" href="34">Executable installers are vulnerable^WEVIL (case 44):	SoftMaker&apos;s FlexiPDF installers allow escalation of privilege</a> <em>Stefan Kanthak (Jan 15)</em></li>
<li><a name="35" href="35">Huawei Flybox B660 - (POST SMS) CSRF Web Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>
<li><a name="36" href="36">Salesforce (Event Registration) - Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>
<li><a name="37" href="37">Apple (iTunes Notify) - Filter Bypass &amp; Persistent Web	Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>
<li><a name="38" href="38">Security BSides Ljubljana 0x7E1 CFP - March 10, 2017</a> <em>Andraz Sraka (Jan 17)</em></li>
<li><a name="39" href="39">New exploit for new vulnerability in WordPress Plugin +	tutorial</a> <em>Diego (Jan 17)</em></li>
<li><a name="40" href="40">Multiple RCE in ZyXEL / Billion / TrueOnline routers</a> <em>Pedro Ribeiro (Jan 17)</em></li>
<li><a name="41" href="41">Reflected Cross-Site Scripting (XSS) in Atlassian Jira Software</a> <em>Roberto Soares (Jan 17)</em></li>
<li><a name="42" href="42">EuskalHack Security Congress CFP</a> <em>Joxean Koret (Jan 17)</em></li>
<li><a name="43" href="43">SEC Consult SA-20170117-0 :: XSS in Recommend Page extension for TYPO3 CMS (pb_recommend_page)</a> <em>SEC Consult Vulnerability Lab (Jan 17)</em></li>
<li><a name="44" href="44">Announce Keypatch v2.1, a better assembler for IDA Pro!</a> <em>Nguyen Anh Quynh (Jan 18)</em></li>
<li><a name="45" href="45">APPLE-SA-2017-01-18-1 GarageBand 10.1.5</a> <em>Apple Product Security (Jan 19)</em></li>
<li><a name="46" href="46">APPLE-SA-2017-01-18-2 Logic Pro X 10.3</a> <em>Apple Product Security (Jan 19)</em></li>
<li><a name="47" href="47">[ERPSCAN-16-036] SAP ASE ODATA SERVER - DENIAL OF SERVICE</a> <em>ERPScan inc (Jan 19)</em></li>
<li><a name="50" href="50">[ERPSCAN-16-037] SAP NetWeaver AS JAVA P4 - INFORMATION	DISCLOSURE</a> <em>ERPScan inc (Jan 19)</em></li>
<li><a name="48" href="48">CALL FOR PAPERS - br3aking c0de</a> <em>Estación Informática (Jan 19)</em></li>
<li><a name="49" href="49">Persistent XSS in Ghost 0.11.3</a> <em>Patrick (Jan 19)</em></li>
<li><a name="51" href="51">[RCESEC-2016-012] Mattermost &lt;= 3.5.1 &quot;/error&quot; Unauthenticated Reflected Cross-Site Scripting / Content Injection</a> <em>Julien Ahrens (Jan 19)</em></li>
<li><a name="52" href="52">Tap &apos;n&apos; Sniff</a> <em>Curesec Research Team (CRT) (Jan 19)</em></li>
<li><a name="53" href="53">Apple iOS 10.2 (Notify - iTunes) - Filter Bypass &amp; Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 20)</em></li>
<li><a name="54" href="54">Executable installers are vulnerable^WEVIL (case 46): Pelles C	allows arbitrary code execution</a> <em>Stefan Kanthak (Jan 22)</em>
<ul>
<li><a name="65" href="65">Re: Executable installers are vulnerable^WEVIL (case 46): Pelles C allows arbitrary code execution</a> <em>Ding Dong (Jan 23)</em>
<ul>
<li><a name="69" href="69">Re: Executable installers are vulnerable^WEVIL (case 46):	Pelles C allows arbitrary code execution</a> <em>Stefan Kanthak (Jan 24)</em>
</li>
</ul>
</li>
</ul>
</li>
<li><a name="55" href="55">GMP Deserialization Type Confusion Vulnerability [MyBB &lt;=	1.8.3 RCE Vulnerability]</a> <em>Taoguang Chen (Jan 22)</em></li>
<li><a name="56" href="56">RVAsec 2017 Call for Presentations</a> <em>Sullo (Jan 23)</em></li>
<li><a name="57" href="57">[ERPSCAN-17-005] Oracle PeopleSoft - XSS vulnerability	CVE-2017-3300</a> <em>ERPScan inc (Jan 23)</em></li>
<li><a name="58" href="58">CVE-2017-3241 - [ERPSCAN-17-006] Oracle OpenJDK - Java	Serialization DoS</a> <em>ERPScan inc (Jan 23)</em></li>
<li><a name="59" href="59">APPLE-SA-2017-01-23-1 iOS 10.2.1</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="60" href="60">APPLE-SA-2017-01-23-2 macOS 10.12.3</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="61" href="61">APPLE-SA-2017-01-23-3 watchOS 3.1.3</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="62" href="62">APPLE-SA-2017-01-23-4 tvOS 10.1.1</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="63" href="63">APPLE-SA-2017-01-23-5 Safari 10.0.3</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="64" href="64">APPLE-SA-2017-01-23-6 iCloud for Windows 6.1.1</a> <em>Apple Product Security (Jan 23)</em></li>
<li><a name="66" href="66">New mailing-list on IoT hacking</a> <em>Fernando Gont (Jan 23)</em></li>
<li><a name="67" href="67">WD My Cloud Mirror 2.11.153 RCE and Authentication Bypass</a> <em>Kacper Szurek (Jan 24)</em></li>
<li><a name="68" href="68">APPLE-SA-2017-01-23-7 iTunes for Windows 12.5.5</a> <em>Apple Product Security (Jan 24)</em></li>
<li><a name="70" href="70">Google Forms WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>
<li><a name="71" href="71">CMS Commander Client WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>
<li><a name="72" href="72">InfiniteWP Client WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>
<li><a name="73" href="73">Call for Papers: DigitalSec2017 in Kuala Lumpur, Malaysia on July 11-13, 2017</a> <em>Sandra Evans (Jan 26)</em></li>
<li><a name="74" href="74">Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>Daniel Elebash (Jan 27)</em>
<ul>
<li><a name="84" href="84">Re: Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>gp (Jan 30)</em>
</li>
<li><a name="85" href="85">Re: Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>Daniel Elebash (Jan 30)</em>
</li>
</ul>
</li>
<li><a name="75" href="75">Privilege Escalation in VirtualBox (CVE-2017-3316)</a> <em>Wolfgang (Jan 27)</em></li>
<li><a name="76" href="76">Cross-Site Request Forgery vulnerability in FormBuilder WordPress Plugin allows plugin permissions modification</a> <em>Summer of Pwnage (Jan 28)</em></li>
<li><a name="77" href="77">Multiple blind SQL injection vulnerabilities in FormBuilder WordPress Plugin</a> <em>Summer of Pwnage (Jan 28)</em></li>
<li><a name="78" href="78">Persistent Cross-Site Scripting vulnerability in User Access Manager WordPress Plugin</a> <em>Summer of Pwnage (Jan 28)</em></li>
<li><a name="79" href="79">SEC Consult SA-20170130-0 :: XSS &amp; CSRF in multiple Ubiquiti Networks products</a> <em>SEC Consult Vulnerability Lab (Jan 30)</em></li>
<li><a name="80" href="80">New BlackArch Linux ISOs (2017.01.28) released!</a> <em>Black Arch (Jan 30)</em></li>
<li><a name="81" href="81">BSidesHannover 2017!</a> <em>Daniel Busch (Jan 30)</em></li>
<li><a name="82" href="82">secuvera-SA-2017-01: Privilege escalation in an OPSI Managed Client environment (&quot;rise of the machines&quot;)</a> <em>Simon Bieber (Jan 30)</em></li>
<li><a name="83" href="83">Re: Announcing NorthSec 2017 CFP + Reg - Montreal, May 16-21</a> <em>Olivier Bilodeau (Jan 30)</em></li>
<li><a name="86" href="86">Free ebook to learn ethical hacking techniques</a> <em>Sparc Flow (Jan 30)</em></li>
<li><a name="87" href="87">Sophos Web Appliance - Block &amp; Unblock IPs Remote Command Injection (CVE-2016-9553)</a> <em>Russell Sanford (Jan 30)</em></li>
<li><a name="88" href="88">Re: [0-day] RCE and admin credential disclosure in NETGEAR	WNR2000</a> <em>Pedro Ribeiro (Jan 30)</em>
<ul>
<li><a name="97" href="97">Re: [0-day] RCE and admin credential disclosure in NETGEAR	WNR2000</a> <em>Netgear Security (Jan 31)</em>
</li>
</ul>
</li>
<li><a name="89" href="89">Hacking Printers Advisory 1/6: PostScript printers vulnerable to print job capture</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="90" href="90">Hacking Printers Advisory 2/6: Various HP/OKI/Konica printers file/password disclosure via PostScript/PJL</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="91" href="91">Hacking Printers Advisory 5/6: HP printers restoring factory defaults through PML commands</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="92" href="92">Hacking Printers Advisory 4/6: Multiple vendors buffer overflow in LPD daemon and PJL interpreter</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="93" href="93">Hacking Printers Advisory 3/6: Brother printers vulnerable to memory access via PJL commands</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="94" href="94">Hacking Printers Advisory 6/6: Multiple vendors physical NVRAM damage via PJL commands</a> <em>Jens Müller (Jan 30)</em></li>
<li><a name="95" href="95">PEAR Base System v1.10.1 Arbitrary File Download</a> <em>hyp3rlinx (Jan 31)</em></li>
<li><a name="96" href="96">[REVIVE-SA-2017-001] Revive Adserver - Multiple vulnerabilities</a> <em>Matteo Beccati (Jan 31)</em></li>
<li><a name="98" href="98">Executable installers are vulnerable^WEVIL (case 47): Heimdal	Security&apos;s SetupLauncher vulnerable to DLL hijacking</a> <em>Stefan Kanthak (Jan 31)</em></li>
</ul>
<a name="end">
<table border="0" width="100%">
<tr>
<td align="left" valign="middle"><a href="/cgi-bin/nav.cgi?from=fulldisclosure%2F2017%2FJan&goto=prev&view=thread"><img src="/images/left-icon-16x16.png" border=0 width=16 height=16>Previous period</a></td>
<td align="right" valign="middle"><a href="/cgi-bin/nav.cgi?from=fulldisclosure%2F2017%2FJan&goto=next&view=thread">Next period<img src="/images/right-icon-16x16.png" border=0 width=16 height=16></a></td>
</tr>
</table>
</TD></TR>
</TABLE>
</TD></TR>
<TR><TD></TD><TD ALIGN="center">
<FONT COLOR="#FFFFFF">
[ <A HREF="https://nmap.org"><FONT COLOR="#FFFFFF">Nmap</FONT></A> |
  <A HREF="http://sectools.org"><FONT COLOR="#FFFFFF">Sec Tools</FONT></A> |
  <A HREF="http://seclists.org/"><FONT COLOR="#FFFFFF">Mailing Lists</FONT></A> |
  <A HREF="http://insecure.org/"><FONT COLOR="#FFFFFF">Site News</FONT></A> |
  <A HREF="http://insecure.org/fyodor/"><FONT COLOR="#FFFFFF">About/Contact</FONT></A> |
  <A HREF="http://insecure.org/advertising.html"><FONT COLOR="#FFFFFF">Advertising</FONT></A> |
  <A HREF="http://insecure.org/privacy.html"><FONT COLOR="#FFFFFF">Privacy</FONT></A> ]<BR>
</FONT>

<!-- SiteSearch Google -->
<div class="gcse-searchbox-only" data-resultsUrl="https://nmap.org/search.html"></div>
<!-- End SiteSearch Google -->

<!-- Bottom Banner -->
<!-- Adsense -->
<script async src="//pagead2.googlesyndication.com/pagead/js/adsbygoogle.js"></script>
<!-- PageBottom728x90 -->
<ins class="adsbygoogle"
     style="display:inline-block;width:728px;height:90px"
     data-ad-client="ca-pub-0078565546631069"
     data-ad-slot="2743510915"></ins>
<script>
(adsbygoogle = window.adsbygoogle || []).push({});
</script>
<!-- End Bottom Banner -->
</TD></TR>
</TABLE>
</BODY>
</HTML>


<!-- MHonArc v2.6.19 -->

The generated code from seclists.org contains an unterminated anchor tag, so to make things easier for BeautifulSoup's parser, we'll just replace this manually. This particular tag is a good tag to use as a locator for the messages section, so it's good to make sure that it's valid html.

Our soup "query" basically consists of:

  1. Find me the tag with name = "begin"
  2. Find me the first ul after this tag

In [2]:
raw = raw.replace('<a name="begin">', '<a name="begin"></a>')
soup = BeautifulSoup(raw, 'html5lib')

begin = soup.find(attrs={'name':'begin'}) #beginning of msg links
items = begin.find_next('ul').find_all('li', recursive=False)

pp.pprint(items)


[   <li><a href="0" name="0">Zend Framework / zend-mail &lt; 2.4.11 Remote Code Execution	(CVE-2016-10034)</a> <em>Dawid Golunski (Jan 03)</em></li>,
    <li><a href="1" name="1">CINtruder v0.3 released...</a> <em>psy (Jan 03)</em></li>,
    <li><a href="2" name="2">Advisories Unsafe Dll in Audacity, telegram and Akamai</a> <em>filipe (Jan 03)</em></li>,
    <li><a href="3" name="3">Persisted Cross-Site Scripting (XSS) in Confluence Jira	Software</a> <em>jlss (Jan 03)</em>
<ul>
<li><a href="9" name="9">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>Moritz Naumann (Jan 04)</em>
<ul>
<li><a href="12" name="12">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>jlss (Jan 06)</em>
</li>
</ul>
</li>
<li>&lt;Possible follow-ups&gt;</li>
<li><a href="11" name="11">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira	Software</a> <em>David Black (Jan 06)</em>
</li>
 </ul>
</li>,
    <li><a href="4" name="4">0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
<ul>
<li><a href="5" name="5">Re: 0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
<ul>
<li><a href="6" name="6">Re: 0-day: QNAP NAS Devices suffer of heap overflow</a> <em>bashis (Jan 03)</em>
</li>
</ul>
</li>
</ul>
</li>,
    <li><a href="7" name="7">Executable installers are vulnerable^WEVIL (case 43):	SoftMaker's Office service pack installers allow escalation	of privilege</a> <em>Stefan Kanthak (Jan 03)</em></li>,
    <li><a href="8" name="8">Executable installers are vulnerable^WEVIL (case 45):	ReadPDF's installers allow escalation of privilege</a> <em>Stefan Kanthak (Jan 03)</em></li>,
    <li><a href="10" name="10">Stop User Enumeration does not stop user enumeration	(WordPress plugin)</a> <em>dxw Security (Jan 04)</em></li>,
    <li><a href="13" name="13">CarolinaCon-13 - May 2017 - Call for Papers/Presenters and Attendees</a> <em>Vic Vandal (Jan 06)</em></li>,
    <li><a href="14" name="14">Trango Altum AC600 Default root Login</a> <em>Ian Ling (Jan 06)</em></li>,
    <li><a href="15" name="15">YSTS 11th Edition - CFP</a> <em>Luiz Eduardo (Jan 06)</em></li>,
    <li><a href="16" name="16">BSides Las Vegas 2017 CFP is open.</a> <em>Daemon Tamer (Jan 09)</em></li>,
    <li><a href="17" name="17">Hotlinking Vulnerability in PHProxy 0.5b2</a> <em>Celso Bento (Jan 09)</em></li>,
    <li><a href="18" name="18">enigma2-plugin-extensions-webadmin Remote Code Execution (IoT)</a> <em>Fabian Fingerle (Jan 09)</em></li>,
    <li><a href="19" name="19">pev 0.80 released</a> <em>Fernando Mercês (Jan 09)</em></li>,
    <li><a href="20" name="20">CSRF/XSS in Responsive Poll allows unauthenticated attackers	to do almost anything an admin can (WordPress plugin)</a> <em>dxw Security (Jan 10)</em></li>,
    <li><a href="21" name="21">Docker 1.12.6 - Security Advisory</a> <em>Nathan McCauley (Jan 10)</em>
<ul>
<li><a href="22" name="22">Re: [oss-security] Docker 1.12.6 - Security Advisory</a> <em>Kurt Seifried (Jan 10)</em>
<ul>
<li><a href="29" name="29">Re: [oss-security] Docker 1.12.6 - Security Advisory</a> <em>Andreas Stieger (Jan 11)</em>
</li>
</ul>
</li>
</ul>
</li>,
    <li><a href="23" name="23">Blackboard LMS 9.1 SP14 - (Profile) Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="24" name="24">BlackBoard LMS 9.1 SP14 - (Title) Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="25" name="25">Bit Defender #39 - Auth Token Bypass Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="26" name="26">Huawei Flybox B660 - (POST Reboot) CSRF Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="27" name="27">Boxoft Wav v1.1.0.0 - Buffer Overflow Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="28" name="28">Cobi Tools v1.0.8 iOS - Persistent Web Vulnerability</a> <em>Vulnerability Lab (Jan 11)</em></li>,
    <li><a href="30" name="30">[CVE-2016-3403] [Zimbra] Multiple CSRF in Administration interface - all versions</a> <em>Sysdream Labs (Jan 12)</em></li>,
    <li><a href="31" name="31">Multiple vulnerabilities in cPanel  &lt;= 60.0.34</a> <em>Open Security (Jan 12)</em></li>,
    <li><a href="32" name="32">ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP peers)</a> <em>Fernando Gont (Jan 12)</em></li>,
    <li><a href="33" name="33">Nginx (Debian-based + Gentoo distros) - Root Privilege Escalation [CVE-2016-1247 UPDATE]</a> <em>Dawid Golunski (Jan 13)</em></li>,
    <li><a href="34" name="34">Executable installers are vulnerable^WEVIL (case 44):	SoftMaker's FlexiPDF installers allow escalation of privilege</a> <em>Stefan Kanthak (Jan 15)</em></li>,
    <li><a href="35" name="35">Huawei Flybox B660 - (POST SMS) CSRF Web Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>,
    <li><a href="36" name="36">Salesforce (Event Registration) - Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>,
    <li><a href="37" name="37">Apple (iTunes Notify) - Filter Bypass &amp; Persistent Web	Vulnerability</a> <em>Vulnerability Lab (Jan 16)</em></li>,
    <li><a href="38" name="38">Security BSides Ljubljana 0x7E1 CFP - March 10, 2017</a> <em>Andraz Sraka (Jan 17)</em></li>,
    <li><a href="39" name="39">New exploit for new vulnerability in WordPress Plugin +	tutorial</a> <em>Diego (Jan 17)</em></li>,
    <li><a href="40" name="40">Multiple RCE in ZyXEL / Billion / TrueOnline routers</a> <em>Pedro Ribeiro (Jan 17)</em></li>,
    <li><a href="41" name="41">Reflected Cross-Site Scripting (XSS) in Atlassian Jira Software</a> <em>Roberto Soares (Jan 17)</em></li>,
    <li><a href="42" name="42">EuskalHack Security Congress CFP</a> <em>Joxean Koret (Jan 17)</em></li>,
    <li><a href="43" name="43">SEC Consult SA-20170117-0 :: XSS in Recommend Page extension for TYPO3 CMS (pb_recommend_page)</a> <em>SEC Consult Vulnerability Lab (Jan 17)</em></li>,
    <li><a href="44" name="44">Announce Keypatch v2.1, a better assembler for IDA Pro!</a> <em>Nguyen Anh Quynh (Jan 18)</em></li>,
    <li><a href="45" name="45">APPLE-SA-2017-01-18-1 GarageBand 10.1.5</a> <em>Apple Product Security (Jan 19)</em></li>,
    <li><a href="46" name="46">APPLE-SA-2017-01-18-2 Logic Pro X 10.3</a> <em>Apple Product Security (Jan 19)</em></li>,
    <li><a href="47" name="47">[ERPSCAN-16-036] SAP ASE ODATA SERVER - DENIAL OF SERVICE</a> <em>ERPScan inc (Jan 19)</em></li>,
    <li><a href="50" name="50">[ERPSCAN-16-037] SAP NetWeaver AS JAVA P4 - INFORMATION	DISCLOSURE</a> <em>ERPScan inc (Jan 19)</em></li>,
    <li><a href="48" name="48">CALL FOR PAPERS - br3aking c0de</a> <em>Estación Informática (Jan 19)</em></li>,
    <li><a href="49" name="49">Persistent XSS in Ghost 0.11.3</a> <em>Patrick (Jan 19)</em></li>,
    <li><a href="51" name="51">[RCESEC-2016-012] Mattermost &lt;= 3.5.1 "/error" Unauthenticated Reflected Cross-Site Scripting / Content Injection</a> <em>Julien Ahrens (Jan 19)</em></li>,
    <li><a href="52" name="52">Tap 'n' Sniff</a> <em>Curesec Research Team (CRT) (Jan 19)</em></li>,
    <li><a href="53" name="53">Apple iOS 10.2 (Notify - iTunes) - Filter Bypass &amp; Persistent Vulnerability</a> <em>Vulnerability Lab (Jan 20)</em></li>,
    <li><a href="54" name="54">Executable installers are vulnerable^WEVIL (case 46): Pelles C	allows arbitrary code execution</a> <em>Stefan Kanthak (Jan 22)</em>
<ul>
<li><a href="65" name="65">Re: Executable installers are vulnerable^WEVIL (case 46): Pelles C allows arbitrary code execution</a> <em>Ding Dong (Jan 23)</em>
<ul>
<li><a href="69" name="69">Re: Executable installers are vulnerable^WEVIL (case 46):	Pelles C allows arbitrary code execution</a> <em>Stefan Kanthak (Jan 24)</em>
</li>
</ul>
</li>
</ul>
</li>,
    <li><a href="55" name="55">GMP Deserialization Type Confusion Vulnerability [MyBB &lt;=	1.8.3 RCE Vulnerability]</a> <em>Taoguang Chen (Jan 22)</em></li>,
    <li><a href="56" name="56">RVAsec 2017 Call for Presentations</a> <em>Sullo (Jan 23)</em></li>,
    <li><a href="57" name="57">[ERPSCAN-17-005] Oracle PeopleSoft - XSS vulnerability	CVE-2017-3300</a> <em>ERPScan inc (Jan 23)</em></li>,
    <li><a href="58" name="58">CVE-2017-3241 - [ERPSCAN-17-006] Oracle OpenJDK - Java	Serialization DoS</a> <em>ERPScan inc (Jan 23)</em></li>,
    <li><a href="59" name="59">APPLE-SA-2017-01-23-1 iOS 10.2.1</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="60" name="60">APPLE-SA-2017-01-23-2 macOS 10.12.3</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="61" name="61">APPLE-SA-2017-01-23-3 watchOS 3.1.3</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="62" name="62">APPLE-SA-2017-01-23-4 tvOS 10.1.1</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="63" name="63">APPLE-SA-2017-01-23-5 Safari 10.0.3</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="64" name="64">APPLE-SA-2017-01-23-6 iCloud for Windows 6.1.1</a> <em>Apple Product Security (Jan 23)</em></li>,
    <li><a href="66" name="66">New mailing-list on IoT hacking</a> <em>Fernando Gont (Jan 23)</em></li>,
    <li><a href="67" name="67">WD My Cloud Mirror 2.11.153 RCE and Authentication Bypass</a> <em>Kacper Szurek (Jan 24)</em></li>,
    <li><a href="68" name="68">APPLE-SA-2017-01-23-7 iTunes for Windows 12.5.5</a> <em>Apple Product Security (Jan 24)</em></li>,
    <li><a href="70" name="70">Google Forms WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>,
    <li><a href="71" name="71">CMS Commander Client WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>,
    <li><a href="72" name="72">InfiniteWP Client WordPress Plugin unauthenticated PHP Object injection vulnerability</a> <em>Summer of Pwnage (Jan 25)</em></li>,
    <li><a href="73" name="73">Call for Papers: DigitalSec2017 in Kuala Lumpur, Malaysia on July 11-13, 2017</a> <em>Sandra Evans (Jan 26)</em></li>,
    <li><a href="74" name="74">Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>Daniel Elebash (Jan 27)</em>
<ul>
<li><a href="84" name="84">Re: Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>gp (Jan 30)</em>
</li>
<li><a href="85" name="85">Re: Digital Ocean ssh key authentication security risk -- password authentication is re-enabled</a> <em>Daniel Elebash (Jan 30)</em>
</li>
</ul>
</li>,
    <li><a href="75" name="75">Privilege Escalation in VirtualBox (CVE-2017-3316)</a> <em>Wolfgang (Jan 27)</em></li>,
    <li><a href="76" name="76">Cross-Site Request Forgery vulnerability in FormBuilder WordPress Plugin allows plugin permissions modification</a> <em>Summer of Pwnage (Jan 28)</em></li>,
    <li><a href="77" name="77">Multiple blind SQL injection vulnerabilities in FormBuilder WordPress Plugin</a> <em>Summer of Pwnage (Jan 28)</em></li>,
    <li><a href="78" name="78">Persistent Cross-Site Scripting vulnerability in User Access Manager WordPress Plugin</a> <em>Summer of Pwnage (Jan 28)</em></li>,
    <li><a href="79" name="79">SEC Consult SA-20170130-0 :: XSS &amp; CSRF in multiple Ubiquiti Networks products</a> <em>SEC Consult Vulnerability Lab (Jan 30)</em></li>,
    <li><a href="80" name="80">New BlackArch Linux ISOs (2017.01.28) released!</a> <em>Black Arch (Jan 30)</em></li>,
    <li><a href="81" name="81">BSidesHannover 2017!</a> <em>Daniel Busch (Jan 30)</em></li>,
    <li><a href="82" name="82">secuvera-SA-2017-01: Privilege escalation in an OPSI Managed Client environment ("rise of the machines")</a> <em>Simon Bieber (Jan 30)</em></li>,
    <li><a href="83" name="83">Re: Announcing NorthSec 2017 CFP + Reg - Montreal, May 16-21</a> <em>Olivier Bilodeau (Jan 30)</em></li>,
    <li><a href="86" name="86">Free ebook to learn ethical hacking techniques</a> <em>Sparc Flow (Jan 30)</em></li>,
    <li><a href="87" name="87">Sophos Web Appliance - Block &amp; Unblock IPs Remote Command Injection (CVE-2016-9553)</a> <em>Russell Sanford (Jan 30)</em></li>,
    <li><a href="88" name="88">Re: [0-day] RCE and admin credential disclosure in NETGEAR	WNR2000</a> <em>Pedro Ribeiro (Jan 30)</em>
<ul>
<li><a href="97" name="97">Re: [0-day] RCE and admin credential disclosure in NETGEAR	WNR2000</a> <em>Netgear Security (Jan 31)</em>
</li>
</ul>
</li>,
    <li><a href="89" name="89">Hacking Printers Advisory 1/6: PostScript printers vulnerable to print job capture</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="90" name="90">Hacking Printers Advisory 2/6: Various HP/OKI/Konica printers file/password disclosure via PostScript/PJL</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="91" name="91">Hacking Printers Advisory 5/6: HP printers restoring factory defaults through PML commands</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="92" name="92">Hacking Printers Advisory 4/6: Multiple vendors buffer overflow in LPD daemon and PJL interpreter</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="93" name="93">Hacking Printers Advisory 3/6: Brother printers vulnerable to memory access via PJL commands</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="94" name="94">Hacking Printers Advisory 6/6: Multiple vendors physical NVRAM damage via PJL commands</a> <em>Jens Müller (Jan 30)</em></li>,
    <li><a href="95" name="95">PEAR Base System v1.10.1 Arbitrary File Download</a> <em>hyp3rlinx (Jan 31)</em></li>,
    <li><a href="96" name="96">[REVIVE-SA-2017-001] Revive Adserver - Multiple vulnerabilities</a> <em>Matteo Beccati (Jan 31)</em></li>,
    <li><a href="98" name="98">Executable installers are vulnerable^WEVIL (case 47): Heimdal	Security's SetupLauncher vulnerable to DLL hijacking</a> <em>Stefan Kanthak (Jan 31)</em></li>]

We end up with an array of <li> tags, but note that children tags are encoded as embedded <ul> portions.

Example:

<li><a href="3" name="3">Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>jlss (Jan 03)</em> <ul> <li><a href="9" name="9">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>Moritz Naumann (Jan 04)</em> <ul> <li><a href="12" name="12">Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software</a> <em>jlss (Jan 06)</em> </li>

We'll have to use recursion to be able to run through each layer, so lets define a function

Args:

  • items: <li> tags at our given thread level
  • messages: array used to hold our results
  • idroot:
  • parent: id of the parent message

In [3]:
import re

def read_messages(items, messages, idroot, parent):
    for li in items:
        msg = li.find('a')
        if msg == None:
            #some messages just read "Possible follow-ups" with no link--skip
            continue
        id = idroot + msg['href']
        title = msg.text
        
        whowhen = li.find('em').text
        rx = re.compile('(.+) \((.+)\)')
        m = rx.search(whowhen)
        who = m.group(1)
        when = m.group(2)

        messages.append({
            'index': msg['href'],
            'id': id,
            'title': title,
            'parent': parent,
            'author': who,
            'date': when
        })
        
        replies = li.find('ul')
        if replies != None:
            read_messages(replies.find_all('li', recursive=False), messages, idroot, id)

    return messages

messages = []
idroot = '2017_Jan_'
read_messages(items, messages, idroot, None)
pp.pprint(messages)


[   {   'author': 'Dawid Golunski',
        'date': 'Jan 03',
        'id': '2017_Jan_0',
        'index': '0',
        'parent': None,
        'title': 'Zend Framework / zend-mail < 2.4.11 Remote Code '
                 'Execution\t(CVE-2016-10034)'},
    {   'author': 'psy',
        'date': 'Jan 03',
        'id': '2017_Jan_1',
        'index': '1',
        'parent': None,
        'title': 'CINtruder v0.3 released...'},
    {   'author': 'filipe',
        'date': 'Jan 03',
        'id': '2017_Jan_2',
        'index': '2',
        'parent': None,
        'title': 'Advisories Unsafe Dll in Audacity, telegram and Akamai'},
    {   'author': 'jlss',
        'date': 'Jan 03',
        'id': '2017_Jan_3',
        'index': '3',
        'parent': None,
        'title': 'Persisted Cross-Site Scripting (XSS) in Confluence Jira\t'
                 'Software'},
    {   'author': 'Moritz Naumann',
        'date': 'Jan 04',
        'id': '2017_Jan_9',
        'index': '9',
        'parent': '2017_Jan_3',
        'title': 'Re: Persisted Cross-Site Scripting (XSS) in Confluence '
                 'Jira Software'},
    {   'author': 'jlss',
        'date': 'Jan 06',
        'id': '2017_Jan_12',
        'index': '12',
        'parent': '2017_Jan_9',
        'title': 'Re: Persisted Cross-Site Scripting (XSS) in Confluence '
                 'Jira Software'},
    {   'author': 'David Black',
        'date': 'Jan 06',
        'id': '2017_Jan_11',
        'index': '11',
        'parent': '2017_Jan_3',
        'title': 'Re: Persisted Cross-Site Scripting (XSS) in Confluence '
                 'Jira\tSoftware'},
    {   'author': 'bashis',
        'date': 'Jan 03',
        'id': '2017_Jan_4',
        'index': '4',
        'parent': None,
        'title': '0-day: QNAP NAS Devices suffer of heap overflow'},
    {   'author': 'bashis',
        'date': 'Jan 03',
        'id': '2017_Jan_5',
        'index': '5',
        'parent': '2017_Jan_4',
        'title': 'Re: 0-day: QNAP NAS Devices suffer of heap overflow'},
    {   'author': 'bashis',
        'date': 'Jan 03',
        'id': '2017_Jan_6',
        'index': '6',
        'parent': '2017_Jan_5',
        'title': 'Re: 0-day: QNAP NAS Devices suffer of heap overflow'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 03',
        'id': '2017_Jan_7',
        'index': '7',
        'parent': None,
        'title': 'Executable installers are vulnerable^WEVIL (case 43):\t'
                 "SoftMaker's Office service pack installers allow "
                 'escalation\tof privilege'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 03',
        'id': '2017_Jan_8',
        'index': '8',
        'parent': None,
        'title': 'Executable installers are vulnerable^WEVIL (case 45):\t'
                 "ReadPDF's installers allow escalation of privilege"},
    {   'author': 'dxw Security',
        'date': 'Jan 04',
        'id': '2017_Jan_10',
        'index': '10',
        'parent': None,
        'title': 'Stop User Enumeration does not stop user enumeration\t'
                 '(WordPress plugin)'},
    {   'author': 'Vic Vandal',
        'date': 'Jan 06',
        'id': '2017_Jan_13',
        'index': '13',
        'parent': None,
        'title': 'CarolinaCon-13 - May 2017 - Call for Papers/Presenters '
                 'and Attendees'},
    {   'author': 'Ian Ling',
        'date': 'Jan 06',
        'id': '2017_Jan_14',
        'index': '14',
        'parent': None,
        'title': 'Trango Altum AC600 Default root Login'},
    {   'author': 'Luiz Eduardo',
        'date': 'Jan 06',
        'id': '2017_Jan_15',
        'index': '15',
        'parent': None,
        'title': 'YSTS 11th Edition - CFP'},
    {   'author': 'Daemon Tamer',
        'date': 'Jan 09',
        'id': '2017_Jan_16',
        'index': '16',
        'parent': None,
        'title': 'BSides Las Vegas 2017 CFP is open.'},
    {   'author': 'Celso Bento',
        'date': 'Jan 09',
        'id': '2017_Jan_17',
        'index': '17',
        'parent': None,
        'title': 'Hotlinking Vulnerability in PHProxy 0.5b2'},
    {   'author': 'Fabian Fingerle',
        'date': 'Jan 09',
        'id': '2017_Jan_18',
        'index': '18',
        'parent': None,
        'title': 'enigma2-plugin-extensions-webadmin Remote Code Execution '
                 '(IoT)'},
    {   'author': 'Fernando Mercês',
        'date': 'Jan 09',
        'id': '2017_Jan_19',
        'index': '19',
        'parent': None,
        'title': 'pev 0.80 released'},
    {   'author': 'dxw Security',
        'date': 'Jan 10',
        'id': '2017_Jan_20',
        'index': '20',
        'parent': None,
        'title': 'CSRF/XSS in Responsive Poll allows unauthenticated '
                 'attackers\tto do almost anything an admin can (WordPress '
                 'plugin)'},
    {   'author': 'Nathan McCauley',
        'date': 'Jan 10',
        'id': '2017_Jan_21',
        'index': '21',
        'parent': None,
        'title': 'Docker 1.12.6 - Security Advisory'},
    {   'author': 'Kurt Seifried',
        'date': 'Jan 10',
        'id': '2017_Jan_22',
        'index': '22',
        'parent': '2017_Jan_21',
        'title': 'Re: [oss-security] Docker 1.12.6 - Security Advisory'},
    {   'author': 'Andreas Stieger',
        'date': 'Jan 11',
        'id': '2017_Jan_29',
        'index': '29',
        'parent': '2017_Jan_22',
        'title': 'Re: [oss-security] Docker 1.12.6 - Security Advisory'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_23',
        'index': '23',
        'parent': None,
        'title': 'Blackboard LMS 9.1 SP14 - (Profile) Persistent '
                 'Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_24',
        'index': '24',
        'parent': None,
        'title': 'BlackBoard LMS 9.1 SP14 - (Title) Persistent Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_25',
        'index': '25',
        'parent': None,
        'title': 'Bit Defender #39 - Auth Token Bypass Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_26',
        'index': '26',
        'parent': None,
        'title': 'Huawei Flybox B660 - (POST Reboot) CSRF Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_27',
        'index': '27',
        'parent': None,
        'title': 'Boxoft Wav v1.1.0.0 - Buffer Overflow Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 11',
        'id': '2017_Jan_28',
        'index': '28',
        'parent': None,
        'title': 'Cobi Tools v1.0.8 iOS - Persistent Web Vulnerability'},
    {   'author': 'Sysdream Labs',
        'date': 'Jan 12',
        'id': '2017_Jan_30',
        'index': '30',
        'parent': None,
        'title': '[CVE-2016-3403] [Zimbra] Multiple CSRF in Administration '
                 'interface - all versions'},
    {   'author': 'Open Security',
        'date': 'Jan 12',
        'id': '2017_Jan_31',
        'index': '31',
        'parent': None,
        'title': 'Multiple vulnerabilities in cPanel  <= 60.0.34'},
    {   'author': 'Fernando Gont',
        'date': 'Jan 12',
        'id': '2017_Jan_32',
        'index': '32',
        'parent': None,
        'title': 'ICMPv6 PTBs and IPv6 frag filtering (particularly at BGP '
                 'peers)'},
    {   'author': 'Dawid Golunski',
        'date': 'Jan 13',
        'id': '2017_Jan_33',
        'index': '33',
        'parent': None,
        'title': 'Nginx (Debian-based + Gentoo distros) - Root Privilege '
                 'Escalation [CVE-2016-1247 UPDATE]'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 15',
        'id': '2017_Jan_34',
        'index': '34',
        'parent': None,
        'title': 'Executable installers are vulnerable^WEVIL (case 44):\t'
                 "SoftMaker's FlexiPDF installers allow escalation of "
                 'privilege'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 16',
        'id': '2017_Jan_35',
        'index': '35',
        'parent': None,
        'title': 'Huawei Flybox B660 - (POST SMS) CSRF Web Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 16',
        'id': '2017_Jan_36',
        'index': '36',
        'parent': None,
        'title': 'Salesforce (Event Registration) - Persistent Vulnerability'},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 16',
        'id': '2017_Jan_37',
        'index': '37',
        'parent': None,
        'title': 'Apple (iTunes Notify) - Filter Bypass & Persistent Web\t'
                 'Vulnerability'},
    {   'author': 'Andraz Sraka',
        'date': 'Jan 17',
        'id': '2017_Jan_38',
        'index': '38',
        'parent': None,
        'title': 'Security BSides Ljubljana 0x7E1 CFP - March 10, 2017'},
    {   'author': 'Diego',
        'date': 'Jan 17',
        'id': '2017_Jan_39',
        'index': '39',
        'parent': None,
        'title': 'New exploit for new vulnerability in WordPress Plugin +\t'
                 'tutorial'},
    {   'author': 'Pedro Ribeiro',
        'date': 'Jan 17',
        'id': '2017_Jan_40',
        'index': '40',
        'parent': None,
        'title': 'Multiple RCE in ZyXEL / Billion / TrueOnline routers'},
    {   'author': 'Roberto Soares',
        'date': 'Jan 17',
        'id': '2017_Jan_41',
        'index': '41',
        'parent': None,
        'title': 'Reflected Cross-Site Scripting (XSS) in Atlassian Jira '
                 'Software'},
    {   'author': 'Joxean Koret',
        'date': 'Jan 17',
        'id': '2017_Jan_42',
        'index': '42',
        'parent': None,
        'title': 'EuskalHack Security Congress CFP'},
    {   'author': 'SEC Consult Vulnerability Lab',
        'date': 'Jan 17',
        'id': '2017_Jan_43',
        'index': '43',
        'parent': None,
        'title': 'SEC Consult SA-20170117-0 :: XSS in Recommend Page '
                 'extension for TYPO3 CMS (pb_recommend_page)'},
    {   'author': 'Nguyen Anh Quynh',
        'date': 'Jan 18',
        'id': '2017_Jan_44',
        'index': '44',
        'parent': None,
        'title': 'Announce Keypatch v2.1, a better assembler for IDA Pro!'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 19',
        'id': '2017_Jan_45',
        'index': '45',
        'parent': None,
        'title': 'APPLE-SA-2017-01-18-1 GarageBand 10.1.5'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 19',
        'id': '2017_Jan_46',
        'index': '46',
        'parent': None,
        'title': 'APPLE-SA-2017-01-18-2 Logic Pro X 10.3'},
    {   'author': 'ERPScan inc',
        'date': 'Jan 19',
        'id': '2017_Jan_47',
        'index': '47',
        'parent': None,
        'title': '[ERPSCAN-16-036] SAP ASE ODATA SERVER - DENIAL OF SERVICE'},
    {   'author': 'ERPScan inc',
        'date': 'Jan 19',
        'id': '2017_Jan_50',
        'index': '50',
        'parent': None,
        'title': '[ERPSCAN-16-037] SAP NetWeaver AS JAVA P4 - INFORMATION\t'
                 'DISCLOSURE'},
    {   'author': 'Estación Informática',
        'date': 'Jan 19',
        'id': '2017_Jan_48',
        'index': '48',
        'parent': None,
        'title': 'CALL FOR PAPERS - br3aking c0de'},
    {   'author': 'Patrick',
        'date': 'Jan 19',
        'id': '2017_Jan_49',
        'index': '49',
        'parent': None,
        'title': 'Persistent XSS in Ghost 0.11.3'},
    {   'author': 'Julien Ahrens',
        'date': 'Jan 19',
        'id': '2017_Jan_51',
        'index': '51',
        'parent': None,
        'title': '[RCESEC-2016-012] Mattermost <= 3.5.1 "/error" '
                 'Unauthenticated Reflected Cross-Site Scripting / Content '
                 'Injection'},
    {   'author': 'Curesec Research Team (CRT)',
        'date': 'Jan 19',
        'id': '2017_Jan_52',
        'index': '52',
        'parent': None,
        'title': "Tap 'n' Sniff"},
    {   'author': 'Vulnerability Lab',
        'date': 'Jan 20',
        'id': '2017_Jan_53',
        'index': '53',
        'parent': None,
        'title': 'Apple iOS 10.2 (Notify - iTunes) - Filter Bypass & '
                 'Persistent Vulnerability'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 22',
        'id': '2017_Jan_54',
        'index': '54',
        'parent': None,
        'title': 'Executable installers are vulnerable^WEVIL (case 46): '
                 'Pelles C\tallows arbitrary code execution'},
    {   'author': 'Ding Dong',
        'date': 'Jan 23',
        'id': '2017_Jan_65',
        'index': '65',
        'parent': '2017_Jan_54',
        'title': 'Re: Executable installers are vulnerable^WEVIL (case 46): '
                 'Pelles C allows arbitrary code execution'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 24',
        'id': '2017_Jan_69',
        'index': '69',
        'parent': '2017_Jan_65',
        'title': 'Re: Executable installers are vulnerable^WEVIL (case '
                 '46):\tPelles C allows arbitrary code execution'},
    {   'author': 'Taoguang Chen',
        'date': 'Jan 22',
        'id': '2017_Jan_55',
        'index': '55',
        'parent': None,
        'title': 'GMP Deserialization Type Confusion Vulnerability [MyBB '
                 '<=\t1.8.3 RCE Vulnerability]'},
    {   'author': 'Sullo',
        'date': 'Jan 23',
        'id': '2017_Jan_56',
        'index': '56',
        'parent': None,
        'title': 'RVAsec 2017 Call for Presentations'},
    {   'author': 'ERPScan inc',
        'date': 'Jan 23',
        'id': '2017_Jan_57',
        'index': '57',
        'parent': None,
        'title': '[ERPSCAN-17-005] Oracle PeopleSoft - XSS vulnerability\t'
                 'CVE-2017-3300'},
    {   'author': 'ERPScan inc',
        'date': 'Jan 23',
        'id': '2017_Jan_58',
        'index': '58',
        'parent': None,
        'title': 'CVE-2017-3241 - [ERPSCAN-17-006] Oracle OpenJDK - Java\t'
                 'Serialization DoS'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_59',
        'index': '59',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-1 iOS 10.2.1'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_60',
        'index': '60',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-2 macOS 10.12.3'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_61',
        'index': '61',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-3 watchOS 3.1.3'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_62',
        'index': '62',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-4 tvOS 10.1.1'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_63',
        'index': '63',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-5 Safari 10.0.3'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 23',
        'id': '2017_Jan_64',
        'index': '64',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-6 iCloud for Windows 6.1.1'},
    {   'author': 'Fernando Gont',
        'date': 'Jan 23',
        'id': '2017_Jan_66',
        'index': '66',
        'parent': None,
        'title': 'New mailing-list on IoT hacking'},
    {   'author': 'Kacper Szurek',
        'date': 'Jan 24',
        'id': '2017_Jan_67',
        'index': '67',
        'parent': None,
        'title': 'WD My Cloud Mirror 2.11.153 RCE and Authentication Bypass'},
    {   'author': 'Apple Product Security',
        'date': 'Jan 24',
        'id': '2017_Jan_68',
        'index': '68',
        'parent': None,
        'title': 'APPLE-SA-2017-01-23-7 iTunes for Windows 12.5.5'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 25',
        'id': '2017_Jan_70',
        'index': '70',
        'parent': None,
        'title': 'Google Forms WordPress Plugin unauthenticated PHP Object '
                 'injection vulnerability'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 25',
        'id': '2017_Jan_71',
        'index': '71',
        'parent': None,
        'title': 'CMS Commander Client WordPress Plugin unauthenticated PHP '
                 'Object injection vulnerability'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 25',
        'id': '2017_Jan_72',
        'index': '72',
        'parent': None,
        'title': 'InfiniteWP Client WordPress Plugin unauthenticated PHP '
                 'Object injection vulnerability'},
    {   'author': 'Sandra Evans',
        'date': 'Jan 26',
        'id': '2017_Jan_73',
        'index': '73',
        'parent': None,
        'title': 'Call for Papers: DigitalSec2017 in Kuala Lumpur, Malaysia '
                 'on July 11-13, 2017'},
    {   'author': 'Daniel Elebash',
        'date': 'Jan 27',
        'id': '2017_Jan_74',
        'index': '74',
        'parent': None,
        'title': 'Digital Ocean ssh key authentication security risk -- '
                 'password authentication is re-enabled'},
    {   'author': 'gp',
        'date': 'Jan 30',
        'id': '2017_Jan_84',
        'index': '84',
        'parent': '2017_Jan_74',
        'title': 'Re: Digital Ocean ssh key authentication security risk -- '
                 'password authentication is re-enabled'},
    {   'author': 'Daniel Elebash',
        'date': 'Jan 30',
        'id': '2017_Jan_85',
        'index': '85',
        'parent': '2017_Jan_74',
        'title': 'Re: Digital Ocean ssh key authentication security risk -- '
                 'password authentication is re-enabled'},
    {   'author': 'Wolfgang',
        'date': 'Jan 27',
        'id': '2017_Jan_75',
        'index': '75',
        'parent': None,
        'title': 'Privilege Escalation in VirtualBox (CVE-2017-3316)'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 28',
        'id': '2017_Jan_76',
        'index': '76',
        'parent': None,
        'title': 'Cross-Site Request Forgery vulnerability in FormBuilder '
                 'WordPress Plugin allows plugin permissions modification'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 28',
        'id': '2017_Jan_77',
        'index': '77',
        'parent': None,
        'title': 'Multiple blind SQL injection vulnerabilities in '
                 'FormBuilder WordPress Plugin'},
    {   'author': 'Summer of Pwnage',
        'date': 'Jan 28',
        'id': '2017_Jan_78',
        'index': '78',
        'parent': None,
        'title': 'Persistent Cross-Site Scripting vulnerability in User '
                 'Access Manager WordPress Plugin'},
    {   'author': 'SEC Consult Vulnerability Lab',
        'date': 'Jan 30',
        'id': '2017_Jan_79',
        'index': '79',
        'parent': None,
        'title': 'SEC Consult SA-20170130-0 :: XSS & CSRF in multiple '
                 'Ubiquiti Networks products'},
    {   'author': 'Black Arch',
        'date': 'Jan 30',
        'id': '2017_Jan_80',
        'index': '80',
        'parent': None,
        'title': 'New BlackArch Linux ISOs (2017.01.28) released!'},
    {   'author': 'Daniel Busch',
        'date': 'Jan 30',
        'id': '2017_Jan_81',
        'index': '81',
        'parent': None,
        'title': 'BSidesHannover 2017!'},
    {   'author': 'Simon Bieber',
        'date': 'Jan 30',
        'id': '2017_Jan_82',
        'index': '82',
        'parent': None,
        'title': 'secuvera-SA-2017-01: Privilege escalation in an OPSI '
                 'Managed Client environment ("rise of the machines")'},
    {   'author': 'Olivier Bilodeau',
        'date': 'Jan 30',
        'id': '2017_Jan_83',
        'index': '83',
        'parent': None,
        'title': 'Re: Announcing NorthSec 2017 CFP + Reg - Montreal, May '
                 '16-21'},
    {   'author': 'Sparc Flow',
        'date': 'Jan 30',
        'id': '2017_Jan_86',
        'index': '86',
        'parent': None,
        'title': 'Free ebook to learn ethical hacking techniques'},
    {   'author': 'Russell Sanford',
        'date': 'Jan 30',
        'id': '2017_Jan_87',
        'index': '87',
        'parent': None,
        'title': 'Sophos Web Appliance - Block & Unblock IPs Remote Command '
                 'Injection (CVE-2016-9553)'},
    {   'author': 'Pedro Ribeiro',
        'date': 'Jan 30',
        'id': '2017_Jan_88',
        'index': '88',
        'parent': None,
        'title': 'Re: [0-day] RCE and admin credential disclosure in '
                 'NETGEAR\tWNR2000'},
    {   'author': 'Netgear Security',
        'date': 'Jan 31',
        'id': '2017_Jan_97',
        'index': '97',
        'parent': '2017_Jan_88',
        'title': 'Re: [0-day] RCE and admin credential disclosure in '
                 'NETGEAR\tWNR2000'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_89',
        'index': '89',
        'parent': None,
        'title': 'Hacking Printers Advisory 1/6: PostScript printers '
                 'vulnerable to print job capture'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_90',
        'index': '90',
        'parent': None,
        'title': 'Hacking Printers Advisory 2/6: Various HP/OKI/Konica '
                 'printers file/password disclosure via PostScript/PJL'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_91',
        'index': '91',
        'parent': None,
        'title': 'Hacking Printers Advisory 5/6: HP printers restoring '
                 'factory defaults through PML commands'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_92',
        'index': '92',
        'parent': None,
        'title': 'Hacking Printers Advisory 4/6: Multiple vendors buffer '
                 'overflow in LPD daemon and PJL interpreter'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_93',
        'index': '93',
        'parent': None,
        'title': 'Hacking Printers Advisory 3/6: Brother printers '
                 'vulnerable to memory access via PJL commands'},
    {   'author': 'Jens Müller',
        'date': 'Jan 30',
        'id': '2017_Jan_94',
        'index': '94',
        'parent': None,
        'title': 'Hacking Printers Advisory 6/6: Multiple vendors physical '
                 'NVRAM damage via PJL commands'},
    {   'author': 'hyp3rlinx',
        'date': 'Jan 31',
        'id': '2017_Jan_95',
        'index': '95',
        'parent': None,
        'title': 'PEAR Base System v1.10.1 Arbitrary File Download'},
    {   'author': 'Matteo Beccati',
        'date': 'Jan 31',
        'id': '2017_Jan_96',
        'index': '96',
        'parent': None,
        'title': '[REVIVE-SA-2017-001] Revive Adserver - Multiple '
                 'vulnerabilities'},
    {   'author': 'Stefan Kanthak',
        'date': 'Jan 31',
        'id': '2017_Jan_98',
        'index': '98',
        'parent': None,
        'title': 'Executable installers are vulnerable^WEVIL (case 47): '
                 "Heimdal\tSecurity's SetupLauncher vulnerable to DLL "
                 'hijacking'}]

The index file summarizes the reply's details, but we would like the full author with email, and a complete timestamp. To do this, we'll need to delve into the actual raw.html. For this notebook, we'll download the file again, but in actual usage, we would just open the existing file.

The full author and timestamp are contained between the X-Head-of-Message comments.


In [9]:
import pendulum

message = messages[4]
reply_url = url + '/' + message['index']
r = requests.get(reply_url)
reply = r.text

start = reply.index('<!--X-Head-of-Message-->') + 24
end = reply.index('<!--X-Head-of-Message-End-->')

head = reply[start:end]
soup = BeautifulSoup(head, 'html5lib')
ems = soup.find_all('em')

for em in ems:
    if em.text == 'From':
        author = em.next_sibling
        #list obfuscates email by replacing @ with ' () ' and removing periods from domain name
        if author.startswith(': '):
            author = author[2:]
        author = author.replace(' () ', '@')
        at = author.find('@')
        author = author[:at] + author[at:].replace(' ', '.') 
        message['author'] = author
    elif em.text == 'Date':
        date = em.next_sibling
        if date.startswith(': '):
            date = date[2:]
        message['date'] = str(pendulum.parse(date).in_timezone('UTC'))

print(message)


{'id': '2017_Jan_9', 'author': 'Moritz Naumann <moritz.naumann@unbelievable-machine.com>', 'index': '9', 'parent': '2017_Jan_3', 'title': 'Re: Persisted Cross-Site Scripting (XSS) in Confluence Jira Software', 'date': '2017-01-04T11:57:15+00:00'}

Finally, let's put this data into csv format.


In [ ]:
import csv
import sys

output = csv.writer(sys.stdout)
output.writerow(['id', 'title', 'date', 'author', 'parent'])
for x in messages:
    output.writerow([x['id'],
                     x['title'],
                     x['date'],
                     x['author'],
                     x['parent']])