{"id":3584,"date":"2013-01-18T05:19:21","date_gmt":"2013-01-18T04:19:21","guid":{"rendered":"http:\/\/emilkirkegaard.dk\/en\/?p=3584"},"modified":"2013-01-18T05:19:54","modified_gmt":"2013-01-18T04:19:54","slug":"ripping-threads-from-able2know-forums","status":"publish","type":"post","link":"https:\/\/emilkirkegaard.dk\/en\/2013\/01\/ripping-threads-from-able2know-forums\/","title":{"rendered":"Ripping threads from able2know forums"},"content":{"rendered":"

So, i thought that is a good idea to take a copy of all the interesting threads on varius forums, just in case they shut down. doing it manually is waste of time. so, i went coding and make a crawler. after spending a couple of hours, i have now made a crawler that reads lines from a txt file, and downloads pages into folders from that.<\/p>\n

code is here: Forum crawler.py<\/a><\/p>\n

or here:<\/p>\n

import urllib2\r\nimport re\r\nimport os\r\n\r\ndef isodd(num):\r\n\u00a0\u00a0\u00a0 return num & 1 and True or False\r\n\r\n#open data about what to rip\r\nfile0 = open(\"Forum threads\/threads.txt\",'r')\r\n\r\n#assign data to var\r\ndata0 = file0.readlines()\r\n\r\n#hard var\r\nnumber = 0\r\n\r\n#skip odd numbers\r\nfor line in data0:\r\n\r\n\u00a0\u00a0\u00a0 #if it is even, then set first var, and continue\r\n\u00a0\u00a0\u00a0 if not isodd(number):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 outputfolder = line\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 outputfolder = outputfolder[:-1]\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 number = number+1\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 continue\r\n\r\n\u00a0\u00a0\u00a0 #get thread url and remove last chars (linebreak +\r\n\u00a0\u00a0\u00a0 if isodd(number):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 threadurl = line\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 threadurl = threadurl[:-2]\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 number = number+1\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print \"starting to crawl thread \"+threadurl\r\n\r\n\u00a0\u00a0\u00a0 #create folder\r\n\u00a0\u00a0\u00a0 if not os.path.isdir(\"Forum threads\/\"+outputfolder):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 os.makedirs(\"Forum threads\/\"+outputfolder)\r\n\r\n\u00a0\u00a0\u00a0 #var introduction\r\n\u00a0\u00a0\u00a0 lastdata2 = \"ijdsfijkds\"\r\n\u00a0\u00a0\u00a0 lastdata = \"kjdsfsa\"\r\n\r\n\u00a0\u00a0\u00a0 #looping over all the pages\r\n\u00a0\u00a0\u00a0 for page in range(999):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #range starts at 0, so +1 is needed\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 response = urllib2.urlopen(threadurl+str(page+1))\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #assign the data to a var\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 page_source = response.read()\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #used for detection of identical output\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #replace data in var2\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 lastdata2 = lastdata\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #load new data into var1\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 lastdata = page_source\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #check if they are identical\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if page>0:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if lastdata == lastdata2:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print \"data identical, stopping loop\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 break\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #alternative check, check len\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if page>0:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 if len(lastdata) == len(lastdata2):\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print \"length identical, stopping loop\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 break\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #used for detection of identical output\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #create a file for each page, and save the data in it\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 output = open(\"Forum threads\/\"+outputfolder+\"\/\"+str(page+1)+\".html\",'w')\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 output.write(page_source)\r\n\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 #progress\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print \"wrote page \"+str(page+1)+\" in \"+outputfolder+\"\/\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 print \"length of file is \"+str(len(page_source))\r\n\r\n\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
So, i thought that is a good idea to take a copy of all the interesting threads on varius forums, just in case they shut down. doing it manually is waste of time. so, i went coding and make a crawler. after spending a couple of hours, i have now made a crawler that reads […]<\/p>\n","protected":false},"author":17,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[1931],"class_list":["post-3584","post","type-post","status-publish","format-standard","hentry","tag-programming","entry"],"_links":{"self":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/3584","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/users\/17"}],"replies":[{"embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/comments?post=3584"}],"version-history":[{"count":2,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/3584\/revisions"}],"predecessor-version":[{"id":3587,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/posts\/3584\/revisions\/3587"}],"wp:attachment":[{"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/media?parent=3584"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/categories?post=3584"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/emilkirkegaard.dk\/en\/wp-json\/wp\/v2\/tags?post=3584"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}