Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

7 votes149 views3 answers
How to install library in python
Python
0 vote139 views1 answer
Button onclick message box in python tkinter
Python
213 votes367 views6 answers
Python iterate through dictionary
Python
2 votes147 views1 answer
Get statistics from list python
Python
0 vote182 views2 answers
Get ContentType with django get_model
Python Django
0 vote150 views1 answer
Print start time in python
Python
12 votes144 views3 answers
Python delete all files in directory
Python
0 vote156 views2 answers
Python suppress exponential notation
Python
1 vote165 views2 answers
Mp4 to mp3 in python
Python
0 vote149 views1 answer
Python cv2 blob detection seg fault
Python