Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

21 votes200 views3 answers
Skip rows in pandas read excel
Python
7 votes197 views1 answer
Get inverse dict python
Python
0 vote192 views1 answer
Python branchless programming
Python
146 votes188 views6 answers
Install pip python
Python
-3 votes229 views5 answers
Multiplication table in python
Python
0 vote225 views1 answer
Django model specify table name
Python Django
1 vote141 views1 answer
Create a response object python
Python
4 votes236 views2 answers
Python get all images in directory
Python
0 vote184 views1 answer
Pandas dataframe display cell size
Python
0 vote188 views1 answer
How to make default kwargs in python
Python