Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

-1 vote131 views1 answer
Pyside2 qt designer tab widget example
Python
0 vote224 views1 answer
Django route accept params with character
Python Django
0 vote184 views1 answer
List files in http directory python
Python
1 vote176 views1 answer
Python auto reload module ipython
Python
0 vote240 views1 answer
Django save another class data while saving a class
Python Django
1 vote190 views1 answer
Cursor python
Python
0 vote220 views1 answer
Query first 5 element in django
Python Django
0 vote200 views1 answer
Python instagram login
Python
8 votes239 views6 answers
Django filters
Python Django
0 vote187 views1 answer
Read .fasta python
Python