Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote60 views1 answer
FunÃ§Ã£o anonima python
Python
0 vote96 views1 answer
Updating many to many field in django using pk
Python
19 votes98 views5 answers
Python floor
Python
0 vote155 views1 answer
Typecasting python
Python
0 vote104 views1 answer
Print all objects in list python
Python
0 vote136 views1 answer
Django generate openapi schema command line
Python Django
2 votes106 views1 answer
Python pi value
Python
2 votes133 views1 answer
Auto datetime in django models
Python Django
1 vote99 views1 answer
Python function remove all whitespace from all character columns in dataframe
Python
2 votes95 views3 answers
Delete unnamed 0 columns
Python