Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote173 views1 answer
How to pairwise permute in python
Python
2 votes167 views1 answer
Not scientific notation python
Python
2 votes172 views2 answers
Format binary string python
Python
0 vote163 views1 answer
Python loop array 0,101/100
Python
1 vote176 views1 answer
How to create a custom callback function in keras while training the model
Python
3 votes158 views2 answers
Otp generation in python
Python
0 vote159 views1 answer
Python list to tuple pairs adjacent
Python
7 votes176 views3 answers
Opencv2 python show
Python
0 vote164 views1 answer
How to check if column has na python
Python
4 votes233 views2 answers
Primary key django model
Python Django