Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

10 votes267 views3 answers
Convert a dictionary into dataframe python
Python
1 vote175 views1 answer
Python open file relative to script location
Python
0 vote195 views1 answer
Python paramiko
Python
2 votes175 views2 answers
Python replace newline
Python
0 vote164 views1 answer
Stores number in set using input in python
Python
11 votes175 views3 answers
How to convert small letters to capital letters in python
Python
-1 vote225 views1 answer
Install grepper for edge
Python
0 vote199 views1 answer
How to read json file from s3 bucket into aws glue job
Python
0 vote206 views1 answer
Mechanize python #2
Python
0 vote173 views1 answer
Python replace nth occurrence in string
Python