Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

42 votes202 views6 answers
If key in dictionary python
Python
24 votes199 views3 answers
Flask vs django
Python
0 vote221 views1 answer
How to add number in tuple
Python
0 vote203 views1 answer
Fastest way to check odd or even in python
Python
3 votes201 views1 answer
N random numbers python
Python
3 votes231 views2 answers
Open multiple plots python
Python
0 vote193 views2 answers
Qq plot using seaborn with regression line
Python
0 vote203 views4 answers
How to convert array of arrays into single array with unique values in numpy
Python
212 votes653 views6 answers
Generate random number python
Python
24 votes198 views6 answers
Python string list to list
Python