Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote251 views1 answer
Smtp django
Python Django
3 votes246 views1 answer
If condition
Python
0 vote203 views1 answer
Strategy forex with python
Python
2 votes220 views1 answer
Python class get attribute by name
Python
2 votes198 views3 answers
Python rotate list
Python
6 votes163 views1 answer
Business logic in django
Python
2 votes186 views1 answer
Lcm math python library
Python
18 votes158 views1 answer
Is odd python
Python
1 vote225 views1 answer
Remove single and double quotes from string python
Python
1 vote346 views1 answer
Python sha256 of file
Python