Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

1 vote122 views1 answer
Xaxis matplotlib
Python
1 vote128 views1 answer
Python try/except loop
Python
0 vote188 views1 answer
How to get only non-blank entry of list in python
Python
3 votes116 views3 answers
NameError: name ‘pd’ is not defined
Python
-1 vote130 views2 answers
Shutil remove
Python
1 vote119 views2 answers
Add new row to numpy array
Python
8 votes122 views2 answers
Python get public ip address
Python
20 votes133 views6 answers
Extract tar
Python
2 votes131 views1 answer
How to multiply inputs in python
Python
10 votes121 views3 answers
Create a role with discord.py
Python