Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote111 views2 answers
Python find index of closest value in list
Python
2 votes107 views1 answer
Sns import python
Python
123 votes117 views6 answers
Input python
Python
6 votes112 views4 answers
What is a python module
Python
1 vote132 views1 answer
Remove non-alphabetic pandas python
Python
0 vote133 views1 answer
AttributeError: 'psycopg2.extensions.Column' object has no attribute '_asdict'
Python
0 vote108 views1 answer
Multivariable traces f(x, y) = sin(x)cos(y)
Python
0 vote157 views2 answers
Create blog in django
Python Django
0 vote111 views1 answer
How to capture video in google colab with python
Python
5 votes108 views2 answers
How to empty a text file in python
Python