Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote141 views1 answer
Ajouter élément liste python
Python
10 votes142 views5 answers
Find nan values in a column pandas
Python
0 vote135 views1 answer
Argumrnt with reverse django
Python
0 vote130 views1 answer
Pytube sample script
Python
1 vote133 views1 answer
__call__() python
Python
0 vote130 views1 answer
Inicio programacao python
Python
8 votes152 views3 answers
Normalize data python
Python
-1 vote132 views2 answers
How to set pandas dataframe as global
Python
0 vote106 views1 answer
How to search over a notebook in python
Python
0 vote144 views1 answer
Spyder new instance
Python