Python web crawler

import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('.post-header>h2'):
            yield {'title': title.css('a ::text').get()}

        for next_page in response.css('a.next-posts-link'):
            yield response.follow(next_page, self.parse)

Source: scrapy.org

Similar codes

0 vote147 views1 answer
Created by and updated by in django
Python
-4 votes135 views4 answers
Latest version of django
Python
0 vote140 views1 answer
Inicio programacao python
Python
0 vote213 views1 answer
Where is tensorflow slim
Python
0 vote153 views1 answer
Pandas backward fill after upsampling
Python
12 votes143 views10 answers
Print in python
Python
0 vote162 views1 answer
Python thunks
Python
0 vote200 views2 answers
How to create template folder in django
Python Django
0 vote167 views4 answers
Create a python for loop that sums the numbers from 100 to 200
Python
1 vote148 views2 answers
How to swap two variables without using third variable python
Python