Como dice el título estoy haciendo una araña, es la primera vez que hago algo como esto. hasta ahora mi araña ya puede seguir los enlaces y traerse el texto. Es una araña sencilla.
Pero el problema esta que me repite los datos cuando los exporto a un Json.
Mi duda es ¿Qué tengo mal? o ¿Por qué mi araña repite los datos? Estoy usando Python 3.8
import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
class WitsiSpider(CrawlSpider):
name = 'witsi'
allowed_domains = ['www.quotes.toscrape.com']
start_urls = ['http://quotes.toscrape.com']
base_url = 'http://quotes.toscrape.com'
rules = (
Rule(LinkExtractor(allow=r'page/'),callback = 'parse', follow=True ),
)
def parse(self, response):
citas = response.xpath('//*[@class="quote"]')
for cita in citas:
texto = cita.xpath('.//span[@class="text"]/text()').extract_first()
autor = cita.xpath('.//*[@class="author"]/text()').extract_first()
palabras_claves = cita.xpath('.//*[@itemprop="keywords"]/@content').extract_first()
yield{ 'Texto' : texto,
'Autor' : autor,
'Palabras Claves' : palabras_claves }
La Salida por consola es esta
2020-09-09 14:02:36 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 43886,
'downloader/request_count': 153,
'downloader/request_method_count/GET': 153,
'downloader/response_bytes': 217935,
'downloader/response_count': 153,
'downloader/response_status_count/200': 153,
'dupefilter/filtered': 794,
'elapsed_time_seconds': 6.949397,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 9, 9, 20, 2, 36, 813941),
'item_scraped_count': 332,
'log_count/DEBUG': 486,
'log_count/INFO': 11,
'request_depth_max': 11,
'response_received_count': 153,
'scheduler/dequeued': 153,
'scheduler/dequeued/memory': 153,
'scheduler/enqueued': 153,
'scheduler/enqueued/memory': 153,
'start_time': datetime.datetime(2020, 9, 9, 20, 2, 29, 864544)}
2020-09-09 14:02:36 [scrapy.core.engine] INFO: Spider closed (finished)
y esta es la salidad de los datos
{"Texto": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "Autor": "Albert Einstein", "Palabras Claves": "inspirational,life,live,miracle,miracles"}, {"Texto": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "Autor": "Albert Einstein", "Palabras Claves": "inspirational,life,live,miracle,miracles"}, {"Texto": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "Autor": "Albert Einstein", "Palabras Claves": "inspirational,life,live,miracle,miracles"}, {"Texto": "\u201cThere are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.\u201d", "Autor": "Albert Einstein", "Palabras Claves": "inspirational,life,live,miracle,miracles"}, {"Texto": "\u201cIt is better to be hated for what you are than to be loved for what you are not.\u201d", "Autor": "Andr\u00e9 Gide", "Palabras Claves": "life,love"}, {"Texto": "\u201cThis life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.\u201d", "Autor": "Marilyn Monroe", "Palabras Claves": "friends,heartbreak,inspirational,life,love,sisters"}, {"Texto": "\u201cI may not have gone where I intended to go, but I think I have ended up where I needed to be.\u201d", "Autor": "Douglas Adams", "Palabras Claves": "life,navigation"}, {"Texto": "\u201cGood friends, good books, and a sleepy conscience: this is the ideal life.\u201d", "Autor": "Mark Twain", "Palabras Claves": "books,contentment,friends,friendship,life"}, {"Texto": "\u201cLife is what happens to us while we are making other plans.\u201d", "Autor": "Allen Saunders", "Palabras Claves": "fate,life,misattributed-john-lennon,planning,plans"}, {"Texto": "\u201cToday you are You, that is truer than true. There is no one alive who is Youer than You.\u201d", "Autor": "Dr. Seuss", "Palabras Claves": "comedy,life,yourself"}, {"Texto": "\u201cLife is like riding a bicycle. To keep your balance, you must keep moving.\u201d", "Autor": "Albert Einstein", "Palabras Claves": "life,simile"}, {"Texto": "\u201cLife isn't about finding your
Se supone que solo son 10 páginas y 10 citas por páginas,dando un total de 100 citas pero mi Spider me exporta 334 citas y hay citas repetidas.