soy principiante en esto de la programación y estoy ahora aprendiendo Python. Una de las utilidades es poder obtener datos de Webs mediante Scrapy. La cuestión es que he probado en distintas páginas y me ha ido bien. Al probar sobre esta pagina www.sxyprn.com y querer obtener el enlace que guarda el video, la salida es un espacio en blanco. Explico un poco el código que tengo.
El spider es el siguiente:
import scrapy
import requests
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from Sxyprn.items import SxyprnItem
class SpdvideosSpider(CrawlSpider):
name = 'spdvideos'
allowed_domains = ['sxyprn.com']
start_urls = ['https://www.sxyprn.com']
rules = {
Rule(LinkExtractor(restrict_xpaths=('//div[@id="content_div"]/div[3]/div[3]//div[2]/a')), follow= True,
callback = 'parse_item')
}
def parse_item(self, response):
item = SxyprnItem()
item['videos'] = str(response.xpath('//video[@id="player_el"]/@src').get())
yield item
La regla hace que llegue a todas las paginas donde se encuentran los videos y en el parse_item guardo el valor del atributo src de cada video.
Al correr la araña este es el resultado (solo he puesto una muestra, pero saca 35 valores mas o menos) donde se puede ver que guarda un espacio en blanco.
2020-03-22 14:11:32 [scrapy.core.engine] INFO: Spider opened
2020-03-22 14:11:32 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-03-22 14:11:32 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-03-22 14:11:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/robots.txt> (referer: None)
2020-03-22 14:11:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com> (referer: None)
2020-03-22 14:11:32 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.sxyprn.com/post/5e76138156bc8.html> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2020-03-22 14:11:32 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e76138156bc8.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e7637ca30adc.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sxyprn.com/post/5e76138156bc8.html>
{'videos': ''}
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e766d272f600.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e7678e92ac35.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e771450b748f.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e76738bf4d4d.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e765ef97259b.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.sxyprn.com/post/5e768662770e1.html> (referer: https://www.sxyprn.com)
2020-03-22 14:11:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sxyprn.com/post/5e7637ca30adc.html>
{'videos': ''}
2020-03-22 14:11:33 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.sxyprn.com/post/5e766d272f600.html>
{'videos': ''}
Con el inspector reviso como esta estructurada la pagina y el atributo src hay un valor como muestro a continuación.
<video id="player_el" class="player_el player_el_nc" data-postid="5e7650c300983" src="/cdn8/c8/5d2r107jzk1n9g85z4189a22zd280l9/CAFp6C-S3l_mMsnFLxgFAg/1584887869/rp2x4a551q6c1m7r059l5v9k7y7a1u7/w5me27z6y5k0ycx380w009a8k3u.vid" poster="//s7.trafficdeposit.com/blog/img/245161709597717/5e7650c300983/0.jpg" controls="controls" playsinline=""></video>
Si cambio el atributo a data-postid por ejemplo, si que me devuelve el valor.
Qué estoy haciendo mal? Muchas gracias