Estoy utilizando scrapy en una p'agina d Amazon de las laptops que mas se han vendido
Quiero Seleccionar el precio y el producto, por ejemplo de este framgmento de HTML
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=<a class="a-link-normal a-text-normal" href="/Apple-MacBook-13-inch-256GB-Storage/dp/B0863D4XJW/ref=zg_bs_565108_1?_encoding=UTF8&psc=1&refRID=XGMP498DGDMYEQ4GX0HW">
<span class="a-color-secondary">2 offers from <span class="a-size-base a-color-price">
<span class='p13n-sc-price' >$979.00</span>
</span>
</span>
</a>
</span>
</div>
</span>
</li>
<li class="zg-item-immersion" role="gridcell">
<span class="a-list-item">
<div class="a-section a-spacing-none aok-relative">
<div class="a-row a-spacing-none aok-inline-block">
<span class="a-size-small aok-float-left zg-badge-body zg-badge-color">
<span class="zg-badge-text">#2</span>
</span>
<span class="aok-float-left zg-badge-triangle zg-badge-color">
</span>
</div>
<span class="aok-inline-block zg-item">
<a class="a-link-normal" href="/Acer-Flagship-CB3-532-Premium-Chromebook/dp/B06XD3LXXK/ref=zg_bs_565108_2?_encoding=UTF8&psc=1&refRID=XGMP498DGDMYEQ4GX0HW">
<span class="zg-text-center-align">
<div class="a-section a-spacing-small">
<img alt="(Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS" src="https://images-na.ssl-images-amazon.com/images/I/51840t%2BtcFL._AC_UL200_SR200,200_.jpg" height="200" width="200">
</div>
</span>
<div class="p13n-sc-truncate p13n-sc-line-clamp-2" aria-hidden="true" data-rows="2">
(Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS
</div>
</a>, initial-scale=1.0">
<title>Document</title>
</head>
lei la documentacion de la pagina https://devhints.io/xpath y encontre que puede obtener de esta forma el precio :
./div[@class="p13-sc-truncate p13-sc-line-clamp-2"]/text()
y el producto con sus caracteristicas:
./spam[@class="p13n-sc-price"]/text()
(puedo estar equivocado claro, y haber malentendido de alguna forma el articulo)
el codigo que estoy utilizando es el siguiente:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "amazon"
def start_requests(self):
urls = [
'https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108'
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
'''
page = response.url.split("/")[-2]
filename = 'amazonBestLaptops-%s.html' % page
with open(filename, 'wb') as f:
f.write(response.body)
self.log('Saved file %s' % filename)
'''
SET_SELECTOR = '.set'
for amazon_info in response.css(SET_SELECTOR):
PRICE_SELECTOR = './div[@class="p13-sc-truncate p13-sc-line-clamp-2"]/text()'
PRODUCT_SELECTOR = './spam[@class="p13n-sc-price"]/text()'
yield {
'product': amazon_info.xpath(PRODUCT_SELECTOR),
'price': amazon_info.xpath(PRICE_SELECTOR),
}
y el error que estoy obteniendo(porque no estoy obteniendo lo que se supone:)
D:\maykel\computacion\scrapy\amazonLaptopsBest\amazonLaptopsBest\spiders>scrapy runspider amazon_spyder.py
2020-05-11 20:57:12 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: amazonLaptopsBest)
2020-05-11 20:57:12 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2020-05-11 20:57:12 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-05-11 20:57:12 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'amazonLaptopsBest',
'NEWSPIDER_MODULE': 'amazonLaptopsBest.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_LOADER_WARN_ONLY': True,
'SPIDER_MODULES': ['amazonLaptopsBest.spiders']}
2020-05-11 20:57:12 [scrapy.extensions.telnet] INFO: Telnet Password: e701d9bca26e2f99
2020-05-11 20:57:12 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-05-11 20:57:13 [scrapy.core.engine] INFO: Spider opened
2020-05-11 20:57:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-05-11 20:57:13 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-05-11 20:57:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/robots.txt> (referer: None)
2020-05-11 20:57:14 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108> (failed 1 times): 503 Service Unavailable
2020-05-11 20:57:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108> (referer: None)
2020-05-11 20:57:15 [scrapy.core.engine] INFO: Closing spider (finished)
2020-05-11 20:57:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1058,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 48322,
'downloader/response_count': 3,
'downloader/response_status_count/200': 2,
'downloader/response_status_count/503': 1,
'elapsed_time_seconds': 2.011807,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 5, 12, 0, 57, 15, 691287),
'log_count/DEBUG': 3,
'log_count/INFO': 10,
'response_received_count': 2,
'retry/count': 1,
'retry/reason_count/503 Service Unavailable': 1,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/200': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'start_time': datetime.datetime(2020, 5, 12, 0, 57, 13, 679480)}
2020-05-11 20:57:15 [scrapy.core.engine] INFO: Spider closed (finished)
me pueden ayudar imagino que sean lso xpath que no estan bien implementados y no obtienen lo que deseo,por ejemplo:
$979.00
(Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS
y asi sucesivamente
Gracias por su tiempo.