0

Estoy utilizando scrapy en una p'agina d Amazon de las laptops que mas se han vendido

Quiero Seleccionar el precio y el producto, por ejemplo de este framgmento de HTML

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=<a class="a-link-normal a-text-normal" href="/Apple-MacBook-13-inch-256GB-Storage/dp/B0863D4XJW/ref=zg_bs_565108_1?_encoding=UTF8&psc=1&refRID=XGMP498DGDMYEQ4GX0HW">
        <span class="a-color-secondary">2 offers from <span class="a-size-base a-color-price">
            <span class='p13n-sc-price' >$979.00</span>
        </span>
    </span>
</a>
</span>
</div>
</span>
</li>
<li class="zg-item-immersion" role="gridcell">
    <span class="a-list-item">
        <div class="a-section a-spacing-none aok-relative">
            <div class="a-row a-spacing-none aok-inline-block">
                <span class="a-size-small aok-float-left zg-badge-body zg-badge-color">
                    <span class="zg-badge-text">#2</span>
                </span>
                <span class="aok-float-left zg-badge-triangle zg-badge-color">

                </span>
            </div>
            <span class="aok-inline-block zg-item">
                <a class="a-link-normal" href="/Acer-Flagship-CB3-532-Premium-Chromebook/dp/B06XD3LXXK/ref=zg_bs_565108_2?_encoding=UTF8&psc=1&refRID=XGMP498DGDMYEQ4GX0HW">
                    <span class="zg-text-center-align">
                        <div class="a-section a-spacing-small">
                            <img alt="(Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS" src="https://images-na.ssl-images-amazon.com/images/I/51840t%2BtcFL._AC_UL200_SR200,200_.jpg" height="200" width="200">
                        </div>
                    </span>
        <div class="p13n-sc-truncate p13n-sc-line-clamp-2" aria-hidden="true" data-rows="2">
            (Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS
        </div>
    </a>, initial-scale=1.0">
    <title>Document</title>
</head>

lei la documentacion de la pagina https://devhints.io/xpath y encontre que puede obtener de esta forma el precio :

./div[@class="p13-sc-truncate p13-sc-line-clamp-2"]/text()

y el producto con sus caracteristicas:

./spam[@class="p13n-sc-price"]/text()

(puedo estar equivocado claro, y haber malentendido de alguna forma el articulo)

el codigo que estoy utilizando es el siguiente:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "amazon"

    def start_requests(self):
        urls = [

            'https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108'

        ]

        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        '''
        page = response.url.split("/")[-2]
        filename = 'amazonBestLaptops-%s.html' % page
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log('Saved file %s' % filename)
        '''

        SET_SELECTOR = '.set'
        for amazon_info in response.css(SET_SELECTOR):


            PRICE_SELECTOR = './div[@class="p13-sc-truncate p13-sc-line-clamp-2"]/text()'
            PRODUCT_SELECTOR = './spam[@class="p13n-sc-price"]/text()'

            yield {

                'product': amazon_info.xpath(PRODUCT_SELECTOR),
                'price': amazon_info.xpath(PRICE_SELECTOR),

            }

y el error que estoy obteniendo(porque no estoy obteniendo lo que se supone:)

D:\maykel\computacion\scrapy\amazonLaptopsBest\amazonLaptopsBest\spiders>scrapy runspider amazon_spyder.py
2020-05-11 20:57:12 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: amazonLaptopsBest)
2020-05-11 20:57:12 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 19:29:22) [MSC v.1916 32 bit (Intel)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Windows-10-10.0.18362-SP0
2020-05-11 20:57:12 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2020-05-11 20:57:12 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'amazonLaptopsBest',
 'NEWSPIDER_MODULE': 'amazonLaptopsBest.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_LOADER_WARN_ONLY': True,
 'SPIDER_MODULES': ['amazonLaptopsBest.spiders']}
2020-05-11 20:57:12 [scrapy.extensions.telnet] INFO: Telnet Password: e701d9bca26e2f99
2020-05-11 20:57:12 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-05-11 20:57:13 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-05-11 20:57:13 [scrapy.core.engine] INFO: Spider opened
2020-05-11 20:57:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-05-11 20:57:13 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-05-11 20:57:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/robots.txt> (referer: None)
2020-05-11 20:57:14 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108> (failed 1 times): 503 Service Unavailable
2020-05-11 20:57:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108> (referer: None)
2020-05-11 20:57:15 [scrapy.core.engine] INFO: Closing spider (finished)
2020-05-11 20:57:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1058,
 'downloader/request_count': 3,
 'downloader/request_method_count/GET': 3,
 'downloader/response_bytes': 48322,
 'downloader/response_count': 3,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/503': 1,
 'elapsed_time_seconds': 2.011807,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 5, 12, 0, 57, 15, 691287),
 'log_count/DEBUG': 3,
 'log_count/INFO': 10,
 'response_received_count': 2,
 'retry/count': 1,
 'retry/reason_count/503 Service Unavailable': 1,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2020, 5, 12, 0, 57, 13, 679480)}
2020-05-11 20:57:15 [scrapy.core.engine] INFO: Spider closed (finished)

me pueden ayudar imagino que sean lso xpath que no estan bien implementados y no obtienen lo que deseo,por ejemplo:

$979.00

(Renewed) Acer Flagship CB3-532 15.6 inches HD Premium Chromebook - Intel Dual-Core Celeron N3060 up to 2.48GH.z, 2GB RAM, 16GB SSD, Wireless AC, HDMI, USB 3.0, Webcam, Chrome OS

y asi sucesivamente

Gracias por su tiempo.

1
  • Termine instalando una extension --> addons.mozilla.org/es/firefox/addon/xpath_finder/--y ahora obuve ambos xpath ...sin embargo, sigo sin obtener el resultado deseado..Las direcciones que obtuve fueron estas: /html/body/div[1]/div[2]/div/div/div[1]/div/ol/li[1]/span/div/span/a[1]/div y /html/body/div[1]/div[2]/div/div/div[1]/div/ol/li[1]/span/div/span/a[2]/span/span/span Commented el 12 may. 2020 a las 3:01

1 respuesta 1

0

Creo que quiere la informacion de cada producto

import scrapy

class QuotesSpider(scrapy.Spider):
name = "amazon"

def start_requests(self):
    start_urls = [
        'https://www.amazon.com/gp/bestsellers/electronics/565108?ref_=Oct_s9_apbd_obs_hd_bw_b2N0e_S&pf_rd_r=5WY8SVEKCHZKM8YHZXQW&pf_rd_p=079802f4-7ae7-5199-a84a-6dbb4331abf2&pf_rd_s=merchandised-search-11&pf_rd_t=BROWSE&pf_rd_i=565108'
    ]
    for url in start_urls:
        print("Start Url ", url)
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
    '''
    page = response.url.split("/")[-2]
    filename = 'amazonBestLaptops-%s.html' % page
    with open(filename, 'wb') as f:
        f.write(response.body)
    self.log('Saved file %s' % filename)
    '''
    all_computers = response.xpath('//span/a[@class="a-link-normal"]/@href').getall()
    for laptop in all_computers:
        # no esta la url completa
        laptop = 'https://www.amazon.com' +laptop
        print("enlace:\t", laptop)
        yield scrapy.Request(laptop, callback=self.parse_detalle)

def parse_detalle(self, response):
    print("entra equipo")
    NAME_SELECTOR = 'span#productTitle::text' # css UNICO
    PRICE_SELECTOR = 'span#priceblock_ourprice::text' # css UNICO
    CAPACITY_SELECTOR = "//label[@class='a-form-label']/following-sibling::span[1]/text()" # XPath
    COLOR_SELECTOR = 'div#variation_color_name>div>span::text' # este no
    CONFIG_SELECTOR = 'div#variation_configuration>div>span::text' # este no
    yield {
        'NAME_SELECTOR': response.css(NAME_SELECTOR).extract_first(),
        'PRICE_SELECTOR': response.css(PRICE_SELECTOR).extract_first(),
        'CAPACITY_SELECTOR': response.xpath(CAPACITY_SELECTOR).extract_first(),
        'COLOR_SELECTOR': response.css(COLOR_SELECTOR).extract_first(),
        'CONFIG_SELECTOR': response.css(CONFIG_SELECTOR).extract_first(),

    }

Como veras trae

{'NAME_SELECTOR': '\n                \n                    \n                    \n                \n\n                \n                    \n                    \n                        (Renewed )Samsung Chromebook XE303C12-A01 11.6-inch, Exynos 5250, 2GB RAM, 16GB SSD, Silver\n                    \n                \n\n                \n                    \n                    \n                \n            ', 'PRICE_SELECTOR': '$180.00', 'CAPACITY_SELECTOR': 'Silver', 'COLOR_SELECTOR': 'Silver', 'CONFIG_SELECTOR': None}

Ahora estoy trabajando en scrapy pero aqui me complico un poco con los texto de cada cosa AMAZON le pone mucho a su sitio, espero puedes sustituir los xpath y lo de pasar de pagina tenia el codigo pero ahora mismo no lo encuentro

7
  • Ten cuidado de ser baneado del sitio. puedes usar una extension como XPath Helper (Chrome) o Ruto x-path Finder (Fire Fox)
    – Cristian
    Commented el 12 may. 2020 a las 13:52
  • A que se refiere Cristian con esto: Ten cuidado de ser baneado del sitio...Si estoy haciendo indevido por favor, hagamelo saber Commented el 13 may. 2020 a las 12:42
  • por otra parte sigo queriendo saber como obtener la informacion de una etiqueta con una clase unica, como la quepuse en mi ejemplo que no me funciona Commented el 13 may. 2020 a las 12:43
  • pues algunos sitios te bloquean porque si bien no es ilegal tampoco lo ven con buenos ojos y te bloquean el acceso por unos dias o puede que un mes, no eh sabido que sea mas. Con respecto a lo de las clases unicas pues hacer lo como el de PRECIO usando el css, no suelo usar lo porque es el mas propenso a cambiar, eliminas estilo antes que el maquetado. Espero haber despejado tus dudas
    – Cristian
    Commented el 13 may. 2020 a las 15:07
  • gracias..sin embargo, como lo harias. ayudame a obtener ese conocimiento Commented el 14 may. 2020 a las 18:00

Tu Respuesta

By clicking “Publica tu respuesta”, you agree to our terms of service and acknowledge you have read our privacy policy.

¿No es la respuesta que buscas? Examina otras preguntas con la etiqueta o formula tu propia pregunta.