Estoy tratando de raspar las preguntas y respuestas de [formularios Google cuyas urls están en un archivo csv][1]. Aquí hay un extracto: `links_y_temas.csv`: Link,Task https://docs.google.com/forms/d/1MYSjxAMCXMXB02GPXqkewVx35_ptzv0XO7GRZQGwyGE/edit?usp=sharing,Hotel ABC https://docs.google.com/forms/d/1VQpzX1GqsnI92J1trlP37v3GjVgNMp1h-1Rh_n8orII/edit?usp=sharing,Airline XYZ https://docs.google.com/forms/d/1z-qyHp7O4eTI848b6L8_vh59IJ-y0bFpAo--zwPKJxY/edit?usp=sharing,Airline XYZ https://docs.google.com/forms/d/1IoDbsif5qorINuUrF1Dl9iMtIdnAsTvVA3vMVsVHjy8/edit?usp=sharing,Airline XYZ Trato de obtener las preguntas y respuestas de los cuestionarios/urls para meterlas en un archivo csv que se parece al siguiente: pickle_file_name,id,question,answer_1,answer_2,answer_3,answer_4,answer_5,answer_6,answer_7,answer_8,answer_9,answer_10,answer_11,answer_12,answer_13,answer_14 applicantHotel_ABC_c,1,How do you feel about your next vacation after COVID-19?,,,,,,,,,,,,,, applicantHotel_ABC_c,2,When do you think your next vacation can start?,In next 3 months,In next 6 months,In next 1 year,Only once COVID-19 is under control,Only once COVID-19 vaccine is developed,,,,,,,,, applicantHotel_ABC_c,3,What are your preferences regarding medical treatment policy (with additional cost)?,Doctor's availability in hotel,Ventilator availability in hotel,Tie-ups with nearby hospitals,Availability of medical rooms with primary first aid care,,,,,,,,,, applicantHotel_ABC_c,4,What is your preferences of complementary breakfast?,Buffet breakfast with social distancing,Buffet breakfast replaced with Ala-carte with limited options,Breakfast to be delivered in room with limited options (chargeable),Packaged breakfast only,,,,,,,,,, Sin embargo, hay dos tipos de cuestionarios, los que se publican (los formularios de Google que enfrentamos en la vida real) y aquellos para los que tenemos acceso al backend. Si puedo hojear ambos tipos de cuestionarios, no puedo obtener las preguntas de las preguntas publicadas. En efecto obtengo los siguientes tipos de exceptiones: - `UnexpectedAlertPresentException`: es porque parece que hay el boton para modificar el `location: https://docs.google.com/forms/d/1HAMUvDpYiz-SpQpKwUOxyHqn3Be7OV9vLER3K5ltrxg/edit?usp=sharing UnexpectedAlertPresentException` - `content_area.get_attribute("aria-label")`: Me siento como si estuviera repitiendo una y otra vez sobre la misma cosa. ¡Basta de hablar! Aquí está mi código que te permite obtener un poco más del 5% de los formularios de Google: from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC import time import pandas as pd from selenium.common.exceptions import ElementNotInteractableException, NoSuchElementException from selenium.webdriver.chrome.options import Options from selenium.webdriver.support.ui import WebDriverWait from selenium.common import exceptions import pickle WDWTIME = 20 USER = '[email protected]' # ciertos google forms le requieren PWD = "yourpassword" def setup_chromedriver(): chrome_options = Options() chrome_options.add_argument("--headless") driver = webdriver.Chrome("C:\Programs\chromedriver.exe") """Algunos de los formularios de Google necesitan un acceso""" url = 'https://www.google.com/accounts/' driver.get(url) # Encontrar el login login_field = WebDriverWait(driver, WDWTIME).until( EC.presence_of_element_located((By.ID, 'identifierId'))) login_field.send_keys(USER) # Haga clic en el botón siguiente driver.find_element_by_id('identifierNext').click() # Encontrar el campo de la contraseña time.sleep(4) driver.set_page_load_timeout(50) driver.set_script_timeout(50) password_field = WebDriverWait(driver, WDWTIME).until( EC.presence_of_element_located((By.ID, 'password'))) password_field = password_field.find_element_by_tag_name('input') password_field.send_keys(PWD) # Haz click sobre "next" button driver.find_element_by_id('passwordNext').click() driver.set_page_load_timeout(30) driver.set_script_timeout(30) return driver def load_data(): df = pd.read_csv("research_assistant_intern_recruitment_an.csv") filter_col = ["Link"] return df, filter_col def get_published_questionnaire(): print("published questionnaire") questionnaire = {} btns = driver.find_elements_by_css_selector(".appsMaterialWizButtonEl") # el botón "Siguiente", *advertencia* "Solicitar acceso de edición" también está activado. next_btns = driver.find_elements_by_class_name("appsMaterialWizButtonPaperbuttonContent.exportButtonContent") if next_btns: next_btns[-1].click() next_btns = driver.find_elements_by_class_name("appsMaterialWizButtonPaperbuttonContent.exportButtonContent") while next_btns != []: containers = driver.find_elements_by_class_name( "freebirdFormviewerViewNumberedItemContainer" ) len_containers = len(containers) for container in containers: len_containers -=1 print("len_containers: ", len_containers) try: question = container.find_element_by_class_name( "freebirdFormviewerViewItemsItemItemTitle.exportItemTitle.freebirdCustomFont" ) except NoSuchElementException: print("No question, NoSuchElementException") continue except exceptions.StaleElementReferenceException: print("No question, StaleElementReferenceException") continue responses = container.find_elements_by_class_name( "docssharedWizToggleLabeledLabelText" ) extracted_text = [response.text for response in responses] questionnaire[question.text] = extracted_text content_areas = driver.find_elements_by_class_name( "quantumWizTextinputSimpleinputInput.exportInput" ) for content_area in content_areas: skip = ["Document title", "Titre du document", "Adresse e-mail valide"] if content_area.get_attribute("aria-label") in skip and not content_area.get_attribute("aria-label").isspace(): print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label")) else: print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label")) content_area.send_keys("10102015") content_areas = driver.find_elements_by_class_name( "quantumWizTextinputPaperinputInput.exportInput" ) for content_area in content_areas: if content_area.get_attribute("type") == "date" and not content_area.get_attribute("type").isspace(): condition = content_area.get_attribute("type") if condition == "date": content_area.send_keys("10102015") elif content_area.get_attribute("max") and not content_area.get_attribute("max").isspace(): max = content_area.get_attribute("max") content_area.send_keys(max) elif content_area.get_attribute("aria-label") and not content_area.get_attribute("aria-label").isspace(): condition = content_area.get_attribute("aria-label") print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label")) if condition == "State (Two letter Abbreviation)": content_area.send_keys("CA") else: content_area.send_keys("10102015") for content_area in content_areas: skip = ["Document title", "Titre du document", "Adresse e-mail valide"] if content_area.get_attribute("aria-label") in skip and not content_area.get_attribute("aria-label").isspace(): print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label")) else: print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label")) content_area.send_keys("10102015") btns_answers = driver.find_elements_by_css_selector(".appsMaterialWizToggleRadiogroupElContainer") for btn_answer in btns_answers: try: driver.execute_script('arguments[0].scrollIntoView(true);', btn_answer) btn_answer.click() except ElementNotInteractableException: pass except exceptions.ElementClickInterceptedException: continue # long answers content_areas = driver.find_elements_by_class_name( "quantumWizTextinputPapertextareaInput.exportTextarea" ) for content_area in content_areas: content_area.send_keys("This restaurant is really good! Me and my boyfriend went there on our holiday \ we had dinner there at 3 of February food was 100% And the service vas 150% And i really want to thank " "\Asie for a really good service as for his coworkers. We highly recommended \ this restaurant!") # check boxes btn_check_boxes = driver.find_elements_by_class_name( "docssharedWizToggleLabeledContainer.freebirdFormviewerViewItemsCheckboxContainer" ) for btn_check_box in btn_check_boxes: btn_check_box.click() break # btn_check_box[-1].click() # # other weird check boxes btn_check_boxes = driver.find_elements_by_class_name( "docssharedWizToggleLabeledLabelText.exportLabel.freebirdFormviewerViewItemsCheckboxLabel" ) for btn_check_box in btn_check_boxes: btn_check_box.click() break # btns[-1].click() next_btns = driver.find_elements_by_class_name( "appsMaterialWizButtonPaperbuttonContent.exportButtonContent") if next_btns != []: next_btns[-1].click() next_btns = [] else: continue print("questionnaire: ", questionnaire) return questionnaire def get_backend_questionnaire(): print("backend questionnaire") # a veces empezamos con algo que parece una página publicada con un botón "siguiente" # if driver.find_element_by_id('identifierNext'): # driver.find_element_by_id('identifierNext').click() questionnaire = {} # Obtengo todas las cartas con preguntas y respuestas dentro de ellas containers = driver.find_elements_by_class_name( "freebirdFormeditorViewItemContentWrapper" ) driver.set_page_load_timeout(30) driver.set_script_timeout(30) # para cada carta for container in containers: try: question = container.find_element_by_css_selector(".exportTextarea[aria-label='Intitulé de la question']") except NoSuchElementException: print("NoSuchElementException in " + str(container)) continue # Obtener las respuestas responses = container.find_elements_by_css_selector( ".quantumWizTextinputSimpleinputInput.exportInput" ) extracted_responses = [response.get_attribute("data-initial-value") for response in responses] questionnaire[question.text] = extracted_responses driver.set_page_load_timeout(30) driver.set_script_timeout(30) print("questionnaire backend: ", questionnaire) return questionnaire def extract(driver, df, survey): count_questionnaires = 0 result = [] count_not_empty = 0.0 print("survey: ", survey) for location, task in zip(df.Link, df.Task): if task == survey: print("location: ", location) questionnaire = {} if "docs.google.com" in str(location): count_questionnaires +=1.0 driver.get(location) # test if it is a published version try: ask_access_btn = driver.find_elements_by_class_name( "freebirdFormviewerViewNavigationHeaderButtonContent" ) except exceptions.UnexpectedAlertPresentException: print("UnexpectedAlertPresentException") get_published_questionnaire if ask_access_btn: questionnaire = get_published_questionnaire() else: questionnaire = get_backend_questionnaire() if questionnaire not in [{}, {'': ''}]: count_not_empty += 1.0 print(questionnaire) result.append({str(count_questionnaires): questionnaire}) count_questionnaires += 1 print("count_questionnaires: ", count_questionnaires) if count_questionnaires != 0: print("count_not_empty/count_questionnaires: ", count_not_empty/count_questionnaires) return result if __name__ == '__main__': """ Necesita acceder a la cuenta de Google para acceder a ciertos cuestionarios. También configurar chromedriver para que se ejecute en estado sin cabeza """ driver = setup_chromedriver() published_questionnaires = [] # tracking published ones """ Cargar CSV de los formularios de Google """ df, columns = load_data() surveys = ['Hotel ABC', "Airline XYZ", "The Ministry of Tourism of France"] for survey in surveys: result = extract(driver, df, survey) survey = survey.replace(" ", "_") pickle_out = open("applicant" + survey + "_c.p", "wb") pickle.dump(result, pickle_out) pickle_out.close() print("published_questionnaires: ", published_questionnaires) El output esta: C:\Users\antoi\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python3.7.exe C:/Users/antoi/Documents/Programming/Scraping/Python/new_scraper.py survey: Hotel ABC location: https://docs.google.com/forms/d/1MYSjxAMCXMXB02GPXqkewVx35_ptzv0XO7GRZQGwyGE/edit?usp=sharing backend questionnaire NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="7271d9f5-478a-4dfc-8f9e-6d9248da5bcc")> ... NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="3ff21bcf-6e8d-4b88-b64d-10cf5a11b18f")> NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="16e5fa0b-3502-4ab7-a7f7-a47b1b8fa388")> NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="fdc1bafb-51c9-4cc7-8f43-675ea9061338")> questionnaire backend: {'How do you feel about your next vacation after COVID-19?': [], 'When do you think your next vacation can start?': ['In next 3 months', 'In next 6 months', 'In next 1 year', 'Only once COVID-19 is under control', 'Only once COVID-19 vaccine is developed', ''], ... , 'Education Level': ['No higher education', 'Diploma', "Bachelor's", "Master's", 'PhD', "Other's", ''], 'Annual Income': ['< £ 30,000', '£ 30,000 to £ 50,000', '£ 50,000 to £ 80,000', '£ 80,000 to £ 120,000', '> £ 120,000', ''], 'Feedback / Comments': [''], 'Email (Optional)': ['']} location: https://docs.google.com/forms/d/1_iRBtfJANF5MGWqoIMQUxBdeuAa4ePMltdIsVRmdY5Y/edit?usp=sharing published questionnaire len_containers: 9 No question, NoSuchElementException len_containers: 8 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response len_containers: 7 No question, StaleElementReferenceException len_containers: 6 No question, StaleElementReferenceException len_containers: 5 No question, StaleElementReferenceException len_containers: 4 No question, StaleElementReferenceException len_containers: 3 No question, StaleElementReferenceException len_containers: 2 No question, StaleElementReferenceException len_containers: 1 No question, StaleElementReferenceException len_containers: 0 No question, StaleElementReferenceException questionnaire: {'Age': ['Under 18', '18-24', '25-34', '35-44', '45-54', 'Over 55']} {'Age': ['Under 18', '18-24', '25-34', '35-44', '45-54', 'Over 55']} location: https://docs.google.com/forms/d/1j0nk_Oo-_pfJBM4UcWITDPXT97-qX5mZpb3uVyKS3CA/edit?usp=sharing published questionnaire len_containers: 13 No question, NoSuchElementException len_containers: 12 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response len_containers : ... ... content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response len_containers: 0 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response questionnaire: {'On average, how many times per year do you travel for 2 days or more? *': ['1 to 2', '3 to 4', '5 or more'], ... , 'What are your expectations from the accommodations’ Pinterest page? (Please select all that apply) *': ['See pictures of customer service staff', 'See pictures of staff in general', 'See pictures of the destination', 'See pictures of all the types of rooms', 'See pictures of services available', 'See pictures that are not on the hotel’s website', "I don't have a Pinterest account", "I don't use Pinterest for these purposes"]} location: https://docs.google.com/forms/d/1kq5dhHvftF6tWmRk_7cG4H0Mkzr6xnUhpTV0j2xIYeE/edit?usp=sharing UnexpectedAlertPresentException published questionnaire len_containers: 13 No question, NoSuchElementException len_containers: 12 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response len_containers: 11 ... len_containers: 1 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response len_containers: 0 content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response content_area.get_attribute("aria-label"): Other response questionnaire: {'On average, how many times per year do you travel for 2 days or more? *': ['1 to 2', '3 to 4', '5 or more'], ... 'What are your expectations from the accommodations’ Pinterest page? (Please select all that apply) *': ['See pictures of customer service staff', 'See pictures of staff in general', 'See pictures of the destination', 'See pictures of all the types of rooms', 'See pictures of services available', 'See pictures that are not on the hotel’s website', "I don't have a Pinterest account", "I don't use Pinterest for these purposes"]} location: https://docs.google.com/forms/d/1IFqdsm9yO8h17JsJPN4c84vpQP06PxIquWfTmRN-TVw/edit?usp=sharing Así que parece que estamos perdidos en un bucle... [1]: https://docs.google.com/spreadsheets/d/1kv9T1bSVhx2Gloy04Xd3AbECYlEhLJ4DiqTGREf1XlQ/edit?usp=sharing