El problema se divide en dos partes:
Obtener los enlaces de los formularios la hoja de cálculo de Google Docs
La forma más simple y la que Google sin duda prefiere es usar la propia API que ofrece Google, lo podemos hacer con gspread por ejemplo.
Dicho ésto, dado que la pregunta se enfoca en el scraping y Selenium, voy a intentar dar una solución en base a ellos, aunque con algo de trampa.
Digo "trampa" porque en éste caso en vez de plantear un scraping de la propia página de Google Docs en si para obtener los enlaces, se me ocurre lo siguiente:
- Usamos Selenium para abrir la el documento en Google Docs.
- Hacemos click en Archivo -> Descargar -> Valores separado por comas
et voilà!, tenemos un csv con el contenido de la hoja muy sencillo de parsear con Python par obtener los links. Obviamente perdemos cosas como los estilos y demás, pero en éste caso esos es irrelevante.
Lo que sigue es obtener los links del csv y ponernos a procesar cada enlace.
Obtener los datos de cada formulario
En cuanto a cómo obtener las preguntas y respuestas, previamente había respondido a ello en ésta pregunta:
Simplemente deberemos iterar sobre los enlaces a los formularios y proceder como se explica en la respuesta.
No obstante, hay algunos cambios en la forma de hacer el scraping en los formularios, con respecto a la solución anterior:
En éste caso tenemos algunos formularios en las que hay que seleccionar una respuesta en todos los casos o escribir una en un textarea
(campos obligatorios). Eso no se contemplaba en la respuesta anterior, por lo que hay que solventarlo haciendo click aleatorio en los radiobutons
y escribiendo texto en los textarea
oportunos.
He modificado la forma de detectar cuando un formulario se ha terminado, antes simplemente se enviaba, pero dentro de lo cuestionable éticamente del scraping (aunque Google viva de ello XD), enviar un formulario rellenado por una máquina aleatoriamente me parece menos ético aún.
He optado por detectar la presencia de un botón con texto "Enviar" para saber cuándo estamos ante la última página del formulario, el problema es que si nuestro idioma no es el Español, el texto del botón cambiará, por lo que hay que modificar esa parte del código de forma acorde, en el código final mostrado abajo:
Linea 136
.
if btn.text == "Enviar":
Todo junto
import csv
import glob
import pathlib
import random
import shutil
import sys
import time
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common import keys
location = "https://docs.google.com/spreadsheets/d/1iLqEFRaHPYxpJKU05VXt3HUCQ2OQUAg8FfWlyFbvaXc/edit?usp=sharing"
# Creamos directorio "temporal" en directorio padre del script
file_dir = pathlib.Path(__file__).absolute().parent / "temp_files"
shutil.rmtree(file_dir.as_posix(), ignore_errors=True)
file_dir.mkdir(parents=True, exist_ok=True)
# Permitimos que el navegador descaargue archivos csv sin preguntar
options = webdriver.FirefoxProfile()
options.set_preference(
"browser.helperApps.neverAsk.saveToDisk",
"text/csv"
)
options.set_preference("browser.download.folderList", 2)
options.set_preference("browser.helperApps.alwaysAsk.force", False)
options.set_preference("browser.download.manager.showWhenStarting", False)
options.set_preference("browser.download.dir", file_dir.as_posix())
driver = webdriver.Firefox(options)
driver.get(location)
menu_archivo = driver.find_element_by_id("docs-file-menu")
menu_archivo.click()
submenu_descargar = driver.find_element_by_css_selector(
"#\:2z > div:nth-child(1) > span:nth-child(1)"
)
action = ActionChains(driver)
action.move_to_element(submenu_descargar)
action.perform()
action.send_keys(keys.Keys.DOWN)
action.perform()
submenus = driver.find_elements_by_class_name('goog-menuitem')
for submenu in submenus:
try:
label = submenu.find_element_by_class_name("goog-menuitem-label")
aria = label.get_attribute("aria-label")
if ".csv" in aria:
action = ActionChains(driver)
action.move_to_element(submenu)
action.perform()
action.send_keys(keys.Keys.ENTER)
action.perform()
break
except Exception:
pass
else:
print("Descarga de csv no disponible")
shutil.rmtree(file_dir.as_posix(), ignore_errors=True)
sys.exit(1)
for _ in range(60):
files = glob.glob((file_dir / "*.csv").as_posix())
if files:
csv_path = files[0]
break
time.sleep(1)
with open(csv_path) as file:
reader = csv.reader(file)
form_links = [
col for row in reader for col in row
if col.startswith("https://docs.google.com/forms")
]
if not form_links:
print("No se encontraron formularios")
shutil.rmtree(file_dir.as_posix(), ignore_errors=True)
sys.exit(2)
data = {}
for form_link in form_links:
driver.get(form_link)
title = driver.find_element_by_class_name(
"freebirdFormviewerViewHeaderTitleRow"
).text
data[title] = {}
fin = False
while not fin:
containers = driver.find_elements_by_class_name(
"freebirdFormviewerViewNumberedItemContainer"
)
btns = driver.find_elements_by_css_selector(".appsMaterialWizButtonEl")
for container in containers:
try:
question = container.find_element_by_class_name(
"freebirdFormviewerViewItemsItemItemTitle"
)
except NoSuchElementException:
continue
try:
radiobtn_cont = container.find_elements_by_class_name(
"freebirdFormviewerViewItemsRadioOptionContainer"
)
except NoSuchElementException:
pass
else:
responses = container.find_elements_by_class_name(
"docssharedWizToggleLabeledLabelText"
)
data[title][question.text] = [
response.text for response in responses
]
rdbtns = container.find_elements_by_class_name(
"appsMaterialWizToggleRadiogroupRadioButtonContainer"
)
if rdbtns:
random.choice(rdbtns).click()
continue
try:
text = container.find_element_by_xpath("//textarea")
except NoSuchElementException:
pass
else:
text.send_keys("No sabría decir")
for btn in btns:
if btn.text == "Enviar":
fin = True
break
else:
btns[-1].click()
driver.quit()
shutil.rmtree(file_dir.as_posix(), ignore_errors=True)
print(data)
Resultado
{
"Air Travel": {
"How often did you fly before the Covid-19 epidemic?": [
"Once per year",
"Between two and five times per year",
"More than five times per year"
],
"What did you dislike the most about Pre-Covid 19 air travel?": [
"Long waits at security",
"Layovers",
"Entertainment on flight",
"Cramped seating on the flight",
"Otro:"
],
"Did you feel safe flying before the Covid-19 pandemic?": [
"Yes",
"No"
],
"How long will you wait before flying again, after Covid-19?": [
"30 Days",
"30 - 90 Days",
"90 - 180 Days",
"More than 180 Days"
],
"Will you fly for pleasure or only when absolutely necessary?": [
"Pleasure",
"Only When Absolutely Necessary"
],
"Would a flight with a guaranteed empty seat between you and the person sitting next to you make you more comfortable?": [
"Yes",
"No"
],
"What can we do to make you feel as safe as possible while flying with us?": [],
"Will you be more interested in the cheapest flight or the airline that offers the highest level of protection from disease?": [
"Cheapest Flight",
"Level of Protection From Disease",
"Mix of Both"
],
"Would you take advantage of a new class of seating that offered improved social distancing during the flight as well as complimentary personal protection equipment?": [
"Yes",
"No"
],
"Which of these options would you like to see the most of your next flight?": [
"The latest movies",
"Free hand sanitizer and wipes",
"Free meal for longer flight or free drinks on shorter ones",
"Scheduled check-in time so you can avoid waiting in a large crowd",
"Otro:"
]
},
"Travel willingness": {
"Would you be comfortable travelling within country by airplane? *": [
"Very uncomfortable",
"Moderate uncomfortable",
"Slightly uncomfortable",
"Neutral",
"Slightly comfortable",
"Moderate comfortable",
"Very comfortable"
],
"Would you be comfortable travelling internationally by airplane? *": [
"Very uncomfortable",
"Moderate uncomfortable",
"Slightly uncomfortable",
"Neutral",
"Slightly comfortable",
"Moderate comfortable",
"Very comfortable"
],
"Would you be comfortable travelling due to business by airplane? *": [
"Very uncomfortable",
"Moderate uncomfortable",
"Slightly uncomfortable",
"Neutral",
"Slightly comfortable",
"Moderate comfortable",
"Very comfortable"
],
"Would you be comfortable travelling for leisure by airplane? *": [
"Very uncomfortable",
"Moderate uncomfortable",
"Slightly uncomfortable",
"Neutral",
"Slightly comfortable",
"Moderate comfortable",
"Very comfortable"
],
"How do you expect the flight ticket price to be changed compared to pre Covid-19 times? *": [
"Much expensive than before",
"More expensive than before",
"Slightly more expensive than before",
"Same as before",
"Slightly cheaper than before",
"Moderate cheaper than before",
"Much cheaper than before"
],
"If wearing mask is required on airplane, would it make you want to travel by airplane less? *": [
"Yes",
"No",
"Maybe"
],
"Would you expect your body temperature to be taken at the airport? *": [
"Yes",
"No",
"Maybe"
],
"Are you ok with body temperature being taken a couple of times during the flight? *": [
"Yes",
"No",
"Maybe"
],
"Would you expect the flight attendants to hand out hand sanitizers on the flight? *": [
"Yes",
"No",
"Maybe"
],
"What are somethings that airline companies can do to make you feel safer and more comfortable to travel? *": []
},
"Untitled form": {
"Have you traveled with Airline XYZ pre-pandemic?": [
"Yes",
"No"
],
"If you have traveled with Airline XYZ pre-pandemic, did you choose low-cost or exclusive flight options?": [
"low-cost",
"exclusive",
"neither"
],
"If you chose low-cost or exclusive flight options, what were your reasons for doing so?": [],
"Have you traveled with Airline XYZ post-pandemic?": [
"Yes",
"No"
],
"If you have traveled with Airline XYZ post-pandemic, are you more or less likely to choose a low-cost option?": [
"much less likely",
"less likely",
"no change",
"more likely",
"much more likely"
],
"If you have traveled with Airline XYZ post-pandemic, are you more or less likely to choose an exclusive option?": [
"much less likely",
"less likely",
"no change",
"more likely",
"much more likely"
],
"\"I have lost a primary source of income due to COVID-19.\" How true is this for you?": [
"Not true at all",
"Mostly untrue",
"Somewhat true",
"Mostly true",
"True"
],
"\"I will be reducing travel to protect myself from COVID-19.\" How true is this for you?": [
"Not true at all",
"Mostly untrue",
"Somewhat true",
"Mostly true",
"True"
],
"What role do you believe private business should play regarding the COVID-19 pandemic?": [
"No role at all",
"Some role",
"Large role",
"Unsure"
],
"To what extent do you believe that Airline XYZ is catering to it's customers during this crisis?": [
"No extent at all",
"Mostly no extent",
"Some extent",
"A great extent",
"Unsure"
],
"What offerings, if any, would you be interested in during this crisis?": [
"more low-cost flights",
"more exclusive flights",
"discount on exclusive flights",
"discount on low-cost flights",
"Otro:"
],
"What channels would you be interested in hearing future updates about Airline XYZ from?": [
"Facebook",
"Twitter",
"Email",
"Text",
"Instagram",
"Snapchat",
"Tumblr",
"Youtube",
"TikTok"
]
}
}
El código es seguro optimizable en varios puntos, uno de ellos es la optención del submenú del csv, otro la mencionada forma de detectar el fin del formulario. Así mismo, solo está testeado con la hoja de ésta pregunta y sus formularios, seguramente necesitaría más testeo para detectar posibles problemas no tenidos en cuenta.