Tengo un bucket S3 con
- json de perfumes de una marca o
- carpetas de una marca con perfumes en formato json.
Sé cómo obtener su índice, pero me gustaría insertar estos objetos en mi base de datos documentdb, en colecciones correspondientes a su marca.
¿Saben cómo puedo hacer eso?
Hasta hoy intenté:
import boto3
import pymongo
import sys
def iterate_bucket_items(bucket):
"""
Generator that iterates over all objects in a given s3 bucket
See http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2
for return data format
:param bucket: name of s3 bucket
:return: dict of metadata for an object
"""
client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')
page_iterator = paginator.paginate(Bucket=bucket)
for page in page_iterator:
if page['KeyCount'] > 0:
for item in page['Contents']:
yield item
##Create a MongoDB client, open a connection to Amazon DocumentDB as a replica set and specify the read preference as secondary preferred
client = pymongo.MongoClient('mongodb://user:[email protected]:27017/?ssl=true&ssl_ca_certs=rds-combined-ca-bundle.pem&replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false')
##Specify the database to be used
db = client.perfumes
c = 0
for i in iterate_bucket_items(bucket='datahubpredicity'):
pathname = i['Key']
if pathname.contains('/'):
collection = pathname.split[0]
# col = db[collection]
# col.insert_one(i)
c+=1
if c >= 10:
break
# ##Specify the collection to be used
# col = db.sample_collection
# ##Insert a single document
# col.insert_one(i)
# ##Print the result to the screen
# print(x)
##Close the connection
client.close()
Pero devuelve:
pymongo.errors.ServerSelectionTimeoutError:
datahub.cluster-1.eu-west-3.docdb.amazonaws.com:27017:
timed out, Timeout: 30s,
Topology Description: <TopologyDescription id: 6254472217824b192df5665d,
topology_type: ReplicaSetNoPrimary,
servers: [<ServerDescription ('datahub.cluster-1.eu-west-3.docdb.amazonaws.com', 27017) server_type: Unknown,
rtt: None, error=NetworkTimeout('datahub.cluster-1.eu-west-3.docdb.amazonaws.com:27017: timed out')>]>
mongodb://user:paswword ...
debería ser:mongodb+srv://user:paswword ...
. Prueba eso y comenta el resultado.replicaSet=rs0
, ¿tu Replica Set se llama así?