2

quiero entrenar mi propio modelo de reconocimiento de objetos para lo cual estoy intentando usar el script model_main.py que se encuntra en la carpeta del api /models/research/object_detection pero al ejecutarlo de la siguiente manera:

python3 model_main.py --logtostderr --model_dir=/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco_2018_03_29/ --pipeline_config_path=/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco.config

me genera lo siguiente y no empieza el entrenamiento:

WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0111 23:57:18.099197 139762259621696 model_lib.py:793] Forced number of epochs for all eval validations to be 1.
INFO:tensorflow:Maybe overwriting train_steps: None
I0111 23:57:18.099363 139762259621696 config_util.py:552] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0111 23:57:18.099413 139762259621696 config_util.py:552] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0111 23:57:18.099469 139762259621696 config_util.py:552] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0111 23:57:18.099525 139762259621696 config_util.py:552] Maybe overwriting eval_num_epochs: 1
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
W0111 23:57:18.099602 139762259621696 model_lib.py:809] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu None
I0111 23:57:18.099665 139762259621696 model_lib.py:846] create_estimator_and_inputs: use_tpu False, export_to_tpu None
INFO:tensorflow:Using config: {'_model_dir': '/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco_2018_03_29/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f1cb2d19310>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0111 23:57:18.100012 139762259621696 estimator.py:209] Using config: {'_model_dir': '/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco_2018_03_29/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
  rewrite_options {
    meta_optimizer_iterations: ONE
  }
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f1cb2d19310>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f1cb2d09440>) includes params argument, but params are not passed to Estimator.
W0111 23:57:18.100254 139762259621696 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f1cb2d09440>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0111 23:57:18.100585 139762259621696 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0111 23:57:18.100709 139762259621696 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0111 23:57:18.100878 139762259621696 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
INFO:tensorflow:Skipping training since max_steps has already saved.
I0111 23:57:18.103509 139762259621696 estimator.py:360] Skipping training since max_steps has already saved.

Tambien intente ejecutarlo con el scrip train.py que se encuentra en el directorio /models/research/object_detection/legacy, lo ejecute de la siguiente manera python3 train.py --logtostderr --train_dir=/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco_2018_03_29/ --pipeline_config_path=/home/alexander/Documentos/proyecto/Training/ssd_mobilenet_v2_coco.config no obstante de esta forma me aparece lo siguiente:

File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "train.py", line 182, in main
    graph_hook_fn=graph_rewriter_fn)
  File "/home/alexander/Documentos/proyecto/tf/models/research/object_detection/legacy/trainer.py", line 376, in train
    keep_checkpoint_every_n_hours=keep_checkpoint_every_n_hours)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 825, in __init__
    self.build()
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 837, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 875, in _build
    build_restore=build_restore)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
    restore_sequentially, reshape)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 328, in _AddRestoreOps
    restore_sequentially)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/training/saver.py", line 575, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1696, in restore_v2
    name=name)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
    op_def=op_def)
  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
  File "train.py", line 186, in <module>
    tf.app.run()  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/absl/app.py", line 325, in run
    raise  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)  File "train.py", line 182, in main
    graph_hook_fn=graph_rewriter_fn)  File "/home/alexander/Documentos/proyecto/tf/models/research/object_detection/legacy/trainer.py", line 415, in train
    saver=saver)  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tf_slim/learning.py", line 788, in train
    should_retry = True  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/tf_should_use.py", line 193, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
==================================
E0111 23:47:56.769966 139791945934656 tf_should_use.py:71] ==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'init_ops/report_uninitialized_variables/boolean_mask/GatherV2:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
  File "train.py", line 186, in <module>
    tf.app.run()  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/absl/app.py", line 325, in run
    raise  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)  File "train.py", line 182, in main
    graph_hook_fn=graph_rewriter_fn)  File "/home/alexander/Documentos/proyecto/tf/models/research/object_detection/legacy/trainer.py", line 415, in train
    saver=saver)  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tf_slim/learning.py", line 788, in train
    should_retry = True  File "/home/alexander/anaconda3/envs/TF/lib/python3.7/site-packages/tensorflow/python/util/tf_should_use.py", line 193, in wrapped
    return _add_should_use_warning(fn(*args, **kwargs))
==================================

La version de TF que estoy usando es la 1.14.0, si alguien sabe que puede estar pasando le agradeceria si me puede ayudar, gracias.

1 respuesta 1

0

Vale, voy por partes.

Primera traza

Lo primero no es una traza de error, es información sobre el procesado que te lanza el modelo sobre distintas cosas, configuración de hiperparámetros, que no estás utilizando el entrenamiento distribuido, etc.

También tienes algún warnings (que no errores) que el mismo resuelve y te dice como lo ha resuelto. Por ejemplo en el archivo de configuración encuentra que el número de epocas es 0 y lo cambia a uno.

Por último todo finaliza diciendo:

Skipping training since max_steps has already saved.

Basicamente te dice que tu le has puesto al modelo un número de pasos, y que para ese número de pasos ya lo tienes guardado. Es decir ya has entrenado un modelo previamente, por lo que no tiene sentido volver a entrenarlo.

Tienes dos opciones para solucionar esto:

  1. Borrar el modelo que ya has entrenado para empezar de cero. Para ello ves donde has definido el directorio de tu modelo, model_dir y borra lo que hay dentro de la carpeta checkpoint. Ojo, que estás borrando el modelo entrenado
  2. Definir un directorio distinto model_dir en el que se hará el entrenamiento.

Segunda traza.

Aunque no pone nada de que indique que es un Error por experiencia parece que si lo es (por favor si has cortado parte de la traza agregalá al completo).

Por lo que sea Tensorflow parece que no es capaz de iniciar el grafo de computación y por tanto no puede llevar a cabo el entrenamiento.

Puede ser por diversos problemas:

  • Un bug en la versión de Tensorflow, estás en el versión 1.14 y la actual es la 2.4, esta versión tiene cerca de dos años

  • Problema con librerías o dependencias

  • El mismo problema que en la traza anterior, que ya lo tienes entrenado.

1
  • Hola Alberto gracias por comentar, actualice la segunda traza con un poco mas de contenido, gracias Commented el 13 ene. 2021 a las 0:05

Tu Respuesta

By clicking “Publica tu respuesta”, you agree to our terms of service and acknowledge you have read our privacy policy.

¿No es la respuesta que buscas? Examina otras preguntas con la etiqueta o formula tu propia pregunta.