tensorflow Found more than one graph event per run

Question

I am loading a tensorboard for my ml engine experiment that is running in local mode and got the following warning:

"Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W0825 19:26:12.435613 Reloader event_accumulator.py:311] Found more than one metagraph event per run. Overwriting the metagraph with the newest event."

Originally, I suspected that this was because I had not cleared my --logdir=$OUTPUT_PATH (as other posts suggested -- however, even if I performed rm -rf $OUTPUT_PATH/* I am still getting this error for a local train. Could this error be indicative of a larger issue in my graph?

Hallowmas

Answer 1

It looks like you may have already come across this post, but without more information, it's the best information I can provide:

This is a known issue, TensorBoard doesn't like it when you write multiple event files from separate runs in the same directory. It will be fixed if you use a new subdirectory for every run (new hyperparameters = new subdirectory).

You may be inadvertently writing multiple event files in the same directory (e.g. training and eval?).

Also, be sure you are returning an appropriate tf.estimator.EstimatorSpec when in modes.EVAL. From the census sample:

if mode == Modes.EVAL:
  labels_one_hot = tf.one_hot(
      label_indices_vector,
      depth=label_values.shape[0],
      on_value=True,
      off_value=False,
      dtype=tf.bool
  )
  eval_metric_ops = {
      'accuracy': tf.metrics.accuracy(label_indices, predicted_indices),
      'auroc': tf.metrics.auc(labels_one_hot, probabilities)
  }
  return tf.estimator.EstimatorSpec(
      mode, loss=loss, eval_metric_ops=eval_metric_ops)

Carnage

I think that is certainly possible. I am writing scalars in both Modes.EVAL and Modes.TRAIN and the scalars in Modes.EVAL are never showing up in tensorboard. I guess I thought that there were always two event files (eval/event and train/event) but I'm now thinking that it's not quite as simple. Is there a way to write scalars for both train/eval within the same model_fn so that they don't generate multiple event files in the same directory? – Hallowmas

What are you setting as your model_dir (e.g. run_config=tf.contrib.learn.RunConfig(model_dir=args.job_dir))? Can you provide a directory listing of that model_dir? – Carnage

Updated the post with info about EstimatorSpec. Does that help? – Carnage

It's evident that I must be writing multiple event files, but I'm not exactly sure how this is happening. I am deleting the args.job_dir every time with 'rm -rf $JOB_DIR' and then I am still ending up with this error once I start a new run. Here is the directory listing of that model_dir.. $ ls output checkpoint events.out.tfevents.1504204297.Users-MacBook-Pro.local graph.pbtxt model.ckpt-1.data-00000-of-00001 model.ckpt-1.index model.ckpt-1.meta and as far as I can tell there is only the single events file, but the tensorboard warning is still yelling at me? – Hallowmas

There is however now another subdirectory eval which does contain another events file, but the warning was showing up before that was even the case.. – Hallowmas

Now that the eval is there, are things working correctly? If you don't explicitly set model_dir, I wonder if the summaries are going to some default dir which you aren't cleaning out all the time. – Carnage

I did specify run_config as you described tf.contrib.learn.RunConfig(model_dir=args.job_dir -- Now that eval is there I got some additional warnings: WARNING:tensorflow:Deleting accumulator 'eval' WARNING:tensorflow:Deleting accumulator '.' WARNING:tensorflow:Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event. WARNING:tensorflow:Found more than one metagraph event per run. Overwriting the metagraph with the newest event. – Hallowmas