Is there a way to see estimated time for training a model in Google AutoML Vision?

Question

I'm attempting to train a model to identify certain tags in images. I tried using the 1 hour free version and after an hour the training ended. The results weren't as accurate as I would like, so I took the plunge and selected the option which did not define a specific time limit for training the model.

At this time, it is showing "training vision classification model" and "Training can take 15 minutes to several hours or more, depending on the compute hours assigned. In the meantime, you can close this window. You will be emailed once training completes."

It's been running for over 24 hours now. Google charges $20 per training hour so I would like to know if there is a way to see how much longer it would take to fully train the model.

I'm using it using 2594 images with only single labels.

I understand Google AutoML Vision is still in beta phase but at this time, there is no way to know how long this training will last. I'm under a strict budget so I will stop the training if it exceeds the time -- but I would still like to know if it was near completion.

Vermiculate

Answer 1

The time required to train your model will completely depends and vary based on the amount of data included within your dataset and the accuracy desired for your model; therefore, I think it is not possible to determine the number of hours needed to complete the training phase. The GCP pricing documentation mentions the following suggestion:

Many customers find that one hour is sufficient to build an experimental model and use additional training hours to increase accuracy to a production level.

Note: Keep in mind that the accuracy of your model generally depends on how long you allow it to train and the quality of your training dataset.

On the other hand, in case you have an specific budget for your model, it is recommended to use the training budget parameter in order to specify the number of hours of training to use; In this way, AutoML Vision can guarantee that the actual training time will be less than or equal to the training budget. I suggest you to take a look on the AutoML Vision API Tutorial where you can find an example of the usage of the training model property.

Additionally, it is important to note that, when training a model, a compute hour represents internal compute usage, which means that this time might not exactly match an actual hour on the clock, as mentioned in the Training costs. Based on this, I recommend you to consider this aspect when calculating the your budget in order to avoid exceeding it.

Voiceful

"In this way, AutoML Vision can guarantee that the actual training time will be less than or equal to the training budget." - I think this is currently not the case. Yesterday I extended the training for 2 more custom hours and this is still running since 10 hours now. Also when requesting the processing operation via API, the field progressPercent is only visible under metadata when the training has completely finished. – Nixie

(even if you consider the statement "an actual hour on the clock" :-( But I think this will improve when it hits an non beta state) – Nixie