How to update a previous run into MLFlow?

Question

I would like to update previous runs done with MLFlow, ie. changing/updating a parameter value to accommodate a change in the implementation. Typical uses cases:

• Log runs using a parameter A, and much later, log parameters A and B. It would be useful to update the value of parameter B of previous runs using its default value.
• "Specialize" a parameter. Implement a model using a boolean flag as a parameter. Update the implementation to take a string instead. Now we need to update the values of the parameter for the previous runs so that it stays consistent with the new behavior.
• Correct a wrong parameter value loggued in the previous runs.

It is not always easy to trash the whole experiment as I need to keep the previous runs for statistical purpose. I would like also not to generate new experiments just for a single new parameter, to keep a single database of runs.

What is the best way to do this?

Haemoid

Did you have error INVALID_PARAMETER_VALUE when trying to update an existing parameter value? – Meteor

@panc I don't remember exactly, but it's probable – Haemoid

Answer 1

To add or correct a parameter, metric or artifact of an existing run, pass run_id instead of experiment_id to mlflow.start_run function

with mlflow.start_run(run_id="your_run_id") as run:
    mlflow.log_param("p1","your_corrected_value")
    mlflow.log_metric("m1",42.0) # your corrected metrics
    mlflow.log_artifact("data_sample.html") # your corrected artifact file

You can correct, add to, or delete any MLflow run any time after it is complete. Get the run_id either from the UI or by using mlflow.search_runs.

Source: https://towardsdatascience.com/5-tips-for-mlflow-experiment-tracking-c70ae117b03f

Improvident

This doesn't seem to work for me on MLflow 1.13.1. When I tried to correct an existing parameter, it issues the error RestException: INVALID_PARAMETER_VALUE: Changing param values is not allowed. Param with key='test_rmse' was already logged with value='3.061878' for run ID='fe03293adb7e4c79a716c11fc938c044'. Attempted logging new value '0'. – Meteor

@panc same issue here. Did you find a solution ? – One

@cyberjoac Not yet. I started a thread on Github github.com/mlflow/mlflow/issues/3999, but no reply yet. – Meteor

Answer 2

MLflow API does not support updating an existing parameter value, see this.

However, there are backdoors you can use to achieve the goal of rewriting an existing parameter's value. But use with caution.

Meteor

Answer 3

Well you can't update it but there is another solution which is getting the run, edit the value that you want, and then log the run as a whole new run.

import mlflow 
import os 
os.environ["MLFLOW_EXPERIMENT_NAME"] = "REPLACE_YOUR_EXPERIEMNTE_NAME"

original_run_id = 'REPLACE_YOUR_RUN_ID'
original_run = mlflow.get_run(original_run_id)
original_run_name = original_run.data.tags.get('mlflow.runName')

with mlflow.start_run(run_name=original_run_name) as new_run:
    for key, value in original_run.data.params.items():
        if key == 'REPLACE_WITH_YOUR_KEY':
            new_value = REPLACE_WITH_YOUR_VALUE
            mlflow.log_param(key, new_value)
        else:
            mlflow.log_param(key, value)

    for key, value in original_run.data.metrics.items():
        mlflow.log_metric(key, value)

setting the epxeriemnt name and run name are optional, the run_id should grab the exact run that you want. But setting the run name will help you create a name the same as before. This snippet currently updates a param, you can change it to update a metric.

Macgregor