How to get a Hydra config without using @hydra.main()

Question

Let's say we have following setup (copied & shortened from the Hydra docs):

Configuration file: config.yaml

db:
  driver: mysql
  user: omry
  pass: secret

Python file: my_app.py

import hydra
@hydra.main(config_path="config.yaml")
def my_app(cfg):
    print(cfg.pretty())
if __name__ == "__main__":
    my_app()

This works well when we can use a decorator on the function my_app. Now I would like (for small scripts and testing purposes, but that is not important) to get this cfg object outside of any function, just in a plain python script. From what I understand how decorators work, it should be possible to call

import hydra
cfg = hydra.main(config_path="config.yaml")(lambda x: x)()
print(cfg.pretty())

but then cfg is just None and not the desired configuration object. So it seems that the decorator does not pass on the returned values. Is there another way to get to that cfg ?

Paquette

Answer 1

Use the Compose API:

from hydra import compose, initialize
from omegaconf import OmegaConf

with initialize(version_base=None, config_path="conf", job_name="test_app"):
    cfg = compose(config_name="config", overrides=["db=mysql", "db.user=me"])

print(OmegaConf.to_yaml(cfg))

This will only compose the config and will not have side effects like changing the working directory or configuring the Python logging system.

Sudanic

@OmryYadan Is this possible to update the solution. I find signatures have changed. And is it possible to construct a cfg without refer to some "config.yml"? – Maceio

Updated, check the API link for more info. (Yes, you can omit the config name). – Sudanic

Hi, @OmryYadan thanks for your response! I'm trying to use this approach and am getting this error. Was there a change in the way to import compose? hydra-core==1.1.0.dev5 ImportError: cannot import name 'compose' from 'hydra' – Bitternut

Figured it out. We need to use hydra.experimental If this is correct perhaps we can update the solution above! – Bitternut

Answer 2

None of the above solutions worked for me. They gave errors:

'builtin_function_or_method' object has no attribute 'code'

and

GlobalHydra is already initialized, call Globalhydra.instance().clear() if you want to re-initialize

I dug further into hydra and realised I could just use OmegaConf to load the file directly. You don't get overrides but I'm not fussed about this.

import omegaconf
cfg = omegaconf.OmegaConf.load(path)

Urbanus

This answer works for regular config files but lets say if you have different structures and paths/dirs/configs for stage/production/development configuration then you have to have these in one file which is tedious to maintain Hydra resolves this. Else if you have only one config then go for this better and faster solution. – Dripdry

Answer 3

I found a rather ugly answer but it works - if anyone finds a more elegant solution please let us know!

We can use a closure or some mutable object. In this example we define a list outside and append the config object:

For hydra >= 1.0.0 you have to use config_name instead, see documentation.

import hydra
c = []
hydra.main(config_name="config.yaml")(lambda x:c.append(x))()
cfg = c[0]
print(cfg)

For older versions:

import hydra
c = []
hydra.main(config_path="config.yaml")(c.append)()
cfg = c[0]
print(cfg.pretty())

Paquette

I haven't seen this one before, creative :) – Sudanic

@hahnec You have to use config_name now instead of config_path`, check out the change log! – Paquette

Answer 4

The currently accepted answer, using the compose API, seems to be the intended way to address this use case.

However, it seems that using it disables overrides from the command line, and all of hydra's own flags

So here is a hacky alternative I've come up with, which still lets hydra parse all command line flags:

import hydra
from omegaconf import OmegaConf
def my_app(cfg):
    print(cfg)
if __name__ == "__main__":
    cfg_holder=[]
    @hydra.main(config_name="config.yaml")
    def parse(cfg):
        OmegaConf.resolve(cfg) # If you need resolving, it needs to be done here
        cfg_holder.append(cfg)
    parse()
    my_app(cfg_holder[0])

You may fairly point out that I'm still using @hydra.main(). But note that it no longer wraps around the actual app. In my use case, this was sufficient to counteract the undesirable effect I was observing from @hydra.main()

(note that I had to make some extra small changes to the original example to make my code work in hydra 1.1.2)

Answer

Answer 5

anther ugly answer, but author said this may be crush in next version

Blockquote

from omegaconf import DictConfig
from hydra.utils import instantiate
from hydra._internal.utils import _strict_mode_strategy, split_config_path, create_automatic_config_search_path
from hydra._internal.hydra import Hydra
from hydra.utils import get_class 

class SomeThing:
...
def load_from_yaml(self, config_path, strict=True):
    config_dir, config_file = split_config_path(config_path)
    strict = _strict_mode_strategy(strict, config_file)
    search_path = create_automatic_config_search_path(
        config_file, None, config_dir
    )
    hydra = Hydra.create_main_hydra2(
        task_name='sdfs', config_search_path=search_path, strict=strict
    )
    config = hydra.compose_config(config_file, [])
    config.pop('hydra')
    self.config = config
    print(self.config.pretty())

Sunderland

Any time you use anything in an _internal package, know that there will be no attempts to maintain backward compatibility or document changes. Use the compose API, it's simpler to use to is maintained like an API. Although it's still experimental it's much more stable than internal things. – Sudanic

Answer 6

This is my solution

from omegaconf import OmegaConf

class MakeObj(object):
    """ dictionary to object. 
    Thanks to https://mcmap.net/q/20740/-how-to-convert-a-nested-python-dict-to-object

    Args:
        object ([type]): [description]
    """
    def __init__(self, d):
        for a, b in d.items():
            if isinstance(b, (list, tuple)):
                setattr(self, a, [MakeObj(x) if isinstance(x, dict) else x for x in b])
            else:
                setattr(self, a, MakeObj(b) if isinstance(b, dict) else b)


def read_yaml(path):
    x_dict =  OmegaConf.load(path)
    x_yamlstr = OmegaConf.to_yaml(x_dict)
    x_obj = MakeObj(x_dict)
    return x_yamlstr, x_dict, x_obj
    
x_yamlstr, x_dict, x_obj = read_yaml('config/train.yaml')
print(x_yamlstr)
print(x_dict)
print(x_obj)
print(dir(x_obj))
               

Cheston

You should check Hydra instantiate support. hydra.cc/docs/advanced/instantiate_objects/overview – Sudanic