Manager / Container class, how to?

Question

I am currently designing a software which needs to manage a certain hardware setup.

The hardware setup is as following :

System - The system contains two identical devices, and has certain functionality relative to the entire system.

Device - Each device contains two identical sub devices, and has certain functionality relative to both sub devices.

Sub device - Each sub device has 4 configurable entities (Controlled via the same hardware command - thus I don't count them as a sub-sub device).

What I want to achieve :

I want to control all configurable entities via the system manager (the entities are counted in a serial way), meaning I would be able to do the following :

system_instance = system_manager_class(some_params)
system_instance.some_func(0) # configure device_manager[0].sub_device_manager[0].entity[0]
system_instance.some_func(5) # configure device_manager[0].sub_device_manager[1].entity[1]
system_instance.some_func(8) # configure device_manager[1].sub_device_manager[1].entity[0]

What I have thought of doing :

I was thinking of creating an abstract class, which contains all sub device functions (with a call to a conversion function) and have the system_manager, device_manager and sub_device_manager inherit it. Thus all classes will have the same function name and I will be able to access them via the system manager. Something around these lines :

class abs_sub_device():
    @staticmethod
    def convert_entity(self):
        sub_manager = None
        sub_entity_num = None
        pass

    def set_entity_to_2(entity_num):
        sub_manager, sub_manager_entity_num = self.convert_entity(entity_num)
        sub_manager.some_func(sub_manager_entity_num)


class system_manager(abs_sub_device):
    def __init__(self):
        self.device_manager_list = [] # Initiliaze device list
        self.device_manager_list.append(device_manager())
        self.device_manager_list.append(device_manager())

    def convert_entity(self, entity_num):
        relevant_device_manager = self.device_manager_list[entity_num // 4]
        relevant_entity         = entity_num % 4
        return relevant_device_manage, relevant_entity

class device_manager(abs_sub_device):
    def __init__(self):
        self.sub_device_manager_list = [] # Initiliaze sub device list
        self.sub_device_manager_list.append(sub_device_manager())
        self.sub_device_manager_list.append(sub_device_manager())        

    def convert_entity(self, entity_num):
        relevant_sub_device_manager = self.sub_device_manager_list[entity_num // 4]
        relevant_entity         = entity_num % 4
        return relevant_sub_device_manager, relevant_entity

class sub_device_manager(abs_sub_device):
    def __init__(self): 
        self.entity_list = [0] * 4

    def set_entity_to_2(self, entity_num):
        self.entity_list[entity_num] = 2

• The code is for generic understanding of my design, not for actual functionality.

The problem :

It seems to me that the system I am trying to design is really generic and that there must be a built-in python way to do this, or that my entire object oriented look at it is wrong.

I would really like to know if some one has a better way of doing this.

Musketry

why do you want to call some_func(5) instead of some_func({"device": 0, "sub_device": 1, "entity": 1})? if it's input from the hardware, why the entity cannot be accessed directly by index 5 and/or the system manager cannot decode how to access the entity without intermediate device and sub-device managers? – Quire

@Aprillion, I am not sure I fully understood your question so ill try my best to answer. The reason for me wanting to call the function as I mentioned is because its easier to read, iterate over and reduces a lot of overhead (why would I want to call the function with a full dict every time when I can simply send an integer?). Secondly as I stated, each layer has its own set of functions (For example : the system has a parameter which does not directly affect the entities, and the sub device has an IO controlling all 8 sub entities beneath it). – Musketry

is the magic number easier to read just for you or for all people working with this hardware? my reason for suggesting the dictionary was that "Explicit is better than implicit." and for suggesting to put all functions into 1 class "Flat is better than nested." (from the zen of python) so you might want to provide more details why you need the hierarchy as opposed to a flat list of methods – Quire

@Quire The magic number is easier for me and my team to read and to use (I can see why a dict would be better in some instances, but not on this one). And I did explicitly explain why I need nested over flat, each layer is an instance by it self regardless of its entities (Has its own parameters and functionality in addition to containing entities). Also there is a certain connectivity between entities (certain hardware commands will affect a group of entities and not one entity directly). – Musketry

Answer 1

After much thinking, I think I found a pretty generic way to solve the issue, using a combination of decorators, inheritance and dynamic function creation.

The main idea is as following :

1) Each layer dynamically creates all sub layer relevant functions for it self (Inside the init function, using a decorator on the init function)

2) Each function created dynamically converts the entity value according to a convert function (which is a static function of the abs_container_class), and calls the lowers layer function with the same name (see make_convert_function_method).

3) This basically causes all sub layer function to be implemented on the higher level with zero code duplication.

def get_relevant_class_method_list(class_instance):
    method_list = [func for func in dir(class_instance) if callable(getattr(class_instance, func)) and not func.startswith("__") and not func.startswith("_")]
    return method_list

def make_convert_function_method(name):
    def _method(self, entity_num, *args):
        sub_manager, sub_manager_entity_num = self._convert_entity(entity_num)
        function_to_call = getattr(sub_manager, name)
        function_to_call(sub_manager_entity_num, *args)        
    return _method


def container_class_init_decorator(function_object):
    def new_init_function(self, *args):
        # Call the init function :
        function_object(self, *args)
        # Get all relevant methods (Of one sub class is enough)
        method_list = get_relevant_class_method_list(self.container_list[0])
        # Dynamically create all sub layer functions :
        for method_name in method_list:
            _method = make_convert_function_method(method_name)
            setattr(type(self), method_name, _method)

    return new_init_function


class abs_container_class():
    @staticmethod
    def _convert_entity(self):
        sub_manager = None
        sub_entity_num = None
        pass

class system_manager(abs_container_class):
    @container_class_init_decorator
    def __init__(self):
        self.device_manager_list = [] # Initiliaze device list
        self.device_manager_list.append(device_manager())
        self.device_manager_list.append(device_manager())
        self.container_list = self.device_manager_list

    def _convert_entity(self, entity_num):
        relevant_device_manager = self.device_manager_list[entity_num // 4]
        relevant_entity         = entity_num % 4
        return relevant_device_manager, relevant_entity

class device_manager(abs_container_class):
    @container_class_init_decorator
    def __init__(self):
        self.sub_device_manager_list = [] # Initiliaze sub device list
        self.sub_device_manager_list.append(sub_device_manager())
        self.sub_device_manager_list.append(sub_device_manager())    
        self.container_list = self.sub_device_manager_list

    def _convert_entity(self, entity_num):
        relevant_sub_device_manager = self.sub_device_manager_list[entity_num // 4]
        relevant_entity         = entity_num % 4
        return relevant_sub_device_manager, relevant_entity

class sub_device_manager():
    def __init__(self): 
        self.entity_list = [0] * 4

    def set_entity_to_value(self, entity_num, required_value):
        self.entity_list[entity_num] = required_value
        print("I set the entity to : {}".format(required_value))

# This is used for auto completion purposes (Using pep convention)
class auto_complete_class(system_manager, device_manager, sub_device_manager):
    pass


system_instance = system_manager() # type: auto_complete_class
system_instance.set_entity_to_value(0, 3)

There is still a little issue with this solution, auto-completion would not work since the highest level class has almost no static implemented function. In order to solve this I cheated a bit, I created an empty class which inherited from all layers and stated to the IDE using pep convention that it is the type of the instance being created (# type: auto_complete_class).

Musketry

Mm, now I'm wondering how many methods these classes share. I feel DRY code isn't worth so much complication, to be honest. That said, could you just pass around function objects instead of looking through the classes' dir? So: system_instance.apply_to_entity(entity_num=0, lambda entity: entity.set_value(3)). This would decouple code for selecting an entity and calling functions on them, and give you back auto-completion. Finally, I think this question fits better in: code review. – Adorl

@BrianRodriguez I dont really feel its over complicated, and the problem with code duplication is that for every function added to any sub layer you would have to update the higher layer (which is a pain really). But im still leaving the question open cause im hoping for a maybe beter solution – Musketry

Answer 2

Does this solve your Problem?

class EndDevice:
    def __init__(self, entities_num):
        self.entities = list(range(entities_num))

    @property
    def count_entities(self):
        return len(self.entities)

    def get_entity(self, i):
        return str(i)


class Device:
    def __init__(self, sub_devices):
        self.sub_devices = sub_devices

    @property
    def count_entities(self):
        return sum(sd.count_entities for sd in self.sub_devices)

    def get_entity(self, i):
        c = 0
        for index, sd in enumerate(self.sub_devices):
            if c <= i < sd.count_entities + c:
                return str(index) + " " + sd.get_entity(i - c)
            c += sd.count_entities
        raise IndexError(i)


SystemManager = Device # Are the exact same. This also means you can stack that infinite

sub_devices1 = [EndDevice(4) for _ in range(2)]
sub_devices2 = [EndDevice(4) for _ in range(2)]
system_manager = SystemManager([Device(sub_devices1), Device(sub_devices2)])

print(system_manager.get_entity(0))
print(system_manager.get_entity(5))
print(system_manager.get_entity(15))

Lacy

I cant really avoid creating the system manager because it has its own set of functions (Its an instance of itself regarding its entities). A 'recursive' creating of what you offered could work, but it doesnt feel to me any more 'pythonic' then my original suggestion is. – Musketry

I posted my own solution for the problem, I would appreciate it if you could review it and post your thoughts. – Musketry

I can see the benefit of class names beign capitalised here! – Pope

Answer 3

I can't think of a better way to do this than OOP, but inheritance will only give you one set of low-level functions for the system manager, so it wil be like having one device manager and one sub-device manager. A better thing to do will be, a bit like tkinter widgets, to have one system manager and initialise all the other managers like children in a tree, so:

system = SystemManager()
device1 = DeviceManager(system)
subDevice1 = SubDeviceManager(device1)
device2 = DeviceManager(system)
subDevice2 = SubDeviceManager(device2)

#to execute some_func on subDevice1
system.some_func(0, 0, *someParams)

We can do this by keeping a list of 'children' of the higher-level managers and having functions which reference the children.

class SystemManager:
  def __init__(self):
    self.children = []
  def some_func(self, child, *params):
    self.children[child].some_func(*params)

class DeviceManager:
  def __init__(self, parent):
    parent.children.append(self)
    self.children = []
  def some_func(self, child, *params):
    self.children[child].some_func(*params)

class SubDeviceManager:
  def __init__(self, parent):
    parent.children.append(self)
    #this may or may not have sub-objects, if it does we need to make it its own children list.
  def some_func(self, *params):
    #do some important stuff

Unfortunately, this does mean that if we want to call a function of a sub-device manager from the system manager without having lots of dots, we will have to define it again again in the system manager. What you can do instead is use the built-in exec() function, which will take in a string input and run it using the Python interpreter:

class SystemManager:
  ...
  def execute(self, child, function, *args):
    exec("self.children[child]."+function+"(*args)")

(and keep the device manager the same)

You would then write in the main program:

system.execute(0, "some_func", 0, *someArgs)

Which would call

device1.some_func(0, someArgs)

Bubo

Hey thanks for answering. I am afraid I dont really see an added value to your solution over my initial solution, and I have two major problems with what you offered. 1) I dont know if its acceptable or not, but it feels really wrong that the sub class would edit the higher layer attirbutes (Feels wrong to me). 2) I would have to copy each and every function of all the sublayers to the higher layer and implement them (The higher the layers the more code duplication). Of course I could do this with a decorator but it would still leave me with actually writing the function header. – Musketry

@Musketry 1) You can make a addchild() function which will add the object to the children list, but itis simpler to just do a straight-up edit. It's only doing it once, when initialised. 2) Yes this is quite cumbersome, but you can try using the eval() function as I explained in the second part. – Bubo

Using the exec function will require me to type down the actual function name (which is very uncomfortable) and will mainly destroy any chance of IDE auto completion (Which is a massive requirement in our environment) - which I would have in my suggested solution. – Musketry

I posted my own solution for the problem, I would appreciate it if you could review it and post your thoughts. – Musketry

I suggest posting this to code review, since you have working code but are looking for improvements. I've also updated my answer with the idea i mentioned, maybe it can give you more ideas. – Adorl

Answer 4

Here's what I'm thinking:

SystemManager().apply_to_entity(entity_num=7, lambda e: e.value = 2)

 

class EntitySuperManagerMixin():
    """Mixin to handle logic for managing entity managers."""

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)  # Supports any kind of __init__ call.
        self._entity_manager_list = []

    def apply_to_entity(self, entity_num, action):
        relevant_entity_manager = self._entity_manager_list[index // 4]
        relevant_entity_num = index % 4
        return relevant_entity_manager.apply_to_entity(
            relevant_entity_num, action)


class SystemManager(EntitySuperManagerMixin):

    def __init__(self):
        super().__init__()
        # An alias for _entity_manager_list to improve readability.
        self.device_manager_list = self._entity_manager_list
        self.device_manager_list.extend(DeviceManager() for _ in range(4))


class DeviceManager(EntitySuperManagerMixin):

    def __init__(self):
        super().__init__()
        # An alias for _entity_manager_list to improve readability.
        self.sub_device_manager_list = self._entity_manager_list
        self.sub_device_manager_list.extend(SubDeviceManager() for _ in range(4))


class SubDeviceManager():
    """Manages entities, not entity managers, thus doesn't inherit the mixin."""

    def __init__(self):
        # Entities need to be classes for this idea to work.
        self._entity_list = [Entity() for _ in range(4)]

    def apply_to_entity(self, entity_num, action):
        return action(self._entity_list[entity_num])


class Entity():
    def __init__(self, initial_value=0):
        self.value = initial_value

With this structure:

• Entity-specific functions can stay bound to the Entity class (where it belongs).
• Manager-specific code needs to be updated in two places: EntitySuperManagerMixin and the lowest level manager (which would need custom behavior anyway since it deals with the actual entities, not other managers).

Adorl

Hey thanks for the answer! I don't really have an issue calculating the index, there is a constant logic to the amount of entities per sub device (Thus the /4, %4) in my example. Also even if there wasn't it would of been much efficient to have a length variable for each sub device and go on from there. – Musketry

They will stay the same, the real issue in my question doesn't derive from an indexing issue (I can solve that easily using lists and the /4 %4, as I described earlier). Its more the fact that the system / sub device manager have to implement all function of the entities, even though all they do is a conversion (thats why I offered solving it with inheritance - which feels really off to me). – Musketry

This has the same problem as digital hamsters solution. This would require me to duplicate every sub layers function to the higher layer (The higher the layer the more code duplication), even though all of my function would actually do the same thing (convert and call the sub layer instance). I managed to solve this issue with my offered solution (But it wont solve the problem that the lowest sub layer will have implementation for functions of a higher sub layer). – Musketry

@Musketry alright, I made another attempt. I think this should solve it rather nicely, let me know what you think. – Adorl

Its very similar to what I did in my offered solution and still requires code duplication. I actually managed to design a solution with zero code duplication and a strict object oriented approach ill post it later and would be glad if you could look at it. – Musketry

I posted my own solution for the problem, I would appreciate it if you could review it and post your thoughts. – Musketry

Answer 5

The way i see it if you want to dynamically configure different part of system you need some sort of addressing so if you input an ID or address with some parameter the system will know with address on which sub sistem you are talking about and then configure that system with parameter.

OOP is quite ok for that and then you can easily manipulate such data via bitwise operators.

So basic addressing is done via binary system , so to do that in python you need first to implement an address static attribute to your class with perhaps some basic further detailing if system grows.

Basic implementation of addres systems is as follows:

bin(71)
1010 1011
and if we divide it into nibbles 
1010 - device manager 10
1011 - sub device manager 11

So in this example we have system of 15 device managers and 15 sub device menagers, and every device and sub device manager has its integer address.So let's say you want to access device manager no10 with sub device manager no11. You would need their address which is in binary 71 and you would go with:

system.config(address, parameter )

Where system.config funcion would look like this:

def config(self,address, parameter):  
    device_manager = (address&0xF0)>>4 #10  
    sub_device_manager = address&0xf  # 11  
    if device_manager not in range(self.devices): raise LookupError("device manager not found")
    if sub_device_manager not in range(self.devices[device_manager].device): raise LookupError("sub device manager not found")

    self.devices[device_manager].device[sub_device_manager].implement(parameter)  

In layman you would tell system that sub_device 11 from device 10 needs configuration with this parameter.

So how would this setup look in python inheritance class of some base class of system that could be then composited/inherited to different classes:

class systems(object):

    parent = None #global parent element, defaults to None well for simplicity

    def __init__(self):
        self.addrMASK = 0xf # address mask for that nibble
        self.addr = 0x1 # default address of that element
        self.devices = [] # list of instances of device 
        self.data = {  #some arbitrary data
            "param1":"param_val",
            "param2":"param_val",
            "param3":"param_val",
        }


    def addSubSystem(self,sub_system): # connects elements to eachother

        # checks for valiability
        if not isinstance(sub_system,systems):
            raise TypeError("defined input is not a system type") # to prevent passing an integer or something

        # appends a device to system data 
        self.devices.append(sub_system)

        # search parent variables from sub device manager to system
        obj = self
        while 1:
            if obj.parent is not None:
                obj.parent.addrMASK<<=4 #bitshifts 4 bits
                obj.parent.addr <<=4 #bitshifts 4 bits
                obj = obj.parent
            else:break

        #self management , i am lazy guy so i added this part so i wouldn't have to reset addresses manualy
        self.addrMASK <<=4 #bitshifts 4 bits
        self.addr <<=4 #bitshifts 4 bits

        # this element is added so the obj address is coresponding to place in list, this could be done more eloquently but i didn't know what are your limitations
        if not self.devices:
            self.devices[ len(self.devices)-1 ].addr +=1        
        self.devices[ len(self.devices)-1 ].parent = self

    # helpful for checking data ... gives the address of system
    def __repr__(self):

        return "system at {0:X}, {1:0X}".format(self.addr,self.addrMASK)

    # extra helpful lists data as well
    def __str__(self):
        data = [ '{} : {}\n'.format(k,v) for k,v in self.data.items() ]
        return " ".join([ repr(self),'\n',*data ])

    #checking for data, skips looping over sub systems
    def __contains__(self,system_index):

        return system_index-1 in range(len(self.data))

    # applying parameter change  -- just an example
    def apply(self,par_dict):
        if not isinstance(par_dict,dict): 
            raise TypeError("parameter must be a dict type")
        if any( key in self.data.keys() for key in par_dict.keys() ):

            for k,v in par_dict.items():
                if k in self.data.keys():
                    self.data[k]=v
                else:pass

        else:pass

    # implementing parameters trough addresses
    def implement(self,address,parameter_dictionary):

        if address&self.addrMASK==self.addr:

            if address-self.addr!=0:
                item = (address-self.addr)>>4
                self.devices[item-1].implement( address-self.addr,parameter_dictionary )
            else:
                self.apply(parameter_dictionary)






a = systems()
b = systems()
a.addSubSystem(b)
c = systems()
b.addSubSystem(c)

print('a')
print(a)
print('')

print('b')
print(b)
print('')

print('c')
print(c)
print('')

a.implement(0x100,{"param1":"a"})
a.implement(0x110,{"param1":"b"})
a.implement(0x111,{"param1":"c"})

print('a')
print(a)
print('')

print('b')
print(b)
print('')

print('c')
print(c)
print('')

Barram

this type of solution is grabbed from IP protocol, if you want more elements than 15 you can use bigger masks ( jump from 0xF to 0xFF) or add actually virtual masks like IPv6 – Barram

Hey, thanks for the answer! As I stated in the original question and also in the numerous comments on the post, the problem is not converting the arguments in order to understand to which sub device I am accessing. But rather how do I call functions only defined in the lowest layer through the highest layer (I managed to solve this in my answer by creating dynamic functions). – Musketry

Robin, you should then answer your own question and close it. That way everybody has a solution if they come up with similar problem... And if q is answered ppl will stop hunting for bounty :) – Barram

I answered the question and accepted it, I am not aware of any way I can close a question is there one? – Musketry

Yes you can, you have under the question the share edit close delete flag click on close – Barram

I don't have the close button, I guess that since its still an open bounty (I didn't give it to anyone) I cant close the question until the bounty period has expired (+24 hours). – Musketry

Yap it is possible. I've never given bounty on my own question. Then just if it is possible award it to yourself , and be done with it ... or if that isn't possible wait for bounty to expire. Also, don't get me wrong but i think you get it now that bounties are often used for others question that have more computer science topic than normal programming ... to avoid this issue, + i kinda think you also figured out that more than often you find solution to your problem by yourself. Albeit this was an interesting question :D – Barram

Also sorry for typo , I typed Rohi .... and i don't know how it got corrected to Robin. – Barram