Erlang supervisor dynamic change to restart intensity
Asked Answered
L

2

11

My question is, can one modify the restart intensity thresholds of an already running supervisor, apart from in a release upgrade scenario, and if so, how?

It's never come up before, but running a supervisor with initially no children, so that another process starts children by way of supervisor:start_child/2, so my sup init/1 being like this:

init([]) ->
    RestartSt = {simple_one_for_one, 10, 10},
    ChSpec = [foo, {foo,start_link,[]}, transient, 1000, worker, [foo]}],
    {ok, {RestartSt, ChSpec}}.

At the time of supervisor start, the likely number of children is unknown; certainly it could vary dramatically from 10, to 10,000, or more.

A restart intensity of say 20 is generous enough for 10 children, but for say 10,000 children I would like to be able to increase it... and decrease it as the number of children drops due to normal terminations.

Lots answered 18/10, 2015 at 22:57 Comment(0)
E
9

There's no API for doing this, so I believe you're stuck with the upgrade approach unless you want to propose a new API for this to the OTP team by submitting a pull request providing a complete patch with code changes, new tests, and documentation changes.

There's also a really dirty hack way of doing this that involves manipulating internal supervisor state, and so it's absolutely not something I would recommend for a production system but I think it's still interesting to look at. A supervisor stores restart intensity in its internal loop state. You can see this state by calling sys:get_state/1,2 on a supervisor process. For example, here's the state of a supervisor in the Yaws web server:

1> rr(supervisor).
[child,state]
2> sys:get_state(yaws_sup).
#state{name = {local,yaws_sup},
       strategy = one_for_all,
       children = [#child{pid = <0.67.0>,name = yaws_sup_restarts,
                          mfargs = {yaws_sup_restarts,start_link,[]},
                          restart_type = transient,shutdown = infinity,
                          child_type = supervisor,
                          modules = [yaws_sup_restarts]},
                   #child{pid = <0.42.0>,name = yaws_server,
                          mfargs = {yaws_server,start_link,
                                                [{env,true,false,false,false,false,false,"default"}]},
                          restart_type = permanent,shutdown = 120000,
                          child_type = worker,
                          modules = [yaws_server]},
                   #child{pid = <0.39.0>,name = yaws_trace,
                          mfargs = {yaws_trace,start_link,[]},
                          restart_type = permanent,shutdown = 5000,
                          child_type = worker,
                          modules = [yaws_trace]},
                   #child{pid = <0.36.0>,name = yaws_log,
                          mfargs = {yaws_log,start_link,[]},
                          restart_type = permanent,shutdown = 5000,
                          child_type = worker,
                          modules = [yaws_log]}],
       dynamics = undefined,intensity = 0,period = 1,restarts = [],
       module = yaws_sup,args = []}

The initial rr command retrieves the record definitions from supervisor so we can see the field names when we get the state from yaws_sup, otherwise we would just get a tuple full of anonymous values.

The retrieved state shows the intensity in this case to be 0. We can change it using sys:replace_state/2,3:

3> sys:replace_state(yaws_sup, fun(S) -> S#state{intensity=2} end).
#state{name = {local,yaws_sup},
       strategy = one_for_all,
       children = [#child{pid = <0.67.0>,name = yaws_sup_restarts,
                          mfargs = {yaws_sup_restarts,start_link,[]},
                          restart_type = transient,shutdown = infinity,
                          child_type = supervisor,
                          modules = [yaws_sup_restarts]},
                   #child{pid = <0.42.0>,name = yaws_server,
                          mfargs = {yaws_server,start_link,
                                                [{env,true,false,false,false,false,false,"default"}]},
                          restart_type = permanent,shutdown = 120000,
                          child_type = worker,
                          modules = [yaws_server]},
                   #child{pid = <0.39.0>,name = yaws_trace,
                          mfargs = {yaws_trace,start_link,[]},
                          restart_type = permanent,shutdown = 5000,
                          child_type = worker,
                          modules = [yaws_trace]},
                   #child{pid = <0.36.0>,name = yaws_log,
                          mfargs = {yaws_log,start_link,[]},
                          restart_type = permanent,shutdown = 5000,
                          child_type = worker,
                          modules = [yaws_log]}],
       dynamics = undefined,intensity = 2,period = 1,restarts = [],
       module = yaws_sup,args = []}

Our second argument to sys:replace_state/2 takes a state record as an argument and changes its intensity field to 2. The sys:replace_state/2,3 functions return the new state, and as you can see near the end of the result here, intensity is now 2 instead of 0.

As the sys:replace_state/2,3 documentation explains, these functions are intended only for debugging purposes, so using them to do this in a production system is definitely not something I recommend. The second argument to replace_state here shows that this approach requires knowledge of the details of the internal state record of supervisor, which we obtained here via the rr shell command, so if that record ever changes, this code may stop working. Even more fragile would be treating the supervisor state record as a tuple and counting on the intensity field to be in a particular tuple position so you can change its value. Therefore, if you really want this functionality of changing a supervisor's restart intensity, you're best off in the long run proposing to the OTP team that it be added; if you're going to take that route, I recommend first proposing the idea on the erlang-questions mailing list to gauge interest.

Enroot answered 19/10, 2015 at 17:17 Comment(1)
That's great, thanks, it's an answer, if not a solution, if you discount this rather interesting but ugly gem you've mined for us! I think we all agree this is one of those gems best left in the ground, or perhaps safely reburied! I will consider your suggestion of posting the feature suggestion and a pull request, etc. It seems like a perfectly valid feature to me really, it's clear no static restart intensity and interval could ever be optimal if you assume variable homogeneous with equal chances of death, where the total criticality is shared across the running set. Thanks for your time!Lots
W
1

One solution would be to nest your supervisors. But the main question is what do you want to achieve by this restart intensities. The intensity when you want to kill the supervisor needs to be a indication for something very wrong e.g. a needed resource unexpectedly not being available.

Woermer answered 19/10, 2015 at 11:15 Comment(9)
Thanks for replying, but I think I would have to say, with respect, this should be a comment - it isn't an answer, and a dynamic tree of supervisors would be FAR from trivial to implement well - things like supervisor:which_children/2 would not work any more; one would have to write code to recurse up the sup tree to find all the children, choosing a supervisor in the tree to spawn from is a job, and I don't want to even think about the restart specs of all these supervisors. I think it would IMO be MUCH easier to write a supervisor that did what one wanted than shoe horn supervisor this way.Lots
Another problem with a tree of supervisors to handle a homogeneous set of processes, is, for example if you have 15 supervisors, and there were 100 crashes across 10 of them, that might be enough for the supervisors to crash, and maybe the top level supervisor would crash... but if the 100 crashes was spread across 15 instead of 10, the top level might survive... The behaviour of the whole thing would become inconsistent.Lots
@Michael, actually I believe Ergodic hypothesis presumes that the behaviour would be consistent if top level crash on every child crash (and other issues are solved with just gproc or simple ets table I think)Knowall
@Knowall Hah! The ergodic hypothesis is statistical though, so for some use cases it may be that a sup tree with certain intensity and period settings with children equally likely to fail over time MAY have a high PROBABILITY of yielding the same result as a single supervisor. Some cases MAY have a good probability of different behaviour though. It would be interesting to peruse a general maths proof of the probabilities involved here, but I'm not going to attempt that myself and I'd rather not have this sort of indeterministic behaviour in my supervisor tree.Lots
@Knowall Take 10 supervisors (restart intensity 10) supervising 10 children. A supervisor death due to restart intensity being exceeded could occur after 11 deaths, or no supervisor deaths could occur after 100 deaths. Since not all children may be equally likely to crash (it's perfectly feasible that long running processes started at a similar time could be grouped on same supervisors), so the ergodic hypothesis would in this case only be valid over a VERY long time, which is what it is designed for. I'm far from a genius mathematician though so I'd be interested to hear if you disagree.Lots
@Michael, I actually assume that all process have same crash probability which is function of worker itself (if every worker execute the same code and connected to the same input source it can crash with certain probability anytime and if it didn't crash for some long amount of time - it means... nothing. (There is common misconception about it). While I'm also only amateur in math we could ask math.stackexchange.com may be %)Knowall
@Knowall Yes, completely agree, I would generally make the same assumption, because apart from anything else it's clear we don't know in advance the reasons for crashes or they wouldn't happen, so what else can we do? Sometimes a bug may occur or be more likely to occur over time due to state development, and workers that do the same functionally may deal with different data (different input sources if you like), so ultimately I want my supervision (the one thing that must cater well for the unknown) not to have any possibility of suboptimal behaviour because of the nature of that unknown.Lots
@Michael, sure, you should just understand that you get trade off here. Either you will apply dirty hack or deal with some possible behaviour inconsistence. Probably you should measure first before selecting actual soultionKnowall
@Knowall I won't apply the dirty hack that Steve Vinoski illustrated, I don't consider that a solution any more than a sup tree, or any more than Steve Vinoski would consider it a solution either I would suggest... I do consider the update to supervisor so it can accept changes to restart intensity a solution, but it's obviously not a solution that actually exists currently. I did propose the change but there doesn't appear to be much immediate interest I have to say. Thus, probably I will write a specialist supervisor for my case.Lots

© 2022 - 2024 — McMap. All rights reserved.