The right way to dynamically add Django formset instances and POST usign HTMX?

Question

I'm making a form with a nested dynamic formset using htmx i (want to evade usign JS, but if there's no choice...) to instance more formset fields in order to make a dynamic nested form, however when i POST, only the data from 1 instance of the Chlid formset (the last one) is POSTed, the rest of the form POSTs correctly and the Child model gets the relation to the Parent model

I read the django documentation on how to POST formset instances and tried to apply it to my code, also i got right how to POST both Parent and Child at the same time. For the formsets i'm making a htmx get request hx-get to a partial template that contains the child formset and that works great, the only problem is that this always returns a form-0 formset to the client side, so for the POST the data repeats x times per field and only takes the data placed in the last instance, however i tried to change the extra=int value on my formset to get more forms upright, this gave the expected result, one Child instance per form in extra=int, so my problem is up with htmx and the way i'm calling the new Child formset instances.

here's my code. (i plan to nest more child formsets inside this form so i call this sformset for conveniece)

****views.py****

def createPlan(request):#Requst for the Parent form 

    form = PlanForm(request.POST or None)
    sformset = StructureFormset(request.POST or None) #Nesting the Child formset

    context = {
        'form':form,
        'sformset':sformset,
        }

    if request.method == 'POST':

        print(request.POST)
        if form.is_valid() and sformset.is_valid():

            plan = form.save(commit=False)
            print(plan)
            plan.save()
             
            sform = sformset.save(commit=False)     
            for structure in sform:

                structure.plan = plan
                structure.save()

    return render(request, 'app/plan_forms.html', context)


def addStructure(request):

    sformset = StructureFormset(queryset=Structure.objects.none())#add a empty formset instance 
    
    context = {"sformset":sformset}

    return render(request, 'app/formsets/structure_form.html', context)

****forms.py****

StructureFormset = modelformset_factory(Structure,
        fields = (
            'material_type',
            'weight',
            'thickness',
            'provider'
        ))

****relevant part for plan_forms.html template****

<form method="POST">
  {% csrf_token %}
  <div class="col-12 px-2">
    <div class="row px-3 py-1">
      <div class="col-3 px-1">{{ form.format }}</div>
      <div class="col-3 px-1">{{ form.pc }}</div>
      <div class="col-3 px-1">{{ form.revission }}</div>
      <div class="col-3 px-1">{{ form.rev_date }}</div>
    </div>
    <div class="row px-3 py-1">
      <div class="col-3 px-1">{{ form.client }}</div>
      <div class="col-3 px-1">{{ form.product }}</div>
      <div class="col-3 px-1">{{ form.gp_code }}</div>
      <div class="col-3 px-1">{{ form.code }}</div>
    </div>
  </div>
  <div>
    <table>
      <tbody style="user-select: none;" id="structureforms" hx-sync="closest form:queue">
        <!--Structure formset goes here-->
      </tbody>
      <tfoot>
        <a href="" hx-get="{% url 'structure-form' %}" hx-swap="beforeend" hx-target="#structureforms">
          Add structure <!--Button to call structure formset-->
        </a>
      </tfoot>
    </table>
  </div>
  <div class="col-12 px-2">
    <div class="row px-4 py-1">{{ form.observation }}</div>
    <div class="row px-4 py-1">{{ form.continuation }}</div>
    <div class="row px-4 py-1">{{ form.dispatch_conditions }}</div>
    <div class="row px-3 py-1">
      <div class="col-6 px-1">{{ form.elaborator }}</div>
      <div class="col-6 px-1">{{ form.reviewer }}</div>
    </div>
  </div>
  <button type="submit">Submit</button>
</form>

****formsets/structure_form.html****

<tr>
  <td class="col-12 px-1">
    {{ sformset }}
  </td>
</tr>

**** relevant urls.py****

urlpatterns = [
    path('create_plan/', views.createPlan, name='create_plan'),
    path('htmx/structure-form/', views.addStructure, name='structure-form')]

Additionally, the form that i built in admin.py using fields and inlines is just exactly what i want as the raw product (except for the amount of initial formsets and styles)

Tymes

Answer 1

To summarize the problem: At present, your code successfully brings in the new formset, but each new formset comes with a name attribute of form-0-title (ditto for id and other attributes). In addition, after adding the new formset with hx-get the hidden fields originally created by the ManagementForm will no longer reflect the number of formsets on the page.

What's needed

After a new formset is added to the site, here's what I think needs to happen so Django can process the form submission.

1. Update the value attribute in the input element with id="id_form-TOTAL_FORMS" so the number matches the actual number of formsets on the page after hx-get brings in the new formset.

2. Update the name and id of the new formset from form-0-title to use whatever number reflects the current total number of formsets.

3. Update the labels' for attributes in the same way.

You can do this with Javascript on the client side. Alternatively, you can do effectively the same thing with Django on the server side and then htmx can be the only javascript needed to do the rest. For that, I have used empty_form to create the html content of a formset which can be altered as needed. That work is shown in the build_new_formset() helper, below.

Example

Here's what I have working:

forms.py

from django import forms
from django.forms import formset_factory

class BookForm(forms.Form):
    title = forms.CharField()
    author = forms.CharField()

BookFormSet = formset_factory(BookForm)

views.py

from django.utils.safestring import mark_safe
from app2.forms import BookFormSet


def formset_view(request):
    template = 'formset.html'

    if request.POST:
        formset = BookFormSet(request.POST)
        if formset.is_valid():
            print(f">>>> form is valid. Request.post is {request.POST}")
            return HttpResponseRedirect(reverse('app2:formset_view'))
    else:
        formset = BookFormSet()

    return render(request, template, {'formset': formset})


def add_formset(request, current_total_formsets):
    new_formset = build_new_formset(BookFormSet(), current_total_formsets)
    context = {
        'new_formset': new_formset,
        'new_total_formsets': current_total_formsets + 1,
    }
    return render(request, 'formset_partial.html', context)


# Helper to build the needed formset
def build_new_formset(formset, new_total_formsets):
    html = ""

    for form in formset.empty_form:
        html += form.label_tag().replace('__prefix__', str(new_total_formsets))
        html += str(form).replace('__prefix__', str(new_total_formsets))
    
    return mark_safe(html)

Note re: build_new_formset() helper: formset.empty_form will omit the index numbers that should go on the id, name and label attributes, and will instead use "__prefix__". You want to replace that "__prefix__" part with the appropriate number. For example, if it's the second formset on the page its id should be id_form-1-title (changed from id_form-__prefix__-title).

formset.html

<form action="{% url 'app2:formset_view' %}" method="post">
  {% csrf_token %}
  {{ formset.management_form }}

  {% for form in formset %}
  <p>{{ form }}</p>    
  {% endfor %}

  <button type="button" 
          hx-trigger="click"
          hx-get="{% url 'app2:add_formset' formset.total_form_count %}"
          hx-swap="outerHTML">
    Add formset
  </button> 

  <input type="submit" value="Submit">
</form>

formset_partial.html

<input hx-swap-oob="true" 
       type="hidden" 
       name="form-TOTAL_FORMS" 
       value="{{ new_total_formsets }}" 
       id="id_form-TOTAL_FORMS">

<p>{{ new_formset }}</p>

<button type="button" 
        hx-trigger="click"
        hx-get="{% url 'app2:add_formset' new_total_formsets %}"  
        hx-swap="outerHTML">
  Add formset
</button>

Note re: the hidden input: With every newly added formset, the value of the input element that has id="id_form-TOTAL_FORMS" will no longer reflect the actual number of formsets on the page. You can send a new hidden input with your formset and include hx-swap-oob="true" on it. Htmx will then replace the old one with the new one.

Docs reference: https://docs.djangoproject.com/en/4.1/topics/forms/formsets/

Embezzle

Answer 2

My approach to working with dynamic formsets using the mighty HTMX:

Have a view that renders an empty_form of a formset instance (i.e, formset.empty_form). This will return a form with __prefix__ in the part that normally will contain the form number (i.e, instead of form-0-field_name, it will be form-__prefix__-field_name). The trick here is to simply replace the __prefix__ with the appropriate form number.

@Matt's answer achieves this by iterating over the form and doing the relevant replacement. While this works okay, I think an even easier way is simply to change the formset's prefix after instantiating it. That way, you don't the build_new_formset helper function.

On the client side: Render the sformset.management_form in the plan_forms.html. And include the value of the formset's form-TOTAL_FORMS [hidden input field] in HTMX's GET request using hx-vals attribute (please note I am using a jquery selector, that's just for convenience. Vanilla JS should work too).

plan_forms.html

<form method="POST">
   {% csrf_token %}
   {{sformset.management_form}} 
        ...
      <tbody style="user-select: none;" id="structureforms" hx-sync="closest form:queue">
        <!--Structure formset goes here-->
      </tbody>
      <tfoot>
        <button hx-get="{% url 'structure-form' %}" 
           hx-swap="beforeend" 
           hx-target="#structureforms"
           hx-get="{% url 'structure-form' %}"
           hx-vals='js:{totalForms: $("#id_form-TOTAL_FORMS").val()}' \\ New>
          Add structure <!--Button to call structure formset-->
        </button>
      </tfoot>
        ...
</form>

After the htmx's request, the total forms in formset needs to be updated (incremented at each formset addition). If you want to strictly use HTMX, you can use the hx-on attribute which allows you to listen to events and run inline JS. However, I find hyperscript quite easier for this because of is increment command.

plan_forms.html

<form method="POST">
        ...
      <tbody style="user-select: none;"
             id="structureforms" 
             hx-sync="closest form:queue"
             _="on htmx:afterSettle increment #id_form-TOTAL_FORMS's value">
        <!--Structure formset goes here-->
      </tbody>
       ...
</form>

Then you can handle the request in your view: views.py

      ...
def createPlan(request): #Requst for the Parent form 
    form = PlanForm(request.POST or None)
    sformset = StructureFormset(request.POST or None) #Nesting the Child formset
      ...

def addStructure(request):
    sformset = StructureFormset(queryset=Structure.objects.none()).empty_form # Creates an empty formset
    form_number = int(request.GET.get("totalForms")) - 1 # forms are numbered from 0, so they will always be 1 less than formset's total forms.
    sformset.prefix = sformset.prefix.replace("__prefix__", str(form_number)) # override the formset's prefix, replacing the __prefix__
    context = {"sformset":sformset}
    return render(request, 'app/formsets/structure_form.html', context)

I think that's pretty much it.

Notes Instead of hx-vals you can use hx-include

...
hx-include='[name="form-TOTAL_FORMS"]'
...

And in addStructure view you will have to get the value by request.GET.get("form0TOTAL_FORMS")

Pasta

Thanks you for your suggestion! Can I ask you to publish full demo, somewhere? I can't solve the puzzle yet... – Pollinosis