Not nesting version of @atomic() in Django?

Question

From the docs of atomic()

atomic blocks can be nested

This sound like a great feature, but in my use case I want the opposite: I want the transaction to be durable as soon as the block decorated with @atomic() gets left successfully.

Is there a way to ensure durability in django's transaction handling?

Background

Transaction are ACID. The "D" stands for durability. That's why I think transactions can't be nested without loosing feature "D".

Example: If the inner transaction is successful, but the outer transaction is not, then the outer and the inner transaction get rolled back. The result: The inner transaction was not durable.

I use PostgreSQL, but AFAIK this should not matter much.

Litigant

Answer 1

Even though this exact behaviour is not possible, since django 3.2 there is a durable=True[@transaction.atomic(durable=True)] option to make sure that such a block of code isnt nested, so that by chance if such code is run as nested it results in a RuntimeError error. https://docs.djangoproject.com/en/dev/topics/db/transactions/#django.db.transaction.atomic An article on this issue https://seddonym.me/2020/11/19/trouble-atomic/

Scup

thank you for this update. This is really good news. – Litigant

This should be the accepted answer now. – Sulcus

Answer 2

You can't do that through any API.

Transactions can't be nested while retaining all ACID properties, and not all databases support nested transactions.

Only the outermost atomic block creates a transaction. Inner atomic blocks create a savepoint inside the transaction, and release or roll back the savepoint when exiting the inner block. As such, inner atomic blocks provide atomicity, but as you noted, not e.g. durability.

Since the outermost atomic block creates a transaction, it must provide atomicity, and you can't commit a nested atomic block to the database if the containing transaction is not committed.

The only way to ensure that the inner block is committed, is to make sure that the code in the transaction finishes executing without any errors.

Integumentary

I could not find the answer to my question in your text. You say I need to ensure that the outer atomic block gets committed. Yes, that's true. How to do this in django? – Litigant

@Litigant I updated my answer. There is no API to do that, the only way to achieve this is to make sure the code in the transaction finishes without any errors. – Integumentary

@guettli, if you want to do low-level transaction management in django, you can use use the transaction package directly. For an example, you can see github.com/2ps/djenga/blob/master/djenga/db/… which is something I wrote back when django did not support nested transactions in mysql. – Whom

Answer 3

I agree with knbk's answer that it is not possible: durability is only present at the level of a transaction, and atomic provides that. It does not provide it at the level of save points. Depending on the use case, there may be workarounds.

I'm guessing your use case is something like:

@atomic  # possibly implicit if ATOMIC_REQUESTS is enabled
def my_view():
    run_some_code()  # It's fine if this gets rolled back.
    charge_a_credit_card()  # It's not OK if this gets rolled back.
    run_some_more_code()  # This shouldn't roll back the credit card.

I think you'd want something like:

@transaction.non_atomic_requests
def my_view():
    with atomic():
        run_some_code()
    with atomic():
        charge_a_credit_card()
    with atomic():
        run_some_more_code()

If your use case is for credit cards specifically (as mine was when I had this issue a few years ago), my coworker discovered that credit card processors actually provide mechanisms for handling this. A similar mechanism might work for your use case, depending on the problem structure:

@atomic
def my_view():
    run_some_code()
    result = charge_a_credit_card(capture=False)
    if result.successful:
        transaction.on_commit(lambda: result.capture())
    run_some_more_code()

Another option would be to use a non-transactional persistence mechanism for recording what you're interested in, like a log database, or a redis queue of things to record.

Antitrust

Your link to "credit card processors actually provide mechanisms for handling this" looks like Two-phase commit protocol: en.wikipedia.org/wiki/Two-phase_commit_protocol – Litigant

Answer 4

This type of durability is impossible due to ACID, with one connection. (i.e. that a nested block stays committed while the outer block get rolled back) It is a consequence of ACID, not a problem of Django. Imagine a super database and the case that table B has a foreign key to table A.

CREATE TABLE A (id serial primary key);
CREATE TABLE B (id serial primary key, b_id integer references A (id));
-- transaction
   INSERT INTO A DEFAULT VALUES RETURNING id AS new_a_id
   -- like it would be possible to create an inner transaction
      INSERT INTO B (a_id) VALUES (new_a_id)
   -- commit
-- rollback  (= integrity problem)

If the inner "transaction" should be durable while the (outer) transaction get rolled back then the integrity would be broken. The rollback operation must be always implemented so that it can never fail, therefore no database would implement a nested independent transaction. It would be against the principle of causality and the integrity can not be guarantied after such selective rollback. It is also against atomicity.

The transaction is related to a database connection. If you create two connections then two independent transactions are created. One connection doesn't see uncommitted rows of other transactions (it is possible to set this isolation level, but it depends on the database backend) and no foreign keys to them can be created and the integrity is preserved after rollback by the database backend design.

Django supports multiple databases, therefore multiple connections.

# no ATOMIC_REQUESTS should be set for "other_db" in DATABASES

@transaction.atomic  # atomic for the database "default"
def my_view():
    with atomic():   # or set atomic() here, for the database "default"
        some_code()
        with atomic("other_db"):
            row = OtherModel.objects.using("other_db").create(**kwargs)
        raise DatabaseError

The data in "other_db" stays committed.

It is probably possible in Django to create a trick with two connections to the same database like it would be two databases, with some database backends, but I'm sure that it is untested, it would be prone to mistakes, with problems with migrations, bigger load by the database backend that must create real parallel transactions at every request and it can not be optimized. It is better to use two real databases or to reorganize the code.

The setting DATABASE_ROUTERS is very useful, but I'm not sure yet if you are interested in multiple connections.

Terrain