What is differentiable programming?

Question

Native support for differential programming has been added to Swift for the Swift for Tensorflow project. Julia has similar with Zygote.

What exactly is differentiable programming?

• what does it enable? Wikipedia says

  the programs can be differentiated throughout

  but what does that mean?

• how would one use it (e.g. a simple example)?

• and how does it relate to automatic differentiation (the two seem conflated a lot of the time)?

Hydrus

Answer 1

I like to think about this question in terms of user-facing features (differentiable programming) vs implementation details (automatic differentiation).

From a user's perspective:

• "Differentiable programming" is APIs for differentiation. An example is a def gradient(f) higher-order function for computing the gradient of f. These APIs may be first-class language features, or implemented in and provided by libraries.

• "Automatic differentiation" is an implementation detail for automatically computing derivative functions. There are many techniques (e.g. source code transformation, operator overloading) and multiple modes (e.g. forward-mode, reverse-mode).

Explained in code:

def f(x):
  return x * x * x

∇f = gradient(f)
print(∇f(4)) # 48.0

# Using the `gradient` API:
# ↳ differentiable programming.

# How `gradient` works to compute the gradient of `f`:
# ↳ automatic differentiation.

Teodorateodorico

Answer 2

I never heard the term "differentiable programming" before reading your question, but having used the concepts noted in your references, both from the side of creating code to solve a derivative with Symbolic differentiation and with Automatic differentiation and having written interpreters and compilers, to me this just means that they have made the ability to calculate the numeric value of the derivative of a function easier. I don't know if they made it a First-class citizen, but the new way doesn't require the use of a function/method call; it is done with syntax and the compiler/interpreter hides the translation into calls.

If you look at the Zygote example it clearly shows the use of prime notation

julia> f(10), f'(10)

Most seasoned programmers would guess what I just noted because there was not a research paper explaining it. In other words it is just that obvious.

Another way to think about it is that if you have ever tried to calculate a derivative in a programming language you know how hard it can be at times and then ask yourself why don't they (the language designers and programmers) just add it into the language. In these cases they did.

What surprises me is how long it to took before derivatives became available via syntax instead of calls, but if you have ever worked with scientific code or coded neural networks at at that level then you will understand why this is a concept that is being touted as something of value.

Also I would not view this as another programming paradigm, but I am sure it will be added to the list.

How does it relate to automatic differentiation (the two seem conflated a lot of the time)?

In both cases that you referenced, they use automatic differentiation to calculate the derivative instead of using symbolic differentiation. I do not view differentiable programming and automatic differentiation as being two distinct sets, but instead that differentiable programming has a means of being implemented and the way they chose was to use automatic differentiation, they could have chose symbolic differentiation or some other means.

It seems you are trying to read more into what differential programming is than it really is. It is not a new way of programming, but just a nice feature added for doing derivatives.

Perhaps if they named it differentiable syntax it might have been more clear. The use of the word programming gives it more panache than I think it deserves.

EDIT

After skimming Swift Differentiable Programming Mega-Proposal and trying to compare that with the Julia example using Zygote, I would have to modify the answer into parts that talk about Zygote and then switch gears to talk about Swift. They each took a different path, but the commonality and bottom line is that the languages know something about differentiation which makes the job of coding them easier and hopefully produces less errors.

About the Wikipedia quote that

the programs can be differentiated throughout

At first reading it seems nonsense or at least lacks enough detail to understand it in context which is why I am sure you asked.

In having many years of digging into what others are trying to communicate, one learns that unless the source has been peer reviewed to take it with a grain of salt, and unless it is absolutely necessary to understand, then just ignore it. In this case if you ignore the sentence most of what your reference makes sense. However I take it that you want an answer, so let's try and figure out what it means.

The key word that has me perplexed is throughout, but since you note the statement came from Wikipedia and in Wikipedia they give three references for the statement, a search of the word throughout appears only in one

∂P: A Differentiable Programming System to Bridge Machine Learning and Scientific Computing

Thus, since our ∂P system does not require primitives to handle new types, this means that almost all functions and types defined throughout the language are automatically supported by Zygote, and users can easily accelerate specific functions as they deem necessary.

So my take on this is that by going back to the source, e.g. the paper, you can better understand how that percolated up into Wikipedia, but it seems that the meaning was lost along the way.

In this case if you really want to know the meaning of that statement you should ask on the Wikipedia talk page and ask the author of the statement directly.

Also note that the paper referenced is not peer reviewed, so the statements in there may not have any meaning amongst peers at present. As I said, I would just ignore it and get on with writing wonderful code.

Arrington

Of interest: CSC321 Lecture 10: Automatic Differentiation – Arrington

Of interest: F#: Derivative Function - An example of how to calculate the derivative. Not as useful as Automatic Differentiation or Symbolic Differentiation, but a stepping stone into doing derivatives in programming. – Arrington

Of interest: Swift Differentiable Programming Mega-Proposal – Arrington

Of interest: Symbolic vs Numeric Math - Performance – Arrington

Answer 3

You can guess its definition by application of differentiability. It's been used for optimization i.e. to calculate minimum value or maximum value Many of these problems can be solved by finding the appropriate function and then using techniques to find the maximum or the minimum value required.

Navigate