The only way to have a recursive anonymous function in MATLAB is to pass the function handle to itself as an input. You can then call it from within the anonymous function.
%// The first input f is going to be an anonymous function
myfun = @(f,x,n) f(f, x, n+1);
%// Then use your anonymous function like this (note the first input)
out = myfun(myfun, x, n);
This will obviously result in infinite recursion since you don't have any branching logic. If you want to simulate the branching logic, you would need to implement another anonymous function to do this (iif
function borrowed from here):
%// Anonymous function to simulate if/elseif/else
iif = @(varargin) varargin{2*find([varargin{1:2:end}], 1, 'first')}();
%// Your anonymous function that is recursive AND has branching
myfun = @(f,x,n)iif(n > 1, ... % if n > 1
@()f(f, f(f, x, n-1), 1), ... % Recurse
true, ... % else
@()expression(x)); % Execute expression()
There is a really solid series of blog entries on the Mathworks site that goes over this sort of functional programming using anonymous functions.
A Word of Caution
While this is an interesting exercise, I definitely wouldn't recommend using this if you want anybody to understand your code easily. It is far clearer and easier to debug a standard function. Then if you really need an anonymous function, wrap the call to that function in an anonymous function.
myanonfunc = @(varargin)myfunc(varargin{:});
Or just create a function handle to the function
myfunchandle = @myfunc;