With pelican 4.0.1 and pelican-bootstrap3 it is simply not possible to sort the menu items for both pages and article categories because they are sorted separately in the base template: first the pages (which are sorted), then the categories (which don't seem to be sorted). So the pages are always placed it front of the categories.
Also, in the template the sorting for the page items are controlled by setting the option PAGES_SORT_ATTRIBUTE, so try setting that one in your pelicanconf.py if the PAGE_ORDER_BY option mentioned in the docs doesn't work.
Bit of a shame, but this answer here by jcollado does seem to solve the problem, how many menu items will you have anyway?
But I had to adjust it a bit:
DISPLAY_CATEGORIES_ON_MENU = False
DISPLAY_PAGES_ON_MENU = False
MENUITEMS = (
('Projects', '/category/projects.html'),
('Publications', '/pages/Publications.html'),
('Music', '/category/music.html'),
('About', '/pages/about.html'),
)
Start the URIs with the forward slash, or you might run into some trouble.