As long as you install it as a plugin and not as an extension (such as XPI on firefox) you don't need to restart the browser. On Mac OS X that means putting the .plugin file in either /Library/Internet Plugins/ or in ~/Library/Internet Plugins/. You can instruct the browser to reload the list of plugins like this:
navigator.plugins.refresh(false);
The false tells it not to refresh the page to do it.
Updating is a slightly different problem; the browsers will not notice that a plugin update has occurred without restarting the browser unless the filename is different from the original. My usual solution is to include the version number of the plugin in the .plugin/ bundle filename, something like "FBTestPlugin_1.0.0.plugin", so when it updates it gets named differently.
On windows different NPAPI browsers have different problems with plugin detection (and different versions of those browsers), but the best way I've found to make this work is to install each plugin in a version-specific subdirectory with a version-specific filename; this seems like overkill, but the cases I was addressing were with a plugin that had 50 million installs and it was necessary for all supported browser versions to work. I was also using the registry install method (HKCU/Software/MozillaPlugins).
Hope that helps