One way of customizing an RCP app is by using a plugin_customization.ini file. Like this:
- Create a new file called "plugin_customization.ini" in the root of your project.
- If you have not already done so, create a product configuration and define a Product.
- In your plugin's manifest editor, on the "Extensions" tab, locate the "org.eclipse.core.runtime.products" extension, expand the node, right click on your product node and select "New > property" from the context menu.
- Enter "preferenceCustomization" in the "name" field and "plugin_customization.ini" in the "value" field.
- Select "File > Save" from the main menu.
You can then customize a lot of things simply by editing plugin_customization.ini file. For example, adding the following line
org.eclipse.ui/SHOW_TRADITIONAL_STYLE_TABS=false
will give you rounded tabs on your views and editors (similar to those that Eclipse IDE has) instead of the default ones.
You can find other constants that you can use in IWorkbenchPreferenceConstants javadoc.
Also, check out the Eclipse skins project, if you haven't already.
There is also the possibility to use CSS to change the look & feel of your app, but that is still in development and can be buggy
As for customizing the SWT controls themselves, the only way to do that (apart from small customizations such as changing the background colour of a control etc.) is by creating your own custom controls by extending Canvas or Composite, but you already know that.
Keep in mind that the point of SWT is to use native controls, so if you desire the ability to completely customize your controls, perhaps using Swing would be a better option. However, if you're going to use Swing, it may be better to use another RCP entirely - namely, NetBeans RCP. I haven't tried it out yet, but I know that whereas Eclipse RCP uses SWT as a widget toolkit, NetBeans RCP relies on Swing, so that may save you from some headaches regarding compatibility problems and so on.