The existing answers cover a lot of ground but I thought I would mention one other point.
I use the registry to store system-wide settings. That is, when 2 or more programs need the exact same setting. In other words, a setting shared by several programs.
In all other cases I use a local config file that sits either in the same path as the executable or one level down (in a Configuration directory). The reasons are already covered in other answers (portable, can be edited with a text editor etc).
Why put system-wide settings into the registry? Well, I found that if a setting is shared but you use local config files you end up duplicating settings. This may mean you end up needing to change a setting in multiple places.
For example, say Program A and Program B both point to the same database. You can have a "system-wide" registry setting for the connection string. If you want to point to a different database, you can change the connection string in one place, and both programs will now run against the other database.
Note - there is no point in using the registry in this way if two or more programs don't need to use the same values. Such as, Program A and Program B both needing a database connection string that may be the same, but not always. For example, I want Program B to now use a test database but Program A should carry on using a production database.
With the above example, you could have some local configuration override system-wide settings but it may start getting overly complicated for simple tasks.