Many years ago I had to convert an Installscript MSI project to Basic MSI because of a bug that caused all changes performed by Installscript custom actions to be rolled back on major upgrade. Could that bug still be in there?
I do not know. If I were you, I would immediately convert my setup to Basic MSI to avoid the very problematic Installscript MSI setup type.
To convert the project, you can create a new Basic MSI project file, close it, open the Installscript MSI project file and then export all components by right clicking the components (in the component view) and go "Export Into..." (or something like that) and then browse to your new Basic MSI project file. This way you can migrate all components, but the custom actions and dialog changes you have to migrate yourself - as far as I know.
I would contact Installshield support and ask for their help. Who knows, maybe they have tools to help you with such a conversion. They will certainly have more information on the problem you are experiencing. However, my 2 cents still stand on converting to Basic MSI. It is the right format to use. Installscript MSI is not popular with corporations for corporate deployment.