Ok, here's an answer to why it's happening, but I don't know the solution. If you use the "Other Swift Flag" -driver-show-incremental Xcode will output what it decides it needs to compile based upon it's dependancies. You'll see things like:
Queuing EditProfileViewController.swift because of dependencies discovered later
Queuing ChangePasswordViewController.swift because of dependencies discovered later
Queuing JoinViewController.swift because of dependencies discovered later
Queuing JoinProfileViewController.swift because of dependencies discovered later
Queuing FormViewBuildable.swift because of dependencies discovered later
Queuing RadioTextFormView.swift because of dependencies discovered later
Queuing TextFieldFormView.swift because of dependencies discovered later
Queuing AccountProfileViewController.swift because of dependencies discovered later
I'm wondering if this is a swift 3 problem, because I wasn't having this issue before converting. I made a little example project where
FileA inclues a thing from FileB includes a thing from FileC
and even adding a file private change to FileC that's not used anywhere causes FileA, FileB, and FileC to be queued for compilation due to dependencies. I'm going to be testing this example in Xcode 7 later today to see what happens.
So it looks like swift 3's dependency resolution isn't working very well. I've tested this on 2 other swift 3 projects at work, and the same holds true. Make any change in any file, and every file gets compiled. It doesn't feel slow until you start getting to projects with about 15,000 lines of code or so, which may be why nobody is talking much about this. Unless you have a somewhat medium sized swift 3 app, you probably won't even notice that incremental complication isn't working quite right. I'll update if I learn anything more.