To solve this you need a dependency management solution, because you want changes to the individual projects to be reflected automatically in the overall project. You have decided you can't use frameworks which are the natural choice for this, for practical reasons (the framework class size and access level).
The solution is to find a different dependency management solution for your source code.
The next most obvious option is to use git submodules, because this has been explicitly designed for modularising at the source level. If you use this in conjunction with the solution suggested by @sjwarner for project organisation you should be able to achieve what you need.
https://git-scm.com/book/en/v2/Git-Tools-Submodules
"Submodules allow you to keep a Git repository as a subdirectory of another Git repository. This lets you clone another repository into your project and keep your commits separate."
In other words you can continue to maintain each view controller separately, and the commits can be included into your overall project.
(Note that the submodules do not have to be the individual projects themselves, you could potentially make only the viewcontrollers separate submodules, but this might be a bit more complex to set up and maintain)
a) make each project containing the view controller you want a git repository if it is not already one.
b) create a git repository for your main project (if it is not already one)
c) add each project as a git submodule to your main git repo
d) follow @sjwarners suggestion on source code organisation for the overall project