Just do an upgrade and see if anything breaks for you.
5 to 6 was a very different upgrade than 6 to 7. The vast majority of 6 code will work in 7. The big thing with 5 to 6 was the pipe operator where functions were moved out of the main observable class into tree shakeable includes so there was a need for the compatibility library. Using the upgrade compatibility library was not a good approach back then, the best approach was to fix your code. You could easily fix a large codebase in a few hours.
Here are the breaking changes in version 7, you will have to work out which ones affect your library and if you can make any incompatibilities work with 6 on a case by case basis.
https://rxjs.dev/deprecations/breaking-changes
It really all depends on the code you have in your codebase. My personal libraries didn't need any work when 7 first came out, they just worked without refactoring a single line of code. I just did an npm upgrade rxjs and it built and all my unit tests passed. Happy times.