Finally got it working and thought I would share what worked as there are a lot of this same question with no replies.
So to answer my question, yes, remDr$open() should open a browser, but in a vnc container that you already have up and running. Here's my process to get there.
First, make sure everything is up-to-date. So many problems could have been solved by making sure my programs, packages and library's where the newest. With that said, I read somewhere that a program (can't remember which) was having issues with Big Sur...
Second, this resource was great about explaining how to setup Docker+Selenium on windows/linux machines and how to scrape using RSelenium. In it he warns that there are some security issues with setting up the ports in the method below. I don't know how to solve that so be warned that you run the code at your own risk!
https://www.youtube.com/watch?v=OxbvFiYxEzI&t=4838s
Ok now to get started... you need the app Docker on your system and up and running. On my mac it's called Docker Desktop.
On the Docker website you use Docker Hub to find the selenium files you need. It will give you a link to copy and paste into your terminal to download the files. I used standalone-firefox and standalone-firefox-debug. I also think you need selenium-server-standalone-xxx.xx.jar but I can't remember what for or the process. It's sitting in my Rproject folder... don't know if that matters.
If you want to see a window with the pages on it you would run in terminal:
docker run -d -p 4445:4444 -p 5901:5900 selenium/standalone-firefox-debug
This will set up a new project in Docker.
You now need to use a vnc to create a container to view the browser in. Mac has one built in via ScreenShare that you can easily access one of two ways... cmd+k and enter vnc://127.0.0.1:5901 or via terminal:
open vnc://127.0.0.1:5901
Password is "secret" !!!!
Now your vnc is up and running you can run your code in RStudio and yes, $open will open firefox in the vnc/screen share window!
library(RSelenium)
remDr <- remoteDriver(port=4445L) ##4445 for mac. windows can use 4444
remDr$open()
Finally note. Once you have tested your code and everything is working properly it is recommended that you run a headless browser instead of the debug one. So your code will work, you just won't need to open the vnc and watch, which slows everything down. Here is the code to run instead:
docker run -d -p 4445:4444 selenium/standalone-firefox
