You are trying to update nunit-console that ships with Mono? You cannot update that just by using nuget.
Instead I would just download the NUnit console runner you want to use from nuget and just call that using Mono. A simple way to get the console runner would be to run:
nuget install NUnit.ConsoleRunner
This would download the latest NUnit console runner and extract it to:
NUnit.ConsoleRunner.3.7.0/tools/nunit3-console.exe
Then either use the above nunit3-console.exe runner directly with mono:
mono full/path/to/NUnit.ConsoleRunner.3.7.0/tools/nunit3-console.exe
Or if you want to have a nunit3-console command then I would look at the existing nunit-console script /Library/Frameworks/Mono.framework/Versions/Current/Commands/nunit-console
#!/bin/sh
exec /Library/Frameworks/Mono.framework/Versions/5.8.0/bin/mono --debug $MONO_OPTIONS /Library/Frameworks/Mono.framework/Versions/5.8.0/lib/mono/4.5/nunit-console.exe "$@"
Then create your own based on this script but have it use the new NUnit console runner you downloaded.
You may want to put this script somewhere else instead of /Library/Frameworks/Mono.framework/Versions/Current/Commands/
to avoid it being removed when Mono is updated.