Okay i got it, but it does not work without editing the csproj file manually. Here is a quick explanation for anyone else struggling with running their tests for a xamarin.android/ios project using reactiveui on a mac in combination with xamarin studio.
Install xamarin.mac using the xamarin.installer
Open your solution and add a new Library project from the "Mac" "Library" - "Unified API" tree.
Delete all unneeded files like the Main.cs or AppDeletage.cs
Open the Options of the added solution and switch the Target Framework from "Xamarin.Mac Mobile Framework" to "Xamarin.Mac .Net 4.5 Framework" under "Build" - "General".
Add your packages. Dont forget Nunit and reactiveui-testing.
Open the csproj file of the new project and change the path part from Net45 to Xamarin.Mac10 for the Splat and ReactiveUI references. See the example code.
Run your tests and be happy
Broken references:
<Reference Include="ReactiveUI">
<HintPath>..\packages\reactiveui-core.6.5.0\lib\Net45\ReactiveUI.dll</HintPath>
</Reference>
<Reference Include="Splat">
<HintPath>..\packages\Splat.1.6.2\lib\Net45\Splat.dll</HintPath>
</Reference>`
Fixed references:
<Reference Include="ReactiveUI">
<HintPath>..\packages\reactiveui-core.6.5.0\lib\Xamarin.Mac10\ReactiveUI.dll</HintPath>
</Reference>
<Reference Include="Splat">
<HintPath>..\packages\Splat.1.6.2\lib\Xamarin.Mac10\Splat.dll</HintPath>
</Reference>
For reference, this is our working csproj file https://gist.github.com/omares/e4bccf5a90524f2482f9