How do I set up TeamCity CI so that it unpacks Xamarin components?

Question

In Visual Studio everything works and a Components directory is created with the appropriate dlls. However, TeamCity is not able to retrieve the Android Support Library dlls because the trigger for the restore is a Xamarin VS plugin that runs when loading the solution. The equivalent of nuget package restore for Xamarin is xamarin-component. I have placed the xamarin-component.exe in my C:\Windows directory. To configure TeamCity, I prepended a Command Line build step with

Command executable: xamarin-component 
Command parameters: restore mysolution.sln

TeamCity runs as NT Authority\System. So using PsExec,

psexec -i -s %SystemRoot%\system32\cmd.exe

If I then run 'xamarin-component login'

INFO (login): Computed cookie jar path: C:\Windows\system32\config\systemprofile\.xamarin-credentials 
INFO (login): Computed cookie jar path: C:\Windows\system32\config\systemprofile\.xamarin-credentials 
INFO (login): Credentials successfully stored.

When I go to my solution in cmd and attempt the restore, I get an attempt to download the componet, and then a Json parsing error. This is the same error I get in TeamCity.

I get the error if I use 'Administrator' (which stores the credential in C:\Users\Administrator. Earlier when I was using my personal account, it did work. However, once I deleted the C:\Users\tim\AppData\Local\Xamarin\Cache\Components, the same issue emerged. Fiddler shows that rather than getting Json back (as we do when we enter an invalid token) we are getting a 302 redirect that says Object moved here. And here is the xamarin login page - obviously not Json.

Tried. 1. Set COOKIE_JAR_PATH to C:\Users\tim.xamarin-credentials - xpkg picks up but same error 2. Copy .xamarin-credentials from Config\system32 to D:\, set COOKIE_JAR_PATH to D:.xamarin-credentials - xpkg picks up but same error 3. Move .xamarin-credentials to C:\, set COOKIE_JAR_PATH - same error 4. Re-login in NT Authority with COOKIE_JAR_PATH to C:.xamarin-credentials - same error

My temporary idea now is to figure out where the NT Authority xamarin-component looks for Cache and put the files there.

C:\Windows\system32\config\systemprofile\AppData\Local\Xamarin\Cache\Components\xamandroidsupportv4-18-4.18.1.xam

The version of my xamarin-component is 0.99 - for 100, we try harder...

Mariande

Answer 1

I’ve had trouble actually getting the cookie jar to load correctly from the system32 path. I think this is a path virtualization issue that I just don't understand well enough to make heads or tails of.

I ended up adding an environment variable that the tool will read from (I'm its principal author at Xamarin :-) that specifies the cookie jar path to read from, and this solved the problem for others using TeamCity. The environment variable is COOKIE_JAR_PATH.

You can set it from TeamCity's environment settings to point to a cookie jar path outside of the system32 profile directory (I think in my original testing, I put it in the root of the C: drive, but it can be anywhere, really).

Builtin

Thank you Bojan. I was hoping to hear from you. I'll give it a try. Suggest that the warning is built into the next version - or maybe an option to specify cookie jar location. – Mariande

Yeah, I've got to spend some time on improving the error handling/documentation on the tool. – Builtin

Hmm, pointing to my personal one C:\Users\tim\.xamarin-credentials yields the same error though the tool does pick up on the change. – Mariande

No luck copying the NT Authority generated version from Config\systemprofile to D:\. From Fiddler it looks like I'm getting redirected instead of getting JSON. – Mariande

You can create the cookie jar on another machine and then copy it--I wouldn't attempt copying the one from Config\systemprofile somewhere else. It's a simple text file that you can create via Notepad. – Builtin

This is interesting. When not using COOKIE_JAR_PATH, the log is a little bit different, though both end in the Json error. When using the system32 path, three times it "computed cookie jar path", but only once when using the one in C:\.xamarin-credentials – Mariande

I deleted my user Xamarin cache and now I get the same error from the user side. AT least it is consistent! – Mariande

Bojan, any more insight on this? It seems like the link called by the util is incorrect or has been modified server-side. – Mariande

I'm struggling with the same issue on OSX, but setting the cookie jar path does not seem to have an impact. Any hints? – Havener

Answer 2

As a hack, I copied the Cache folder from

C:\Users\tim\AppData\Local\Xamarin 

to

C:\Windows\system32\config\systemprofile\AppData\Local\Xamarin\

That bypassed communication with the Xamarin server.

Update. I suspect it might be a bad link or setup on their server side. When xamarin-component restore is called, a call is made to

GET /api/available_versions?alias=xamandroidsupportv4-18 HTTP/1.1

which returns "Object moved to here" where "here" is nowhere.

If you start Visual Studio after deleting the Cache and Components folder (next to the solution), Xamarin makes a call to

GET /api/download/xamandroidsupportv4-18/4.18.1 HTTP/1.0

which has a similar looking Object moved to, but this time it directs you to xamarin-components.s3.amazonaws.com/

GET /fdca922d2b77799fe208a08c9f3444fe/xamandroidsupportv4-18-4.18.1.xam HTTP/1.0

Perhaps something changed, or the available_versions API has changed.

Mariande

great news! I reported this error to Xamarin and they have identified the problem. They expect that the fixed API will be available by the end of the week. – Trifoliate

Answer 3

Thanks very much for this question and your answers to it. I didn't really like the idea of storing an auth cookie on the build node or having to copy a cache there manually, so I came up with my own solution so I hacked around this problem with a quick Powershell script that mimics the behaviour of the xamarin-component.exe restore action:

param
(
    [Parameter(Mandatory=$true)]
    $authCookie,

    [Parameter(Mandatory=$true)]
    $componentDirectory,

    [Parameter(Mandatory=$true)]
    $project
)

[void]([System.Reflection.Assembly]::LoadWithPartialName('System.IO.Compression.FileSystem'))
$xml = [xml] $(cat $project);
$components = $xml.Project.ItemGroup.XamarinComponentReference | ? { $_.Include.Length -gt 0 } | % { $_.Include };

if (!(test-path $componentDirectory))
{
    echo "$componentDirectory didn't exist, so it was created.";
    [void](mkdir $componentDirectory);
}

foreach ($component in $components)
{
    $source = "http://components.xamarin.com/download/$component";
    $destination = "$componentDirectory\$component.zip";

    if (test-path $destination)
    {
        echo "$destination already exists, skipping...";
        continue;
    }

    echo "Downloading $component from $source to $destination...";

    $client = New-Object System.Net.WebClient
    $client.Headers.Add([System.Net.HttpRequestHeader]::Cookie, "XAM_AUTH=$authCookie");

    try
    {
        $client.DownloadFile($source, $destination);
    }
    catch
    {
        # The error message will be on one of these lines hopefully:
        write-error "Failed to download! Errors are below:";
        write-error $_
        write-error $_.Exception
        write-error $_.Exception.InnerException
        write-error $_.Exception.InnerException.InnerException
        exit 1;
    }

    if (!(test-path $destination))
    {
        write-error "$destination doesn't exist - the download must have failed!";
        exit 1;
    }

    echo "Decompressing $source to $componentDirectory"
    [System.IO.Compression.ZipFile]::ExtractToDirectory($destination, $componentDirectory)
    echo ""
}

echo "Done!";

The -authCookie parameter can be extracted from either the XAM_AUTH cookie in your browser or from the .xamarin-credentials "cookiejar" in your home directory. It's nice to have it parameterised like this so you can store it as a secret variable in TeamCity.

The componentDirectory parameter must be the full path to the component directory - it will be created if it doesn't exist.

The project parameter should be the path to your project that you want to restore packages for - if you have multiple projects that need this then you'll have to execute the script for each one. Don't specify your solution as it won't work.

Unfortunately, this isn't very resilient to Xamarin's whims - a simple API change could render this useless, so obviously the best solution is to wait for Xamarin to fix this. I e-mailed Xamarin support to complain about this problem but I don't imagine I'll get a timely response (they seem very very busy these days). I hope this is useful!

Trifoliate

Answer 4

Create directory and put that directory path in environment variable XAMARIN_CACHEPATH

Shearwater