I am finding a problem with Newtonsoft.Json library throwing a
System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified
when running an app as a Docker container and I'd like to know why this happens and why dependency management does not work smoothly.
I use .NET 5.
I have a library MyLibrary.A that explicitly uses Newtonsoft.Json 13.0.1 for serializing and deserializing json.
I have a different library MyLibrary.B that wraps a MassTransit.AmazonSQS library. This MassTransit library also uses Newtonsoft.Json, but probably a different version.
If I don't do anything explicit, it seems the MassTransit dependency shows the Newtonsoft.Json 11.0.2. If at MyLibrary.B I explicitly add Newtonsoft.Json 13.0.1, even though I don't explicitly use it, then MassTransit seems to be happy with using this newest Newtonsoft.Json 13.0.1
Now, I have a web app MyApp that uses MyLibrary.A and MyLibrary.B. It works fine locally, but I use a CI/CD server to generate a Docker image.
Now I spin up this Docker image as a container locally (as a Docker Compose) and I get the error
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
It complains about a version that does not even exist. There is no 13.0.0.0, This library seems to go from 12.0.3 to 13.0.1.
I am now going through all my libraries and making sure they all use Newtonsoft.Json 13.0.1 explicitly. And when I detect that some of them use some third party that relies on Newtonsoft.Json I add explicitly the very same version, so that I can get everywhere the 13.0.1 version.
UPDATE 1: My workaround didn't work. I don't know what else to try.
I have even added the Newtonsoft.Json 13.0.1 explicitly to my webapp so I was hoping that at least at runtime it has it available.
Also, if I run my web up locally as a standard kestrel AspNetCore app (.NET 5) it launches properly. What is going on? Why is my docker container complaining about Newtonsoft.Json 13.0.0.0 not being found?
These are the traces when attempting to run as a Docker container
docker run -p 8080:80 \
> -e ASPNETCORE_ENVIRONMENT=Production \
> registry.gitlab.com/sample/foo-integration-service:latest
Unable to find image 'registry.gitlab.com/sample/foo-integration-service:latest' locally
latest: Pulling from sample/foo-integration-service
07aded7c29c6: Pull complete
97aff7269a5a: Pull complete
633b89d569a5: Pull complete
bd0e639a2ac9: Pull complete
a9a5571a369e: Pull complete
9569d825ee3a: Pull complete
Digest: sha256:5499b40392512f1731890ccf1ee13507769b733ee2f30c95d281f0550f7a892e
Status: Downloaded newer image for registry.gitlab.com/sample/foo-integration-service:latest
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
File name: 'Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitDisposeCache(ServiceCallSite transientCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitIEnumerable(IEnumerableCallSite enumerableCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitCache(ServiceCallSite callSite, RuntimeResolverContext context, ServiceProviderEngineScope serviceProviderEngine, RuntimeResolverLock lockType)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite singletonCallSite, RuntimeResolverContext context)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.DynamicServiceProviderEngine.<>c__DisplayClass1_0.<RealizeService>b__0(ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngine.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at foo.ItgService.Program.Main(String[] args) in /builds/sample/foo-integration-service/src/foo.ItgService/Program.cs:line 10
UPDATE 2: I decided to match MassTransit dependency Newtonsoft.Json 11.0.2 everywhere in my libraries.
The problem remains.
The error is now
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. The system cannot find the file specified.
I don't get it. All my dependencies of Newtonsoft.Json are now 11.0.2 and still it complains. I'll add this version explicitly (even though I don't directly need it) at the web app main assembly and see if it still complains about it.
UPDATE 3: Still same problem after adding Newtonsoft.Json 11.0.2 to all my libraries and after adding that very same dependency to my web app assembly as a dependency.
As per Chris' comment I have now a .dockerignore.
bin/
obj/
The way I build the image is, with GitLab, with a standard dotnet build, dotnet publish and then copying all the contents of publish folder into the Docker image like this
FROM mcr.microsoft.com/dotnet/aspnet:5.0
COPY publish/ .
EXPOSE 80
ENTRYPOINT ["dotnet", "MyCompany.ItgService.dll"]
More specific, I use Kaniko and this is my .gitlab-ci.yml
image: mcr.microsoft.com/dotnet/sdk:5.0
variables:
GIT_DEPTH: 1000
PUBLISH_OUTPUT_DIR: publish
ENTRYPOINT_DLL: ReplaceMe.dll
CLUSTER_NAME: ReplaceMe
SERVICE_NAME: ReplaceMe
stages:
- build
- test
- publish
- delivery
build:
stage: build
script:
- dotnet restore --no-cache --force
- dotnet build --configuration Release --no-restore
artifacts:
paths:
- test
expire_in: 8 hour
rules:
- if: $CI_COMMIT_TAG
when: never
- when: always
test:
stage: test
services:
- name: localstack/localstack:0.12.17.5
alias: localstack
variables:
# Localstack with SNS and SQS
AWS_DEFAULT_REGION: "us-east-1"
EDGE_PORT: "4566"
SERVICES: "sns,sqs"
before_script:
- rounds=10;
while [ $rounds -gt 0 ]; do
curl http://localstack:4566 && echo OK && break || echo FAIL
rounds=$rounds - 1;
sleep 5;
done;
script: dotnet test --blame --configuration Release
rules:
- if: $CI_COMMIT_TAG
when: never
- exists:
- test/**/*Tests.csproj
publish:
stage: publish
before_script:
- export PATH=$PATH:/root/.dotnet/tools
- dotnet tool install --global GitVersion.Tool --version 5.7.0
- dotnet gitversion
- SEMVER=$(dotnet gitversion -showvariable semver)
- mkdir version
- echo "${SEMVER}" > ./version/semver
- APP_VERSION=$(cat ./version/semver)
script:
- dotnet publish -c Release -o $PUBLISH_OUTPUT_DIR -p:Version=$APP_VERSION
artifacts:
paths:
- $PUBLISH_OUTPUT_DIR/
- version/
expire_in: 8 hour
rules:
- if: $CI_COMMIT_TAG
when: never
- when: always
container_registry:
stage: delivery
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
before_script:
- IMAGE_TAG=$(cat ./version/semver)
- echo "bin/" > $CI_PROJECT_DIR/.dockerignore
- echo "obj/" > $CI_PROJECT_DIR/.dockerignore
- echo "FROM mcr.microsoft.com/dotnet/aspnet:5.0" > $CI_PROJECT_DIR/Dockerfile
- echo "COPY $PUBLISH_OUTPUT_DIR/ ." >> $CI_PROJECT_DIR/Dockerfile
- echo "EXPOSE 80" >> $CI_PROJECT_DIR/Dockerfile
- echo "ENTRYPOINT [\"dotnet\", \"$ENTRYPOINT_DLL\"]" >> $CI_PROJECT_DIR/Dockerfile
- cat $CI_PROJECT_DIR/Dockerfile
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- cat /kaniko/.docker/config.json
script:
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:latest --destination $CI_REGISTRY_IMAGE:$IMAGE_TAG
rules:
- if: $CI_COMMIT_TAG
when: never
- when: always
I don't know if the dotnet publish is messing something up for docker or whether there is some flaw in my process (it works fine for all other services). What might be the problem?
I can see Newtonsoft.Json.dll among the files produced by dotnet publish along with MassTransit.AmazonSqsTransport and all others that are supposedly being copied properly to the Docker image.
Also, If I open the .deps.json I can see all references to Newtonsoft.Json are 11.0.2, so no version conflict anymore (I think!).
I'm out of ideas.
UPDATE 4 I've just cleared all my local nuget packages (using Ubuntu)
dotnet nuget locals all --clear
Clearing NuGet HTTP cache: /home/diegosasw/.local/share/NuGet/v3-cache
Clearing NuGet global packages folder: /home/diegosasw/.nuget/packages/
Clearing NuGet Temp cache: /tmp/NuGetScratch
Clearing NuGet plugins cache: /home/diegosasw/.local/share/NuGet/plugins-cache
Local resources cleared.
An interesting thing is that when I restore dependencies after that on my project
dotnet restore
I can see at /home/diegosasw/.nuget/packages/newtonsoft.json
that there are versions 11.0.2 and 9.0.1
So I guess somewhere somehow a sub-dependency is using Newtonsoft 9.0.1, even though there is no trace of a Newtonsoft.Json 9.0.1 in my the *.deps.json that my dotnet publish produces, and I'm wondering whether this is related to my problem, maybe because that's the assembly being loaded and the 11 is being ignored?
UPDATE 5
I've just seen that the Newtonsoft.Json 9.0.1 is being used by some test project, because I can trace it at coverlet.collector.deps.json under coverlet.core 1.0.0 and the Microsoft.Extensions.DependencyModel 2.1.0
I guess it's not the cause of my problem then.
Also I've verified with docker export $(docker ps -lq) -o foo.tar that the container has the Newtonsoft.Json.dll.
I would like to understand why is this happening and learn how to better troubleshoot these kind of things.
UPDATE 6 (4 Oct)
I don't think the problem is with Docker. I think the problem is likely with dotnet publish or something I'm missing or doing wrong.
I have left Docker outside since I didn't see anything wrong there. I tried to simply do a
dotnet publish -c Release -o publish
and execute the app in that
publish folder with dotnet MyCompany.ItgService.dll to reproduce the exception.
But before publishing, when I run the application with a
dotnet run -c Release --project src/Rubiko.ItgService
I don't get that exception.
See https://github.com/dotnet/sdk/issues/21716 for full details, traces, tree structure, etc.
Summary
The questions are:
Why
dotnet publishdoes not seem to produce everything my app needs to run?Why does it complain at runtime about an assembly that is there?
$ ls publish/ | grep Newtonsoft Newtonsoft.Json.Bson.dll* Newtonsoft.Json.dll*
FINAL UPDATE: Problem solved. See my own response where info on how to properly troubleshoot these kind of issues and how I solved it by ensuring my test projects (that use different version of the library) don't publish artifacts and overwrite the desired dependency assembly.
dotnet publish, what it generates and/or the way I expect it to work. Question and title updated to reflect this. – Mesoglea