Login/Register
Error accessing Key Vault while executing Azure Synapse Notebook via Synapse Pipelines
Asked Answered
runtime-errorazure-data-factoryazure-keyvaultazure-synapse
U

2

6

I'm trying to execute an Azure Synapse Notebook using Notebook Activity in Synapse Pipelines and it keeps coming up with errors while debuging the Pipelines, the Notebook is using TokenLibrary.getSecret() and it looks like accessing key vault is the problem.

The notebook runs fine when executed its when added as pipeline activity the error comes up.

The key vault access policies are set to give both me and the synapse app get and list on secrets.

Many thanks for any help in advance. Mariusz

Error msg for ref:

Microsoft Azure
Synapse Analytics

Search

/
feature/fixError6002 branch


Develop


Filter resources by name
SQL scripts
1

Notebooks
40

Data flows
1

Activities
Search activities
Synapse
Move & transform
Azure Data Explorer
Azure Function
Batch Service
Databricks
Data Lake Analytics
General
HDInsight
Iteration & conditionals
Machine Learning
Pipeline run ID:
58ff181a-37e6-47b8-b3d8-fe94295b9ec7


View debug run consumption
Name

Type

Run start

Duration

Status

Integration runtime

Run ID

Invoke Notebook
Notebook
2021-09-29T10:44:24.3680765Z
00:02:04
Failed
DefaultIntegrationRuntime (UK South)
9a6773d8-69e7-46f6-bc99-25878637b7b7
Log Pipeline Start
Azure Function
2021-09-29T10:44:19.9601216Z
00:00:03
Succeeded
IR-AZ-Runtime (UK South)
8280f852-1915-49ab-a6a8-35bf22a716bb
Error
{
    "errorCode": "6002",
    "message": "Py4JJavaError: An error occurred while calling z:com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getSecret.\n: java.lang.Exception: Access token couldn't be obtained {\"result\":\"DependencyError\",\"errorId\":\"BadRequest\",\"errorMessage\":\"LSRServiceException is [{\\\"StatusCode\\\":400,\\\"ErrorResponse\\\":{\\\"code\\\":\\\"LSRResolveFailure\\\",\\\"message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"target\\\":\\\"Vault\\\"},\\\"Message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"Data\\\":{},\\\"InnerException\\\":null,\\\"StackTrace\\\":\\\"   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.CheckForFailures(HttpResponseMessage response, String responseContent) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 272\\\\r\\\\n   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken, String traceId) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 288\\\\r\\\\n   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.ResolveAudienceAsync(String audience, ResolveAudienceRequest request, String traceId, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 164\\\\r\\\\n   at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetToken(Boolean isLinkedService, String audience, String sessionToken, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 140\\\\r\\\\n   at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetTokenForAudienceAsync(Boolean isLinkedService, String audience, String account, String sessionToken, SignaturePayload signaturePayload, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 67\\\\r\\\\n   at Microsoft.Marlin.TokenService.Controllers.TokenController.GetTokenAsync(TokenRequest request, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Controllers\\\\\\\\TokenController.cs:line 67\\\\r\\\\n   at lambda_method(Closure , Object )\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\\\\r\\\\n   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)\\\",\\\"HelpLink\\\":null,\\\"Source\\\":\\\"Microsoft.Marlin.Common.ADF\\\",\\\"HResult\\\":-2146233088}]. TraceId : 8158a0b1-4612-48de-aaa5-c443c564d25b. Error Component : LSR\"}\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:382)\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:374)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1808)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1807)\n\tat com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240)\n\tat com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:215)\n\tat com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91)\n\tat com.twitter.util.Promise$WaitQueue$$anon$4.run(Promise.scala:86)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109)\n\tat com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:778)\n\tat com.twitter.util.Promise.update(Promise.scala:750)\n\tat com.twitter.util.Promise.setValue(Promise.scala:726)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)\n\tat com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\nTraceback (most recent call last):\n\n  File \"<stdin>\", line 9, in get_cloudinary_data\n\n  File \"/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py\", line 1257, in __call__\n    answer, self.gateway_client, self.target_id, self.name)\n\n  File \"/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py\", line 69, in deco\n    return f(*a, **kw)\n\n  File \"/opt/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py\", line 328, in get_return_value\n    format(target_id, \".\", name), value)\n\npy4j.protocol.Py4JJavaError: An error occurred while calling z:com.microsoft.azure.synapse.tokenlibrary.TokenLibrary.getSecret.\n: java.lang.Exception: Access token couldn't be obtained {\"result\":\"DependencyError\",\"errorId\":\"BadRequest\",\"errorMessage\":\"LSRServiceException is [{\\\"StatusCode\\\":400,\\\"ErrorResponse\\\":{\\\"code\\\":\\\"LSRResolveFailure\\\",\\\"message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"target\\\":\\\"Vault\\\"},\\\"Message\\\":\\\"Cannot acquire MSI token for a Vault audience.\\\",\\\"Data\\\":{},\\\"InnerException\\\":null,\\\"StackTrace\\\":\\\"   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.CheckForFailures(HttpResponseMessage response, String responseContent) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 272\\\\r\\\\n   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken, String traceId) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 288\\\\r\\\\n   at Microsoft.Marlin.Common.ADF.Impl.LSRClient.ResolveAudienceAsync(String audience, ResolveAudienceRequest request, String traceId, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\Common\\\\\\\\Microsoft.Marlin.Common.ADF\\\\\\\\Impl\\\\\\\\LSRClient.cs:line 164\\\\r\\\\n   at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetToken(Boolean isLinkedService, String audience, String sessionToken, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 140\\\\r\\\\n   at Microsoft.Marlin.TokenService.Token.LSRAudienceTokenProvider.GetTokenForAudienceAsync(Boolean isLinkedService, String audience, String account, String sessionToken, SignaturePayload signaturePayload, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Token\\\\\\\\LSRAudienceTokenProvider.cs:line 67\\\\r\\\\n   at Microsoft.Marlin.TokenService.Controllers.TokenController.GetTokenAsync(TokenRequest request, CancellationToken cancellationToken) in C:\\\\\\\\source\\\\\\\\TokenService\\\\\\\\Microsoft.Marlin.TokenService\\\\\\\\Controllers\\\\\\\\TokenController.cs:line 67\\\\r\\\\n   at lambda_method(Closure , Object )\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()\\\\r\\\\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)\\\\r\\\\n   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)\\\\r\\\\n   at Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)\\\\r\\\\n   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.Invoke(HttpContext context)\\\",\\\"HelpLink\\\":null,\\\"Source\\\":\\\"Microsoft.Marlin.Common.ADF\\\",\\\"HResult\\\":-2146233088}]. TraceId : 8158a0b1-4612-48de-aaa5-c443c564d25b. Error Component : LSR\"}\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:382)\n\tat com.microsoft.azure.synapse.tokenlibrary.TokenLibrary$$anonfun$9.apply(TokenLibrary.scala:374)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1808)\n\tat com.twitter.util.Future$$anonfun$flatMap$1.apply(Future.scala:1807)\n\tat com.twitter.util.Promise$FutureTransformer.liftedTree1$1(Promise.scala:240)\n\tat com.twitter.util.Promise$FutureTransformer.k(Promise.scala:240)\n\tat com.twitter.util.Promise$Transformer.apply(Promise.scala:215)\n\tat com.twitter.util.Promise$WaitQueue.com$twitter$util$Promise$WaitQueue$$run(Promise.scala:91)\n\tat com.twitter.util.Promise$WaitQueue$$anon$4.run(Promise.scala:86)\n\tat com.twitter.concurrent.LocalScheduler$Activation.run(Scheduler.scala:198)\n\tat com.twitter.concurrent.LocalScheduler$Activation.submit(Scheduler.scala:157)\n\tat com.twitter.concurrent.LocalScheduler.submit(Scheduler.scala:274)\n\tat com.twitter.concurrent.Scheduler$.submit(Scheduler.scala:109)\n\tat com.twitter.util.Promise$WaitQueue.runInScheduler(Promise.scala:86)\n\tat com.twitter.util.Promise.updateIfEmpty(Promise.scala:778)\n\tat com.twitter.util.Promise.update(Promise.scala:750)\n\tat com.twitter.util.Promise.setValue(Promise.scala:726)\n\tat com.twitter.concurrent.AsyncQueue.offer(AsyncQueue.scala:123)\n\tat com.twitter.finagle.netty4.transport.ChannelTransport$$anon$2.channelRead(ChannelTransport.scala:168)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.UnpoolHttpHandler$.channelRead(UnpoolHttpHandler.scala:32)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.ClientExceptionMapper$.channelRead(ClientExceptionMapper.scala:35)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat com.twitter.finagle.netty4.http.handler.HeaderValidatorHandler$.channelRead(HeaderValidatorHandler.scala:51)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)\n\tat io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat com.twitter.finagle.util.BlockingTimeTrackingThreadFactory$$anon$1.run(BlockingTimeTrackingThreadFactory.scala:23)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\n\n",
    "failureType": "UserError",
    "target": "Invoke Notebook",
    "details": []
}
Uzziel answered 29/9, 2021 at 10:54 Comment(1)
Can you post a snippet of the actual code that is running please? – Stanza
H
3

Please make sure to enable MSI(managed service identity) under identity > system /user assigned(ON) and make sure to add access policies properly to the KeyVault secrets, and also add a LinkedService to the Azure Synapse pointing to the KeyVault.

(Then restart again)

If your url is something like below

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString")

Try to also linked service as a third parameter like below

url = TokenLibrary.getSecret("mykeyvault", "ConnectionString", "AzureKeyVaultLinkedServiceName")

References:

  1. secure-credentials-with-tokenlibrary | Microsoft docs
  2. access-secret-from-vault-using-synapse | microsoft Q&A
  3. securely-access-Article-dzone
Helve answered 1/10, 2021 at 11:23 Comment(0)
M
1

Running it in parallel, maybe with a for loop that is not sequentialmight be problematic. I seem to have a similar issue, when running the same notebook multiple times in parallel (each sending requests to the keyvault at the same time).

Miler answered 14/9, 2022 at 9:43 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.