SQL Server CLR Threading
Asked Answered
S

1

8

I have been struggling with a SQL Server CLR stored procedure.

Background:

We are using SQL Server 2014 and a CLR stored procedure has been implemented which calls a customer's web service.

The threading was initially used not to slow the main thread of SQL Server CLR.

Although, now, I know that using threading under CLR is no best idea, it has been working correctly for 6 years (since SQL Server 2008). It has been migrated to SQL Server 2014 recently.

The problem

On my development machine, same as on test system we have no problem with the solution.

On the customer system, the thread, which calls the web service, is never executed for some reason.

I can see from the log files that everything is working correctly till the thread execution.

There is no specific error, nothing.

We have been trying to change the permissions, but without a success. Therefore I think its not a permission issue.

Questions

  1. Does anyone know how to change the behavior? We couldn't find any configuration which might does the trick.

  2. Would it be good idea to remove the threading completely, and having the calling of web services directly on SQL Server main thread?

Thank you for any advice, Petr

Suksukarno answered 31/1, 2016 at 17:0 Comment(2)
Add logging to the thread to see if it starts at all and where it stops making progress. Do you have any .NET level error handling that might suppress errors?Scandium
Thats what I did already. The thread does not start at all.Suksukarno
S
5

Not sure about Question #1, though it might not matter given the recommendation for Question #2. Still, one difference between SQL Server 2008 (where it is working) and SQL Server 2014 (where it is not working) is the CLR version that SQL Server is linked to. SQL Server 2005 / 2008 / 2008 R2 are linked to CLR v2.0 while SQL Server 2012 and newer are linked to CLR v 4.0. Since you are not seeing the error and your client is, I would make sure that their system has been updated to the same .NET Framework version that you are running.

For Question #2, I would recommend removing the multi-threading. That has too much potential for problems, and requires the Assembly to be UNSAFE. If you remove the threading, you can set the Assembly to EXTERNAL_ACCESS.

If you want to reduce contention, then assuming the Web Service calls are to the same URI, then you need to increase the number of allowed concurrent web requests. That can be done by setting the ServicePointManager.DefaultConnectionLimit Property. The default value is 2. Which means, any additional requests will wait and wait until one of the current 2 is closed.

Also, be sure to properly Dispose of the WebRequest.


The concern about making external calls (i.e. the Web Service) that can potentially not complete quickly is that SQL Server uses Cooperative Multitasking wherein each thread is responsible for "yielding" control back to the Scheduler (effectively pausing it) at various points so that the Scheduler can shuffle things around and run other things that are currently "sleeping". This concern with regards to SQLCLR code can typically be mitigated by doing at least one of the following:

  • Perform data access / querying the instance
  • Calling thread.sleep(0);

However, an external call is not doing data access, and you cannot easily call thread.sleep(0) while waiting for the WebResponse to complete. Yes, you can call the WebService on a separate thread and while waiting for it to finish, assuming you are just looping and checking, the sleep(x) will allow for the yield.

But is doing the Web Service call asynchronously necessary? It certainly has the downside of requiring the Assembly to be marked as WITH PERMISSION_SET = UNSAFE. It greatly depends on how long the call usually takes, and how frequently it is being called. The more frequent the call, the more likely it is that any delays are, at least in part, caused by the low default value for how many concurrent connections are allowed per each URI. This relates to the recommendation I made at the top.

But if you want to see how SQL Server actually works, this should be fairly easy to test. On my laptop, I went to the Server "Properties" in Object Explorer, went to "Processors", unchecked the "automatically set processor affinity..." option, selected only a single CPU under "Processor Affinity" in the tree view in the middle of the dialog, clicked "OK", and then restarted the service. I then set up a web page that did nothing but call "sleep" for 60 seconds. I have a SQLCLR TVF that calls web pages so I ran that concurrently in two different tabs / sessions. In a 3rd tab / session, I ran:

SELECT SUM(so1.[schema_id]), so1.[type_desc], so2.[type_desc]
FROM sys.objects so1
CROSS JOIN sys.objects so2
CROSS JOIN sys.objects so3
CROSS JOIN sys.objects so4
CROSS JOIN sys.objects so5
WHERE so3.[create_date] <> so4.[modify_date]
GROUP BY so1.[type_desc], so2.[type_desc], so5.[name]
ORDER BY so2.[type_desc], so5.[name] DESC;

And finally, in a 4th tab, after kicking off the first 3, I ran the following to monitor the system:

SELECT * FROM sys.dm_os_schedulers WHERE [scheduler_id] = 0;

SELECT *
FROM sys.dm_exec_requests
WHERE [scheduler_id] = 0
AND [status] <> N'background'
ORDER BY [status] DESC, session_id;

The status for the 2 sessions running the SQLCLR function was always "running" and the status for the session running that ugly query in tab 3 was always "runnable". But just to be sure, running that ugly query again, when neither of the SQLCLR functions was executing, took the same 1 minute and 14 seconds that it did when running concurrently with the 2 sessions running the SQLCLR call to the web page that was sleeping for 60 seconds.

Please do not infer that there is no cost to running the SQLCLR code to make the web calls. Since those threads were busy the whole time, if the system was busy then it would have reduced the ability for SQL Server to allocate those threads to complete other queries faster. But it does seem safe to conclude that, at least on systems with low to moderate load, the benefit gained by adding the threading doesn't seem to be worth the cost of increased complexity (especially since now there is a not-yet-reproducable problem to debug).

Surra answered 31/1, 2016 at 17:23 Comment(13)
My worries are about removing the threading and leaving all the logic within the main thread of MSSQL CLR. Which I believe has more important things to do than calling some web services. If I understand correctly, what you suggest has a different purpose. Or am I wrong?Suksukarno
@PetrB. What "main" thread? You have a set of schedulers, usually 1 per logical CPU. You can see them using SELECT * FROM sys.dm_os_schedulers;. What I am suggesting is to remove the main bottleneck to your Web Service calls from completing sooner. It depends on how long the call should take and how frequently those calls are made, but only allowing 2 active WebRequests at a time can easily make those calls take longer given that all Sessions are sharing that AppDomain. Be sure to Dispose of the WebRequests and increase the DefaultConnectionLimit.Surra
@PetrB. After that, you can easily test this. Just create a simple WebService that does nothing but Thread.Sleep(x) and pass in the value for x. Execute that code in one tab in SSMS, passing in a high value for x, and then open more tabs and run queries. Check SELECT * FROM sys.dm_exec_requests and pay attention to the scheduler_id field. Find the scheduler_id for the session_id running the Web Service call, then see if that scheduler_id is running anything else. Check the status field to see if any other session_id ever shows running for that same scheduler_id.Surra
@strutzky Do I understand it correctly, that each time the CLR SP is executed it runs on its own thread anyway? Therefore it does not make sense to use threading at all? I think I read somewhere that these stored procedures are running under "main" thread of MSSQL.Suksukarno
@PetrB. Where did you read about stored procedures, or maybe just SQLCLR, running under a "main" thread? And if you aren't sure, then why do you trust it? I just updated my answer with more details.Surra
Also make sure the web request is wrapped in Thread.BeginThreadAffinity() / Thread.EndThreadAfinity() to avoid CLR quantum punishment.Suzan
@RemusRusanu Thanks for mentioning that. Is it something to worry about only if you notice forced_yield_count being > 0? I ask because BeginThreadAffinity and EndThreadAffinity require the Assembly to be UNSAFE. Beyond avoiding "CLR Quantum Punishment", is there a reason to mark the thread affinity? Could something go wrong in the process if it is moved between threads? I would assume so given that they have this feature to prevent that move, but I have never come across it, I don't think. I want to add this info to my answer but want to understand it better first.Surra
@RemusRusanu Also, is this still an issue on more recent versions of SQL Server? Or has it decreased with changes made in 2008 and 2012? I found this ( spotlightessentials.com/waitopedia/waits/… ) but not sure how reliable the numbers are. And, how does one go about seeing how much quantum is being used and what the limit is? I did see some additional quantum fields in sys.dm_os_workers. I also see quantum_length_us in sys.dm_os_schedulers but not how usable that value is.Surra
It absolutely still applies to newer releases. You can't move the processing between threads, how would that happen? You do not need to know the quantas. You must declare your 'thread affinity' whenever 1) your own CLR code does lenghty computations (eg. iterates a list of unknown size, processes an unknown size HTML response etc) or 2) the CLR code is calling some code that has the potential to block without SQL seeing it as 'blocked'.Suzan
@RemusRusanu I asked about it still applying because the article you referenced is one of the only articles to mention it and it was from 2006 (a few others copy from it, and your article on CodePlex mentions it). But good to know that it is still relevant. And regarding moving a process between threads, I got that from the BeginThreadAffinity() page that you linked to (in Remarks): "A host that provides its own thread management can move an executing task from one physical operating system thread to another at any time."Surra
any update or hack on this issue? i am facing the same issue, and i need to run my clr sp outside the main thread.Monoplegia
@Edwin How exactly are you doing the threading? Why are you doing it? I might need to see some code in order to attempt to reproduce the issue.Surra
i have created another thread here, with detail explanation of the query, please attend to me there. thanks! #39736246Monoplegia

© 2022 - 2024 — McMap. All rights reserved.