It's that simple: a query that runs in just a few seconds in SQL Developer connecting to Oracle 11g takes 15-25 minutes in SSRS 2008 R2. I haven't tried other versions of SSRS. So far I'm doing all the report execution from VS 2008.
I'm using the OLE DB Provider "OraOLEDB.Oracle.1" which in the past has seemed to give me better results than using the Oracle provider.
Here's what I've been able to determine so far:
• The delay is during the DataSet execution phase and has nothing to do with the result set or rendering time. (Proving by selecting the same rowset directly from a table I inserted it to.)
• SSRS itself is not hung up. It is truly waiting for Oracle which is where the delay is (proven by terminating the DB session from the Oracle side, which resulted in a prompt error in SSRS about the session being killed).
• I have tried direct queries with parameters in the form :Parameter. Very early versions of my query that were more simple worked okay for direct querying, but it seemed like past a certain complexity, the query would start taking forever from SSRS.
• I then switched to executing an SP that inserts the query results to a table or global temp table. This helped for a little while, getting me farther than direct querying, but again, it almost seems like increased query complexity or length eventually broke this method, too. Note: running a table-populating SP works because with option "use single transaction" checked in the DataSource options, DataSets are then run in the order of their appearance in the rdl file. DataSets that return no Fields are still run, as long as all their parameters are satisfied.
• I just tried a table-returning function and this still made no improvement, even though direct calls with literal parameters in SQL Developer return in 1-5 seconds.
• The database in question does not have statistics. It is part of a product created by a vendor and we have not had the time or management buy-in to create/update statistics. I played with the DYNAMIC_SAMPLING hint to calculate statistics on the fly and got a better execution plan: without statistics the cost-based optimizer had been poorly using a LOOP join instead of a HASH join, causing similar many-minute execution times. Thus I put in query hints to force join order and also to cause it to use the strategic hash join, bringing the execution time back down to just seconds. I did not go back and try direct querying in SSRS using these execution hints.
• I got some help from our Oracle DBA who set up a trace (or whatever the Oracle equivalent is) and he was able to see things being run, but he hasn't found anything useful so far. Unfortunately his time is limited and we haven't been able to really dig in to find out what's being executed server-side. I don't have the experience to do this quickly or the time to study up on how to do this myself. Suggestions on what to do to determine what's going on would be appreciated.
My only hypotheses are:
• The query is somehow getting a bad execution plan. E.g., improperly using a LOOP join instead of a HASH join when there are tens of thousands of "left" or outer-loop rows rather than just a few hundred.
• SSRS could be submitting the parameters as nvarchar(4000) or something instead of something reasonable, and as Oracle SP & function parameters don't have length specifications but get their execution lengths from the query call, then some process such as parameter sniffing is messing up the execution plan as in the previous point.
• The query is somehow being rewritten by SSRS/the provider. I AM using a multi-valued parameter, but not as is: the parameter is being submitted as expression Join(Parameters!MultiValuedParameter.Value, ",") so it shouldn't need any rewriting. Just a simple binding and submitting. I don't see how this could be true in the SP and the function calls, but gosh, what else could it be?
I realize it is a very complicated and lengthy query, but it does exactly what I need. It runs in 1-5 seconds depending on how much data is asked for. Some of the reasons for the complexity are:
- Properly handling the comma-separated cost center list parameter
- Allowing the weekly breakdown to be optional and if included, ensuring all the weeks in a month are shown even if there is no data for them.
- Showing "No Invoices" when appropriate.
- Allowing a variable number of summary months.
- Having an optional YTD total.
- Including previous/historical comparison data means I can't simply use this month's vendors, I have to show all the vendors that will be in any historical column.
Anyway, so here's the query, SP version (though I don't think it will be much help).
create or replace
PROCEDURE VendorInvoiceSummary (
FromDate IN date,
ToDate IN date,
CostCenterList IN varchar2,
IncludeWeekly IN varchar2,
ComparisonMonths IN number,
IncludeYTD IN varchar2
)
AS
BEGIN
INSERT INTO InvoiceSummary (Mo, CostCenter, Vendor, VendorName, Section, TimeUnit, Amt)
SELECT
Mo,
CostCenter,
Vendor,
VendorName,
Section,
TimeUnit,
Amt
FROM (
WITH CostCenters AS (
SELECT Substr(REGEXP_SUBSTR(CostCenterList, '[^,]+', 1, LEVEL) || ' ', 1, 15) CostCenter
FROM DUAL
CONNECT BY LEVEL <= Length(CostCenterList) - Length(Replace(CostCenterList, ',', '')) + 1
), Invoices AS (
SELECT /*+ORDERED USE_HASH(D)*/
TRUNC(I.Invoice_Dte, 'YYYY') Yr,
TRUNC(I.Invoice_Dte, 'MM') Mo,
D.Dis_Acct_Unit CostCenter,
I.Vendor,
V.Vendor_VName,
CASE
WHEN I.Invoice_Dte >= FromDate AND I.Invoice_Dte < ToDate
THEN (TRUNC(I.Invoice_Dte, 'W') - TRUNC(I.Invoice_Dte, 'MM')) / 7 + 1
ELSE 0
END WkNum,
Sum(D.To_Base_Amt) To_Base_Amt
FROM
ICCompany C
INNER JOIN APInvoice I
ON C.Company = I.Company
INNER JOIN APDistrib D
ON C.Company = D.Company
AND I.Invoice = D.Invoice
AND I.Vendor = D.Vendor
AND I.Suffix = D.Suffix
INNER JOIN CostCenters CC
ON D.Dis_Acct_Unit = CC.CostCenter
INNER JOIN APVenMast V ON I.Vendor = V.Vendor
WHERE
D.Cancel_Seq = 0
AND I.Cancel_Seq = 0
AND I.Invoice_Dte >= Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY'))
AND I.Invoice_Dte < ToDate
AND V.Vendor_Group = '1 ' -- index help
GROUP BY
TRUNC(I.Invoice_Dte, 'YYYY'),
TRUNC(I.Invoice_Dte, 'MM'),
D.Dis_Acct_Unit,
I.Vendor,
V.Vendor_VName,
CASE
WHEN I.Invoice_Dte >= FromDate AND I.Invoice_Dte < ToDate
THEN (TRUNC(I.Invoice_Dte, 'W') - TRUNC(I.Invoice_Dte, 'MM')) / 7 + 1
ELSE 0
END
), Months AS (
SELECT ADD_MONTHS(Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY')), LEVEL - 1) Mo
FROM DUAL
CONNECT BY LEVEL <= MONTHS_BETWEEN(ToDate, Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY')))
), Sections AS (
SELECT 1 Section, 1 StartUnit, 5 EndUnit FROM DUAL
UNION ALL SELECT 2, 0, ComparisonMonths FROM DUAL
UNION ALL SELECT 3, 1, 1 FROM DUAL WHERE IncludeYTD = 'Y'
), Vals AS (
SELECT LEVEL - 1 TimeUnit
FROM DUAL
CONNECT BY LEVEL <= (SELECT Max(EndUnit) FROM Sections) + 1
), TimeUnits AS (
SELECT S.Section, V.TimeUnit
FROM
Sections S
INNER JOIN Vals V
ON V.TimeUnit BETWEEN S.StartUnit AND S.EndUnit
), Names AS (
SELECT DISTINCT
M.Mo,
Coalesce(I.Vendor, '0') Vendor,
Coalesce(I.Vendor_VName, 'No Paid Invoices') Vendor_VName,
Coalesce(I.CostCenter, ' ') CostCenter
FROM
Months M
LEFT JOIN Invoices I
ON Least(ADD_MONTHS(M.Mo, -ComparisonMonths), TRUNC(M.Mo, 'YYYY')) < I.Mo
AND M.Mo >= I.Mo
WHERE
M.Mo >= FromDate
AND M.Mo < ToDate
)
SELECT
N.Mo,
N.CostCenter,
N.Vendor,
N.Vendor_VName VendorName,
T.Section,
T.TimeUnit,
Sum(I.To_Base_Amt) Amt
FROM
Names N
CROSS JOIN TimeUnits T
LEFT JOIN Invoices I
ON N.CostCenter = I.CostCenter
AND N.Vendor = I.Vendor
AND (
(
T.Section = 1 -- Weeks for current month
AND N.Mo = I.Mo
AND T.TimeUnit = I.WkNum
) OR (
T.Section = 2 -- Summary months
AND ADD_MONTHS(N.Mo, -T.TimeUnit) = I.Mo
) OR (
T.Section = 3 -- YTD
AND I.Mo BETWEEN TRUNC(N.Mo, 'YYYY') AND N.Mo
)
)
WHERE
N.Mo >= FromDate
AND N.Mo < ToDate
AND NOT ( -- Only 4 weeks when a month is less than 28 days long
T.Section = 2
AND T.TimeUnit = 5
AND TRUNC(N.Mo + 28, 'MM') <> N.Mo
AND I.CostCenter IS NULL
) AND (
T.Section <> 1
OR IncludeWeekly = 'Y'
)
GROUP BY
N.Mo,
N.CostCenter,
N.Vendor,
N.Vendor_VName,
T.Section,
T.TimeUnit
) X;
COMMIT;
END;
UPDATE
Even after learning all about Oracle execution plans and hints (to translate my SQL Server knowledge), I still could not get the query to run quickly in SSRS until I made it run in two steps, first to put the real table results into a GLOBAL TEMPORARY TABLE and then second to extract the data from that. DYNAMIC_SAMPLING gave me a good execution plan, which I then copied using join and access hints. Here's the final SP (it couldn't be a function because in Oracle you can't do DML in a function when that function is called inside of a SELECT statement):
Sometimes I swear it was ignoring my join hints such as swap_join_inputs and no_swap_join_inputs but from my reading apparently Oracle only ignores hints when they can't actually be used or you're doing something wrong. Fortunately, the tables swapped appropriately (as in the case of USE_NL(CC) it reliably puts the CC table as the swapped, left input, even though it's joined last).
CREATE OR REPLACE
PROCEDURE VendorInvoicesSummary (
FromDate IN date,
ToDate IN date,
CostCenterList IN varchar2,
IncludeWeekly IN varchar2,
ComparisonMonths IN number,
IncludeYTD IN varchar2
)
AS
BEGIN
INSERT INTO InvoiceTemp (Yr, Mo, CostCenter, Vendor, WkNum, Amt) -- A global temporary table
SELECT /*+LEADING(C I D CC) USE_HASH(I D) USE_NL(CC)*/
TRUNC(I.Invoice_Dte, 'YYYY') Yr,
TRUNC(I.Invoice_Dte, 'MM') Mo,
D.Dis_Acct_Unit CostCenter,
I.Vendor,
CASE
WHEN I.Invoice_Dte >= FromDate AND I.Invoice_Dte < ToDate
THEN (TRUNC(I.Invoice_Dte, 'W') - TRUNC(I.Invoice_Dte, 'MM')) / 7 + 1
ELSE 0
END WkNum,
Sum(D.To_Base_Amt) To_Base_Amt
FROM
ICCompany C
INNER JOIN APInvoice I
ON C.Company = I.Company
INNER JOIN APDistrib D
ON C.Company = D.Company
AND I.Invoice = D.Invoice
AND I.Vendor = D.Vendor
AND I.Suffix = D.Suffix
INNER JOIN (
SELECT Substr(REGEXP_SUBSTR(CostCenterList, '[^,]+', 1, LEVEL) || ' ', 1, 15) CostCenter
FROM DUAL
CONNECT BY LEVEL <= Length(CostCenterList) - Length(Replace(CostCenterList, ',', '')) + 1
) CC ON D.Dis_Acct_Unit = CC.CostCenter
WHERE
D.Cancel_Seq = 0
AND I.Cancel_Seq = 0
AND I.Invoice_Dte >= Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY'))
AND I.Invoice_Dte < ToDate
GROUP BY
TRUNC(I.Invoice_Dte, 'YYYY'),
TRUNC(I.Invoice_Dte, 'MM'),
D.Dis_Acct_Unit,
I.Vendor,
CASE
WHEN I.Invoice_Dte >= FromDate AND I.Invoice_Dte < ToDate
THEN (TRUNC(I.Invoice_Dte, 'W') - TRUNC(I.Invoice_Dte, 'MM')) / 7 + 1
ELSE 0
END;
INSERT INTO InvoiceSummary (Mo, CostCenter, Vendor, VendorName, Section, TimeUnit, Amt)
SELECT
Mo,
CostCenter,
Vendor,
VendorName,
Section,
TimeUnit,
Amt
FROM (
WITH Months AS (
SELECT ADD_MONTHS(Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY')), LEVEL - 1) Mo
FROM DUAL
CONNECT BY LEVEL <= MONTHS_BETWEEN(ToDate, Least(ADD_MONTHS(FromDate, -ComparisonMonths), TRUNC(FromDate, 'YYYY')))
), Sections AS (
SELECT 1 Section, 1 StartUnit, 5 EndUnit FROM DUAL
UNION ALL SELECT 2, 0, ComparisonMonths FROM DUAL
UNION ALL SELECT 3, 1, 1 FROM DUAL WHERE IncludeYTD = 'Y'
), Vals AS (
SELECT LEVEL - 1 TimeUnit
FROM DUAL
CONNECT BY LEVEL <= (SELECT Max(EndUnit) FROM Sections) + 1
), TimeUnits AS (
SELECT S.Section, V.TimeUnit
FROM
Sections S
INNER JOIN Vals V
ON V.TimeUnit BETWEEN S.StartUnit AND S.EndUnit
), Names AS (
SELECT DISTINCT
M.Mo,
Coalesce(I.Vendor, '0') Vendor,
Coalesce(I.CostCenter, ' ') CostCenter
FROM
Months M
LEFT JOIN InvoiceTemp I
ON Least(ADD_MONTHS(M.Mo, -ComparisonMonths), TRUNC(M.Mo, 'YYYY')) <= I.Mo
AND I.Mo <= M.Mo
WHERE
M.Mo >= FromDate
AND M.Mo < ToDate
)
SELECT
N.Mo,
N.CostCenter,
N.Vendor,
Coalesce(V.Vendor_VName, 'No Paid Invoices') VendorName,
T.Section,
T.TimeUnit,
Sum(I.Amt) Amt
FROM
Names N
INNER JOIN APVenMast V ON N.Vendor = V.Vendor
CROSS JOIN TimeUnits T
LEFT JOIN InvoiceTemp I
ON N.CostCenter = I.CostCenter
AND N.Vendor = I.Vendor
AND (
(
T.Section = 1 -- Weeks for current month
AND N.Mo = I.Mo
AND T.TimeUnit = I.WkNum
) OR (
T.Section = 2 -- Summary months
AND ADD_MONTHS(N.Mo, -T.TimeUnit) = I.Mo
) OR (
T.Section = 3 -- YTD
AND I.Mo BETWEEN TRUNC(N.Mo, 'YYYY') AND N.Mo
)
)
WHERE
N.Mo >= FromDate
AND N.Mo < ToDate
AND V.Vendor_Group = '1 '
AND NOT ( -- Only 4 weeks when a month is less than 28 days long
T.Section = 2
AND T.TimeUnit = 5
AND TRUNC(N.Mo + 28, 'MM') <> N.Mo
AND I.CostCenter IS NULL
) AND (
T.Section <> 1
OR IncludeWeekly = 'Y'
)
GROUP BY
N.Mo,
N.CostCenter,
N.Vendor,
V.Vendor_VName,
T.Section,
T.TimeUnit
) X;
COMMIT;
END;
This has been a long, painful ride, but if there's one thing I've learned it's that working in a database without properly updated statistics (which I'm going to look into getting our DBA to add even though the vendor doesn't care about them) can be a real disaster for someone who wants to get things done in a reasonable amount of time.