How do you compose query expressions in F#?

Question

I've been looking at query expressions here http://msdn.microsoft.com/en-us/library/vstudio/hh225374.aspx

And I've been wondering why the following is legitimate

let testQuery = query {
        for number in netflix.Titles do
        where (number.Name.Contains("Test"))
    }

But you can't really do something like this

let christmasPredicate = fun (x:Catalog.ServiceTypes.Title) -> x.Name.Contains("Christmas")
let testQuery = query {
        for number in netflix.Titles do
        where christmasPredicate 
    }

Surely F# allows composability like this so you can reuse a predicate?? What if I wanted Christmas titles combined with another predicate like before a specific date? I have to copy and paste my entire query? C# is completely unlike this and has several ways to build and combine predicates

Wheen

C# is completely like this, you can't do where predicate. You can do Where(predicate), but I think you could do the same thing here too. – Endothelioma

I never said otherwise. But from what I can see so far, you can't do this in a query expression. I thought the same thing but can't seem to get anything to even compile. Also there was no literature on the internet about this. Also, where can only take a bool, not an expression so I think you're wrong :) Prove it? where(expression<predicate>) is good because there are things like LinqKit that allow you to manipulate your expressions like crazy. – Wheen

I've an issue with my F# 3 interactive, but won't where (christmasPredicate number) work? – Pangenesis

Nice Ramon; that makes it so I can compile it, but apparently the way the query expressions are translated, it won't allow the query to be translated and throws an exception. I won't give up yet because we seem to be getting somewhere. It seems like expression trees are just more flexible than computational expressions when it comes to ORMs, but like I said, I could be completely wrong. – Wheen

I'm not sure it can translate your general function to SQL, that's true. There's some magic in the query, I believe (but I did not dig well enough into F# 3 internals, out of a lack of interest). – Pangenesis

I wasn't expecting it to. My beef is that LINQ to Entities can most definitely can take an arbitrary Expression<predicate> (possibly built of other expressions) in a where clause as long as the function calls INSIDE the expression are translatable and it makes it much more flexible and composable. – Wheen

Answer 1

This was quite easy to do with the F# 2.0 version of queries which required explicit quotations (I wrote a blog post about it). There is a way to achieve similar thing in C# (another blog post) and I think similar tricks could be played with F# 3.0.

If you do not mind uglier syntax, then you can use explicit quotations in F# 3.0 too. When you write
query { .. } the compiler actually generates something like:

query.Run(<@ ... @>)

where the code inside <@ .. @> is quoted F# code - that is, code stored in an Expr type that represents the source code and can be translated to LINQ expressions and thus to SQL.

Here is an example that I tested with the SqlDataConnection type provider:

let db = Nwind.GetDataContext()

let predicate = <@ fun (p:Nwind.ServiceTypes.Products) -> 
  p.UnitPrice.Value > 50.0M @>

let test () =
  <@ query.Select
      ( query.Where(query.Source(db.Products), %predicate), 
        fun p -> p.ProductName) @>
  |> query.Run
  |> Seq.iter (printfn "%s")

The key trick is that, when you use explicit quotations (using <@ .. @>) you can use the % operator for quotation slicing. This means that the quotation of predicate is put into the quotation of the query (in test) in place where you write %predicate.

The code is quite ugly compared to the nice query expression, but I suspect you could make it nicer by writing some DSL on top of this or by pre-processing the quotation.

EDIT: With a bit more effort, it is actually possible to use the query { .. } syntax again. You can quote the entire query expression and write <@ query { .. } @> - this will not directly work, but you can then take the quotation and extract the actual body of the query and pass it to query.Run directly. Here is a sample that works for the above example:

open System.Linq
open Microsoft.FSharp.Quotations
open Microsoft.FSharp.Quotations.Patterns

let runQuery (q:Expr<IQueryable<'T>>) = 
  match q with
  | Application(Lambda(builder, Call(Some builder2, miRun, [Quote body])), queryObj) ->
      query.Run(Expr.Cast<Microsoft.FSharp.Linq.QuerySource<'T, IQueryable>>(body))
  | _ -> failwith "Wrong argument"

let test () =
  <@ query { for p in db.Products do
             where ((%predicate) p)
             select p.ProductName } @>
  |> runQuery
  |> Seq.iter (printfn "%s")

Capitalist

Are you a wizard? Very nicely done. This was going to be a dealbreaker for me. I'm surprised this question doesn't come up more often. – Wheen

You can actually splice directly into a query, though I'm not sure if that's a feature or a bug... – Baliol

@Baliol I get a runtime error when using a direct splicing (because it doesn't actually splice, but rather call the operator). – Pangenesis

@TomasPetricek Awsome code, have been searching for this for literally days, thank you very much! See my question: #16801214. You can have the bounty. – Fluellen

Answer 2

Naively, in the original example one could try to quote the predicate, and then splice it in:

let christmasPredicate = <@ fun (x:Catalog.ServiceTypes.Title) -> 
                             x.Name.Contains("Christmas") @>
let testQuery = query {
        for number in netflix.Titles do
        where ((%christmasPredicate) number) 
        select number
    }

(I've taken the liberty of cleaning up the original example slightly)

Examples such as this (with simple, first-order lambda-abstractions) often do work in F# anyway, but in general, there is no guarantee that F#'s default QueryBuilder will normalize away the resulting applications of lambda-abstractions in the quoted term. This can result in strange error messages, or in queries with poor performance (e.g. querying one table and then generating one query on another table per row of the first table, instead of doing a single query join).

We recently developed a library called FSharpComposableQuery that (if opened) overloads the query operator to perform normalization (and to do some other helpful things). It provides a strong guarantee to generate a single query for a nontrivial subset of F# query expressions. Using FSharpComposableQuery's version of query, the above naive composition works. We have also tested extensively to try to ensure that FSharpComposableQuery doesn't break existing query code.

Similarly, for example, using FSharpComposableQuery, Tomas's example does not require the special RunQuery function. Instead, one can simply do:

open FSharpComposableQuery

let predicate = <@ fun (p:Nwind.ServiceTypes.Product) -> 
                     p.UnitPrice.Value > 50.0M @>
let test () =
  query { for p in db.Products do
          where ((%predicate) p)
          select p.ProductName }
  |> Seq.iter (printfn "%s")

(Caveat: I have only tested the above code only with the OData version of Northwind, not the SQL type provider, but we have tested a large number of similar and more complex examples. The OData version fails with a mysterious error from OData, but this seems orthogonal to the matter at hand.)

FSharpComposableQuery is now available from NuGet here: https://www.nuget.org/packages/FSharpComposableQuery

and more information (including examples and a small tutorial, demonstrating more complex forms of composition) can be found here:

http://fsprojects.github.io/FSharp.Linq.ComposableQuery/

[EDIT: Changed the above links to remove the word "Experimental", since the project name has changed.]

Sollows