Email Conversations / Email Threading support in EWS Managed Api (against Exchange 2010 or so)

Question

The EWS Managed API has a handful of functions for retrieving and managing email conversations (aka email threads). Unfortunately, a large part of them work only against new versions of Exchange (2013 etc.)

Outlook does implement email threading against older versions of Exchange. Perhaps it does this by managing the threads by itself (Outlook is a desktop application, the emails are copied on the local machine, therefore they can be easily grouped by Conversation Topic etc.).

Now, how can email threading be supported within a web application? What is usually done for supporting this feature in an Exchange client? By supported I mean:

• retrieve first 10 conversations snapshots, then retrieve the next 10 conversations snapshots - that is, support for paging (retrieve pages on demand) - this data would be used to build a master view
• retrieve all emails within a conversation - that is, retrieve the children of a conversation on demand) - this data would be used to build a detail view of a conversation.

Issues with EWS Managed Api etc.:

• there is no Conversation.Bind(conversationId) in EWS Managed API
• ExchangeService.FindItems(filter for ConversationTopic == "some topic") is by no means reliable (because there may be different conversations having the same topic)
• ExchangeService.FindItems(filter for ConversationId == "QWERYUIO") - I could not figure out how to use this :) Is it possible to search for emails by ConversationId?
• functions like ExchangeService.GetConversationItems() are only "applicable for clients that target Exchange Online and versions of Exchange starting with Exchange Server 2013."

What I am using now (as a workaround):

1. retrieve (on demand) a page of conversations using ExchangeService.FindConversation()
2. for each conversation in the retrieved page, read the Conversation.GlobalIds property
3. build an aggregate (an array) containing the values from all GlobalIds - by concatenating Conversation.GlobalIds of all the conversations
4. make an Exchange call to bind the ids to emails (ExchangeService.BindToItems)
5. perform a group-by operation of the emails (conceptually, it is a grouping operation, but implementation is not a trivial group-by call - emails cannot be grouped by ConversationId, as that property is not available when working against Exchange 2010, though the documentation does not specify this)
6. use the data to build the UI in one step (the list of conversations for the master view, the groups of emails for the detail view of each conversation) etc.

Some issues with the implementation described above

• I am retrieving a lot of data from the server when calling the ExchangeService.BindToItems operation - the performance is not excellent, but it is not quite bad either. Of course, it would have been better to retrieve the emails only when the user wants to access the detail view of a specific conversation. A possible hack: hold the GlobalIds array somewhere in a hidden field, then use it to fetch the emails in order to build the detail view. I know that a GET request is limited in size, but however...

On email conversations / email threading, nobody knows which supports what:

• Here it says that FindConversation(ViewBase, FolderId) is applicable for clients that target Exchange Online and versions of Exchange starting with Exchange Server 2013. On the other hand, here is written that the ExchangeService.FindConversation() function can be used for Versions of Exchange starting with Exchange 2010, including Exchange Online.
• This is funny, too: Applies to: EWS Managed API | Exchange Server 2010 Service Pack 1 (SP1), BUT Ensure that you have an Exchange 2013 or Exchange Online service account with a major version of 15 or higher. :)
• Here it says that the Item.ConversationId property is available in Versions of Exchange starting with Exchange 2010, including Exchange Online. But it is not :)

Note: I am not very sure about the support of the Item.ConversationId, as I do not have the code at hand and cannot perform a test right now. Therefore, please forgive me if that property is available after all when using EWS Managed API against Exchange 2010.

All in all, do you have any ideas for implementing the email conversations / email threading feature in a web application, using the EWS Managed API against an Exchange 2010 server?

Thank you a lot for having the patience to read such a long post :)

Some references:

http://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.exchangeservice_methods%28v=exchg.80%29.aspx http://msdn.microsoft.com/en-us/library/microsoft.exchange.webservices.data.conversation_members%28v=exchg.80%29.aspx http://msdn.microsoft.com/en-us/library/office/gg274407%28v=exchg.80%29.aspx http://msdn.microsoft.com/en-us/library/office/jj220497%28v=exchg.80%29.aspx

Implementing Outlook 2010's group by conversation using EWS and Exchange 2007 Exchange Webservice Managed API - Find items by extended properties

Haply

Microsoft's documentation has always been holey. But it sounds like you want features of Exchange Server 2013, but for the 2010 version. If such features unavailable in 2010 (not implemented) or unsatisfactory/unreliable, you won't have much choice other than to reimplement them yourself. In particular, detecting message threads could use the JWZ threading algorithm - hopefully you'll find a .NET implementation and add it to your project. – Argile

I don't have a complete answer for you, but I can straighten out some of your confusion over the list of documentation issues you have in your "On email conversations / email threading, nobody knows which supports what:" list. When reading the documentation you have to pay attention to whether it's referring to the EWS technology, in which case the name is "FindConversation operation", or if it's referring to the EWS Managed API (EWSMA) technology, in which case it's called the FindOperation method. Your last bullet point goes to a page about the EWS functionality, which was available in 2010. – Patronage

Answer 1

I've addressed some of the documentation questions you had in the comments, so I'm going to attempt to answer your real coding questions here.

To get your master view, ExchangeService.FindConversation is the right method to use. It does support paging by limiting the results to the number of conversations specified by the view parameter. You could call it on demand to get older and older results.

To get your detailed view, because ExchangeService.GetConversationItems isn't available on Ex2010, you can use ExchangeService.FindItems with an IsEqualTo SearchFilter that searches for items with a matching ConversationId (see code below). There's more information about search filters here: How to: Use search filters with EWS in Exchange.

In the following method, I limited the properties of the FindItems call by specifying a property set, and not returning all the properties. If you wanted to return all the properties, you would just remove the line that sets the PropertySet.

static void forumFindConversationItem(ExchangeService service)
    {
        ItemView view = new ItemView(10);

        //Remove the following line if you want to get all the properties for each message. This will limit the properties returned in your results (and save time).
        view.PropertySet = new PropertySet(EmailMessageSchema.Subject, EmailMessageSchema.DateTimeReceived);

        SearchFilter.IsEqualTo conversationFilter =
            new SearchFilter.IsEqualTo(EmailMessageSchema.ConversationId, "AAQkADIwM2ZlM2ZlLWMwYjctNDg2Ny04MDU0LTVkMTFmM2IxY2ZjZQAQANEDR7V/30dphLiNOLSTuxE=");

        FindItemsResults<Item> results = service.FindItems(WellKnownFolderName.Inbox, conversationFilter, view);
    }

Once you have each ItemID (returned by the code above) you could use the Bind method to get all the properties on each item.

Hope that helps. I'll follow up when the versioning issues for the methods on MSDN have been updated.

Patronage

Thank you, I think I'll have the chance to try this tomorrow :) – Haply

Filtering by ConversationId works against Exchange2013, but not against Exchange2010_SP2, as it delivers this error: "The property cannot be used with this type of restriction.". It seems that the service doesn't like to see a filter on the EmailMessageSchema.ConversationId property :( – Haply

Andrei- I wanted to let you know that the documentation has been updated. ExchangeService methods and the reference pages it links to now indicate that only one of the FindConversation methods was supported in Exchange 2010 SP1 (FindConversation(ViewBase, FolderId)). Also, the applies to line in How to: Get conversation items has been updated (Exchange 2010 was removed). – Patronage

And How to: Work with conversations by using EWS in Exchange has a Version Differences section (at the bottom of the page) that lists current differences from previous versions. – Patronage

Answer 2

More generally, there seems to be an issue with the ConversationID as it is popularly understood and actually used in Exchange itself. This could impact your (or anyone's) development.

I have been working on an email ticketing program in production and was using ConversationID to cluster emails together in a single thread for easy viewing.

But it now appears that there are emails that are not on the same thread -- yes, they share the same Subject and From Address and are sent around the same date/time -- but they do not occur in citation of each other (e.g. one email body in response to an older one is how many construe an "email thread") -- yet these disparate emails have the same exact ConversationID, even though one body has nothing to do with another.

In fact, even if the date/time is not close, for example, a "Thought of the Day" email from the same person, those will sometimes be grouped with the same ConversationID. This may sound useful in that case but is not so useful in the business case of a payroll person sending "RE: 401k".

To be clear, this is not a case-sensitive overmatch, an oversight I had when using Item.ExchangeID before (which is unique if you account for case). Even accounting for case, wholly different email threads have the exact same ConversationID.

This suggests to me one cannot depend upon ConversationID as a GROUP BY clause and must use some additional, custom-created code.

Lastminute