EWS reports SendItem as succeeding but message is still in Drafts folder

Question

I'm using the EWS Managed API to send emails via exchange. When sending an item with attachments I first do a CreateItem and then a SendItem. Mostly this works fine but sometimes items are left in the Drafts folder, even though EWS reports the SendItem succeeds. How can I tell what's going on here?

The EWS messages I see from tracing are first CreateItem:

  <?xml version="1.0" encoding="utf-8"?>
  <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Header>
      <t:RequestServerVersion Version="Exchange2013" />
    </soap:Header>
    <soap:Body>
      <m:CreateItem MessageDisposition="SaveOnly">
        <m:SavedItemFolderId>
          <t:DistinguishedFolderId Id="drafts">
            <t:Mailbox>
              <t:EmailAddress>[email protected]</t:EmailAddress>
            </t:Mailbox>
          </t:DistinguishedFolderId>
        </m:SavedItemFolderId>

        ...

      </m:CreateItem>
    </soap:Body>
  </soap:Envelope>
</Trace>

I get a Success response:

  <?xml version="1.0" encoding="utf-8"?>
  <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Header>
      <h:ServerVersionInfo MajorVersion="15" MinorVersion="0" MajorBuildNumber="1104" MinorBuildNumber="3" Version="V2_22" xmlns:h="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" />
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <m:CreateItemResponse xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
        <m:ResponseMessages>
          <m:CreateItemResponseMessage ResponseClass="Success">
            <m:ResponseCode>NoError</m:ResponseCode>
            <m:Items>
              <t:Message>
                <t:ItemId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAA=" ChangeKey="CQAAABYAAACJZGY3HQ5zQpbF9WoRXD0sAAA6JC4z" />
                <t:Attachments>
                  <t:FileAttachment>
                    <t:AttachmentId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAABEgAQAGiKX5lgtuFLgfIBgyg4IwM=" />
                  </t:FileAttachment>
                  <t:FileAttachment>
                    <t:AttachmentId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAABEgAQADkNd4D6H0NAgxCM7uH6MGo=" />
                  </t:FileAttachment>
                  <t:FileAttachment>
                    <t:AttachmentId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAABEgAQAKu2RdYFlBVIiLyPxshNCZQ=" />
                  </t:FileAttachment>
                  <t:FileAttachment>
                    <t:AttachmentId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAABEgAQAJ2C2tyz2iJHm8XC9LmSkcA=" />
                  </t:FileAttachment>
                </t:Attachments>
              </t:Message>
            </m:Items>
          </m:CreateItemResponseMessage>
        </m:ResponseMessages>
      </m:CreateItemResponse>
    </s:Body>
  </s:Envelope>
</Trace>

I then ask to send it and move to SentItems:

  <?xml version="1.0" encoding="utf-8"?>
  <soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Header>
      <t:RequestServerVersion Version="Exchange2013" />
    </soap:Header>
    <soap:Body>
      <m:SendItem SaveItemToFolder="true">
        <m:ItemIds>
          <t:ItemId Id="AAMkADExMmY0MTgzLWJmZmUtNDcxNi1iOTk0LWMzZTU3M2I3NjBkMwBGAAAAAABL/dBAdDByRbztYn12Cy/4BwCJZGY3HQ5zQpbF9WoRXD0sAAAAAAEPAACJZGY3HQ5zQpbF9WoRXD0sAAA6HNVuAAA=" ChangeKey="CQAAABYAAACJZGY3HQ5zQpbF9WoRXD0sAAA6JC4z" />
        </m:ItemIds>
        <m:SavedItemFolderId>
          <t:DistinguishedFolderId Id="sentitems">
            <t:Mailbox>
              <t:EmailAddress>[email protected]</t:EmailAddress>
            </t:Mailbox>
          </t:DistinguishedFolderId>
        </m:SavedItemFolderId>
      </m:SendItem>
    </soap:Body>
  </soap:Envelope>
</Trace>

And I get a Success NoError response:

  <?xml version="1.0" encoding="utf-8"?>
  <s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Header>
      <h:ServerVersionInfo MajorVersion="15" MinorVersion="0" MajorBuildNumber="1104" MinorBuildNumber="3" Version="V2_22" xmlns:h="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" />
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <m:SendItemResponse xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
        <m:ResponseMessages>
          <m:SendItemResponseMessage ResponseClass="Success">
            <m:ResponseCode>NoError</m:ResponseCode>
          </m:SendItemResponseMessage>
        </m:ResponseMessages>
      </m:SendItemResponse>
    </s:Body>
  </s:Envelope>
</Trace>

These are all traced by the same managed ThreadId so I'm pretty sure they all correspond to the same item.

I'm not using exchange impersonation. I'm authenticating with username & password auth with credentials for the mailbox that is sending the items.

EDIT (22 May):

I've added X-AnchorMailbox header to my requests and upgraded to EWS Managed API 2.2 (was using 2.1, although since the SOAP messages looked right I don't think this was an issue). The problem is still happening.

I added code to re-retrieve the item after sending, to check if it's still in Drafts folder. I do this through EmailMessage.Bind() with the Id of the item just sent from drafts. When I run against Exchange on Office 365 that always raises an exception because the item no longer exists, which is expected since the ItemId changes when the item moves folder. When I run against customer's Exchange server it finds the item by that Id. I then try to resend it and get a ServiceResponseException with message Access is denied. Check credentials and try again. If I try to resend a message against Office 365 I get a SendItemResponse with message The specified object was not found in the store., The process failed to get the correct properties. (SOAP message is SendItemResponse with ResponseCode ErrorItemNotFound.

I've raised a support request with Microsoft (which oddly hasn't been responded to in over 48 hours ... maybe there's some trick to raising support requests).

EDIT (24 May)

When repeating the tests I sometimes saw items that remain in Drafts BUT are also received by the recipient! When inspecting their Properties > Internet Headers via outlook they both have the same Message-ID. More indication that something is messed up on the Exchange side of things?

EDIT (30 May)

I heard from Microsoft support who explained a bit about how sending in exchange works and what they think is happening:

a No error response from EWS means that the message was successfully flagged for submissions (PR_MESSAGE_FLAGS > MSGFLAG_SUBMIT). When the submission flag is set, the Mail Submission service on Exchange notifies the Transport Service that there are messages awaiting to be picked up in the Drafts (Outbox if Outlook). If any client reads the message after it’s been flagged for submission but before it’s picked up by the Transport service, the MSGFLAG_SUBMIT flag is removed which leads to the message being stuck in the Drafts folder.

  In order to determine if this is the case, you can sign up for notifications using Streaming Notifications (and a tool the likes of EWSEditor or EwsStreaming) and check for any item Modified events and check for any PR_MESSAGE_FLAGS changes.

  As discussed, the message could have been accessed by any client or API before the transport service went to pick it up, such as OWA, Outlook, EWS Managed API, Outlook add-ins or any mobile apps.

So next task is twofold:

1. Identify if anything is reading my messages immediately after they've flagged to be sent, resetting the MSGFLAG_SUBMIT flag. If so, ask them to please stop.

2. Include a process to re-send items that I think have been sent that I later find in Drafts.

I didn't get any information about why I'd get an Access Denied when trying to resend, but perhaps that's consistent with something else modifying the item.

They didn't think there was any chance this is a problem in EWS, where EWS is incorrectly reporting success when actually the message hasn't been flagged to be sent. Which is reassuring.

EDIT (20 June):

Introducing a process to re-send items that are stuck in Drafts didn't work very well. It turns out that sometimes when items are left in Drafts they've actually been sent and received by the recipient, so re-sending them means the recipient gets a duplicate of the email. Or several, if the same problem happens again. Sometimes the item isn't sent but is still removed from Drafts and moved to Sent Items - again it's not possible for the sender to work out whether it's sent or not. Totally weird. I've asked another question here about specifically whether Exchange message sending is expected to be atomic and reliable.

---

Also posed on Exchange forums

Cusped

Are you setting the X-AnchorMailbox header to the Mailbox your trying to access ? – Amerind

No but I'm not using EWS Impersonation so didn't think that was necessary. – Cusped

You should be setting it on every request regardless of impersonation as it affects the way requests are routed when your accessing a mailbox other then the security principal your using. I would suggest you give that a try as this sounds like a routing issue to me. – Amerind

ok, sounds like it's good to add so I will. But in this case I'm logging in with username & password as the mailbox owner so presumably it shouldn't affect things? – Cusped

Any idea how I can investigate what's going on at the Exchange end of things? – Cusped

Check the EWSLog on the CAS server also consider updating to the latest CU it maybe happening due to similar bug that was fixed in CU10 support.microsoft.com/en-us/kb/3076257 you appear to be a CU9 – Amerind

@GlenScales thanks! The exchange environment is owned by my customer so it'll be difficult to update. I'll get EWSLogs, see if X-AnchorMailbox helps, and raise a support request from MS to see if they've had any other reports since it's similar to that bug kb3076257. In response headers I see X-FEServer varies between responses, is that significant and is it affected by X-AnchorMailbox? – Cusped

You need to add the MAPI extended property 'MessageFlags' when creating the item and set it to 'Read' - thus your item won't appear as draft. Can share the code later if needed - on my mobile right now. – Alby

@SergeSemenov - that'd be great, although I'm fine with the item being shown as Draft initially, it's when I've asked for SendAndSaveCopy that it should no longer be Draft. So it doesn't sound like setting MessageFlags is what I need but I'm open to all options right now. – Cusped

Do you need to do this in two steps? If not I'd suggest calling CreateItem with MessageDisposition="SendAndSaveCopy". – Oldster

@Oldster - Yeah, if there's attachments then that's an illegal operation: you have to save as draft first and then send. – Cusped

Can you test to set a timeout between the CreateItem success response and the send action. Exchange might not have saved the new items attachement completely despite you receiving a success response. – Godhead

@MarcusH - yeah, I was assuming because EWS isn't amazingly fast that the network latency would itself provide a little timeout. But (as per my question edit just now) it looks like something else is messing with my messages, so as well as retrying them I'll try a little timeout to give whatever it is some time to play with my draft before I send it. – Cusped

@MarcusH - I tried sending all messages in a two-step process via Drafts instead of just doing that when necessary because of attachments. I think the same happens - i.e. messages stuck in drafts - even for messages with no attachments, so it's not caused by that. – Cusped

Ok..What happens if you try to resend a message that didn't get sent the first time? Does all messages work then? – Godhead

The problems only happen in a customer's on-premise environment, where I get a ServiceResponseException with message Access is denied. Check credentials and try again. when I try to resend a message I've just sent. I haven't yet been able to try pausing between Save and Send or between Send and Resend. Will do tomorrow. – Cusped

Answer 1

I didn't solve this, except to ensure as far as possible that no other apps are interacting with the mailbox at the same time as advised by MS support. We changed permissions on the mailbox to stop some people who had access to the shared mailbox and would have had it available through outlook. This totally fixed the problem for the main customer who experienced the problem. BUT it's still totally weird to me that EWS/exchange can be so unreliable and I'd never choose to use it again given this sort of problem. It also doesn't explain the cases where a message was delivered but remained in the Drafts folder, meaning it's not possible to be sure of whether sending succeeded or failed.

Cusped

Answer 2

Taken from MSDN documentation on SendItem:

An additional scenario to consider is when a delegate creates an e-mail message and saves it to the Drafts folder of the delegate's mailbox. If the delegate tries to send the item and save a copy to the principal's Sent Items distinguished folder, the message is sent correctly, the draft message remains in the delegate's Drafts folder, the sent message does not appear in either the delegate's or principal's Sent Items folder, and the response is a success.

The documentation seems to say you're out of luck... The best work around may be to perform the send and move to sent items as two separate functions as well.

Oldster

Hmmm, I hadn't seen that but I think it doesn't explain this case. Firstly my results are that it sometimes all works fine and sometimes doesn't, so it doesn't feel like expected behaviour. But also I'm logging in as the mailbox Principal and sending from that same mailbox (not Send on behalf of or Sending as). Does the 'delegate' term in that doc refer to someone else with permissions to the mailbox but not the owner? – Cusped

oh, and I'm already doing the 'workaround': to specify the principal's mailbox in the DistinguishedFolderId element (for the SendItem call). – Cusped

I wish i could explain the apparently indeterminate behaviour you're experiencing but sadly i can't... – Oldster

I wish you could too! :) Thanks for that though, it does sound related so I'll explore some variations around it and see if I get different behaviour. – Cusped

I'm awarding the bounty as this was the most helpful answer BUT it's not the reason for the problem occurring. – Cusped

Have you been able to either fix the issue or find a reasonable workaround for this? I am quite interested in understanding this problem and learning about the solution. – Oldster

So far still no solution, but the workaround I'm going for is a) configuration settings that allow me to tweak the wait period between save & send, wait between send and retry sending, and b) a background thread that watches for and resends things in the Drafts folder older than X seconds. But I'd love to determine the root cause. – Cusped

May I suggest that you post your workaround as an answer to your question? – Oldster

Answer 3

Set the 'MessageFlags' MAPI extended property to 'Read', so the item won't be marked as 'Draft'.

Here is the extended property definition:

/// <summary>
/// Definitions of Microsoft's Mail API (MAPI) extended properties that you can get/set on mailbox items via EWS interface
/// </summary>
public static class MapiSchema
{
    /// <summary>
    /// Definition of MAPI property unique identifiers
    /// </summary>
    public static class Tags
    {
        /// <summary>
        /// Message flags (like 'read', 'unsent'; see PR_MESSAGE_FLAGS online) 
        /// </summary>
        public const int MessageFlags = 0x0E07;
    }

    /// <summary>
    /// Message flags (like 'read', 'unsent'; see PR_MESSAGE_FLAGS online) 
    /// </summary>
    public static readonly ExtendedPropertyDefinition MessageFlags = new ExtendedPropertyDefinition(Tags.MessageFlags, MapiPropertyType.Integer);
}

Here are possible values (flags) for this property:

/// <summary>
/// Values for the PR_MESSAGE_FLAGS message property (Microsoft's Message API)
/// </summary>
[Flags]
public enum MessageFlags
{
    /// <summary>
    /// No flags set
    /// </summary>
    None = 0,

    /// <summary>
    /// [MSGFLAG_READ] The message is marked as having been read. This can occur as the result of a call at any time to IMessage::SetReadFlag or IMAPIFolder::SetReadFlags. Clients can also set this flag by calling a message's IMAPIProp::SetProps method before the message has been saved for the first time. This flag is ignored if the MSGFLAG_ASSOCIATED flag is set.
    /// </summary>
    Read = 1,

    /// <summary>
    /// [MSGFLAG_UNMODIFIED] The outgoing message has not been modified since the first time that it was saved; the incoming message has not been modified since it was delivered.
    /// </summary>
    Unmodified = 2,

    /// <summary>
    /// [MSGFLAG_SUBMIT] The message is marked for sending as a result of a call to IMessage::SubmitMessage. Message store providers set this flag; the client has read-only access.
    /// </summary>
    Submit = 4,

    /// <summary>
    /// [MSGFLAG_UNSENT] The message is still being composed. It is saved, but has not been sent. The client or provider has read/write access to this flag until the first IMAPIProp::SaveChanges call and read-only thereafter. If a client doesn't set this flag by the time the message is sent, the message store provider sets it when IMessage::SubmitMessage is called. Typically, this flag is cleared after the message is sent.
    /// </summary>
    Unsent = 8,

    /// <summary>
    /// [MSGFLAG_HASATTACH] The message has at least one attachment. This flag corresponds to the message's PR_HASATTACH (PidTagHasAttachments) property. The client has read-only access to this flag.
    /// </summary>
    HasAttachment = 16,

    /// <summary>
    /// [MSGFLAG_FROMME] The messaging user sending was the messaging user receiving the message. The client or provider has read/write access to this flag until the first IMAPIProp::SaveChanges call and read-only thereafter. This flag is meant to be set by the transport provider.
    /// </summary>
    FromMe = 32,

    /// <summary>
    /// [MSGFLAG_ASSOCIATED] The message is an associated message of a folder. The client or provider has read-only access to this flag. The MSGFLAG_READ flag is ignored for associated messages, which do not retain a read/unread state.
    /// </summary>
    Associated = 64,

    /// <summary>
    /// [MSGFLAG_RESEND] The message includes a request for a resend operation with a nondelivery report. The client or provider has read/write access to this flag until the first IMAPIProp::SaveChanges call and read-only thereafter.
    /// </summary>
    Resend = 128,

    /// <summary>
    /// [MSGFLAG_RN_PENDING] A read report needs to be sent for the message. The client or provider has read-only access to this flag.
    /// </summary>
    ReadReportPending = 256,

    /// <summary>
    /// [MSGFLAG_NRN_PENDING] A nonread report needs to be sent for the message. The client or provider has read-only access to this flag.
    /// </summary>
    NonReadReportPending = 512,

    /// <summary>
    /// [MSGFLAG_ORIGIN_X400] The incoming message arrived over an X.400 link. It originated either outside the organization or from a source the gateway cannot consider trusted. The client should display an appropriate message to the user. Transport providers set this flag; the client has read-only access.
    /// </summary>
    OriginX400 = 4096,

    /// <summary>
    /// [MSGFLAG_ORIGIN_INTERNET] The incoming message arrived over the Internet. It originated either outside the organization or from a source the gateway cannot consider trusted. The client should display an appropriate message to the user. Transport providers set this flag; the client has read-only access.
    /// </summary>
    OriginInternet = 8192,

    /// <summary>
    /// [MSGFLAG_ORIGIN_MISC_EXT] The incoming message arrived over an external link other than X.400 or the Internet. It originated either outside the organization or from a source the gateway cannot consider trusted. The client should display an appropriate message to the user. Transport providers set this flag; the client has read-only access.
    /// </summary>
    OriginOtherExternal = 32768,
}

And here is how you set it before creating an item:

var item = new EmailMessage(service);
var messageFlags = MessageFlags.Read;
item.SetExtendedProperty(MapiSchema.MessageFlags, (int)messageFlags);

Exchange is full of bugs, and I hope this solves the problem.

Alby

Thanks @SergeSemenov, but my problem isn't about it being marked as Read, my problem is that the item remains in Drafts and unsent. I'm instructing EWS to send the item and move it to SentItems, to which is replies Success but actually doesn't do anything. – Cusped

I know that it's not related, I just made a guess - maybe it can solve the problem indirectly, maybe not. At least you can try and see if anything changed. – Alby