JavaMail search by ReceivedDate, doesn't work down to the second?
Asked Answered
A

1

10

In IMAP, message numbering is based on when a message is placed into a folder (i.e. first message in the folder is 1, second message in is 2, etc.). However, this ordering has no regard for a message's received date. What I'm concerned about are messages older than message number 1 of the folder, i.e. a message that's been moved into a folder by a user manually (and not by when it was received into the system).

I cannot just get every message's receivedDate, as this is horrendoulsy inefficent.

I thought I could do a JavaMail search to get all messages with a received date older than the first message's received date, (and likewise then do a search to get all message with a recieved date newer than the first message's received date... which would be many, but while I need to process ALL of the older ones, I only need to process a few of the newer ones, (and by process, I mean download its headers).

However, when I am using the search, it just doesn't seem to work right.

        Message[] messages = csqFolder.getMessages();           

        if (messages != null && messages.length > 0) {

            Date receivedDate = messages[0].getReceivedDate();
            log.trace("Message 1 receivedDate: <" + receivedDate.toString() + ">");
            SearchTerm msgsOlderThanOETFirst = 
                new ReceivedDateTerm(DateTerm.LT, receivedDate);

            SearchTerm msgsNewerThanFirst = 
                new ReceivedDateTerm(DateTerm.GT, receivedDate);

            Message[] oldMsgs = csqFolder.search(msgsOlderThanOETFirst, messages);
            log.trace("Size of oldMsgs: <" + oldMsgs.length + ">");
            Message[] newMsgs = csqFolder.search(msgsNewerThanFirst, messages);
            log.trace("Size of newMsgs: <" + newMsgs.length + ">");

However, when I run these searches, it does not seem to give the correct results.

On a folder with 8 messages, 7 of which have a received date time of around 12:00 pm on Aug 5, but the first message in the folder which is at 4:00 pm on Aug 5, we will see the following:

Message 1 receivedDate: <Fri Aug 05 16:46:57 CDT 2011>
Size of oldMsgs: <0>
Size of newMsgs: <7>

However, ALL of the remaining seven messages are older than the first message... they should ALL be in oldMsgs.

That said, IF I put a message from the previous day (Aug 4) in that folder, then the search will coreectly work... for that ONE message. It's like the search only works on a day-by-day basis, not down to the second...

I should note that in the term msgsOlderThanOETFirst, I originally was using LE (hence the name), however, that would flip the above resutls -- ALL the messages would then be found in oldMsgs.

Can anyone else confirm if this behavior is true, and if so, would it point to a bug in Java?

I tried to look into the source code of search but I think I was getting a command line version of it and not the kind that the javamail package uses....

I'm using MS Exchange 2007 with SP1, and JavaMail 1.4.3.

Thanks for any suggestions.

UPDATE: I don't think my reading of the comparator order is wrong. Look the answer given in the following thread: java imap fetch messages since a date

Suppose we use that answer, and we have our boundaries from 7/15/2011 to 9/15/2011. Then we have the search compare a message with a given date of 8/4/2011. Then we would have the following:

                                              8/4/2011  <     9/15/2011
SearchTerm olderThen = new ReceivedDateTerm(ComparisonTerm.LT, someFutureDate);
SearchTerm newerThen = new ReceivedDateTerm(ComparisonTerm.GT, somePastDate);
                                              8/4/2011  >     7/15/2011

This would evaluate to true in both cases, which is what we expect, since the date is in the desired range.

Likewise, my code is as follows:

    SearchTerm msgsOlderThanOETFirst =    8/4/2011  < 8/5/2011
            new ReceivedDateTerm(DateTerm.LT, receivedDate); --> TRUE
    SearchTerm msgsNewerThanFirst =       8/4/2011  > 8/5/2011
            new ReceivedDateTerm(DateTerm.GT, receivedDate); --> FALSE

The above evaluations to TRUE and FALSE are what I am expecting, and will receive. However, If we take a given message at 8/5/2011 12:00:00, we get the following:

    SearchTerm msgsOlderThanOETFirst =    8/5/2011 12:00:00  < 8/5/2011 16:46:00
            new ReceivedDateTerm(DateTerm.LT, receivedDate); --> TRUE?
    SearchTerm msgsNewerThanFirst =       8/4/2011 12:00:00  > 8/5/2011 16:46:00
            new ReceivedDateTerm(DateTerm.GT, receivedDate); --> FALSE?

Except, no -- we don't get that -- instead, we get the reverse..... it's easy to think yourself in circles at this point, but... I've double-checked this several times. Is there something wrong with JavaMail, or am I utterly confused? Please correct my confustion if it is the latter!

Are answered 5/8, 2011 at 22:26 Comment(7)
In your code sample you are getting the messages for the variable folder but performing a search on the csqFolder. Is this just a typo on your example?Rogan
You could try using a differnt implementation of the JavaMail API for which the source is available so you could debug the issue e.g. jarvana.com/jarvana/archive-details/org/apache/geronimo/specs/…. No guarantees of the same behavior between implementations tho.Rogan
Yes that was a typo. Fixed. Well, I'm leaning toward the notion that this is a bug right now, and if it is, it doesn't pay for me to try and debug. If a fix for this was put in an official release of JavaMail, I could use it, however.Are
Can you provide an output where put display the Recieved Date of each of the messages in the Message[]Rogan
I do have such an output. In my test of 7 messages, it shows all the messages that I am expecting to be in OldMsg to actually be in newMsgs. And obviously oldMsgs is empty. What are you looking for specifically? I can't give you that exact output anymore without creating a new test (messages in example have since been deleted).Are
Can you loop thru the message array and log the recievedDate from each of the messages in the arrary.Rogan
Yes, I did do this. The messages displayed are what I expected -- they're from the folder in question, but, not according the search term. I didn't show this when I made the question, although those messages no longer exist, so I can't go back and display that output. However, there was a roughly 5 hour gap in time between the oldest and newest message.Are
A
8

According to JM developer Bill Shannon, this is a limitation of the IMAP protocol:

from    Bill Shannon 
to  Jonathan Hanson 
cc  [email protected]
date    Wed, Aug 10, 2011 at 11:55 AM
subject Re: Bug with searching by ReceivedDateTerms
mailed-by   oracle.com
    Important mainly because of your interaction with messages in the conversation.

hide details 11:55 AM (16 minutes ago)

The searching is being done by the IMAP server, and you're running into a limitation of the IMAP protocol.
Are answered 10/8, 2011 at 17:14 Comment(3)
The limitation being that IMAP SEARCH BEFORE only accepts dates (without time), not date+time, right? (To quote RFC 3501: "BEFORE <date> - Messages whose internal date (disregarding time and timezone) is earlier than the specified date.")Osset
What is the solution of this problem? I really want to filter mails based on time, I want mails received only in last 2 mins or so, how can to do it?Snicker
The solution is to not use IMAP searching, it is rubbish. The best solution would be to get IMAP protocol updated or else use a different protocol (like MAPI -- which isn't available in JavaMail). If you must use IMAP as is, you're better off working under the understanding that IMAP folders function as a FIFO queue. Keep a memory map of your messages and track new ones every 2 min.Are

© 2022 - 2024 — McMap. All rights reserved.