JavaMail: Keeping IMAPFolder.idle() alive

Question

I am making a program that needs to monitor a Gmail account for new messages, and in order to get them ASAP I am using JavaMail's idle feature. Here is a code snippet from the thread I am using to call folder.idle():

//Run method that waits for idle input. If an exception occurs, end the thread's life.
public void run() {

    IMAPFolder folder = null;

            try {
                folder = getFolder();
                while(true)
                {
                  //If connection has been lost, attempt to restore it
                  if (!folder.isOpen())
                      folder = getFolder();
                  //Wait until something happens in inbox
                  folder.idle(true);
                  //Notify controller of event
                  cont.inboxEventOccured();
                }
            }
            catch (Exception ex) {
                ex.printStackTrace();
            }
             System.out.println("MailIdleWaiter thread ending.");
}

The getFolder() method basically opens the connection to the IMAP server and opens the inbox.

This works for a while, but after 10 minutes or so it stops getting updates (no exception is thrown).

I am looking for suggestions to keep the connection alive. Do I need a second thread whose only role is to sleep and renew the idle() thread every 10 minutes or is there an easier/better way?

Thanks in advance.

Shirlshirlee

I'm planning to do the same. Were you finally able to solve the issue? Currently, I'm polling the folder via 'folder.open/folder.close' every 15 secs, but IDLE would be better of course. I'm planning to use this in an app server environment. – Exile

Sorry for not spotting your comment sooner. I ended up ditching the project, so I never got closer to a solution. But now that this thread has an answer, maybe that will work... although it's based on polling and not idle. – Shirlshirlee

You have to both poll and be idle to do it properly. IDLE has to be terminated and renewed every half-hour according to the spec and more often if a broken NATbox is in the way. The right interval is... well, maybe there isn't a One Correct Value. – Barrelhouse

Any good solutions to this? I am having a similar issue myself. I want to wait idly until a new message is inserted into the inbox folder, while keeping the connection alive. I do not want to use polling. – Gesualdo

Answer 1

A common mistake is to assume an IDLE command will keep posting updates indefinitely. However, the RFC 2177, that defines the IDLE extension states:

The server MAY consider a client inactive if it has an IDLE command running, and if such a server has an inactivity timeout it MAY log the client off implicitly at the end of its timeout period. Because of that, clients using IDLE are advised to terminate the IDLE and re-issue it at least every 29 minutes to avoid being logged off. This still allows a client to receive immediate mailbox updates even though it need only "poll" at half hour intervals.

GMail in particular, has a much lower timeout, as you say, around 10 minutes.

We simply need to reissue the IDLE command every 9 minutes or so for it to work. The javax.mail APIs have no way to set a timeout for the IDLE command, so you will need a second thread to move around this.

A first approach would be to have the second thread interrupt the first one, handling the exception and ignoring it. This however, would allow for no clean way to shutdown the thread, so I won't recomend it. A much cleaner way is to have the second thread issue a NOOP command to the server. This does nothing at all, but is enough to have IDLE abort and be reissued.

I here provide some code to do this:

public void startListening(IMAPFolder imapFolder) {
    // We need to create a new thread to keep alive the connection
    Thread t = new Thread(
        new KeepAliveRunnable(imapFolder), "IdleConnectionKeepAlive"
    );

    t.start();
    
    while (!Thread.interrupted()) {
        LOGGER.debug("Starting IDLE");
        try {
            imapFolder.idle();
        } catch (MessagingException e) {
            LOGGER.warn("Messaging exception during IDLE", e);
            throw new RuntimeException(e);
        }
    }
    
    // Shutdown keep alive thread
    if (t.isAlive()) {
        t.interrupt();
    }
}

/**
 * Runnable used to keep alive the connection to the IMAP server
 * 
 * @author Juan Martín Sotuyo Dodero <[email protected]>
 */
private static class KeepAliveRunnable implements Runnable {

    private static final long KEEP_ALIVE_FREQ = 300000; // 5 minutes

    private IMAPFolder folder;

    public KeepAliveRunnable(IMAPFolder folder) {
        this.folder = folder;
    }

    @Override
    public void run() {
        while (!Thread.interrupted()) {
            try {
                Thread.sleep(KEEP_ALIVE_FREQ);
                
                // Perform a NOOP just to keep alive the connection
                LOGGER.debug("Performing a NOOP to keep alvie the connection");
                folder.doCommand(new IMAPFolder.ProtocolCommand() {
                    public Object doCommand(IMAPProtocol p)
                            throws ProtocolException {
                        p.simpleCommand("NOOP", null);
                        return null;
                    }
                });
            } catch (InterruptedException e) {
                // Ignore, just aborting the thread...
            } catch (MessagingException e) {
                // Shouldn't really happen...
                LOGGER.warn("Unexpected exception while keeping alive the IDLE connection", e);
            }
        }
    }
}

Junji

In a scenario where you use an ExecutorService (in particular in JEE), you may want to use Object.wait() rather than Thread.sleep() so you can interrupt it also without being in control of the Thread. – Intoxicating

catch (InterruptedException e) should contain a break, to avoid a new thread being created at each call of startListening(). The NOOP can be replaced with folder.exists() (this way we have less com.* imports). A MessageCountListener should be added that does the same when the messagesAdded event is fired so that one can also call folder.exists() which results in the idle() to end immediately. And finally folder.getUnreadMessageCount() needs to be called to avoid staying there forever. – Unshackle

Correcting myself: The MessageCountListener isn't needed when calling folder.idle(true). – Unshackle

Answer 2

The suggestion by @user888307 is a dirty hack and eventually fail miserably. There is really only one proper way of doing this.

Call the idle(false) method on the folder that's currently selected. Ideally Inbox because that will receive all messages.

Calling idle(false) will basically hang the runtime of the thread, so better to put idle(false) on a new thread. Then once you receive a new email/notification using messageCountChange, you have to rerun this thread.

This is the only true way of achieving this. I have written a wrapper for your explicit problem as I am writing a program called JavaPushMail. You can find more info on my website (http://www.mofirouz.com/wordpress) or you can grab the application (which is currently in development) on GitHub https://github.com/mofirouz/JavaPushMail

Moccasin

Two questions: 1) is the messageCountChange called on the same idled thread or on a new one? 2) how do you renew the idle to prevents timeouts? Thanks. – Meredi

The javadoc @Mo you don't really supply an answer to the question. You give a link to a github repo and a wordpress blog. Calling idle() and idle(false) will both hang the runtime and need to be run on their own thread. You also say "the above suggestion" but since there is voting on S.O. there is no guarantee that the above answer will remain above. – Brebner

LOL both links no longer work. – Unshackle

Answer 3

Actually the Java Mail samples include an IMAP IDLE example, which is as follows. Besides that, the IdleManager class might be interesting.

/*
 * Copyright (c) 1996-2010 Oracle and/or its affiliates. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Oracle nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

import java.util.*;
import java.io.*;
import javax.mail.*;
import javax.mail.event.*;
import javax.activation.*;

import com.sun.mail.imap.*;

/* Monitors given mailbox for new mail */

public class monitor {

    public static void main(String argv[]) {
    if (argv.length != 5) {
        System.out.println(
        "Usage: monitor <host> <user> <password> <mbox> <freq>");
        System.exit(1);
    }
    System.out.println("\nTesting monitor\n");

    try {
        Properties props = System.getProperties();

        // Get a Session object
        Session session = Session.getInstance(props, null);
        // session.setDebug(true);

        // Get a Store object
        Store store = session.getStore("imap");

        // Connect
        store.connect(argv[0], argv[1], argv[2]);

        // Open a Folder
        Folder folder = store.getFolder(argv[3]);
        if (folder == null || !folder.exists()) {
        System.out.println("Invalid folder");
        System.exit(1);
        }

        folder.open(Folder.READ_WRITE);

        // Add messageCountListener to listen for new messages
        folder.addMessageCountListener(new MessageCountAdapter() {
        public void messagesAdded(MessageCountEvent ev) {
            Message[] msgs = ev.getMessages();
            System.out.println("Got " + msgs.length + " new messages");

            // Just dump out the new messages
            for (int i = 0; i < msgs.length; i++) {
            try {
                System.out.println("-----");
                System.out.println("Message " +
                msgs[i].getMessageNumber() + ":");
                msgs[i].writeTo(System.out);
            } catch (IOException ioex) { 
                ioex.printStackTrace(); 
            } catch (MessagingException mex) {
                mex.printStackTrace();
            }
            }
        }
        });

        // Check mail once in "freq" MILLIseconds
        int freq = Integer.parseInt(argv[4]);
        boolean supportsIdle = false;
        try {
        if (folder instanceof IMAPFolder) {
            IMAPFolder f = (IMAPFolder)folder;
            f.idle();
            supportsIdle = true;
        }
        } catch (FolderClosedException fex) {
        throw fex;
        } catch (MessagingException mex) {
        supportsIdle = false;
        }
        for (;;) {
        if (supportsIdle && folder instanceof IMAPFolder) {
            IMAPFolder f = (IMAPFolder)folder;
            f.idle();
            System.out.println("IDLE done");
        } else {
            Thread.sleep(freq); // sleep for freq milliseconds

            // This is to force the IMAP server to send us
            // EXISTS notifications. 
            folder.getMessageCount();
        }
        }

    } catch (Exception ex) {
        ex.printStackTrace();
    }
    }
}

Gennagennaro

Answer 4

You can register your folder to a connectionListener.

var folder = store.getFolder("<FOLDER_NAME>");
folder.addConnectionListener(new javax.mail.event ConnectionAdapter() {
        public void closed(ConnectionEvent e) {
            try {
                log.info("Folder connection closed. Reconnect to server");
                
                // reopen connection 
                connectToServer();
            } catch (Exception exception) {
                log.error("Could not connect to server={0}", exception);
            }
        }
    });

Elaterite

Answer 5

checking the message count every 5 minutes works for me:

new Thread()
{
    @Override
    public void run()
    {
        startTimer();
    }
    private void startTimer()
    {
        int seconds = 0;
        while (true)
        {
            try
            {
                Thread.sleep(300000);
                int c = folder.getMessageCount();    
            }
            catch (InterruptedException ex)
            {
            }
            catch (MessagingException me)
            {
            }
        }
    }
}.start();

Moreland

This is a poll loop instead of idle. It inefficient, both for the server, and for bandwidth usage, as well as not getting instant notifications. – Override

Also, please NEVER use an empty catch block. Even if you are SURE it would not happen (not this case) it's better to log it. – Toadflax