Paginate on LDAP server which does not support PagedResultsControl

Question

I'm trying to get all entries on an LDAP server using Spring LDAP (version 2.3.2). Within my code, I make use of PagedResultsDirContextProcessor to paginate through all the result. This works fine on the servers which support PagedResultsControl.

However, I now need to connect to an LDAP server which does not support PagedResultsControl. How can I get all entries without using PagedResultsControl?

Gora

I have found that I can use Virtual List View - is this something supported in Spring LDAP? – Gora

Answer 1

You can use VirtualListView via JNDI. You have to retrieve and re-supply the 'contextID' to paginate, as follows:

static final int LIST_SIZE = 20; // Adjust to suit

@Test
public void TestVLV() throws NamingException, IOException
{

    Hashtable<String,Object> env = new Hashtable<>();

    env.put(Context.INITIAL_CONTEXT_FACTORY,
        "com.sun.jndi.ldap.LdapCtxFactory");

    env.put(Context.PROVIDER_URL, "ldap://localhost");
    env.put(Context.SECURITY_AUTHENTICATION, "simple");
    env.put(Context.SECURITY_PRINCIPAL, "cn=XXXXXXX");
    env.put(Context.SECURITY_CREDENTIALS, "YYYYYYY");

    try
    {
        /* Create initial context with no connection request controls */
        LdapContext ctx = new InitialLdapContext(env, null);

        /* Sort Control is required for VLV to work */
        SortKey[] sortKeys =
        {
            // sort by cn 
            new SortKey("cn", true, "caseIgnoreOrderingMatch")
        };
        // Note: the constructors for SortControl that take String or String[]
        // as the first argument produce 'no ordering  rule' errors with OpenLDAP.
        SortControl sctl = new SortControl(
            // "cn",
            //  new String[]{"cn"},
            sortKeys,
            Control.CRITICAL);

        /* VLV that returns the first 20 answers */
        VirtualListViewControl vctl =
            new VirtualListViewControl(1, 0, 0, LIST_SIZE-1, Control.CRITICAL);

        /* Set context's request controls */
        ctx.setRequestControls(new Control[]
            {
                sctl,
                vctl
            });

        int count = 0;
        SearchControls  sc = new SearchControls(SearchControls.SUBTREE_SCOPE, 0, 0, null, false, false);
        for (;;)
        {
            /* Perform search */
//          System.out.println("namespace="+ctx.getNameInNamespace());
//          System.out.println("count limit="+sc.getCountLimit());
//          System.out.println("search scope="+sc.getSearchScope());
            NamingEnumeration<SearchResult> ne =
                ctx.search("ou=Users,dc=xxxx,dc=com", "(objectClass={0})", new String[]{"inetOrgPerson"}, sc);

            /* Enumerate search results */
            while (ne.hasMore())
            {
                count++;
                SearchResult sr = ne.next();
//              System.out.println(i+": "+sr.getName());
                System.out.println(count+": "+sr.getNameInNamespace());
            }

            ne.close();

            // Get the contextID.
            Control[] controls = ctx.getResponseControls();
            VirtualListViewResponseControl vlvrc = null;
            byte[] contextID = null;
            for (int j = 0; j < controls.length; j++)
            {
                if (controls[j] instanceof VirtualListViewResponseControl)
                {
                    vlvrc = (VirtualListViewResponseControl)controls[j];
                    contextID = vlvrc.getContextID();
                    System.out.println("contextID=0x"+new BigInteger(1,contextID).toString(16));
                    if (contextID != null)
                    {
                        vctl = new VirtualListViewControl(vlvrc.getTargetOffset()+LIST_SIZE, 0, 0, LIST_SIZE-1, Control.CRITICAL);
                        vctl.setContextID(contextID);
                        ctx.setRequestControls(new Control[]
                                        {
                                            sctl,
                                            vctl
                                        });
                    }
                    break;  // there should only be one VLV response control, and we're not interested in anything else.
                }
            }
            if (vlvrc != null && contextID != null && count < vlvrc.getListSize())
            {
                System.out.println("Continuing");
            }
            else
            {
                System.out.println("Finished");
                break;
            }
        }

        ctx.close();

    }
    finally
    {
    }
}

Adjust the authentication and search root and filter to suit yourself, of course.

And to test whether it is supported (although an 'unsupported critical control' exception from the above code will tell you just as well):

/**
 * Is VLV Control supported?
 *
 * Query the rootDSE object to find out if VLV Control is supported.
 * @return true if it is supported.
 */
static boolean isVLVControlSupported(LdapContext ctx)
    throws NamingException
{
    String[]    returningAttributes = 
    {
        "supportedControl"
    };

    // Fetch the supportedControl attribute of the rootDSE object.
    Attributes  attrs = ctx.getAttributes("", returningAttributes);
    Attribute   attr = attrs.get("supportedControl");
    System.out.println("supportedControls="+attr);
    if (attr != null)
    {
        // Fast way to check. add() would have been just as good. Does no damage to the DIT.
        return attr.remove(VLV_CONTROL_OID);
    }
    return false;
}

The VirtualListViewControl and VirtualListViewResponseControl are part of the Sun/Oracle LDAP Booster Pack, which you can obtain via Maven as:

    <dependency>
        <groupId>com.sun</groupId>
        <artifactId>ldapbp</artifactId>
        <version>1.0</version>
        <type>jar</type>
    </dependency>

Basel

This is awesome. I ended up using a version of this from spring ldap. github.com/spring-projects/spring-ldap/blob/master/sandbox/src/… – Gora

Well great, but your question says it is not officially supported, and you asked how to do it with the javax package, so I answered that. – Basel

Answer 2

Super frustrating..

I wouldn't really recommend it, but you could pull off something like this where you manually paginate by cn/sn..

List<String> alphabetRange = getAlphabetRange();
for (int i = 0; i < alphabetRange.size() - 1; i++) {
    String filter = "(&(sn>=" + alphabetRange.get(i) + ")" + "(sn<=" + alphabetRange.get(i + 1) + " ))";
    NamingEnumeration<SearchResult> searchResult = context.search(base_dn, filter, controls);
    while (searchResult.hasMore()) {
     // searchResult.next().getAttributes() and do something with it
    }
}

private List<String> getAlphabetRange() {
    List<String> result = new ArrayList<>();
    for (char alph = 'A'; alph <= 'Z'; alph++) {
        if (alph == 'S') {
            result.add("S");
            result.add("Sd");
        } else {
            result.add(String.valueOf(alph));
        }
    }
    result.add("Zz");
    return result;
}

• as soon as you run over (i.e. 1.000) in one "page", you will face the dreaded javax.naming.SizeLimitExceededException. You might add more "pages", like in that example between [S-Sd][Sd-T]

Medora

Answer 3

What exacly servers are you using?

if server has no queue limits you cant try set unlimited search:

SearchControls controls = new SearchControls();
controls.setTimeLimit(0);
controls.setCountLimit(0);

Juliennejuliet

Hard to see how an answer with no pagination at all answers a question about how to implement pagination. – Basel