Efficiently populate combobox in Delphi

Question

Need to add many items (more than 10k) in TComboBox (I know that TComboBox is not supposed to hold many items but its not up to me to change this) without adding duplicates. So I need to search the full list before adding. I want to avoid TComboBox.items.indexof as I need a binary search but the binary find is not available in TStrings.

So I created a temporary Tstringlist, set sorted to true and used find. But now assigning the temporary Tstringlist back to TComboBox.Items

(myCB.Items.AddStrings(myList)) 

is really slow as it copies the whole list. Is there any way to move the list instead of copying it? Or any other way to efficient populate my TComboBox?

Distressful

Just fix the underlying problem. 10k items in a combo box is ridiculous. What version of delphi? – Wallsend

@TomBrunberg Assign is even slower than AddStrings – Distressful

@Wallsend Agree that 10k is ridiculous but this does not change the fact that adding items to combobox for a scenario like this looks really inefficient? Unless I miss something.. Delphi version -> XE3 – Distressful

Are you sure that the slowness is in AddStrings? I've tested it under Delphi 2007 with 10.000 items (having length = 20) from a TStringList and it takes less than half second. – Erasmoerasmus

Has Combobox (or its Items) BeginUpdate method? – Arlyne

@MBo: Yes it has, but it's already used in TStrings.AddStrings method (Delphi 2007) – Erasmoerasmus

@ExDev isn't it? If strings are longer, Coming from C++ half a sec still looks really expensive (maybe im wrong) – Distressful

@Arlyne BeginUpdate is already used in AddStrings – Distressful

The main problem is the unreasonable number of items for a combobox. For 10k items, I think that half second is not so much slow, it would be hard to make it faster. You should use a component which allows loading only visible items and I don't think that TComboBox has such feature. – Erasmoerasmus

@Victoria That would may explain the issue, but I call clear before adding the strings, so.. – Distressful

Sounds like you will have to give up. You won't fix the underlying problem so it's game over. – Semiconductor

I can't try it right now, but you could check if sending a CB_INITSTORAGE message might help. Note that they (MSDN) call 100 a large number, showing that a number of 10,000 items is really exorbitant for a combobox. – Sacerdotal

I am really curious as to what type of user would be willing to use an application that makes him scroll through a 10K item combo to choose some value. Is this VCL or FireMonkey? – Fleshings

@Fleshings I have added filters. I also filter the list from the moment user type a letter. Product owner still insists that user should be able to see the full list in case that does not know which filter should apply while does not even know the first letter of the item user looks for. I disagree, but convincing product owners that a 10k combobox is stupid is not one my strongest skills. Im more interested on the technical side and how to make efficient something that even from a business point of view is wrong. Based on the answer below, moving the list is not possible so I just give up! – Distressful

For this type of functionality you should really (having the right amount of time) find a better component or write your own. You can very well put nice and functional UI elements in a dropdown and make it look like a combo on steroids. – Fleshings

Answer 1

As Rudy Velthuis already mentioned in the comments and assuming you are using VCL, the CB_INITSTORAGE message could be an option:

SendMessage(myCB, CB_INITSTORAGE, myList.Count, 20 * myList.Count*sizeof(Integer));

where 20 is your average string length.

Results (on a i5-7200U and 20K items with random length betwen 1 and 50 chars):

• without CB_INITSTORAGE: ~ 265ms
• with CB_INITSTORAGE: ~215ms

So while you can speed up things a little by preallocating the memory, the bigger issue seems to be the bad user experience. How can a user find the right element in a combobox with such many items?

Toothless

Used a TStopWatch to measure the times. – Toothless

Answer 2

There is no way to "move" the list into the combo box because the combo box's storage belongs to the internal Windows control implementation. It doesn't know any way to directly consume your Delphi TStringList object. All it offers is a command to add one item to the list, which TComboBox then uses to copy each item from the string list into the system control, one by one. The only way to avoid copying the many thousands of items into the combo box is to avoid the issue entirely, such as by using a different kind of control or by reducing the number of items you need to add.

A list view has a "virtual" mode where you only tell it how many items it should have, and then it calls back to your program when it needs to know details about what's visible on the screen. Items that aren't visible don't occupy any space in the list view's implementation, so you avoid the copying. However, system combo boxes don't have a "virtual" mode. You might be able to find some third-party control that offers that ability.

Reducing the number of items you need to put in the combo box is your next best option, but only you and your colleagues have the domain knowledge necessary to figure out the best way to do that.

Boudicca

It is true that the VCL's TComboBox does not have a virtual mode, but it does have owner-draw modes, so you don't have to load the actual strings into it, you could just load empty placeholders and then draw the actual strings when needed. Otherwise, a ComboBox is just a glorified Edit+ListBox combination, which you could simulate manually in a custom TForm, and TListBox does have a virtual mode. – Script

Answer 3

Notwithstanding that 10k items is crazy to keep in a TComboBox, an efficient strategy here would be to keep a cache in a separate object. For example, declare :

    { use a TDictionary just for storing a hashmap }        
    FComboStringsDict : TDictionary<string, integer>;

where

procedure TForm1.FormCreate(Sender: TObject);
var
  i : integer;
  spw : TStopwatch;
begin
  FComboStringsDict := TDictionary<string, integer>.Create;
  spw := TStopwatch.StartNew;
  { add 10k random items }
  for i := 0 to 10000 do begin
    AddComboStringIfNotDuplicate(IntToStr(Floor(20000*Random)));
  end;
  spw.Stop;
  ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;

function TForm1.AddComboStringIfNotDuplicate(AEntry: string) : boolean;
begin
  result := false;
  if not FComboStringsDict.ContainsKey(AEntry) then begin
    FComboStringsDict.Add(AEntry, 0);
    ComboBox1.Items.Add(AEntry);
    result := true;
  end;
end;

Adding 10k items initially takes about 0.5s this way.

{ test adding new items }
procedure TForm1.Button1Click(Sender: TObject);
var
  spw : TStopwatch;
begin
  spw := TStopwatch.StartNew;
  if not AddComboString(IntToStr(Floor(20000*Random))) then
    ListBox1.Items.Add('Did not add duplicate');
  spw.Stop;
  ListBox1.Items.Add(IntToStr(spw.ElapsedMilliseconds));
end;

But adding each subsequent item is very fast <1ms. This is a clumsy implementation, but you could easily wrap this behaviour into a custom class. The idea is to keep your data model as separate from the visual component as possible - keep them in sync when adding or removing items but do your heavy searches on the dictionary where the lookup is fast. Removing items would still rely on .IndexOf.

Wallsend