Delphi TList of records

Question

I need to store a temporary list of records and was thinking that a TList would be a good way to do this? However I am unsure how to do this with a TList and was wondering if this is the best was and also if anyone has any examples of how to do this?

Vachell

Nobody has explicitly suggested Generics.Collections.TList<T>. Worth considering in my view. – Deadhead

@David: That's definately the way to go. I've found Generics.Collections.TList<T> is WAY faster if it's a large read-only list of records. – Putscher

@David: I suggested it. :-) There are problems still with by-reference and by-value semantics, that would cause a lot of data copying, when adding stuff to the Generics list, though. @Putscher - Yes, for a read-only list of records, it's GREAT, unless you needed to "copy in" a whole lot of records from somewhere else. Then the data-copy penalty might hurt you. – Blight

@Warren All depends on how big the records are. – Deadhead

@David: Yep. 4 byte record = No problem. :-) – Blight

If you need to access to a record in generic TList by-reference and do not copy record: use List.List - direct access to array: lList := TList<TTestRec>.Create; [...] lRecP := @lList.List[i]; (lRecP: PTestRec; ) – Bering

Answer 1

The easiest way is to create your own descendant of TList. Here's a quick sample console app to demonstrate:

program Project1;

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes;

type
  PMyRec=^TMyRec;
  TMyRec=record
    Value: Integer;
    AByte: Byte;
  end;

  TMyRecList=class(TList)
  private
    function Get(Index: Integer): PMyRec;
  public
    destructor Destroy; override;
    function Add(Value: PMyRec): Integer;
    property Items[Index: Integer]: PMyRec read Get; default;
  end;

{ TMyRecList }

function TMyRecList.Add(Value: PMyRec): Integer;
begin
  Result := inherited Add(Value);
end;

destructor TMyRecList.Destroy;
var
  i: Integer;
begin
  for i := 0 to Count - 1 do
    FreeMem(Items[i]);
  inherited;
end;

function TMyRecList.Get(Index: Integer): PMyRec;
begin
  Result := PMyRec(inherited Get(Index));
end;

var
  MyRecList: TMyRecList;
  MyRec: PMyRec;
  tmp: Integer;
begin
  MyRecList := TMyRecList.Create;
  for tmp := 0 to 9 do
  begin
    GetMem(MyRec, SizeOf(TMyRec));
    MyRec.Value := tmp;
    MyRec.AByte := Byte(tmp);
    MyRecList.Add(MyRec);
  end;

  for tmp := 0 to MyRecList.Count - 1 do
    Writeln('Value: ', MyRecList[tmp].Value, ' AByte: ', MyRecList[tmp].AByte);
  WriteLn('  Press Enter to free the list');
  ReadLn;
  MyRecList.Free;
end.

This eliminates a couple of things:

• It handles freeing the memory.
• You don't have to typecast everything to use it.

As Remy and Warren both said, it's a little more work because you have to allocate the memory when you add new records.

Ashwell

You still have to allocate memory yourself manually, using GetMem, everywhere. If someone was to declare a record type on a heap somewhere, and add it to a MyRecList, you would get into trouble. Because this would get the obvious beginner into trouble I am down voting this very clever and interesting piece of code. – Blight

@Warren: I pointed that out in my last paragraph, so your point is moot. You've made your opinion clear in several posts here. Downvoting me because I disagreed with you seems pretty petty, but whatever. – Ashwell

It's about thinking about the new kids. Not petty. You pointed out more work, but not the safety aspects of record types and the stack/heap thing. – Blight

What will happen if user executes MyRecList.Delete(i) ? The right way is to override Delete method or to catch Notify(cnRemoved). Moreover if the record contains dynamic-type fields (dynamic arrays, strings), you'll have to Finalize it before freeing to avoid memory leak. – Transitive

@Fr0sT: Where did I say I'd written a full implementation of everything that would be needed in a real-world application? I posted an example, and examples are intended to be small to illustrate a point. It's typical to leave out most error handling and only implement what's needed for that example. If you want me to write a full implementation for you along with unit tests, contact me and I'll send you my hourly rate. :-) – Ashwell

@KenWhite all I wanted to say is that your example teaches children bad things ;) – Transitive

To avoid allocating memory out of this class one might add (if possible) an Add method (overload) with parameters of all the record fields and allocate that record inside such method. Of course, this would be usable only for records with a few fields. – Prepositor

Answer 2

First, if you want to combine a classic TList with Records, you will need to:

1. Allocate your records on the heap, not on the stack. Use GetMem as Remy did.
2. Take the address of the record and add it to the TList.
3. When removing an item from the list, and using it, dereference it:
4. Remember to free and clean up, afterwards.

Combining Lists with Records requires so much "pointers-and-heap-management" work that such a technique would be only within the capabilities of an expert.

Alternatives to what you have asked for that still use something called "TList", include using a generics.collections style TList, with Record types, which would have all the benefits of TList, but would require you to basically do a lot of entire-record-copies to get data into it.

The most idiomatic Delphi ways to do what you ask are to either:

1. use a TList or TObjectList with a Class Types instead of a record. Usually you end up subclassing either TList or TObjectList in this case.

2. Use a dynamic Array of Record Types, but be aware that it's harder to sort an Array type, and that expanding an array type at runtime isn't as speedy as it is with a TList.

3. Use generics.Collections TList with your classes. This lets you avoid subclassing TList or TObjectList each time you want to use a list with a different class.

A code sample showing Dynamic arrays:

 TMyRec = record
    ///
 end;

 TMyRecArray = array of TMyRec;

 procedure Demo;
 var
    myRecArray:TMyRecArray;
 begin
    SetLength(myRecArray,10);
 end;

Now for some background information on why TList is not easy to use with Record types:

TList is better suited for use with Class types, because a variable of type 'TMyClass', where 'type TMyClass = class .... end;' can be easily "referred to" as a pointer value, which is what TList holds.

Variables of type Record are value-Types in Delphi, whereas class values are implicitly by-reference values. You can think of by-reference values as stealth-pointers. You don't have to dereference them to get at their contents, but when you add it to a TList, you're actually just adding a pointer to the TList, not making a copy or allocating any new memory.

The answer by Remy shows you literally you how to do exactly what you want, and I am writing my answer only because I want to warn you about the details of what you are asking, and suggest that you consider alternatives too.

Blight

@Warren: The problem with arrays is that they aren't as easily sorted. I agree with the class type recommendation, though; you can more easily manage them with TObjectList. (BTW, the ^ is almost entirely optional now; the compiler handles most of that for you and you don't need it.) – Ashwell

@Warren: Dynamic arrays are also not as memory-efficient as TList when it comes to adding and removing items to/from the list. TList grows exponentially, and removes efficiently. Arrays do not. You would have to implement those operatons manually when using arrays. – Carinthia

@Warren: The second problem with your answer (as I read it more carefully) is that it doesn't answer the actual question asked. It clearly demonstrates your feelings about pointers and records, but doesn't address the question "How do I store records in a TList". Sorry; I have to down-vote here. If I post the question "I cut my finger off with a kitchen knife while chopping vegetables. How do I stop the bleeding before I bleed out?", your answer shouldn't be "You should be more careful with a knife so you don't cut yourself.". Answer the question, and then post your opinions on other options. – Ashwell

He was thinking of Tlist. Not married with 2 kids with it. Maybe pedantic works for you. Real amd solid and safe answers that try to do something the user asks are not mere opinion. – Blight

@Warren: Has nothing to do with pedantry. A specific question was asked. Having your opinion is great, but doesnt' make it an answer to the question if it isn't. As I said, answer the question first, and then offer alternative (and possibly better) solutions. – Ashwell

I have edited, and removed most of my opinions. I have instead pointed at what is commonly done in this language and environment, and what is not commonly done, and why. – Blight

Another example of a user doing exactly what I did is here, and I happen to agree that this user tried to educate the person answering the question. Note the upvotes. Would you downvote this guy Ken: #5775098 – Blight

@Warren: Relating to the post you linked, no, I would not downvote you there. You answered the question first, and then talked about alternatives. You didn't do that here prior to your edit, and thus my downvote - actually, you didn't do that here after your edit. You just said "here's a lot of things you'll need to figure out", and then "now here's to do it my way instead". IOW, you still didn't answer the question asked. – Ashwell

Well, your answer is accepted, and that's fine. I think that my answer provides a valuable alternative, and as I said, I didn't think it was worth duplicating what Remy did. (You did, however think that, and your answer is accepted, which is fine too.) – Blight

I didn't duplicate what Remy did, unless Remy created a TList descendant that eliminated all of the typecasting and pointer dereferencing, and automatically deallocated the memory in the list. Whatever, Warren. On this we agree to disagree. Time to end this. – Ashwell

Answer 3

You can take a look at our TDynArray wrapper. It's defined in an Open Source unit, working from Delphi 6 up to XE.

With TDynArray, you can access any dynamic array (like TIntegerDynArray = array of integer or TRecordDynArray = array of TMyRecord) using TList-like properties and methods, e.g. Count, Add, Insert, Delete, Clear, IndexOf, Find, Sort and some new methods like LoadFromStream, SaveToStream, LoadFrom and SaveTo which allow fast binary serialization of any dynamic array, even containing strings or records - a CreateOrderedIndex method is also available to create individual index according to the dynamic array content. You can also serialize the array content into JSON, if you wish. Slice, Reverse or Copy methods are also available.

It will handle a dynamic array of records, and even records within records, with strings or other dynamic arrays inside.

When using an external Count variable, you can speed up a lot the adding of elements in the referred dynamic array.

type
  TPerson = packed record
    sCountry: string;
    sFullName: string;
    sAddress: string;
    sCity: string;
    sEmployer: string;
  end;
  TPersons = array of TPerson;
var
  MyPeople: TPersons;

(...)
procedure SavePeopleToStream(Stream: TMemoryStream);
var aPeople: TPerson;
    aDynArray: TDynArray;
begin
  aDynArray.Init(TypeInfo(TPersons),MyPeople);
  aPeople.sCountry := 'France';
  aPeople.sEmployer := 'Republique';
  aDynArray.Add(aPeople);
  aDynArray.SaveToStream(Stream);
end; // no try..finally Free needed here

There is also a TDynArrayHashed class, which allow internal hashing of a dynamic array content. It's very fast and able to hash any kind of data (there are standard hashers for strings, but you can supply your own - even the hash function can be customized).

Note that TDynArray and TDynArrayHashed are just wrappers around an existing dynamic array variable. You can therefore initialize a TDynArray wrapper on need, to access more efficiently any native Delphi dynamic array.

Repp

@Ken White (by anticipation :) ) Yes, this is not a true TList implementation, I know. But it's definitively a TList-like implementation, faster than using a TList (since records are allocated by chunk). TList is NOT meant to store records, but pointers. And in our wrapper, there are methods you don't have with a TList, like hashing, internal saving or loading to memory or stream, sort with an external integer look-up index, Slice methods, and such. We use this e.g. in our framework for instance to store a cache of compiled SQL statements, in a few lines of code. – Repp

Can only this wrapper be downloaded without the whole framework? – Ineligible

You can extract the needed units: SynCommons.pas and SynLZ.pas, plus Synopse.inc. – Repp

Answer 4

You can use TList for that, eg:

type
  pRec = ^sRec;
  sRec = record
    Value: Integer;
    ...
  end;

var
  List: TList;
  Rec: pRec;
  I: Integer;
begin
  List := TList.Create;
  try
    for I := 1 to 5 do begin
      GetMem(Rec);
      try
        Rec^.Value := ...;
        ...
        List.Add(Rec);
      except
        FreeMem(Rec);
        raise;
      end;
    end;
    ...
    for I := 0 to List.Count-1 do
    begin
      Rec := pRec(List[I]);
      ...
    end;
    ...
    for I := 0 to List.Count-1 do
      FreeMem(pRec(List[I]));
    List.Clear;
  finally
    List.Free;
  end;
end;

Carinthia

Note that in this case, Remy is solving the by-Value and by-reference semantic gap by having Rec be declared as pRec, which is a pointer type. I would argue he should call that variable PointerToARecord, or something. Most delphi programmers prefer to avoid directly using pointers where possible, thus the newer TList<Generic> and the classic dynamic-array features of Delphi are usually preferable. – Blight

@Warren: Delphi convention says that PMyRec means pointer to TMyRec. Read any of the RTL/VCL that uses pointers to see examples. Calling it PointerToARecord is redundant and long-winded. We get that you don't like pointers; that doesn't mean they don't have a place in the language. – Ashwell

I would use New and Dispose instead of GetMem, FreeMem. The poster could be using strings, interfaces or some other lifetime managed variable in his records. – Simpson

It's nothing to do with don't like. Pointers and newbies don't mix. As johan said. You're showing the newby a lawnmower and you're asking him to put his hand under the deck while it's running. – Blight

Answer 5

Use Generiс TList from System.Generics.Collections. If you need to access to a record in generic TList by-reference and do not copy record: use List.List - direct access to the array of TList.

MyList := TList<TTestRec>.Create; 
[...] 

var lRecP: PTestRec; // (PTestRec = ^TTestRec)
lRecP := @MyList.List[i]; 

Now you can access to record inside Tlist array without copying it.

Bering

Answer 6

If using an older version of Delphi where generics isn't present, consider inheriting from TList and override Notify method. When adding an item, alloc memory, copy added pointer memory content and override content in list. When removing, just free memory.

  TOwnedList = class(TList)
  private
    FPtrSize: integer;
  protected
    procedure Notify(Ptr: Pointer; Action: TListNotification); override;
  public
    constructor Create(const APtrSize: integer);
  end;

  constructor TOwnedList.Create(const APtrSize: integer);
  begin
    inherited Create();
    FPtrSize := APtrSize;
  end;

  procedure TOwnedList.Notify(Ptr: Pointer; Action: TListNotification);
  var
    LPtr: Pointer;
  begin
    inherited;
    if (Action = lnAdded) then begin
      GetMem(LPtr, FPtrSize);
      CopyMemory(LPtr, Ptr, FPtrSize); //May use another copy kind
      List^[IndexOf(Ptr)] := LPtr;
    end else if (Action = lnDeleted) then begin
      FreeMem(Ptr, FPtrSize);
    end;
  end;

Usage:

...
LList := TOwnedList.Create(SizeOf(*YOUR RECORD TYPE HERE*));
LList.Add(*YOU RECORD POINTER HERE*);
...

• Note that where I did use CopyMemory(LPtr, Ptr, FPtrSize), you may use another copy aproach. My list is intended to store a record with pointer references, so it doesn't manage it's fields memory.

Cuckoo

Answer 7

It all depends on the type of data you want to store.

You might consider using TCollection and TCollectionItem.

Here is (edited) code from a working unit, in which I used TCollection to read a list of report definitions from a folder. Each report consisted of a sort of template and an SQL statement which had to be stored together with a file name.

Since it is edited, and uses some of my own units (TedlFolderRtns reads files into an internal list, to name but one), the example is simple enough to be useful. With a few replace all, you can adapt to whatever your need.

Look up TCollection in the help, you can do a lot with it. And it keeps your code handling nicely grouped together in a class-like structure.

  unit cReports;
  interface
  uses
     SysUtils, Classes, XMLDoc, XMLIntf, Variants,
     // dlib - Edelcom
     eIntList, eProgSettings,eFolder ;
  type

     TReportDefItem = class(TCollectionItem)
     private
        fSql: string;
        fSkeleton: string;
        fFileName: string;
        procedure Load;
        procedure SetFileName(const Value: string);
     public
        constructor Create(Collection:TCollection); override;
        destructor Destroy ; override;

        property FileName: string read fFileName write SetFileName;
        property Sql : string read fSql write fSql;
        property Skeleton : string read fSkeleton write fSkeleton;
     end;

     TReportDefList = class(TCollection)
     private
        function OsReportFolder: string;
        function GetAction(const Index: integer): TReportDefItem;
     public
        constructor Create(ItemClass: TCollectionItemClass);
        destructor Destroy; override;

        procedure LoadList;

        function Add : TReportDefItem;
        property Action [ const Index:integer ]: TReportDefItem read GetAction;
     end;

  implementation

  { TReportDefList }

  constructor TReportDefList.Create(ItemClass: TCollectionItemClass);
  begin
     inherited;
  end;

  destructor TReportDefList.Destroy;
  begin
     inherited;
  end;
  function TReportDefList.Add: TReportDefItem;
  begin
     Result := TReportDefItem( Add() );
  end;

  function TReportDefList.GetAction(const Index: integer): TReportDefItem;
  begin
     if (Index >= 0) and (Index < Count)
     then Result := TReportDefItem( Items[Index] )
     else Result := Nil;
  end;

  procedure TReportDefList.LoadList;
  var Folder : TedlFolderRtns;
      i : integer;
      Itm : TReportDefItem;
  begin
     Folder := TedlFolderRtns.Create;
     try
        Folder.FileList( OsReportFolder,'*.sw.xml', False);
        for i := 0 to Folder.ResultListCount -1 do
        begin
          Itm := Add();
          Itm.FileName := Folder.ResultList[i];
        end;
     finally
        FreeAndNil(Folder);
     end;
  end;

  function TReportDefList.OsReportFolder: string;
  begin
     Result := Application.ExeName + '_RprtDef';
  end;

  { TReportDefItem }

  constructor TReportDefItem.Create(Collection: TCollection);
  begin
     inherited;
     fSql := '';
     fSkeleton := '';
  end;

  destructor TReportDefItem.Destroy;
  begin
    inherited;
  end;

  procedure TReportDefItem.Load;
  var XMLDoc : IXMLDocument;
      TopNode : IXMLNode;
      FileNode : IXmlNode;
      iWebIndex, iRemoteIndex : integer;
      sWebVersion, sRemoteVersion: string;
      sWebFileName: string;
  begin
     if not FileExists(fFileName ) then Exit;

     XMLDoc := TXMLDocument.Create(nil);
     try
        XMLDoc.LoadFromFile( fFileName );
        XMLDoc.Active := True;

        TopNode := XMLDoc.ChildNodes.FindNode('sw-report-def');
        if not Assigned(TopNode) then Exit;

        FileNode := TopNode.ChildNodes.First;
        while Assigned(FileNode) do
        begin
           fSql := VarToStr( FileNode.Attributes['sql'] );
           fSkeleton := VarToStr(  FileNode.Attributes['skeleton'] );
           FileNode := FileNode.NextSibling;
        end;
        XMLDoc.Active := False;
     finally
        XMLDoc := Nil;
     end;
  end;

  procedure TReportDefItem.SetFileName(const Value: string);
  begin
     if fFileName <> Value
     then begin
        fFileName := Value;
        Load;
     end;
  end;
  end.

Use as :

fReports := TReportDefList.Create( TReportDefItem );
fReports.LoadList();

Reasonable

Seems like a good answer, except the item is a class, not a record as per the OP. +1 since it still seems to useful. – Escent

Answer 8

We've just run into a similar issue here with a generic list of records. Hope the following psuedo code helps.

type
  PPat = ^TPat;
  TPat = record
    data: integer;
  end;

...
var
    AList: TList<PPat>;

...
procedure TForm1.Button1Click(Sender: TObject);
var
  obj: PPat;
begin
  obj := AList[0];
  obj.data := 1;
  Assert(obj.data = AList[0].data);  // correct
end;

procedure TForm1.FormCreate(Sender: TObject);
var
  obj: PPat;
begin
  AList := TList<PPat>.Create;
  GetMem(obj, SizeOf(TPat));  // not shown but need to FreeMem when items are removed from the list
  obj.data := 2;
  AList.Add(obj);
end;

Trilemma