"descending" records in delphi?

Question

I know you can't actually descend anything from a record, but I'm not sure how to summarize my problem in one sentence. Edit the title if you do.

What I want to do here is make an array of some generic type, which can be one of X number of types, the array would be filled with those custom types (they have different fields and that's what is important). The easy way is to make just an array of variant records, each variant has it's own type, but obviously can't redeclare identifiers like so:

GenericRec = Record
  case SubTypeName: TSubTypeName of
    type1name: (SubRec: Type1);
    type2name: (SubRec: Type2);
    ...
    typeNname: (SubRec: TypeN);
  end;

Changing SubRec to SubRec1, SubRec2... SubRecN makes referencing painful, but not impossible. And since I started looking for alternative solutions to the above problem, classes came to mind.

The obvious example to demonstrate what I am trying to achieve is TObject, an array of those can be assigned to many different things. That's what I want, but with records (and that's impossible to do), because I want to be able to save the records to file as well as read them back (also because it's something I'm already familiar with). Making my own simple class is not a problem, making a descendant class from that to represent my subtype - I can do that. But what about writing that to file and reading it back? This boils down to serialization, which I have no idea how to do. From what I gather it's not as easy and the class must be descended from TComponent.

TMyClass = Class

Does it make any difference if I make the class like above? It's nothing fancy and has at most 10 fields, including a few custom types.

Setting serialization aside (just because I have a lot of reading to do on that topic), use of classes here also might be out of the question.

At this point, what are my options? Should I abandon records and try this with classes? Or would it be a lot less complicated just to stick to records and deal with the variant "limitation"? I'm all about learning and if exploding the class approach might make me smarter, I'll do it. I've also just looked into TList too (never used it), but it seems that it doesn't mix too well with records, well maybe it can be done, but that might be out of my league at the moment. I'm open to any kind of suggestions. What do i do?

Finished

At a basic level objects aren't that much more than a record with a notion of inheritance and the possibility to add smarts (methods)... I'd rethink the problem in terms of classes and subclasses. – Jansson

+1. The usual way to do this (with records) is to use a byte that indicates the record type, and then when saving write that byte marker and the record. When reading back, you read the byte, and then read the number of bytes corresponding to the record type indicated by the preceeding byte. (Sort of like API calls that accept various definitions of the same structure, and so contains a cbSize member that provides the size of the structure you're actually providing.) – Spherics

@Ken the code in the Q is a data structure exactly as you describe – Vardhamana

@Jansson Records can have methods. – Vardhamana

@DavidHeffernan just saw it, apparently introduced in D2006... They sure do look an awful lot like a class now. – Jansson

@DavidHeffernan: No, it's not. :-) Variant records are handled fairly well in Delphi, but don't lend themselves well to serialization; I was describing something else entirely that's probably familiar to people who used early versions of Delphi (or Turbo Pascal before that, even). But that's OK; I posted a comment instead of an answer intentionally because I wasn't offering a solution. – Spherics

@Ken, converting a record to a string and back is serializing. It's not the only form of serialization, but it's one way. – Hoop

@RobKennedy: Didn't mean to start that discussion again; we've had it (see the linked question - of course, you lose the benefit of the many comments Jerry deleted along the way). Deleting my comment shortly. :-) – Spherics

@ken I don't see why variant records can't be serialized easily. Anyway, I don't really understand what you are getting at. – Vardhamana

@David, if you read the words I wrote, I specifically mention earlier versions of Delphi (pre-new-RTTI), which of course means the vast majority of Delphi licenses currently in use. I'm not sure why you're fixating on it, though; it's a comment, not an answer, and the original one I made wasn't even addressed to you. You misunderstood what I wrote, commented on it, and I responded. And you're now ignoring what I've written twice before (no three times - earlier versions of Delphi). As there's no mentiontion of a Delphi version in the subject, tags, or text here... – Spherics

Answer 1

The "natural" way of handling such data is to use a class, and not a record. It will be much easier to work with, both at definition time and when dealing with implementation: in particular, virtual methods are very powerful to customize a process for a particular kind of class. Then use a TList/TObjectList or a TCollection, or a generic-based array in newer versions of Delphi to store the list.

About serialization, there are several ways to do it. See Delphi: Store data in somekind of structure

In your particular case, the difficulty comes from the "variant" kind of record you are using. IMHO the main drawback is that the compiler will refuse to set any reference-counted kind of variable (e.g. a string) within the "variant" part. So you'll be able to write only "plain" variables (like integer) within this "variant" part. A big limitation IMHO, which reduces the interest of this solution.

Another possibility could be to store the kind of record at the beginning of its definition, e.g. with a RecType: integer or even better with a RecType: TEnumerationType which will be more explicit than a number. But you'll have to write a lot of code by hand, and works with pointers, which is a bit error-prone if you are not very fluent with pointer coding.

So you can also store the type information of the record, accessible via TypeInfo(aRecordVariable). Then you can use FillChar to initialize the record content to zero, just after allocation, then use the following function to finalize the record content, just after disallocation (this is what Dispose() does internally, and you shall call it, otherwise you'll leak memory):

procedure RecordClear(var Dest; TypeInfo: pointer);
asm
  jmp System.@FinalizeRecord
end;

But such an implementation pattern will just reinvent the wheel! It is in fact how class is implemented: the first element of any TObject instance is a pointer to its ClassType:

function TObject.ClassType: TClass;
begin
  Pointer(Result) := PPointer(Self)^;
end;

There is also another structure in Delphi, which is called object. It is some kind of record, but it supports inheritance - see this article. It is the old style of OOP programming in Turbo Pascal 5.5 days, deprecated, but still available. Note that I discovered a weird compilation issue on newer versions of Delphi: sometimes, an object allocated on the stack is not always initialized.

Take a look at our TDynArray wrapper and its associated functions, who is able to serialize any record content, into binary or JSON. See Delphi (win32) serialization libraries question. It will work with variant records, even if they include a string in their unvariant part, whereas a plain "Write/BlockWrite" won't work with reference counted fields.

Ethnography

I am using TKBDynamic, it handles variants fine from what I've tried, but you're right, I'm not very familiar with pointers and so the answer by Remy Lebeau just scares me away from trying that approach (despite that it might work). I am going the class route, now all is left is to learn how to serialize stuff and do all that magic with virtual methods everyone keeps telling me about – Finished

@Finished About serialization of TObject classes, see the link I put above, i.e. #7106495 And do not underestimate my latest proposal in this answer: use a database! You can find light DBs those days (like SQlite3 or even the TClientDataSet). – Ethnography

Answer 2

You're conflating serialization with "writing everything to disk with a single BlockWrite call." You can serialize anything you want, regardless of whether it descends from TComponent or TPersistent.

Although writing everything with a single BlockWrite call looks convenient at first, you'll quickly find it's not really what you want if your desired record types are going to store anything particularly interesting (like strings, dynamic arrays, interfaces, objects, or other reference- or pointer-based types).

You'll probably also find variant records unsatisfying since you'll be coding to the lowest common denominator. You won't be able to access anything in the record without checking the actual contained type, and the size of even the smallest amount of data will occupy the same amount of space as the largest data type.

The question seems to describe polymorphism, so you may as well embrace what the language already provides for that. Use an array (or list, or any other container) of objects. Then you can use virtual methods to treat them all uniformly. You can implement dynamic dispatch for records if you want (e.g., give each record a function pointer that refers to a function that knows how to deal with that record's contained data type), but in the end you'll probably just find yourself reinventing classes.

Hoop

Indeed, have to start embracing classes, thanks for the input. On a sidenote, I am using streams for writing stuff, streams combined with TKBDynamic and I am able to write/read dynamic arrays full of dynamic arrays no problem, probably not interfaces/objects and such, but it works just as I want for simpler "data" as in arrays and stings. – Finished

Answer 3

To do this with records, you would create different record types that have a common field(s) in front, and then put those same field(s) in the generic record. Then you can simply type-cast a pointer to a generic record to a pointer to a specific record when needed. For example:

type
  PGenericRec = ^GenericRec;
  GenericRec = Record 
    RecType: Integer;
  end;

  PType1Rec = ^Type1Rec; 
  Type1Rec = Record 
    RecType: Integer;
    // Type1Rec specific fields...
  end;

  PType2Rec = ^Type2Rec; 
  Type2Rec = Record 
    RecType: Integer;
    // Type2Rec specific fields...
  end;

  PTypeNRec = ^TypeNRec;
  TypeNRec = Record
    RecType: Integer;
    // TypeNRec specific fields...
  end; 

var
  Recs: array of PGenericRec;
  Rec1: PType1Rec; 
  Rec2: PType2Rec; 
  RecN: PTypeNRec;
  I: Integer;
begin
  SetLength(Recs, 3);

  New(Rec1);
  Rec1^.RecType := RecTypeForType1Rec;
  // fill Rec1 fields ...
  Recs[0] := PGenericRec(Rec1);

  New(Rec2);
  Rec2^.RecType := RecTypeForType2Rec;
  // fill Rec2 fields ...
  Recs[1] := PGenericRec(Rec2);

  New(RecN);
  Rec3^.RecType := RecTypeForTypeNRec;
  // fill RecN fields ...
  Recs[2] := PGenericRec(RecN);

  for I := 0 to 2 do
  begin
    case Recs[I]^.RecType of
      RecTypeForType1Rec: begin
        Rec1 := PType1Rec(Recs[I]);
        // use Rec1 as needed...
      end;
      RecTypeForType1Re2: begin
        Rec2 := PType2Rec(Recs[I]);
        // use Rec2 as needed...
      end;
      RecTypeForTypeNRec: begin
        RecN := PTypeNRec(Recs[I]);
        // use RecN as needed...
      end;
    end;
  end;

  for I := 0 to 2 do
  begin
    case Recs[I]^.RecType of
      RecTypeForType1Rec: Dispose(PType1Rec(Recs[I]));
      RecTypeForType2Rec: Dispose(PType2Rec(Recs[I]));
      RecTypeForTypeNRec: Dispose(PTypeNRec(Recs[I]));
    end;
  end;
end;

As for serialization, you do not need TComponent for that. You can serialize records, you just have to do it manually. For writing, write out the RecType value first, then write out the record-specific values next. For reading, read the RecType value first, then create the appropriate record type for that value, then read the record-specific values into it.:

interface

type
  PGenericRec = ^GenericRec;
  GenericRec = Record 
    RecType: Integer;
  end;

  NewRecProc = procedure(var Rec: PGenericRec);
  DisposeRecProc = procedure(Rec: PGenericRec);
  ReadRecProc = procedure(Rec: PGenericRec);
  WriteRecProc = procedure(const Rec: PGenericRec);

function NewRec(ARecType: Integer): PGenericRec;
procedure DisposeRec(var Rec: PGenericRec);
procedure ReadRec(Rec: PGenericRec);
procedure WriteRec(const Rec: PGenericRec);

procedure RegisterRecType(ARecType: Integer; ANewProc: NewRecProc; ADisposeProc: DisposeRecProc; AReadproc: ReadRecFunc; AWriteProc: WriteRecProc);

implementation

type
  TRecTypeReg = record
    RecType: Integer;
    NewProc: NewRecProc;
    DisposeProc: DisposeRecProc;
    ReadProc: ReadRecProc;
    WriteProc: WriteRecProc;
  end;

var
  RecTypes: array of TRecTypeReg;

function NewRec(ARecType: Integer): PGenericRec;
var
  I: Integer;
begin
  Result := nil;
  for I = Low(RecTypes) to High(RecTypes) do
  begin
    with RecTypes[I] do
    begin
      if RecType = ARecType then
      begin
        NewProc(Result);
        Exit;
      end;
    end;
  end;
  raise Exception.Create('RecType not registered');
end;

procedure DisposeRec(var Rec: PGenericRec);
var
  I: Integer;
begin
  for I = Low(RecTypes) to High(RecTypes) do
  begin
    with RecTypes[I] do
    begin
      if RecType = Rec^.RecType then
      begin
        DisposeProc(Rec);
        Rec := nil;
        Exit;
      end;
    end;
  end;
  raise Exception.Create('RecType not registered');
end;

procedure ReadRec(var Rec: PGenericRec);
var
  LRecType: Integer;
  I: Integer;
begin
  Rec := nil;
  LRecType := ReadInteger;
  for I = Low(RecTypes) to High(RecTypes) do
  begin
    with RecTypes[I] do
    begin
      if RecType = LRecType then
      begin
        NewProc(Rec);
        try
          ReadProc(Rec);
        except
          DisposeProc(Rec);
          raise;
        end;
        Exit;
      end;
    end;
  end;
  raise Exception.Create('RecType not registered');
end;

procedure WriteRec(const Rec: PGenericRec);
var
  I: Integer;
begin
  for I = Low(RecTypes) to High(RecTypes) do
  begin
    with RecTypes[I] do
    begin
      if RecType = Rec^.RecType then
      begin
        WriteInteger(Rec^.RecType);
        WriteProc(Rec);
        Exit;
      end;
    end;
  end;
  raise Exception.Create('RecType not registered');
end;

procedure RegisterRecType(ARecType: Integer; ANewProc: NewRecProc; ADisposeProc: DisposeRecProc; AReadproc: ReadRecFunc; AWriteProc: WriteRecProc);
begin
  SetLength(RecTypes, Length(RecTypes)+1);
  with RecTypes[High(RecTypes)] do
  begin
    RecType := ARecType;
    NewProc := ANewProc;
    DisposeProc := ADisposeProc;
    ReadProc := AReadProc;
    WriteProc := AWriteProc;
  end;
end;

end.

.

type
  PType1Rec = ^Type1Rec; 
  Type1Rec = Record 
    RecType: Integer;
    Value: Integer;
  end;

procedure NewRec1(var Rec: PGenericRec);
var
  Rec1: PType1Rec;
begin
  New(Rec1);
  Rec1^.RecType := RecTypeForType1Rec;
  Rec := PGenericRec(Rec1);
end;

procedure DisposeRec1(Rec: PGenericRec);
begin
  Dispose(PType1Rec(Rec));
end;

procedure ReadRec1(Rec: PGenericRec);
begin
  PType1Rec(Rec)^.Value := ReadInteger;
end;

procedure WriteRec1(const Rec: PGenericRec);
begin
  WriteInteger(PType1Rec(Rec)^.Value);
end;

initialization
  RegisterRecType(RecTypeForType1Rec, @NewRec1, @DisposeRec1, @ReadRec1, @WriteRec1);

.

type
  PType2Rec = ^Type2Rec; 
  Type2Rec = Record 
    RecType: Integer;
    Value: Boolean;
  end;

procedure NewRec2(var Rec: PGenericRec);
var
  Rec2: PType2Rec;
begin
  New(Rec2);
  Rec2^.RecType := RecTypeForType2Rec;
  Rec := PGenericRec(Rec2);
end;

procedure DisposeRec2(Rec: PGenericRec);
begin
  Dispose(PType2Rec(Rec));
end;

procedure ReadRec2(Rec: PGenericRec);
begin
  PType2Rec(Rec)^.Value := ReadBoolean;
end;

procedure WriteRec2(const Rec: PGenericRec);
begin
  WriteBoolean(PType2Rec(Rec)^.Value);
end;

initialization
  RegisterRecType(RecTypeForType2Rec, @NewRec2, @DisposeRec2, @ReadRec2, @WriteRec2);

.

type
  PTypeNRec = ^Type2Rec; 
  TypeNRec = Record 
    RecType: Integer;
    Value: String;
  end;

procedure NewRecN(var Rec: PGenericRec);
var
  RecN: PTypeNRec;
begin
  New(RecN);
  RecN^.RecType := RecTypeForTypeNRec;
  Rec := PGenericRec(RecN);
end;

procedure DisposeRecN(Rec: PGenericRec);
begin
  Dispose(PTypeNRec(Rec));
end;

procedure ReadRecN(Rec: PGenericRec);
begin
  PTypeNRec(Rec)^.Value := ReadString;
end;

procedure WriteRecN(const Rec: PGenericRec);
begin
  WriteString(PTypeNRec(Rec)^.Value);
end;

initialization
  RegisterRecType(RecTypeForTypeNRec, @NewRecN, @DisposeRecN, @ReadRecN, @WriteRecN);

.

var
  Recs: array of PGenericRec;

procedure CreateRecs;
begin
  SetLength(Recs, 3);

  NewRec1(Recs[0]);
  PRecType1(Recs[0])^.Value : ...;

  NewRec2(Recs[1]);
  PRecType2(Recs[1])^.Value : ...;

  NewRecN(Recs[2]);
  PRecTypeN(Recs[2])^.Value : ...;
end;

procedure DisposeRecs;
begin
  for I := 0 to High(Recs) do
    DisposeRec(Recs[I]);
  SetLength(Recs, 0);
end;

procedure SaveRecs;
var
  I: Integer;
begin
  WriteInteger(Length(Recs));
  for I := 0 to High(Recs) do
    WriteRec(Recs[I]);
end;

procedure LoadRecs;
var
  I: Integer;
begin
  DisposeRecs;
  SetLength(Recs, ReadInteger);
  for I := 0 to High(Recs) do
    ReadRec(Recs[I]);
end;

Emanation

You did just reinvent the class implementation! A TObject instance start with a pointer to its ClassType: very similar to your RecType. And allocating one record at a time, and storing pointer, is exactly what the class model does. IMHO no interest at all here: it is just painful to do it by hand, a lot of code to write manually in order to emulate what the compiler does by itself. You would have better stored the TypeInfo(...) of the record instead of an integer flag in the beginning of the record, and use low-level RTTI record initialization stuff (e.g. InitializeRecord). – Ethnography

And IMHO using an enumerate instead of an integer for RecType could make code more readable. – Ethnography

I didn't say it would be pretty. Classes and virtual methods definitely work better for this kind of thing. And I chose an Integer because it is more flexible for dynamic registrations. – Emanation

You are right. The better flexibility will be achieved when having TypeInfo() pointer to be stored within the record instead of integer, and use a RTTI based record initialization and finalization, using this TypeInfo() - see RecordClear() in my answer (initialization can be done with a FillChar or is done with a dynamic array when the SetLength() is called). – Ethnography