Delphi 7 compared to 2009 (& 2010) Record sizes

Question

I have a weird issue when converting code from Delphi 7 to 2010. It has to do with records. The record defined below, when sized in D7, is 432 bytes, and in D2009 (and 2010) it's 496. I know, that an easy solution is to make it a packed record, then all versions come out to 426 bytes... However, we have data stored where we streamed the record and now we are trying to read those streams with a newer language.

TToTry = Record
 a,b,c,d : Extended;
 e,f,g,h : Extended;
 i : String[15];
 j,k,l,m,n,o,p,q,r,s,t : Array[1..3] of Extended; End;

In investigating this issue, I created another record, and, for whatever reason, the sizes are the same? The record is smaller, but it has the same data types. but it comes out the same size in all versions of the language.

TMyRecord = Record
Ext1  : Extended;
Ext2  : Extended;
Ext3  : Extended;
Ext4  : Extended;
Ext5  : Extended;
Ext6  : Extended;
Int1  : Integer;
Int2  : Integer;
char1 : AnsiChar;
char2 : AnsiChar;
MyString  : String[15];
Arr1  : Array[1..3] of Extended;
Arr2  : Array[1..3] of Extended; end;

Anybody have any insight as to why one record is so different, and the other is the same? Something to do with byte boundary alignments in Delphi for sure. but what changed so drastically from one version to the next?

Almswoman

I know that the default byte alignment for Delphi records changed in a recent version (2009 I think) but I'm not sure of the details. – Carlacarlee

Answer 1

Well, the first problem is that you stored a non-packed record to disk. Field and array packing is allowed to change between product releases because normally the layout in memory is not visible outside of the process. You've broken that rule.

If the byte padding defaults changed between Delphi 7 and Delphi 2009, find out what the defaults were in D7 and set the defaults to the same in Delphi 2009.

Also check the array packing default. I can't remember if there is a separate setting for this.

Take a look at your record structure in Delphi 2009 in the debug memory view. Some or all of the additional size may be due to padding of the record itself (not the fields within it) so that when the record is used in an array the array elements are on speedy machine boundaries.

If none of this helps, create a temporary packed record type in D2009 and manually insert byte padding fields between the actual data fields until the record size and field alignments match the D7 layout. It's not just size, it's field alignments. Read your old data file in using this temp packed record. Then transfer the data field by field into your "real" record type in D2009 and write out a new file.

And while you're at it, pack that record type in D2009 so this doesn't happen again.

Vedic

This is all assuming you don't have the option to go back to D7 and modify the code there to pack the record structure and write out a file of packed records. That would be less work than manually trying to match the byte padding after the fact. – Vedic

Hind sight is always 20/20... and forward thinking development (and developers) does not always happen. Good idea about the packed record with spaces for 2009/2010. We'll give that a try. – Almswoman

Hindsight should be educational, too. Writing unpacked records to a file should happen only once per career. ;> – Vedic

This doesn't really answer the question. The question asked why record-layout rules that changed between the two Delphi versions affected the first record but not the second. This answer gives suggestions on how to solve the problem resulting from those changes, but that's not what the question asked for. What is it about TToTry that makes it susceptible to different alignment rules whereas TMyRecord appears immune? – Circinate

I can't answer policy questions of why the product dev team decided to make this change or that. As for why the TMyRecord size is stable between releases, has anyone checked the project file to see if alignment is set explicitly? TMyRecord is also a very different structure from TToTry. TToTry is dominated by 8 10-byte Extended fields and 12 arrays of 3 10 byte Extended fields. TMyRecord throws in integers and bytes and a lot fewer 10 byte extendeds. There's a lot less in TMyRecord that might need padding for N-word alignment. – Vedic

@Rob: Forgot you're relatively new to Delphi. Allow me to introduce Danny Thorpe. Danny hasn't been the Chief Engineer on the Delphi compiler for a few years now, and therefore wouldn't know why recent changes were made. He is, however, quite qualified to explain how to solve the problem (and analyze why the problem exists). :-) – Plough

That "relatively new" remark was a joke, right? I know who he is, @Ken. The posted question didn't ask why the change was made, and neither did my follow-up comment. The question did ask why the change would affect one record and not another, and I think Danny's answer (in the comment) to that question was rather vague. And although it's practical information to have, the question did not ask how to fix those differences. – Circinate

@Rob: You didn't see the smiley? I can point it out - it's just before the '- Ken White' at the end of my comment. But my comment still stands - unless you're a member of the compiler team that worked on Delphi 2009's version of the compiler, you can't answer the question of why it changed; therefore, Danny's answer was as good a guess as anyone else could make (and better than the answer you didn't provide <g>), and his first sentence was dead on. (You do see the <g> in my previous sentence here, right? It's just before the closing parenthesis.) – Plough

@Ken, I simply wasn't sure whether the smiley at the end of your comment applied to the joke at the beginning since there was non-joke material between them. Anyway, I know we can't answer why it changed, but so what? No one ever asked that. Danny's comment's first sentence is dead on answering a non-existent question. I considered posting an answer describing how to determine where to add padding bytes to the record, but I checked myself; I realized I wasn't answering what Jason asked. In fact, Sertac is the one who's dead on. – Circinate

Answer 2

I believe you've hit a feature!. I could not get a reasonable size with your TToTry with D2007 so I had to look up field addresses with the debugger;

First, size of the below record,

{$A8}
type
  TToTry = record
    j: array[1..3] of Extended;
    k: array[1..3] of Extended;
    l: array[1..3] of Extended;
    m: array[1..3] of Extended;
  end;

is 128 (32*4). This is expected, since an Extended is 10 bytes, 30 bytes would align on 32 bytes.

But size of this record,

{$A8}
type
  TToTry = record
    j, k, l, m: array[1..3] of Extended;
  end;

is 120 (30*4). This is certainly unexpected - fields should still align on an 8 byte boundary.

(I don't have D7 to verify but my thinking is that:)
So now we know that grouped fields are packed, it follows that the alignment on D7 is 8 bytes and your record is nearly packed;

TToTry = Record 
 a,b,c,d : Extended;    // 40 bytes (8*5)
 e,f,g,h : Extended;    // 40 bytes (8*5)
 i : String[15];        // 16 bytes (8*2)
 j,k,l,m,n,o,p,q,r,s,t: Array[1..3] of Extended; // 330 bytes
End; 

The compiler is padding 6 bytes to the last group to have it a multiple of 8 and then you get 40+40+16+336 = 432 bytes.

With D2009/D2010 you are either declaring each field - without grouping them, or the behavior is changed. Either way pack your record and add a 6 byte array dummy field to the end, and you should be good to go.

If that does not work, look at the field addresses of your record with D7, then create an exact duplicate on D2009 by using a packed record and using dummy fields as necessary, after you've imported your stored data you can drop the dummy fields.

--
I've never known such behavior and I can't find it documented anywhere. Still, it is so much like a feature that I'm hesitant to call it a bug. I don't know the behavior is the same with D2009 or D2010, test to see if it is. If it is, to get expected results - not to have half-packed records - don't be lazy and declare each and every field on its own for non-packed records.

Doehne

The size of the second record in Delphi 2009 is 128 bytes – Median

@Serg - thanks for looking it up, so it's not a feature. In fact I encountered other strange/unexpected alignments while testing with D2007. I'm glad that later versions have it straight! – Doehne

Answer 3

I know this is an old post, but I ran into the same issue with Delphi XE2 yesterday. I have several typed files that were written out using a Turbo Delphi 2006 app and I also made the mistake of not using Packed records. My situation was compounded because I was using variable length records (that's what we call them in the z/OS world, not 100% sure of the Delphi term), so the mis-alignment was causing the record key tags to not end up in the correct field, thus causing none of the sub-records to be in the right place, rendering the entire file useless.

What I discovered is that you can put the alignment compiler directives around just a specific block of code. Along with that, I also discovered this little gem: {$OLDTYPELAYOUT ON}

{$OLDTYPELAYOUT ON}
 RMasterRecord = Record       
    mKey       : word;
    mDeleted   : boolean;
    case mType : ANSIChar of
       'V' : //Info
          (Vers : RVersionLayout);
       […]
{$OLDTYPELAYOUT OFF}

I only put the directive around the master record that gets written and read to the file, not around the sub-record definitions. This solved my issue, and I can now compile in XE2 and read my TD2006 files. Next time I will use Packed records (or better yet, SQLite). But I thought I would share this, since this site has helped me immeasurably for years.

You can read more about $OLDTYPELAYOUT here: http://docwiki.embarcadero.com/RADStudio/XE5/en/Internal_Data_Formats#Record_Types. By the way, the first one that fixed it was {$A4} but when I discovered {$OLDTYPELAYOUT ON} I switched it to use that, since it is more obvious.

Frizzy

Answer 4

I believe the default alignment was made wider. Specify alignment 4 in the later versions and see if it comes out the way you want.

For the future you should make sure that any record that is going to be written out to disk is stored packed so you don't get burned this way.

Edit: Since none of the alignments work (which surprises me) I would go back to the original and figure out how it really is aligned. Fill the record with something like $FF, put the data in and write it out--see where the $FFs survived. Take the new record, make it packed and add fillers to match the padding in the old record.

One thing: Is this actually just one record? In the old days I have used objects as fake records with inheritance--oops, at the point of the inheritance the normal alignment was applied and I couldn't stop it. I ended up having to pad before the data in order to make the forced alignment not break my data. (This was going to an API, it HAD to be right, I couldn't process the fields independently.)

Kalvn

How does one "specify alignment"? – Almswoman

Found it: {$A4}. But it didn't help. The number is now 464. I tried all the other {$A?} possibilities and none match. – Almswoman

Answer 5

Applying {$A8} directive does not mean that all record fields are aligned at 8-byte boundary - the compliler uses a different alignment strategy. For example, the size of

{$A8}
type
  TToTry = record
    a: byte;
    b: word;
    c: longword;
  end;

is 8 bytes in Delphi 2009, because the compiler aligns 2-byte value at 2-byte boundary, 4-byte value at 4-byte boundary, and the only actual alignment in the above example is b field aligned at 2-byte boundary.

As for the original question what changed between Delphi 7 and Delphi 2009 - read Sertac Akyuz answer and my comment to it

Median

Ah, Ok! That's what I meant by 'strange/unexpected' with my comment to your comment on my answer. But is this to be expected? The docs say: "In the {$A8} or {$A+} state, fields in record types that are declared without the packed modifier and fields in class structures are aligned on quad word boundaries." – Doehne

@Sertac: Quad word alignment only applies to data types that are at least 8 bytes in length. There's little to be gained on x86 architecture by putting a byte field on an 8 byte boundary. (It could help with cache line collisions in the L2 cache, but that's really really minor). As Serg said, data types align to their natural boundary (= size of data) up to the $A/n/ size. Note that there are multiple alignment points to consider: offset of fields from the start of the struct, alignment of data within the arrays, and padding of the record itself to align well when it is used in an array. – Vedic

@Vedic - Thanks, all make sense. But I wish the 'structured types' or the 'align fields' topic in the documentation be more explicit on this, especially for those like me who have no knowledge of the underlying architecture.. – Doehne