How should I prepare my 32-bit Delphi programs for an eventual 64-bit compiler? [duplicate]

Question

Possible Duplicate:
How to also prepare for 64-bits when migrating to Delphi 2010 and Unicode

Since I believe that 64bit Delphi compiler will appear soon, I am curious if anybody knows what kind of programs that are now 32bit will compile and work without any changes when using 64bit compiler.

And if there is a general rule what kind of changes should we systematically make in our old programs to be compiled as 64bit?

It is good to be prepared when the 64bit compiler will suddenly be here...

Any suggestion will be much appreciated.

Marchall

Nominated for re-opening as this question appears to be producing infinitely more useful answers than the marked duplicate. – Drumstick

What makes you think that Embarcadero will release a 64-bit Delphi compiler soon? – Desperate

I belive what they promise: edn.embarcadero.com/article/39934 Does anybody have information that Delphi 64 will not be available in 2011? – Marchall

Added a bounty because it's a great question! – Gujral

Answer 1

First up, a disclaimer: although I work for Embarcadero. I can't speak for my employer. What I'm about to write is based on my own opinion of how a hypothetical 64-bit Delphi should work, but there may or may not be competing opinions and other foreseen or unforeseen incompatibilities and events that cause alternative design decisions to be made.

That said:

• There are two integer types, NativeInt and NativeUInt, whose size will float between 32-bit and 64-bit depending on platform. They've been around for quite a few releases. No other integer types will change size depending on bitness of the target.

• Make sure that any place that relies on casting a pointer value to an integer or vice versa is using NativeInt or NativeUInt for the integer type. TComponent.Tag should be NativeInt in later versions of Delphi.

• I'd suggest don't use NativeInt or NativeUInt for non-pointer-based values. Try to keep your code semantically the same between 32-bit and 64-bit. If you need 32 bits of range, use Integer; if you need 64 bits, use Int64. That way your code should run the same on both bitnesses. Only if you're casting to and from a Pointer value of some kind, such as a reference or a THandle, should you use NativeInt.

• Use PByte for pointer arithmetic where possible, in preference to NativeInt or NativeUInt. It will suffice for most purposes, and is more typesafe because it can't be (easily) mistaken for a normal integer type, and vice versa.

• Pointer-like things should follow similar rules to pointers: object references (obviously), but also things like HWND, THandle, etc.

• Don't rely on internal details of strings and dynamic arrays, like their header data.

• Our general policy on API changes for 64-bit should be to keep the same API between 32-bit and 64-bit where possible, even if it means that the 64-bit API does not necessarily take advantage of the machine. For example, TList will probably only handle MaxInt div SizeOf(Pointer) elements, in order to keep Count, indexes etc. as Integer. Because the Integer type won't float (i.e. change size depending on bitness), we don't want to have ripple effects on customer code: any indexes that round-tripped through an Integer-typed variable, or for-loop index, would be truncated and potentially cause subtle bugs.

• Where APIs are extended for 64-bit, they will most likely be done with an extra function / method / property to access the extra data, and this API will also be supported in 32-bit. For example, the Length() standard routine will probably return values of type Integer for arguments of type string or dynamic array; if one wants to deal with very large dynamic arrays, there may be a LongLength() routine as well, whose implementation in 32-bit is the same as Length(). Length() would throw an exception in 64-bit if applied to a dynamic array with more than 2^32 elements.

• Related to this, there will probably be improved error checking for narrowing operations in the language, especially narrowing 64-bit values to 32-bit locations. This would hit the usability of assigning the return value of Length to locations of type Integer if Length(), returned Int64. On the other hand, specifically for compiler-magic functions like Length(), there may be some advantage of the magic taken, to e.g. switch the return type based on context. But advantage can't be similarly taken in non-magic APIs.

• Dynamic arrays will probably support 64-bit indexing. Note that Java arrays are limited to 32-bit indexing, even on 64-bit platforms.

• Strings probably will be limited to 32-bit indexing. We have a hard time coming up with realistic reasons for people wanting 4GB+ strings that really are strings, and not just managed blobs of data, for which dynamic arrays may serve just as well.

• Perhaps a built-in assembler, but with restrictions, like not being able to freely mix with Delphi code; there are also rules around exceptions and stack frame layout that need to be followed on x64.

Zippy

Ouch, that no assembler bit will hurt me plenty. I've a lot of SSE code in my vision apps. Yes I know, I'll probably have to alter them by push/popping regs in x86_64, but an update than finding a different solution. – Ladd

"Length() would throw an exception in 64-bit if applied to a dynamic array with more than 232 elements." Umm... 232? – Confidential

@Marco: +1. No ASM will really hurt a lot of people. – Confidential

I have used my own "NativeInt" and "NativeUInt" (named CpuInt and CpuWord, respectively) for quite some time now, and unless I need a specific bitsize, I always use these, ie. in FOR loops, temporary counters, etc. Are you recommending that I switch to using INTEGER or CARDINAL in these cases? Won't that make inefficient code (using 32-bit variables in 64-bit environment)? I know that using 16-bit variables in 32-bit code makes for inefficient code, so I assume the same to be true for 32-bit variables in 64-bit code - or am I wrong? – Estaestablish

No, in x86_64 typically only pointer and equivalents goes to 64-bit. This because x86_64 supports 32-bits General Purpose regs too – Ladd

And 32-bit supports 16-bits General Purpose registers as well, but using them in 32-bit code is inefficient (larger opcodes, slower execution). Isn't this the same for using 32-bit registers in 64-bit code? – Estaestablish

@Marco: +1 for no ASM. That will also be a big drawback for me as well (an old-time assembler programmer :-)). – Estaestablish

@Estaestablish - The larger concern is semantic differences between 32-bit and 64-bit. The more we can minimize those, the better. 16-bit to 32-bit was different because many applications needed more than 16 bits in many counters; but few applications need more than 32-bits for general purpose integers. There is also the problem of conversions between 32-bit and 64-bit values in the RTL and user code. Assuming most user code uses Integer, it would be painful for the API to use 64-bit Int64 and cause truncation warnings (increase in warnings anticipated). For most APIs, restricting to 32-bit is ok. – Zippy

@Estaestablish - in 32-bit, writing to a 16-bit register writes to half a register, but in x64, writing a 32-bit value to a 32-bit register implicitly clears the upper 32-bits of the 64-bit register, thus writing an entire 64-bit register. Iow, it is designed for use with both 32bit or 64bit values. – Pyromania

There was crippled BASM support for inline subroutines, now about to remove BASM at all. I do appreciate Ribbon support, but i like to write fast code much more. This future sucks. – Mattson

Hmm, and what about the RTL? Does this mean the lower reaches of the 64-bit RTL won't be in Pascal? – Ladd

There is a problem with NativeInt under Delphi 2007: it's equal to int64.. – Kleiman

AFAIK assembler support is removed in Visual C++ 64 bit as well. If assembler code is needed, it has to be compiled externally and then linked. 64 bit requisites for assembler code are strict enough that handwritten assembler may be an issue. The drawback is I guess Embarcadero won't bring TASM back - a separate compiler will be needed. – Rhodia

@Marco: + 1 for no asm, I would REALLY miss it – Britnibrito

I suspect a certain amount of correlation between people who write applications that actually need 64 bits support and people who care about asm. :) – Rosenthal

@Rosenthal I don't. People who need 64-bits generally need it either because they have a lot of data, or because they need to integrate with the Windows shell; then there's a whole crowd who are just old, and remember the 16-bit to 32-bit transition and don't want to be left behind. The difference this time is that many, if not most, desktop apps don't actually need 64-bit. As to people needing assembler: they have to rewrite that anyway. Whether it's in a separate file with a .asm extension, or in an asm block in code, doesn't make much incremental difference to the effort of a rewrite. – Zippy

@Marco I don't get it? System.pas would still exist. Low-level routines would either be pure Pascal, or linked in if they must be in assembler? A lot of the old routines had to be in assembler owing to odd calling conventions (check out _LStrCatN, a kind of bastardized callee-pops cdecl); but x64 has only one calling convention, simplifying things greatly. – Zippy

Barry Kelly: the strong point of the internal assembler IMHO always has been the integration with pascal, e.g. obtaining record offsets, local vars, dealing with (mildly) structured parameters. In the SSE routines I typically call some procedures storing complex things in local vars before starting an asm block that does the tight loop. – Ladd

The remark about Pascal rtl was not so much that it couldn't be done any other way. I just thought I saw a pattern :-) – Ladd

@Barry Kelly... All three sample sets (large data sets, shell integration, old farts) are people who tend to write asm now and then (as opposed to the young hipsters building the business logic) :). Somehow the statement that having to rewrite and maintain asm in separate files (through a separate toolset) is "not much more difficult" is kinda strange coming from someone building IDEs. The I part of that acronym counts, and not just for the pascal part of the application. – Rosenthal

And just for the record, I'm adding "building 64-bits COM AddIns" and "64-bit MAPI applications" to the list of reason we are in the waiting-and-whining line for 64-bits support. Yes, the percentage of 64 bits office packages sold is tiny, but thanks to large numbers it is still a real customer base. – Rosenthal

Another interesting question relating to 64-bit btw: is CONST still guaranteed by reference? – Ladd

@Marco I'm not sure what you mean by CONST guaranteed by reference. Delphi has a kind of half-baked constness built into its type system, via structured constants and const parameters; but likely changes will be minimal to maximize compatibility. (IMO const parameters should never have been put in; it's possible to pass by reference and simulate pass by value if the body of the method modifies the argument; meanwhile, typed constants started out in life fully mutable (ick!) before initializable global vars came in.) – Zippy

Hmm, my bad, that issue is less win64, and more reusing interfaces for Mozilla XPCOM on cdecl archs. wiki.freepascal.org/… – Ladd

$L *.obj is pretty important to me as I'm statically linking quite a bit of C code; thanks a lot for the detailed answer Barry – Bilabial

@Barry: this question has reappeared because a bounty has been added to it. Has anything changed in the past year that can be clarified / added / removed from your answer? – Lir

Answer 2

First of all, FreePascal already offers 64-bits support. It's not Delphi, though.
Second of all, I expect about the same problems that existed in the time Delphi 1 was upgraded to Delphi 2. The biggest problem is is mostly address-space related and the problem here is that pointers will be widened from 4 bytes to 8 bytes. In WIN16 they use to be 2 bytes and a trick was needed to get over the 64KB boundary by using segments and offsets for pointers. (With the possibility to use default segments for several tasks.)
It's also likely that certain datatypes will become bigger than they are now. The integer-type will be 8 bytes, most likely. (Used to be just 2 bytes in Windows 2.) Enumerations will likely become bigger too. But most other datatypes are likely to keep their current size, so not too many changes here.
Another issue will be memory requirements. Since pointers will be 8 bytes long, an application that uses a lot of them will also eat up a lot more memory. A list with 10.000 pointers will increase from 40.000 bytes to 80.000 bytes. You might want to use a bit more memory than on a 32-bit system.
Speed will also change a bit. Since the processor now handles 8 bytes at the same time, it can process data much faster. But since pointers and some data types become bigger, receiving or sending these to some device or memory will be a bit slower. In general, your applications will be slightly faster in general, but some parts might actually become slower!
Finally, changes in the Windows API will require you to use the 64-bits API functions. Maybe the Delphi compiler will do something smart to allow code to call 32-bit API functions, but this would slow down performance because the processor now switches between native 64-bits mode and emulated 32-bits mode.

Lexis

On 64-bit Windows, the model is for int and long to stay 32-bit. Correspondingly, Delphi will follow with Integer and Longint staying 32-bit. Also, in Delphi, enumerations have always had the smallest type capable of representing their range. – Zippy

From Delphi 1 to Delphi 2, the integer type changed from 2 bytes to 4. Since it's the generic integer type, I expect it to increase again, although Embarcadero might keep it 4 bytes. Enumerations will use the smallest size that will fit all values, but you can specify a minimum size for enumeration types by using {$Z1} or {$Z2} or {$Z4} {$MINENUMSIZE 1} or {$MINENUMSIZE 2} or {$MINENUMSIZE 4}. I think they might add a {$Z8} to it too. This is related to (packed) records and their alignment of the record fields. – Lexis

@Workshop Alex - I'm on the Emabarcadero Delphi compiler team; trust me when I say that Integer will stay 32-bit :) – Zippy

@Workshop alex: Delphi 1 to Delphi 2 was a long time ago; they have had plenty of time to reflect on the impact that had. – Lionellionello

IMHO the real reason they need Integer and Cardinal to stay 32 bit long is they never used coherently those types and longint/longword across the VCL. The same issue Microsoft had across Windows APIs where LONG/ULONG and DWORD were used interchangeably. That's why Windows is the only OS to use the LLP model instead of the LP one almost everybody else use. Now it's too late to correct without breaking a lot of code. Anyway if data types get "correct" names is far better for the future. – Rhodia

LLP64 keeps int<->long equivalence, LP64 keeps long<->ptr equivalence. It depends on your codebase what you think is more important, see also stack.nl/~marcov/buildfaq/#toc-Section-5.1 – Ladd

@Barry Kelly, I know you're in the Compiler team. :-) However, is it wise to keep it 4 bits when in the past, it's size changed with the number of bits of the processor? Why use a 4-bit integer when the processor works with 8-bit integers by default? – Lexis

It's kept 4 bytes because less change = less transitional bugs, and because that's the model on 64-bit Windows - LLP64. Integer corresponds to C's int; Longint corresponds to C's long. On 64-bit Windows, both of those types are 4 bytes. – Zippy

True. especially when we get the NativeInt type in return. Still, I'm an old-school Pascal programmer who started using Pascal somewhere in 1985. I've grown up with thinking the size of the Integer type is related to the size of the processor, thus a 64-bit processor would have 64-bits integers. :-) Keeping integer just 32-bits makes sense to reduce possible transitional bugs but it just feels as something horrible against my nature. :-) I would have preferred an Int32 type instead, woth a compiler option to make Integer either 32- or 64-bits. Just like the Real and Real48 types. – Lexis

Thus, int8, int16, int32 and int64 to specify integers with fixed size, or the Integer-type which would be controlled by a compiler directive. Same for byte, Word16, Word32 or DWord, word64 or QWord. Or maybe an array-like option that would make integers of flexible size, like Integer[16] for an integer of 2 bytes and Integer[64] for a 64-bit Integer. (Because arrays are declared as 'Array[] of integer', not 'Integer[]'...) This would actually allow integers of more flexible sizes, like Integer[21]... – Lexis

Answer 3

Depending on your code, you can try to compile it using FreePascal, which supports both 32-bit and 64-bit compilation. The compiler will warn you about possibly erroneous places in your code.

Flied

Answer 4

Many similar questions were asked when it was announced that Delphi 2009 would only create Unicode applications. In the end it turned out that most existing code ran just fine without changes. Tricky parts were code that assumed that SizeOf(Char) = 1 and 3rd party components that might be doing that.

I would expect the move to 64-bit Delphi to be a similar experience. Everything just works out of be box, except for code that plays tricks with pointers and assumes that SizeOf(Pointer) = 4 or SizeOf(Pointer) = SizeOf(Integer). You can already fix such issues today by calling SizeOf(Pointer) rather than hardcoding 4 and using NativeInt or NativeUInt when you need pointer-sized integers.

You should use SizeOf(Pointer) rather than SizeOf(NativeInt) if you want your code to work with Delphi 2007. Delphi 2007 has an unfortunate bug that causes SizeOf(NativeInt) to return 8 instead of 4 as it should. This was fixed in Delphi 2009.

Unsearchable

I did my transformation to Unicode quite fast, but anyway there were – Marchall

quite some changes: all IORoutines (reset, readln, rewrite, writeln) for reading from and writing to files do not work anymore for Unicode, in every procedure writing strings appropriate Unicode font should be possible to select. But altogether the transition was done quite smoothly. – Marchall

Answer 5

The vast majority of simple applications should work just fine. As far as I can see, only applications that manually make use of pointers are at a risk. Indeed, if a pointer now is 64-bit, and you use it in calculations together with integers or cardinals (that are still 32-bit by default), you will get into trouble. I also think it is rather common that declarations for API functions that take pointers as arguments are using cardinals instead of the (unsigned) native integer type.

To make code that works well on any platform, one should use NativeUInts (IIRC, don't have a Deplhi compiler right now) instead of cardinals when working with pointers and integers simultaneously.

Gottschalk

I think that is rather common to declare API that takes pointers as pointers parameters :) Also NativeUInt is a relatively new type - once it was Cardinal to play its role. There are some issue, think about the Tag property (noone knows yet what it will become), the wParam/lParam types in Windows messages, record sizes may change. – Rhodia

@Idsandon: True, but to many people a pointer is just a fancy name of a cardinal, and that "misconception" has been working well for rather long now. So it is a possible issue. So the Tag might become 64-bit? I suppose that will not break any existing code, though... – Gottschalk

twitter.com/#!/kylix_rd "As speculated, the Tag property will become a NativeInt." – Trident

Answer 6

As long as Embarcadero doesn't release official informations about their 64 bit implementation is not easy to tell. You should check any cast to/from Pointer, Integer and Cardinal assuming they are the native platform size, including object properties and references (i.e. storing an Integer in a TObject property, which is a pointer, or using Tag to store references and not numbers).

You must also ensure no code relies on the "wrap-around" effect when incrementing (or decrementing) a value at its maximum (minimum) size.

Check any code in structures that relies on the data size, and don't use SizeOf() correctly, and at large that SizeOf() is always used when the datasize matters. Check code that writes/read data to files, if sizes can change, especially if data need to be exchanged between 32 and 64 bit code.

Check Win64 changes, if the application calls API and manages Windows messages directly. Handcoded ASM code must be checked for 64 bit compatibility (there are far stricter rule to write 64 bit assembler).

Rhodia

TComponent.Tag should be NativeInt to handle the expected case of people casting TObject references and similar in and out of it. Integer and Cardinal should stay the same 32-bit, as that's the general policy on 64-bit Windows; it should also reduce the semantic changes of switching the bitness of the target. 64-bit built-in assembler probably won't exist; hopefully linking support will exist for nasm or similar. – Zippy

+1 mainly for the mention of ASM compatibility, as I have had those exact issues since the release of Delphi XE2 (x64). – Mannerless

Answer 7

Besides the obvious pointer<-> int tasks: (using intptr/nativeint/ptrint etc)

• Anything that you have as a binary blob (DLLs maybe OCX etc) need to be upgraded. This might include old SDKs for dongles etc.
• All tools that do something on binary level (debuggers,profilers, phone home tools) might need updates.
• Nearly all assembler and other very lowlevel tricks (e.g. dependant on VMT layout, debug format (tracebacks) dynamic loading stubs like in Jedi Apilib etc) needs to be updated
• check all own created headers for changes in packing and mistranslations that matter now pointer<>integer. The packing bit must not be underestimated
• Interfacing with Office and other external apps might change
• TComponent.tag is a longint now, and thus might remain longint, meaning that schemes that stuff pointers into component.tag may fail.
• x87 FPU is deprecated on x64, and in general SSE2 will be used for florating point. so floating point and its exception handling might work slightly differently, and extended might not be 80-bit (but 64-bit or, less likely 128-bit). This also relates to the usual rounding (copro controlwork) changes when interfacing wiht C code that expects a different fpu word.

The packing of records problem is something I noticed when porting existing headers to win64.

Ladd

Unless some unexpected incompatibility happens, TComponent.Tag should almost certainly become NativeInt. – Zippy

Curious: any metrics on (a) how many applications built with FPC were relying on Tag to reference an object, and (b) broke with a 64-bit target? – Zippy

Questions from users about this pop up from time to time, but are not that common, and its usage usually is very local and easily solved. OTOH most users on the fringe platforms are people with large serverside codebases (typically ex-kylixers), and they are the type that solve their problems themselves, silenty. I however sometimes try to convert code that people offer me at user group meetings, often D3/D4 hobbyist code , and then .tag use is much more common (and every dirty trick in the book is exploited sooner or later) – Ladd

I wouldn't say that the x87 FPU is deprecated, but it is certainly the case that Microsoft have decided to do their best to make it that way (and they really don't seem to like 80-bit FP values), although it is clearly technically possible to use the FPU/80-bit floats on Win64. – Pyromania

Answer 8

My 2 cents:

• in the old days every asm writer was to pushed to USE BASM

• external asm64 would be acceptable and the using the old inlclude xy.obj code, while any way a complete rewrite is required

• Debugger & CPU64: the question will be is this still there??

• D64 Float Extended: Is this still maintained as 80 bit float??

Hp

Chrysoberyl

Answer 9

As a complete guess, any code that doesn't depend on a specific word size, or can adapt its word size based on what the compiler tells it, will be fine.

Minneapolis

What are you calling a "word"? In Delphi, a word is always a 16 bit value. So I guess you spoke about "NativeInt"... because DWord will always stay 32 bit, such as integer will stay 32 bit also... – Baseborn

I'm using the CE definition, not the Delphi definition. – Minneapolis