ZeroConf/Bonjour Code that works in Delphi 7 not working in 2009

Question

I have the following declaration for DNSServiceRegister:

  function DNSServiceRegister
      (
      var sdRef: TDNSServiceRef;
      const flags: TDNSServiceFlags;
      const interfaceIndex: uint32_t;
      const name: PUTF8String;                    //* may be NULL */
      const regType: PUTF8String;
      const domain: PUTF8String;                  //* may be NULL */
      const host: PUTF8String;                    //* may be NULL */
      const port: uint16_t;
      const txtLen: uint16_t;
      const txtRecord: Pointer;                 //* may be NULL */
      const callBack: TDNSServiceRegisterReply; //* may be NULL */
      const context: Pointer                    //* may be NULL */
      ): TDNSServiceErrorType; stdcall; external DNSSD_DLL;

In my Bonjour framework I have the following response to an announced service being made active (i.e. to actually start announcing itself, via Bonjour):

  procedure TAnnouncedService.Activate;
  var
    flags: Cardinal;
    name: UTF8String;
    svc: UTF8String;
    pn: PUTF8String;
    ps: PUTF8String;
  begin
    fPreAnnouncedServiceName := ServiceName;

    inherited;

    if AutoRename then
      flags := 0
    else
      flags := kDNSServiceFlagsNoAutoRename;  { - do not auto-rename }

    if (ServiceName <> '') then
    begin
      name  := ServiceName;
      pn    := PUTF8String(name);
    end
    else
      pn := NIL;

    svc := ServiceType;
    ps  := PUTF8String(svc);

    CheckAPIResult(DNSServiceRegister(fHandle,
                                      flags,
                                      0 { interfaceID - register on all interfaces },
                                      pn,
                                      ps,
                                      NIL { domain - register in all available },
                                      NIL { hostname - use default },
                                      ReverseBytes(Port),
                                      0   { txtLen },
                                      NIL { txtRecord },
                                      DNSServiceRegisterReply,
                                      self));
    TBonjourEventHandler.Create(fHandle);
  end;

This is more verbose than I think it strictly needs to be, certainly it was working perfectly well in Delphi 7 in a much less verbose form. I have expanded a lot of operations into explicit steps to facilitate debugging, e.g. to be able to identify any implicit transforms of string payloads that may be occuring "under the hood" in Delphi 2009.

Even in this untidy expanded form this code compiles and works perfectly well in Delphi 7, but if I compile and run with Delphi 2009 I get no announcement of my service.

For example, if I run this code as part of a Delphi 7 application to register a _daap._tcp service (an iTunes shared library) I see it pop-up in a running instance of iTunes. If I recompile the exact same application without modification in Delphi 2009 and run it, I do not see my service appearing in iTunes.

I get the same behaviour when monitoring with the dns-sd command line utility. That is, service code compiled with Delphi 7 behaves as I expect, compiled in Delphi 2009 - nothing.

I am not getting any errors from the Bonjour API - the DNSServiceRegisterReply callback is being called with an ErrorCode of 0 (zero), i.e. success, and if I supply a NIL name parameter with AutoRename specified in the flags then my service is allocated the correct default name. But still the service does not show up in iTunes.

I am at a loss as to what is going on.

As you might be able to tell from the expansion of the code, I have been chasing potential errors being introduced by the Unicode implementation in Delphi 2009, but this seems to be leading me nowhere.

The code was originally developed against version 1.0.3 of the Bonjour API/SDK. I've since updated to 1.0.6 in case that was somehow involved, without any success. afaict 1.0.6 merely added a new function for obtaining "properties", which currently supports only a "DaemonVersion" property for obtaining the Bonjour version - this is working perfectly.

NOTE: I'm aware that the code as it stands is not technically UTF8-safe in Delphi 7 - I have eliminated explicit conversions as far as possible so as to keep things as simple as possible for the automatic conversions that Delphi 2009 applies. My aim now is to get this working in Delphi 2009 then work backward from that solution to hopefully find a compatible approach for earlier versions of Delphi.

NOTE ALSO: I originally also had problems with browsing for advertised services, i.e. identifying an actual iTunes shared library on the network. Those issues were caused by the Unicode handling in Delphi 2009 and have been resolved. My Delphi 2009 code is just as capable of identifying an actual iTunes shared library and querying it's TXT records. It's only this service registration that isn't working.

I must be missing something stupid and obvious.

Does anyone have any ideas?!

UPDATE

Having returned to this problem I have now discovered the following:

If I have a pre-D2009 and a D2009+ IDE open (e.g D2006 and D2010) with the same project loaded into both IDE's concurrently:

• Build and run under 2006: It works - my service announcement is picked up by iTunes
• Switch to D2010 and run (without building): It does a minimal compile, runs and works.

• Do a full build in D2010: It stops working

• Switch back to D2006 and run (without building): It doesn't work

• Do a full build in D2006: It works again

Does this give anyone any other ideas?

Umbrian

Please mail me a copy of your working source code; I'm at EKON13/DelphiLive Germany right now, and when my spare laptop has arrived (primary broke this morning: Beeeeeeeep beep beep) and my sessions restored, I'll try to get your stuff reproduced. Almost anything at the pluimers dot com domain ends in my mailbox, but using my firstname usually works best :-) – Nunez

That's a very generous offer, Jeroen. Thanks. I shall prep a framework project with the necessary infrastructure support (I'm using my own threading class etc) and send it as soon as I get the chance. This is driving me round the bend! The intention was to get the code to a point where I could share it with the community so rest assured your efforts won't disappear into my own private code bank. :) – Umbrian

This code should not be calling ReverseBytes. You are trying to convert from host byte order to network byte order. For that you need htons. – Torpedo

@David pt 1: Utterly irrelevant and afaict factually wrong. The problem was not that ReverseBytes() was being called, it was that the Delphi 2009 compiler was calling it differently (wrongly?) as compared to the Delphi 2007 and earlier compilers. Whether this was due to the longword overload rather than the word overload being called, or some conversion of the Integer value to a Word I never bothered to investigate since it mattered not one jot. Fixing the type of the parameter value ensured that the correct, and required, ReverseByte() implementation was invoked as required. – Umbrian

@David pt 2: Furthermore, whether a call to htons() would have been broken by the same type mismatch might be interesting to investigate, but not relevant or important since the code is written for a little endian machine by it's use of the Windows Bonjour implementation that underpins it, and thus isn't portable to any big endian machine that Delphi supports that I am aware of. If I ever contemplate porting the code to such an environment this may become a consideration, so I shall document the code accordingly in anticipation of that day, should it ever arrive. So, thanks for that. :) – Umbrian

The point is to make it clear what the code is doing. The goal is not to reverse bytes. The goal is to convert to network byte order, however that may be affected. It's like the difference between writing *2 and shl 1. End result is the same, but only one of them conveys the true intent. – Torpedo

By the way, what was factually wrong? – Torpedo

"Should not be calling ReverseBytes()". A call to htons() is a call to reversebytes by any other name when compiling for a little endian environment. So the code absolutely did need to call "reversebytes" or some variant there-of. htons() is just once such variant which includes a redundant - in this case - pre-check that the host is not itself big endian. – Umbrian

There's most definitely no runtime check in htons. What makes you think that would be needed? – Torpedo

Did I say "runtime" ? I haven't checked the source of htons() in the Winsock DLL, so I can't say for certain whether the pre-check is in the build process or is indeed a runtime check. I presume you must have seen the source if you can say /for certain/ that the check is not made at runtime. ;) – Umbrian

Oh, you didn't mean runtime. What did you mean? Why would you need to see source code? Object code is enough. Why would anyone implement htons with a runtime check? Compilers know whether the target architecture is little endian or big endian. Obviously. You really should not need me to teach you this sort of thing. – Torpedo

I dunno - why would anyone implement a compiler that would shove 32-bit integers into 16-bit parameters without even mentioning the mis-match? Sometimes people do the unexpected. /You/ aren't teaching /me/ anything. ;) – Umbrian

The Delphi compiler is not what we are talking about. We are talking about how to code allowing for host and network byte order. You are right on one point though. I am not teaching you anything. – Torpedo

/YOU/ are talking about a non-existent need to ensure that this non-portable code be portable. What other non-problems are you going to spend your time solving today ? Geez. – Umbrian

No, I'm talking about a real motivation for function names to describe the true conceptual task that they perform, rather than one specific implementation of that task. But you don't understand that concept. I'm looking at a bigger broader picture than the one you can see. You are consumed by the specific implementation details, but I'm talking about the abstract. We are just seeing this on a different conceptual level. So no, I'm not succeeding in teaching you anything at all. – Torpedo

Answer 1

The answer to this is mind boggling. On the one hand I made a completely stupid, very simple mistake, but on the other hand it should never - as far as I can see - have worked in ANY version of Delphi!

The problem was nothing what-so-ever to do with the Unicode/non-unicodeness of any strings, but was actually due to a type mismatch in the PORT parameter.

I was passing in the result of ReverseBytes(Port) - that parameter expected a uint16_t, i.e. a Word value. My Port property was however declared (lazily) as an Integer!!

Once I fixed this and had Port declared as a Word, it now works on both D2007- and D2009+ versions of Delphi.

Very weird.

I can only think that some other edge-case behaviour of the compiler that might have somehow affected this was changed when Unicode support was introduced.

Umbrian

Can you post the code of the ReverseBytes method? I was doing some C header conversions and porting from D2006 to DXE2 at the same time and wonder about some odd behaviour; if some odd behaviour (not getting any results from an AS400) matches with your observations. – Nunez

Sure. But rather than posting it here I put that code up on my blog. Enjoy. :) deltics.co.nz/blog/?p=1233 – Umbrian

For record: byte swap / rotate operations are usually a single contained x86 cpu command, while pure pascal procedures like that usually take half-dozen of commands and require RAM access. Computation-intensive components are usually have both pure-pascal version and optimized asm version - especially so now, as we have INLINE procedures in Delphi. Ho! i remember inline procedures in Turbo Pascal. And THAT was a trick :-) – Grahamgrahame

For example all database engines would need indexing and searching, like TDBF.sf.net One more example is compressing or crypto, for example code.google.com/p/delphi-spring-framework/issues/detail?id=38 And many more. Just looking inside such libs would give you readymade optimized and checked implementations – Grahamgrahame

16 bit swap: XCHG AH, AL 32 bit swap: BSWAP EAX 64 bit swap: BSWAP RAX – Kenny

I can't get reproduce the behaviour you describe. The only plausible explanation is that the behaviour change did not occur when you switch compilers. But it was introduced when you added the LongWord overload of ReverseBytes. – Torpedo

Sorry, scratch that. My repro attempt was bogus. I can repro. – Torpedo

@MADHatter 1: 16-bit swap is compiler magic docwiki.embarcadero.com/Libraries/en/System.Swap :-) // 2: BSWAP RAX is random generator. 1st parameter in Delphi win64 is RCX AFAIR. // 3: that is too little. You should make 3x3 codes (16,32,64)x(Pure Pascal, x86, x64). Too much for comments. DUnit-tested implementations can be taken from spring4d link above – Grahamgrahame

@Arioch: neverthless, functions return their result in RAX... you may move data in it before or later, but you have to do it. Of course in a comment there was just simple examples, not fully working routines. Why should I more complex "Pure Pascal" implementations I have no need for given those are one instruction functions with a simple and documented behaviour? When there will be an ARM compiler I will think about it if I ever write ARM code. – Kenny

@MadHatter PurePascal serves three functions: 1) they allow the library to be used when optimization is not yet available, like aforementioned ARM or MIPS, in compilers/environments which calling convention were not yet investigated; 2) as a reference implementation for unit tests; 3) as a reference implementation for porting. Pascal code is higher-level than asm code and implement ASM x64 routine reading Pascal code often is simpler than looking at ASM x86. And if someone is from embedded world he may be incapable of reading x86 asm at all. – Grahamgrahame

@Arioch: when you're manipulating low level data a pure Pascal implementation is often unreadable, and as a reference implementation, useless - it may be so complex compared to the assembly code there's a good chance it is buggy while the ASM is not. There are no Delphi compiler for ARM (and guess there will never be one for MIPS...), thereby I just throw a compilation error if it's not an x86 compiler. When ARM will be available, I'll write the needed assembly directly. – Kenny

@Mad: Pascal implementation is usually more or less direct translation of mathematical formulas. So they are conceptually cleaner. That is my personal experience of course, YMMV. – Grahamgrahame

Answer 2

Based on the information that we have available here, the situation is this:

• When calling the DLL with your code in Delphi 2007, it gives one result.
• When calling the same DLL with your code in Delphi 2009, it gives another result.
• The suspicion is, that it is related to the Delphi 2009 compiler.

Logically, the difference must therefore be, that Delphi 2009 sends different values as parameters. In order to make the debugging truly Delphi-independent, you therefore need to create a dummy DLL, which reports the values it gets. Other Delphi-dependent methods may be applied, like looking at the disassembly of the function-call into the DLL, and debugging it so that we know exactly what values are passed, and how, to the DLL, in both compilers.

Branton

Answer 3

I can't find the declaration instruction for the vars "ServiceName" and "ServiceType" in your code sample.

Assuming a String type (thus a unicode string), I guess (yes... no D2009 available to test this) lazy typecasting could be an issue:

name  := ServiceName;

Why not use the following?

name  := PAnsiChar(AnsiString(ServiceName)) 

Anyhow... just my 2 cts.

BTW: I always use the pre defined "EmptyStr", "EmptyWideStr" ... so the test would look like:

if (ServiceName <> EmptyStr) then

which should be safe and avoid the confusion of types.

On the other side, Delphi may interpret the '' as an ANSIChar like the following declaration do:

const
  MyParagraphChar = '§';

Not sure... I'm confused - should go home now ;)

Ioved

ServiceName and ServiceType are properties of the component (TAnnouncedService) and yes, they are String, and therefore (in D2009) Unicode. If explicit casting is required I will be both surprised and disappointed. I thought direct assignment of one string to another was supposed to take care of any transcoding - it's type casting that is - I thought - dangerous. But I shall try it. Thanks for the suggestion. re: EmptyStr, I find that this makes tests for '' look like any other comparison with another string var and harder to spot as a result ('' is always a NIL pointer anyway, isn't it?) – Umbrian

Answer 4

If the DLL is not written using Delphi 2009, you may want to use something else than PUTF8String. The Delphi 2009 Utf8String type is different from Delphi 2007's UTF8String type.

If the DLL was written using C/C++, I strongly suggest to use PAnsiChar() instead of PUtf8String.

Branton

I should perhaps have explained that the code posted is only the latest in a number of iterations. The C function import was originally declared using PUTF8Char, a type that I had explicitly introduced in Delphi 7 (and retained in D2009) as follows: UTF8Char = type ANSIChar; PUTF8Char = ^UTF8Char; When it didn't work I modified the code to use UTF8String, thinking that leveraging the built in, consistent support for transformations between encodings I might eliminate any errors that I had unwittingly introduced. – Umbrian

As far as I can see, the only thing that changed between Delphi 2007 and 2009, that relates to your code, is how utf8strings work. I can only strongly recommend that you do not use utf8string in a DLL api. – Branton

Kindly read previous comment. It does not work even if I do NOT use UTF8String and use my own PUTF8Char instead (effectively an alias for PANSIChar, exactly as you suggested). – Umbrian

I didn't downvote your answer previously because I hadn't been clear in my original answer, but since you've not addressed the fact that your suggestion doesn't work even after I clarified the point I have now downvoted. I shall also update the question this evening to eliminate the possibility for other potential answerers that it may be the use of a UTF8String type in the C fn prototype that is somehow responsible. – Umbrian

Did you try to disassemble the code, or to implement a test-DLL that reports, what you are actually passing as parameters? – Branton

Btw - moderate as you see fit, I don't care. I'm just here to help you out ;-) – Branton

Doing this from work, so will have to be brief. Yes I did compare the CPU window in each compiler. I don't speak x86 though so won't try to pretend that I understood what I saw. From what I recall, the only difference was that where the Delphi 7 code was calling LStrAsg the Delphi 2009 was calling UStrAsg, but I can't remember precisely. When I update the question this evening I shall include the CPU view from each compiler. – Umbrian