I strongly recommend you talk to TIBCO about what you want to do. Adapters are rarely used without a tranformation engine such as TIBCO BusinessWorks. All solutions are different, but a typical integration is shown in the figure below. BusinessWorks operates as a "semantic" adapter to expose an application through a standardised service interface. The application specific Adapter operates as a "technical" adapter to help you work with a specific product, API or technology. Standards have come a long way in the last 10 years and a technical Adapter is not always necessary unless you have some complex or tricky API to integrate with (i.e. BusinessWorks can often do the job by itself).
But to answer your questions...
Q. What are ActiveEnterprise adapters and ActiveEnterprise Schemas? When and why to use them?
Ans. ActiveEnterprise is the (old) brand name of the TIBCO integration suite. Newer products are called ActiveMatrix...but that's just branding. The Adapters operate as a bridge between Rendezvous and various products (Siebel, SAP etc) and technologies (ODBC, COM, CORBA, COBOL etc). An ActiveEnterprise schema is a schema representation of the wire-format for a tree-structured mssage between Adapters and other components. AE Schemas actually predate XML Schemas....most people use XML schemas and an XML wire format these days.
Q. What is difference between using Tibco Randezvous Transport directly and ActiveEnterprise adapter over Tibco Randezvous? I only found that messages send through adapter have more complicated structure defined by AESchema.
Ans. Rendezvous doesn't specify the form or content of your message. AE wire format was developed (before XML) to represent complex messages. AESchema is a representation of the message schemas. The Adapters usually generate AESchema from the product API or object model. E.g. SAP Adapter can generate AESchema definitions from IDocs.
Q. Is it possible to use XSD schemes to define messages send by Tibco Randezvous transport or ActiveEnterprise adapters?
Ans. I think there is a tool to generate an AESchema from XSD, but that is not the usual mode of operation. Usually, an Adapter runs behind TIBCO BusinessWorks which operates as a "semantic adapter" converting AE wire format messages to/from XML wire format messages. BusinessWorks supports messages in AE wire format and in XML wire format (defined using XSD).
Q. Is there anything like WSDL document to describe what messages the service expects / provide?
Ans. BusinessWorks supports SOAP web services defined via WSDLs (either generated within the tool or from an external WSDL).
Q. How to write client for a service using ActiveEnterprise adapters and AESchemas? (better how to write such program in .NET!). I just found only Tibco Adapter SDK which looks like the only option to work with AESchemas. For direct usage of Tibco Randezvous in .NET I found TIBCO.Randezvous library hich works pretty good but it doesn't have support for AESchemas (or I missed it)
Ans. BusinessWorks operates as an AE client, or you can use the Adapter SDK. It depends what you want to do. Rendezvous knows nothing about AE wire format. AE is an adapter level concept. You are better off using BusinessWorks as a SOAP Server and writing your .NET client against that (see figure).
Once again...talk to TIBCO about what you want to do because every application is different and the state-of-the-art and product suite has moved on considerably since the old ActiveEnterprise days.
Finally, if you are interested in Rendezvous vs EMS, you might want to look at this question.