Could Free Pascal benefit of something like Apache Maven?

Question

Apache Maven is a very popular build and dependency management tool in the Java open source ecosphere. I did some tests to find out if it can handle compiled Free Pascal / Delphi units and found it easy to implement. So it would be possible to

• release open source libraries precompiled for Free Pascal (or Delphi) in a public Maven repository
• include metadata in this repository which contains dependency information
• use Maven on the command line to download the open source library from the public repository, and automatically resolve all dependencies
• local repositories, working as proxies, could be used to cache frequently used binaries
• automatic checksum generation and verification (provided by Maven) would reduce the risk of downloading corrupted binaries
• source code and even documentation files could be provided with the binaries
• binaries can be provided with or without debug information
• continuous integration servers like Hudson, TeamCity or CruiseControl can be used to build projects whenever changes have been submitted to the source control system and notify developers about build errors

This way of dependency management could be very beneficial for open source projects which use many third party libraries with complex dependencies. It would avoid typical conflicts caused by using wrong versions.

For the developer, the workflow for editing and building a project would be reduced to a minimum:

• checkout the project source from internal version control system
• edit source file(s)
• run mvn package to automatically download all required third party libraries (precompiled units) if they are not yet in the workstation's local repository
• compile and run

The only additional file for Apache Maven which is required in the project folder is the POM.XML file containing the project information.

Edit: while Maven is usable for some of the required tasks, implementing a solution like Maven in native Free Pascal would have some advantages: no Java SDK required, support for all development platforms where Free Pascal is available, maintenance and plugin development in Pascal.

Usage of a Maven-like tool would not be helpful for open source projects only - commercial projects could access and use the artifacts in public Maven repositories in the same way as well.

Maven features are listed at http://maven.apache.org/maven-features.html

---

Update:

one use case could be the build of Lazarus, where Maven would download all required libraries and invoke the compiler with the necessary build path arguments. Changes in the dependencies on lower levels would be propagated automatically up to the parent build.

Possible benefits:

• less time needed to set up a new work station, no manual installation of third party libraries required
• less errors caused by wrong library versions, detection of version conflicts (for example if two libraries depend on different versions of a third library)
• artifacts which are created inhouse can be added to the local maven repository and shared between developers and project, central storage of all artifacts with metadata
• builds are reproducible, just by using the same source and project metadata file (pom.xml)
• can reduce development time and increase project stability

Update #2: FPMake

the FPMake build system for Free Pascal seems to be a tool with much potential, in many details it is quite similar to Maven:

• FPMake is a pascal based build system developed for and distributed with FPC
• FPMake standardizes the building by defining some limits like standard directories
• the command fppkg <packagename> will look in a database for the package, extract it, and then compile fpmake.pp and run it
• it has standard build targets (clean, build, install, ...)
• it can create a 'manifest' file suitable for import into a repository (like mvn deploy or mvn install), the manifest is an XML file which looks very similar to a pom.xml in Maven:

FPMake manifest file:

      <packages>
        <package name="my-package">
          <version major="0" minor="7" micro="6" build="1"/>
          <filename>my-package-0.7.6-1.zip</filename>
          <author>my name</author>
          <license>GPL</license>
          <homepageurl>http://www.freepascal.org/</homepageurl>
          <email>[email protected]</email>
          <description>this is the package description</description>
          <dependencies>
            <dependency>
              <package packagename="rtl"/>
            </dependency>
          </dependencies>
        </package>    
      </packages>

Logger

In response of the update: I don't understand how this could work. Either releases are already precompiled, or you have to deal with a version matrix (several thousands of revisions for both projects an year) that precludes prebuild binaries. – Thorley

In Maven terminology, the current release is a 'snapshot' version which can be compared with the trunk (or head) revision in Subversion. At the end of a successful build, all artifacts would be deployed to the Maven repository with version id '0.9-SNAPSHOT' until version 0.9 is stable. Then it would be deployed only one time as 0.9 and work on 0.9.1-SNAPSHOT would start. So there would be only a very small number of versions in the repository, one per stable (tagged) release and one snapshot of the current development version. – Logger

Great, I like the idea! Can I find more details somewhere? – Phenylketonuria

Answer 1

Freepascal has been working on a package system of its own in a cross between apt-get and freebsd ports style. (download source/build/install automatically), called fppkg. However work has stalled. People investing time are the bottleneck, not people wanting to choose tools.

As far as Maven goes, I don't like auxilary tools that need installation of huge external runtimes. It might be fine for a big major app (like Open Office), but not for an util.

I also prefer a tool that is designed to the FPC reality and workflow. Documentation tools, build tools, download systems, testsuite systems are already all there, it just need a person that dedicates a lot of time into it to make it happen.

Some typical problems when introducing a new technology in a project as FPC, and why it has a tendency to make its own tools:

• need to train 20+ committers in parttime.
• The only COMMON programming language you can assume is Free Pascal. Even Delphi inner workings can't be taken for granted to be known (many committers came directly to FPC or even still via TP or a Mac Pascal)
  • Obviously that makes something with plugins in a different language annoying.
• Bash script is a close second. (g)make third, but already a magnitude less.
• All servers are *nix-like (FreeBSD, OS X, Linux), but not all run Apache. (e.g. my FreeBSD mirror runs XSHTTPD)
• somebody most knowledgable must be dedicated maintainer for a long time. Fix problems, update/ do migrations etc. Perferably more than one for obvious reasons.
• a major pain are Linux distributions (and FreeBSD to a lesser degree), most maintainers of *nix packages are not capable of more than "./configure;make;make install", and must be spoonfed with a near buildable repository and auxilary files.
  • In-distribution packaging of FPC/Lazarus has always been important, and is still increasing
  • All distributions have their own special rules about metadata, depedancies, and how sources must be published. Particularly Debian/Ubuntu is very bureaucratic and slow.
  • Most don't like third party auto-installers on top of their systems (since that bypasses their dependancy control)

This all leads to the effective practice that own tools in Pascal with minimal scripting work best. Some tools used:

• Gmake is mainly used to parameterise the build process on a per directory level, a successor, fpcmake (not really a make derivative despite the name) has begun, but the migration hasn't completed.
• Latex and a latex to html conversion (tex4ht, but debian uses hevea) are used in the documentation building (the non library documentation)
• The community site (netscape community server which uses TCL scripting, a heavy complex application server) has been a trouble ever since it started, but specially lately since the maintainer became less active.
• Mantis has been a problem (specially the email module would crash or lame the server due to the volume), but it has been whipped into shape during successive updates and hard work of several lazarus devels. Currently it is a decent workhorse.
• lazarus.freepascal.org PHPBB forum OTOH is relatively painless since a lot of younger people know how to deal with it.
• The same goes for subversions (though the more advanced scale needs some adjusting, not everybody is deep into the ins and outs of mergetracking)

If somebody was really serious about Maven, I usually would ask him:

• to CRITICIALLY investigate the use for the project. In a very concrete way, with schedule and time estimates. Birds-eye level "everything's possible" overviews are essentialy worthless.
• Give some thought on future change of used technologies. Every technology is eventually replaced, even the in-house ones, in 18 year+ projects. A new technology must not make migrations of other infrastructural components hard or involved. The new technology to end all new technologies doesn't exist.
• Make a migration plan. Migration is often underrated and underestimated.
• And in the end, there is always the 1000000 Euro question, who will do the daily maintenance?

Keep in mind that in a company you just kick the person responsible for the application server. But in an informal environment this is way harder, specially long term, since people's lives, occupations and time spent on the project vary.

Thorley

Mavens intention is not to replace all existing tools for documentation and testing. Instead the usage of Maven plugins for these tools, executing within the mvn build process, makes it possible to integrate these tools in one workflow. Maven could also pick up docmentation output and integrate it a HTML site, using the 'mvn site' goal. – Logger

Keep in mind that FPC uses a cluster of mirrors and "borrowed" servers/VMs. It's not like in the corporate world where continuous integration is done on a single server where you have full control. It is still not clear what Maven would actually bring to the table, except another overly complex component. – Thorley

Please explain, do you mean the FPC website uses mirrors? Maven is not meant to be run on the FPC website, the repository is nothing more than files placed on a HTTP server. Continuous integration will always run in the developer's system, not in a 'cloud'. – Logger

Yes, mirrors, and even building and testsuites is distributed over several servers. (Due to the high amount of platforms and architectures). I enhanced my post above with a short overview of lessons and issues we encountered over the years. – Thorley

External runtimes are not required if a solution similar to Maven can be built using Free Pascal. Maven is open source so it would be possible to reuse many of its concepts and data structures, like the project object model and the repository layout. I edited my question to make clear that I do not suggest the original Maven. – Logger

From what I hear in the discussions on the various forums that you have started, fppkg already does this. The concepts are not new or unique to Maven. – Thorley

FPMake at wiki.freepascal.org/FPMake is very interesting, the manifest file almost looks like a Maven pom.xml ... – Logger

fpmake and fppkg are a team. fpmake is the build system, fppkg the package repository and downloading part. – Thorley

a) Why do you mention Apache vs. XSHTTPD?? The fact that Maven is published by Apache has nothing to do with their HTTPD server! b) You complain about scarce resources, but vote for writing every tool "yourself". Where's the sense in that? IMHO the software communities better not invent "their own" stack of tools for such purposes over and over again, but better work together on generic tools with plug-ins where necessary. Defining a new module information file similiar to pom.xml is IMHO somewhat absurd – Phenylketonuria

That's the only thing you get from reading the reply. Interesting, you totally miss the point, namely that while Maven is easier, the fact that it is totally foreign to the project means that there are less people able and motivated to work on those plugins. Moreover, generic tools are always more complicated to use than tailored tools. – Thorley

you also could use a similar method by building around this(that is windows but indifferent os you could modify the correct commands) https://mcmap.net/q/89599/-premature-end-of-script-headers – Somato

A lot of infra has been changed to github last summer. Not that everybody is happy with that, but that means the picture above is outdated. – Thorley

Answer 2

Sounds like an interesting plan, but the Delphi community (and FPC even more so, I'd imagine!) values libraries as source far more than precompiled libraries. The general consensus is that anyone who uses a binary-only library is a fool, for two reasons: You can't fix any bugs you find in it, and compiler changes will break compatibility.

Byandby

see my edit - source code and documentation are available in the repository, binaries can be provided in two versions (with or without debug information). Users can choose artifact download with or without source code (for building only). – Logger

given that the FPC compiler is quite slow compared to Delphi, I assume prebuilt libraries are not an unusal way to reduce the turnaround time for FPC developers – Logger

Keep in mind that FPC is nearly twice as fast on Linux/Freebsd as on windows. FPC is mainly more many-files I/O intensive and binary startup-time, something that is roughly a factor two faster on *nix. The problem with binaries is more the matrix of targets and having that hardware available. Daily windows and linux snapshots have been built since 1995-6 – Thorley

FPC+lazarus build on e.g. linux in say 3 minutes (Core2 2GHz+) – Thorley

3 minutes built from source - how fast when built using precompiled libraries? The idea is to make it as easy as possible to use precompiled libraries to reduce the build time. Another option which comes to mind is the usage of archives, to reduce the number of I/O operations. – Logger

Precompiled stuff is version dependant, and increases the matrix of binaries to be released by magnitudes. Releases are already released in percompiled form, and don't need an update tool. Moreover, the core FPC and Lazarus SVNs are the big ones (with millions of lines each), the rest of the packages will all compile in under a few secs. – Thorley