Portable scripting language for a multi-server admin?

Question

Please Note: Portable as in portableapps.com, not in the traditional sense of a language that can be used on multiple architectures or operating systems. Whoever coined this usage of the word portable should be whacked. :)

I'm a DBA and sysadmin, mostly for Windows machines running SQL Server. I'm looking for a programming/scripting language for Windows that doesn't require Admin access or an installer, needing no install process other than expanding it into a folder. My intent is to have a language for automation around which I can standardize.

Up to this point, I've been using a combination of batch files and Unix shell, using sh.exe from UnxUtils but it's far from a perfect solution.

I've evaluated a handful of options, all of them have at least one serious shortcoming or another. I have a strong preference for something open source or dual license, but I'm more interested in finding the right tool than anything else. Not interested that anything that relies on Cygwin or Java, but at this point I'd be fine with something that needs .NET.

Requirements:

• Manageable footprint (1-100 files, under 30 MB installed)
• Run on Windows XP and Server (2003+)
• No installer (exe, msi)
• No reliance on a JVM or Cygwin install
• Works with external pipes, processes, and files
• Support for MS SQL Server or ODBC connections

Bonus Points:

• Open Source
• FFI for calling functions in native DLLs
• GUI support (native or gtk, wx, fltk, etc)
• Linux, AIX, and/or OS X support
• Dynamic, object oriented and/or functional, interpreted or bytecode compiled; interactive development
• Able to package or compile scripts into executables

So far I've tried:

• Ruby: 148 MB on disk, 23000 files
• Portable Python: 54 MB on disk, 2800 files
• Strawberry Perl: 123 MB on disk, 3600 files
• REBOL: Great, except closed source and no MSSQL or ODBC in free version
• Squeak Smalltalk: Great, except poor support for scripting

Timbering

Of the ones that you tried, did any one of them work? Btw, I am a huge Python fan. – Lionhearted

Could this be a Serverfault question? – Shutz

@Hamish: Apologies for not making it clear, but the languages listed were those I tried but fell short for one reason or another. I could elaborate more, but Ruby, Python, and Perl all fell short because of the relative complexity of install, number of files, and size of installation. Other answers have suggested Iron Ruby and Iron Python, options I hadn't really looked at- I come from more of a Unix background and had been evaluating the Windows distros of mainline Ruby and Python. Python was the first language I was excited about, back in 99- and I still enjoy working in it! – Timbering

I know it's bad form to delay in picking an best answer- but I need a little time to compare the front runners: tclkit, some form of lua, Iron Python or Iron Ruby. Tomorrow, I promise! – Timbering

I understand the size requirement but why do you care about number of files in the "engine" ?? – Pomerleau

@Perica: I realize it might seem a bit oblique, but installs with thousands of files have ended up giving me trouble. Something encapsulated in a smaller number of files installs quickly, is easily upgraded, and doesn't usually require extensive registry or environmental changes. I'm a DBA and Unix admin; I don't want to step on the toes of the Windows admins and their installs, deal with global ENV vars set, or fiddle with complicated paths. May seem silly, but the ability to run with nothing more than "copy \\unc\tclkit.exe c:\dba & .\tclkit abc.tcl" is appealing in environment I work in. – Timbering

@Aaron: I do understand your concern about speed of installation. This is indeed point which I'm also trying to improve for next version of Portable Python. Users will be able to select packages to install, instead of installing entire distribution. However, the "registry and environmental changes" are totally not needed if you are using portable application. Portable applications don't, by definition, modify host OS. Easy upgrade should also be no problem as you can install new version side by side. This gives you option to first test all your apps before deleting old version of the "engine" – Pomerleau

Answer 1

Tclkit is a single-file, self-contained Tcl/Tk system. The mac version I have is about 3.8 megs. You can get a version for just about any modern OS. I carry around a thumb drive that has mac, windows and linux binaries so I can run my scripts on any platform. No install is required, just copy one file wherever you want.

The only thing it's missing from your original spec is MS SQL Server / ODBC support out of the box. I know people use tcl for that but I think you'll have to add an extra library or something. See the Tcl'ers wiki entry on MS SQL Server for more information.

Alaster

Some people seem to have a mental block that prevents them from understanding Tcl. I'm not sure why because it really is one of the more readable languages out there. There are only 12 rules that govern the entire language so it's not exactly a difficult language. – Alaster

Tcl works fine for that. You might want to add a few libs to the development environment if your doing windows stuff (tcom for OLE, twapi for good access to Windows APIs and one of the ODBC bindings to access SQL Server). WITS is a nice example what can be done with that stuff: wits.magicsplat.com – Martingale

Tcl is difficult because it is based on textual substitution. Berkeley and Stanford professors should know better than to create programming languages whose evaluation models are based on substitution. (Don Knuth, phone your office.) I will admit that Tcl's interface to C is the simplest and easiest I've ever seen. But it pushes a huge burden out to the programmer of making all the native types representable as strings. – Volition

@Norman: Well, lisp/scheme is the same only the building blocks are lists of symbols instead of lists of text. But lisp gets all the love while tcl tend to get all the hate. – Illegal

@slebetman: The key thing about cons cells is that you can compose them to make more complex data structures. Strings are atomic (and in Tcl, immutable). It wouldn't be that big a deal except Tcl is designed to be embedded, and to map complex C data structures onto Tcl strings is a huge pain in the ass. (I did it many times in the early 1990s. Then I discovered Lua and left Tcl behind forever.) – Volition

@Norman: Don't directly use strings then, that's sooo 90's ;-) These days most tcl code deal with lists (and dicts). Then there is no real difference between tcl and lisp data structures. The only difference is that lisp's lowest level is the cons cell itself, which is similar to tcl's "word" in lists. But in tcl: a dict is just a specially formatted list, a list is just a specially formatted string and a string is just a sequence of characters. I find this unification of data from the ground up comforting. – Illegal

True, Tcl is not the best thing for complex data structures. That doesn't mean it's generally bad -- there is a huge world of applications that tcl is perfectly suitable for. I spent 15 years testing that theory and created many complex, successful tcl apps over that period of time. The language has its warts, but what language doesn't? I honestly believe that most people who dismiss Tcl (and I'm not suggesting that's anyone here per se) simply can't accept that its simplicity is its one of its greatest strengths. – Alaster

I didn't have the time to take a better look at all of the answers today, but this is the answer that I'll likely choose. I've actually used tclkit, but on Windows Mobile a few years back. Frankly, tcl nor tclkit hadn't even occurred to me when I started seeking out the language I wanted to standardize around- I'd looked for portable Python, Perl, and Ruby distros with Tk bindings but not Tcl. I think tcl would do well for my needs and mesh with my affinity for Lisp and Rebol. I mostly deal in automating my tasks, requiring a good deal of str ops on files and pipes. – Timbering

Answer 2

I urge you to try Lua. Regarding your requirements:

• Tiny footprint (56 source files, under 150K compiled)
• Runs everywhere (uses only ANSI C)
• No installer needed; you compile from source (there's also a "batteries included" package that I haven't explored
• Doesn't need JVM and works with any ANSI C compiler, so you can compile with Visual Studio, not Cygwin
• Works with external processes and files but only to the extent supported by ANSI C. If POSIX popen is provided then that is supported also.

And your bonus points:

• Open source (MIT license)
• FFI to C is brilliantly conceived and executed—not quite as simple as Tcl but loads more powerful. Much better integration with C than Python or Ruby.
• GUI support is mixed but there are good bindings for wx widgets. QT support was there at one time but I don't know if it has been maintained.
• Linux is supported
• Language/compiler features:
  • Dynamic
  • Functional
  • Prototype-based objects and inheritance through metamethods (you'll want to see examples in the book below
  • Fastest bytecode compiler in the West
  • Interactive read-eval-print loop; load new code dynamically
• Able to package scripts into executables; either use Luiz de Figueiredo's srlua, or I can send you a 120-line Lua script that converts Lua source to a .c file that you link in with your app and the interpreter to make an executable.

Additional bonus points:

• Very crisp, clean, well-designed language.
• Small enough to master in its entirety and to be productive within a day.
• Superb book Programming in Lua (check out the previous edition free online)

Volition

Excellent suggestion. Especially for Windows, Idle might also be worth a look. It is a proper superset of Lua 5.1 (Every syntactically correct Lua 5.1 program is also a syntactically correct and semantically identical Idle program), but it comes pre-compiled and pre-bundled with some either Windows-specific or hard-to-compile Lua extensions. All the features and advantages listed here equally apply to Idle, with only two caveats: community size and no Roberto Ierusalimschy. – Vachil

@Norman and @Joerg- what is your opinion of murgaLua? That looks to be along similar lines, but it's hard to see if it's headed for the dustbin or not... Not Lua in general, that doesn't look like it's going anywhere. I've some experience with NewtonScript, and Lua reminds me a lot of it. I'll take a look at the libraries and the distros you've recommended! – Timbering

@Aaron: Murga's thing looks dormant. The real advantage seems to be FLTK, but he hasn't kept pace with FLTK. And for XML I have had really good results with lxp and lxp.lom from PUC-Rio. For GUI I would be tempted to try IUP also from PUC-Rio, just because it was designed to work with Lua from the start. (I've used wxLua and Lua bindings for QT, and neither experience left me enthusiastic.) – Volition

Answer 3

There are a couple of options for Python that might fit your bill:

• The first is IronPython, which can be run without an installer and will play nicely with .net APIs. This gives you access to anything with a .net API or a COM typelib that you could build a PIA for. I've used at as a scripting mechanism for precisely this reason - it could be dropped into a directory within the system and did not need to be explicitly installed..
  
  You will have to have an appropriate .Net runtime installed, but .Net 2.0 is installed with SQL Server 2005. SQL Server can be accessed through ADO.net and building GUIs with Winforms is fairly straightforward.

• The second is Portable Python which is designed to be run off a USB key. Although I see you've already tried it, you might elaborate on what the shortcomings were. If something isn't available in the basic install you could always look into building a custom version with it included. TkInter (at least) is bundled.
  
  You can also use Py2EXE to generate standalone python applications with all superfluous junk stripped out. This will give you about 10 files or so (depending on the number of DLLs) that can be run from a single directory, possibly on a USB key.

Running local python installs on Unix-oid OS's is pretty straightforward, so that's pretty much a no brainer. Also, python comes with most linux distros and is available as 'contributed software' from most if not all trad unix vendors. IIRC it's also bundled with MacOS.

Biddle

+1 for elaboration on Portable Python shortcomings. If it is the size you can strip it down and remove packages you don't need. If you need extra modules, elaborate and I will see what I can do for the next release ;-) – Pomerleau

Does it come with the ODBC module in the standard build? I couldn't see anything in the documentation on the site. – Biddle

Answer 4

For tcl, apart from Tclkit, freewrap is another small portable, self-contained interpreter for tcl.

Just rename the freewrap executable to something else will convert it to a stand-alone interpreter. Renaming it back to freewrap will convert it to a script wrapper.

Also, freewrapped apps contain a tcl interpreter. In dire emergencies you can try opening the app as a zip file and edit/replace the tcl code contained within (just remember to make a copy first). This has saved me several times when I'm at a client site without development tools but need to troubleshoot something. I just make a copy of one of my deployed app and presto - instant development environment!

Illegal

Answer 5

Looking at wikipedia's exhaustive list of portable software There's Tiny C compiler, again on Wikipedia here, and its own homepage here.

To summarize by quoting from wikipedia's list of features:

• Small - can compile and execute C code everywhere, for example on rescue disks (about 100KB for x86 TCC executable, including C preprocessor, C compiler, assembler and linker).
• Fast - tcc generates optimized x86 code. No byte code overhead. It compiles, assembles and links about 9 times faster than GCC.
• Any C dynamic library can be used directly. TCC is heading towards full ISOC99 compliance. TCC can of course compile itself.
• Includes an optional memory and bound checker. Bound checked code can be mixed freely with standard code.
• Compile and execute C source directly. No linking or assembly necessary. Full C preprocessor and GNU-like assembler included.
• C script is supported: just add '#!/usr/local/bin/tcc -run' at the first line of your C source, and execute it directly from the command line.
• With libtcc, you can use TCC as a backend for dynamic code generation.
• Few dependencies. It includes its own hand-written lexer, and it is implemented using a recursive descent parser. Thus, building TCC requires few other libraries.
• Its LGPL license permits anyone to use, modify, and/or redistribute the software, and it can be used to develop either open source or proprietary software.

Hope this helps and would be of use, Best regards, Tom.

Lynn

Answer 6

Every somewhat modern Windows version comes pre-installed with both VBScript and JScript. The doesn't meet all your features (compile to an executable comes to mind), but they certainly have an unbeatable advantage with the installation size: it's hard to beat 0.

Powell

Answer 7

In addition to the Lua suggestion, there is also Idle. It is basically a superset of Lua 5.1, with both the language (and libraries) and the implementation based on Lua. It was originally created to be a more complete scripting solution for Windows: because Lua is primarly intended for embedding, it has a rather small standard library and it is usually expected that the embedding application provides a rich library to Lua.

This makes sense for an embedded language, because, after all, there isn't much common functionality between, say Adobe Lightroom, Nginx and World of Warcraft, so there simply is nothing you can put in a standard library. But for a more general purpose OS scripting language, one would want a slightly larger library. Thus, Idle bundles a couple of libraries that are third-party (and sometimes hard to get to work on Windows) in Lua in its standard library.

Some of the things that the Idle standard library adds over Lua are tight Win32 integration, SQLite3 support, networking support, a PEG parser generator and archive support.

Also, Idle has support for embedding Perl and C code into your Idle programs.

Powell