Industry-style practices for increasing productivity in a small scientific environment

Question

I work in a small, independent scientific lab in a university in the United States, and it has come to my notice that, compared with a lot of practices that are ostensibly followed in the industry, like daily checkout into a version control system, use of a single IDE/editor for all languages (like emacs), etc, we follow rather shoddy programming practices.

So, I was thinking of getting together all my programs, scripts, etc, and building a streamlined environment to increase productivity. I'd like suggestions from people on Stack Overflow for the same. Here is my primary plan.: I use MATLAB, C and Python scripts, and I'd like to edit, compile them from a single editor, and ensure correct version control.

(questions/things for which I'd like suggestions are in italics)

1] Install Cygwin, and get it to work well with Windows so I can use git or a similar version control system (is there a DVCS which can work directly from the windows CLI, so I can skip the Cygwin step?).

2] Set up emacs to work with C, Python, and MATLAB files, so I can edit and compile all three at once from a single editor (say, emacs)

(I'm not very familiar with the emacs menu, but is there a way to set the path to the compiler for certain languages? I know I can Google this, but emacs documentation has proved very hard for me to read so far, so I'd appreciate it if someone told me in simple language)

3] Start checking in code at the end of each day or half-day so as to maintain a proper path of progress of my code (two questions),

• can you checkout files directly from emacs?

• is there a way to checkout LabVIEW files into a DVCS like git?

Lastly, I'd like to apologize for the rather vague nature of the question, and hope I shall learn to ask better questions over time. I'd appreciate it if people gave their suggestions, though, and point to any resources which may help me learn.

Paugh

Answer 1

1) Both git and mercurial will work under Windows natively - no need for Cygwin - an environment I would avoid if at all possible.

2) Don't see why you think you have to use emacs.

3) You don't check in code at a particular time of day or with a particular frequency - you check in when you have made a distinct change that has been tested and works. Or if you are on your own branch, whenever it is convenient.

Counterfoil

@ax That was not really my main point. Certainly, if people want to use emacs, they should - I have zero interest or intent in starting another boring flame war. – Counterfoil

i know, i agree, and i don't want to start a boring flame war neither. just +1'in the notion that there are other editors beside emacs. – Exceedingly

Answer 2

It's great you're going this route. Here's a few things to add to that list:

1) Add in testing to your work cycle. That is, once some segment has been written have other people who haven't worked on it review the code. Then write tests to make sure that the new code integrates well with other code and produces the results you'd like to see, and finally make these tests a part of the overnight build process.

2) Get an overnight build process going. Or at least some automated process that rebuilds your code every time someone checks code in. Have it run all your tests on that build.

Auctorial

Answer 3

Trying to follow some sort of best practice will benefit you a lot, but you might find that many people are not really interested in following them. Try to advertize the benefits in subtle ways so you don't intimidate your collegues -- if they like what they see they might eventually adopt some of your suggestions.

Scientific environments where many people only care about graduating fast or getting their funding proposals approved seem to nuture a hit-n-run, cowboy style of coding. If you can make people think about quality and reproducibility at all you have already won.

What I found useful:

• Always try to break your code in small subprojects that you can easily test. Run your tests on every check-in. You can use git's submodules to assemble your "applications".
• If multiple people can commit to the repository use something like astyle to have consistent formatting (probably less an issue for Python code).
• Set up something like gitweb/cgit/... so people can get tarballs of code and get used to the idea that source control is useful.
• Try to find a coding buddy in your group/community. Code review is very hard to do in an environment where people are prone to "just get it done", often throwing good style/common sense over board.

Cesta

Answer 4

As others have mentioned, checking in at regular intervals isn't terribly important. Check in when you've hit a milestone (you've made a change that works with the rest of the code, and it builds properly. Automated builds are much more important. They validate that you've checked in all relevant changes.

I think you may be a little confused about hosting source control vs. clients for source control. Clients for CVS, SVN, Git and others are almost universally available for windows. Hosting is available from many web hosting companies. It may be a good option, since it would also effectively give you an offsite backup of your codebase for ~100$/year. Alternately you can just load the server on a low end pc and put it in a closet somewhere, although the electricity cost is probably as much as you'd pay for hosting.

Using a single IDE for all languages isn't important at all, consistency within projects of the same language is important. All of your projects within one language should probably have a shared coding standard. They should also build on the same compiler, and/or run on the same interpreter/VM and have up to date project files for whatever IDE you use (if applicable).

Ionopause

Sadly, "builds"!="is good". For some quality you want at least a small test suite. These still cannot catch inefficient code or crappy API designs -- for these you need peer review (and it's the hardest part to get). – Cesta

"I think you may be a little confused about hosting source control vs. clients for source control" A clarification : not really, I know that source control clients are available for Windows. The reason for my asking about the Windows CLI was wanting to use a DVCS from within emacs so I didn't have to shift focus to anything else for code checkout. But, if, as people have said, frequent code checkout isn't terribly important (I think I have been blurring the lines between checkout and backup), then that eliminates the need. – Paugh

@drachen: source control can be a kind of backup. Commit whenever something works. Then you can go back later just like you should be able to set up a old measurement from a lab book entry. – Cesta

Answer 5

is there a way to checkout LabVIEW files into a DVCS like git?

There is no reason why LabVIEW files couldn't be in a DVCS, I'm using Mercurial. However since LabVIEW files are treated as binary files the size of your repo will grow fast.

Out of the box LabVIEW only provides 3-way merges, to use a diff function between two revisions you need LV-diff, with a few tweaks to support tools that only change the location of the files (eg. duplicates filenames).

Also it's important to add *.lvlps and *.aliases to your ignore list, these files don't have a usefull meaning on another computer.

Tractile