Is there a need for a "use strict" Python compiler?

Question

There exist static analysis tools for Python, but compile time checks tend to be diametrically opposed to the run-time binding philosophy that Python embraces. It's possible to wrap the standard Python interpreter with a static analysis tool to enforce some "use strict"-like constraints, but we don't see any widespread adoption of such a thing.

Is there something about Python that makes "use strict" behavior unnecessary or especially undesirable?

Alternatively, is the "use strict" behavior unnecessary in Perl, despite its widespread adoption?

Note: By "necessary" I mean "practically necessary", not strictly necessary. Obviously you can write Perl without "use strict," but (from what I've seen) most Perl programmers do use it.

Note: The Python interpreter-wrapper need not require "use strict"-like constraints -- you could use a pseudo-pragma similar to "use strict" that would be ignored by the normal interpreter. I'm not talking about adding a language-level feature.

---

Update: Explaining what "use strict" does in Perl per comments. (Link to official docs is in the first paragraph.)

The "use strict" directive has three distinct components, only two of which are really interesting:

• use strict vars: Statically checks lexically scoped variable usage in your program. (Keep in mind that, in Python, there is basically only global scope and local scope). Many Python linters check for this sort of thing. Since it's the only static analysis that they can do, the linters assume you use straightforward lexical scoping and warn you about things that appear wrong in that sense until you tell them to shut up; i.e.

  FOO = 12
  foo += 3
  

  If you're not doing anything fancy with your namespaces this can be useful to check for typos.

• use strict refs: Prevents symbolic namespace dereferencing. Python's closest analog is using locals() and globals() to do symbolic binding and identifier lookup.

• use strict subs: No real analog in Python.

Subscribe

it would help a bit if you explain what use strict does in perl and why is it needed? and why do many people use it all the time? what would happen without it? – Jimenez

See my answer below on what "use strict" actually does. There seems to be some confusion in this post and the comments about what it really means. Yes Perl programmers are in love with "use strict" but it doesn't make Perl more Java-y. – Kokura

of course python 3 complicates (clarifies?) things with the non-local declaration - as in python.org/dev/peps/pep-3104 – Adolfoadolph

I don't think it complicates things that badly. The constructs that create a new scope are the same, you can just access an intermediate local scope explicitly. Now you have three directly-addressable namespaces instead of two. – Subscribe

Here's two good (IMO) posts in favor of mandatory variable declaration (i.e. use strict 'vars'), one from a Perlmonk and one from a Pythonic perspective: perlmonks.org/?node_id=755790 mail.python.org/pipermail/python-3000/2006-October/003968.html – Worked

Answer 1

"the run-time binding philosophy that Python embraces... makes "use strict" behavior unnecessary [and] especially undesirable"

Pretty good summary. Thanks.

That is essentially it. Static analysis tools don't help Python enough to be worthwhile.

---

Edit

"I'm asking for us to introspect on why we don't need it and, relatedly, why Perl programmers think they do need it."

The reason why is precisely the reason you already gave. We don't need it because it doesn't help. Clearly, you don't like that answer, but there's not much more to be said. Compile-time or pre-compile time checking simply does not help.

However, since you took the time to asked the question again, I'll provide more evidence for the answer you already gave.

I write Java almost as much as I write Python. Java's static type checking does not prevent any logic problems; it doesn't facilitate meeting performance requirements; it doesn't help meet the use cases. It doesn't even reduce the volume of unit testing.

While static type checking does spot the occasional misuse of a method, you find this out just as quickly in Python. In Python you find it at unit test time because it won't run. Note: I'm not saying wrong types are found with lots of clever unit tests, I'm saying most wrong type issues are found through unhandled exceptions where the thing simply won't run far enough to get to test assertions.

The reason why is Pythonistas don't waste time on static checking is simple. We don't need it. It doesn't offer any value. It's a level of analysis that has no economic benefit. It doesn't make me any more able to solve the real problems that real people are having with their real data.

Look at the most popular SO Python questions that are language (not problem domain or library) related.

Is there any difference between "foo is None" and "foo == None"? -- == vs. is. No static checking can help with this. Also, see Is there a difference between `==` and `is` in Python?

What does ** (double star) and * (star) do for parameters? -- *x gives a list, **x gives a dictionary. If you don't know this, your program dies immediately when you try to do something inappropriate for those types. "What if your program never does anything 'inappropriate'". Then your program works. 'nuff said.

How can I represent an 'Enum' in Python? -- this is a plea for some kind of limited-domain type. A class with class-level values pretty much does that job. "What if someone changes the assignment". Easy to build. Override __set__ to raise an exception. Yes static checking might spot this. No, it doesn't happen in practice that someone gets confused about an enum constant and a variable; and when they do, it's easy to spot at run time. "What if the logic never gets executed". Well, that's poor design and poor unit testing. Throwing a compiler error and putting in wrong logic that's never tested is no better than what happens in a dynamic language when it's never tested.

Generator Expressions vs. List Comprehension -- static checking doesn't help resolve this question.

Why does 1+++2 = 3? -- static checking wouldn't spot this. 1+++2 in C is perfectly legal in spite of all the compiler checking. It's not the same thing in Python as it is in C, but just as legal. And just as confusing.

List of lists changes reflected across sublists unexpectedly -- This is entirely conceptual. Static checking can't help solve this problem either. The Java equivalent would also compile and behave badly.

Quarto

Your answer pretty much amounts to, "It just isn't necessary." I understand that lots of Python code has been written successfully without it -- I'm asking for us to introspect on why we don't need it and, relatedly, why Perl programmers think they do need it. – Subscribe

last verse contains a shorter explanation from the Zen of Python once again (on why static compile-time checking like 'use strict' would never work with python): Namespaces are one honking great idea -- let's do more of those! – Adolfoadolph

Again, "use strict" is not a static compile-time check. – Kokura

Yet, the question specifically says "There exist static analysis tools for Python, but..." so I'm lumping static analysis, as mentioned specifically in the question, along with the additional dynamic checks of use strict. In keeping with the original post. – Quarto

@mpeters: use strict at least has components which are compile time checks (use strict vars), which are clearly performed via static analysis. – Subscribe

@S.Lott: Thanks for expounding. Devil's advocate: some program incorrectness is easier to detect than others. I'm not saying we should statically type, but we can do static analysis of lexically scoped identifiers in a file (not attributes), thus preventing our program from inevitably bombing out. – Subscribe

@cdleary: I'll assume you're talking about magically detecting misspellings of identifiers in Python. If your unit tests don't reveal the uninitialized variables or misspelled method or function names, you have bigger issues that static checking cannot hope to resolve. – Quarto

@S.Lott: Most people who write "scripts" (in the strict sense) don't write unit tests, usually because of lots of side-effecty I/O. Unit tests make sense for application/framework programming, but for scripting scenarios a compiler flag to check identifiers seems useful, no? – Subscribe

@cdleary: Undefined identifiers already raise a run-time exception. That was easy. Anything else we can check for? – Quarto

@S.Lott: Right, but you don't want to be half way through one of the aforementioned scripts and have it bomb out because of a NameError that you could have detected at compile time, right? Is it useful to have an interpreter-wrapper that optionally runs this analysis at compile time? – Subscribe

@cdleary: Running an untested script that can do damage is not a use case I can imagine. I can't get past the "you want to run without testing?!?!?" part of this example. – Quarto

@S.Lott: Fair enough -- I find that part hard to understand as well, though I've known people to insist that testing and scripting were somehow at odds. – Subscribe

@cdleary: anyone who claims testing and scripting are at odds is simply refusing to test. They need to start a different career, far away from software. – Quarto

−1, this is just a rant, and objectively wrong. Other languages with similar philosophies (Perl, JavaScript) reap obvious benefits from use strict to the point that it has become an established best practice to always use it. Python would benefit to the same degree. – Mattos

Testing will tell you that there is a problem. But good luck trying to figure out where/why the problem occurs without some kind of static analysis. – Worked

@Worked any decent IDE (e.g. PyDev) performs static analysis, what would be the benefit of "use strict" pragma for pinpointing problems? – Respectability

@deathApril Your comment appears under an answer which claims that all forms of static analysis are pointless, but anyway... use strict would change the language to force the programmer to be more declarative about their intentions, so it can check whether their intentions are violated. A static analyser can't read the programmer's mind. – Worked

While you obviously don't need use strict to ensure that your code is correct, since tests do that, it would definitely help you code more quickly. Imagine catching bugs leading to failed unit tests before even running the tests. Plus, static analysis tools tell you why things are wrong rather than just the fact that they are. – Centavo

Answer 2

Well, I'm not much of a python programmer, but I'd say that the answer is 'YES'.

Any dynamic language that lets you create a variable with any name at any time, could use a 'strict' pragma.

Strict vars (one of the options for strict in Perl, 'use strict' turns them all on at once) in Perl requires that all variables are declared before they are used. Which means that this code:

my $strict_is_good = 'foo';
$strict_iS_good .= 'COMPILE TIME FATAL ERROR';

Generates a fatal error at compile time.

I don't know of a way to get Python to reject this code at compile time:

strict_is_good = 'foo';
strict_iS_good += 'RUN TIME FATAL ERROR';

You will get a run-time exception that strict_iS_good is undefined. But only when the code is executed. If your test suite does not have 100% coverage, you can easily ship this bug.

Any time I work in a language that does not have this behavior (PHP for example), I get nervous. I am not a perfect typist. A simple, but hard to spot, typo can cause your code to fail in ways that may be hard to track down.

So, to reiterate, YES Python could use a 'strict' pragma to turn on compile time checks for things that can be checked at compile time. I can't think of any other checks to add, but a better Python programmer probably could think of some.

Note I focus on the pragmatic effect of stict vars in Perl, and am glossing over some of the details. If you really want to know all the details see the perldoc for strict.

Update: Responses to some comments

Jason Baker : Static checkers like pylint are useful. But they represent an extra step that can be and often is skipped. Building some basic checks into the compiler guarantees that these checks are performed consistently. If these checks are controllable by a pragma, even the objection relating to the cost of the checks becomes moot.

popcnt : I know that python will generate a run time exception. I said as much. I advocate compile time checking where possible. Please reread the post.

mpeters : No computer analysis of code can find all errors--this amounts to solving the halting problem. Worse, to find typos in assignments, your compiler would need to know your intentions and find places where your intentions differ from your code. This is pretty clearly impossible.

However this does not mean that no checking should be done. If there are classes of problems that are easy to detect, then it makes sense to trap them.

I'm not familiar enough with pylint and pychecker to say what classes of errors they will catch. As I said I am very inexperienced with python.

These static analysis programs are useful. However, I believe that unless they duplicate the capabilities of the compiler, the compiler will always be in a position to "know" more about the program than any static checker could. It seems wasteful not to take advantage of this to reduce errors where possible.

Update 2:

cdleary - In theory, I agree with you, a static analyzer can do any validation that the compiler can. And in the case of Python, it should be enough.

However, if your compiler is complex enough (especially if you have lots of pragmas that change how compilation occurs, or if like Perl, you can run code at compile time), then the static analyzer must approach the complexity of the compiler/interpreter to do the analysis.

Heh, all this talk of complex compilers and running code at compile time shows my Perl background.

My understanding is that Python does not have pragmas and can not run arbitrary code at compile time. So, unless I am wrong or these features are added, a relatively simple parser in the static analyzer should suffice. It certainly would be helpful to force these checks at every execution. Of course, the way I'd do this is with a pragma.

Once you add pragmas to the mix, you have started down a slippery slope and the complexity of you analyzer must grow in proportion to the power and flexibility you provide in your pragmas. If you are not careful, you can wind up like Perl, and then "only python can parse Python," a future I wouldn't want to see.

Maybe a command line switch would be a better way to add forced static analysis ;)

(In no way do intend to impugn Python's capabilities when I say that it can't futz with compile time behavior like Perl can. I have a hunch that this is a carefully considered design decision, and I can see the wisdom in it. Perl's extreme flexibility at compile time is, IMHO, a great strength and a terrible weakness of the language; I see the wisdom in this approach as well.)

Rojo

There are static checkers like pylint and pychecker that can do this for you. – Sirdar

python DOES "reject" the code to use your terminology, but at run-time. a decent editor will help to avoid such mistakes, in both perl and python-- which is the point -> avoid including unintended errors in your source code. – Adolfoadolph

see this example from python: >>> strict_is_for_those_with_worthless_code_editors = 'yes' >>> strict_Is_god += 'RUN TIME FATAL ERROR' Traceback (most recent call last): File "<stdin>", line 1, in ? NameError: name 'strict_Is_god' is not defined >>> – Adolfoadolph

but this doesn't catch typos in your assignment. What if you change the value of a variable later on in your code but you typo'd the name. It wouldn't be caught by the interpreter at runtime. Would pylint or pychecker catch it? – Kokura

@Jason Baker: Yes, part of the question is why we don't distribute a version of the Python interpreter with a PyChecker static analysis check flag to find this kind of thing at compile time. – Subscribe

I disagree with the last part of your edit - the static (lexical) analysis that the compiler could perform is identical to a third party inspection of the syntax. This AST functionality is also in the stdlib. The Python compiler just converts a valid syntax tree into stack-machine bytecode. – Subscribe

Python can compile/run arbitrary code at runtime. Check out eval ( docs.python.org/library/functions.html#eval ). Python has pragmas. See future bitflags ( docs.python.org/library/…future ) – Subscribe

Evals make static analysis impossible. We generally don't care about static checking if we use eval - most static checkers will give up if they see one. I'm only talking about optional, per-file, compile-time, scope-based syntax checking in the OP. A subset of what PyLint/PyFlakes do. – Subscribe

That example is valid Python code. If the example is inside a function then strict_iS_good is a global and static analysis can't know that that name won't be in the global scope with the code is executed. – Batish

@PeterGraham but the point here is that in the presence of "use strict" that example would be invalid. That's what "use strict" means. It means you are not allowed to refer to global variables without some extra syntax to say "yes, I really do mean global here". – Worked

@Subscribe eval does not preclude strict. Perl has eval. – Worked

Answer 3

Python does have something that can change script syntax:

from __future__ import print_function

and various other future-features that have syntax implications. It's just that Python's syntax has been stricter, stabler and more well-defined than historical Perl; the kind of things that ‘strict refs’ and ‘strict subs’ prohibit have never existed in Python.

‘strict vars’ is primarily intended to stop typoed references and missed-out ‘my’s from creating accidental globals (well, package variables in Perl terms). This can't happen in Python as bare assignments default to local declaration, and bare unassigned symbols result in an exception.

(There is still the case where users accidentally try to write-through to a global without declaring it with a ‘global’ statement, causing either an accidental local or, more often, an UnboundLocalError. This tends to be learned fairly quickly, but it is an arguable case where having to declare your locals could help. Although few experienced Python programmers would accept the readability burden.)

Other language and library changes that do not involve syntax are handled through the warnings system.

Poison

I like where this answer is going. I figured that using globals()[] or locals()[] would be similar to strict refs, although I agree on strict subs. It seems like there a need for checking in the typo department, though -- why haven't people fallen all over static analysis tools to check their typos? – Subscribe

Answer 4

I think there's some confusion as the what "use strict" does, from the comments I'm seeing. It does not turn on compile time type checks (to be like Java). In that sense, Perl progammers are in agreement with python programmers. As S.Lott says above these types of checks don't protect against logic bugs, don't reduce the number of unit tests you need to write and we're also not big fans of bondage programming.

Here's a list of what "use strict" does do:

1. Using symbolic references is a run-time error. This prevents you from doing crazy (but sometimes useful things like)

   $var = 'foo';

   $foo = 'bar';

   print $$var; # this would contain the contents of $foo unless run under strict

2. Using undeclared variables is a run-time error (this means you need to use "my", "our" or "local" to declare your variable's scope before using it.

3. All barewords are considered compile-time syntax errors. Barewords are words that have not been declared as symbols or subroutines. This is mainly to outlaw something that was historically done but is considered to have been a mistake.

Kokura

"don't reduce the number of unit tests you need to write": Unless you believe that unit-testing every single catch-and-rethrow-with-a-better-message block just to check for trivial typos is a huge waste of a programmer's time. Which, judging by the unit tests most pythonistas I've seen write, is not an uncommon opinion. – Pamela

Answer 5

This original answer is correct, but does not perhaps explain the situation in a practical sense.

There exist static analysis tools for Python, but compile time checks tend to be >diametrically opposed to the run-time binding philosophy that Python embraces.

What 'use strict' provides in Perl is the ability to ensure that a mis-spelled or variable name is (usually) caught at compile-time. This does improve code reliability, and speeds up development. But in order to make such a thing worthwhile, you need to declare variables. And Python style seems to discourage that.

So in Python, you never find out about a mis-spelled variable until you notice at run-time that the assignment you thought you made is not being made, or that an expression seems to resolve to an unexpected value. Catching such errors can be time-consuming, especially as programs get large, and as people are forced to maintain code developed by others.

Java and C/C++ take it a step further, with type checking. The motivation is practical, rather than philosophical. How can you catch as many errors as possible as soon as possible, and be sure that you eliminate all of them before releasing code to production? Each language seems to take a particular strategy and run with it, based upon what they think is important. In a language like Perl, where run-time binding isn't supported, it makes sense to take advantage of 'use strict' to make development easier.

Daron

This hurts much less in python, though, than for example in Java. When making a typo in LongNamedFunctionNeededInJava, you won’t see it easily. If you make a typo in user_name, imported from the hypothetical winter-module (from winter import user_name), you’ll spot that almost at once. And it will be visible at load time, because that’s where you typically import stuff. Comes down to using namespaces: They allow keeping code readable. – Cornerstone

Answer 6

Python has no true lexical scoping, so strict vars wouldn't be very sensible. It has no symbolic references AFAIK, so it has not need for strict refs. It has not barewords, so it has no need for strict vars.

To be honest, it's only lexical scoping I miss. The other two I'd consider warts in Perl.

Cackle

attribute with double underbar (obj.__attr) do not count as lexical scoping to you? – Adolfoadolph

Attributes have little to do with lexical scoping. – Cackle

What do you mean? Python clearly has lexical scoping (at least since 2.1), but I don't see that it has much to do with requiring strict declarations. Are you referring to something else by this? – Experimental

Yes, Python is lexically scoped. I think is referring to the fact a new lexical scope is only created with some particular constructs: modules, classes, and functions. i.e. for loops don't have an independent enclosing scope. If you bind an identifier inside a for loop it stays bound outside of it. – Subscribe

@cdleary: yeah, that's what I meant. – Cackle

python does name mangling with double-underbar attributes-- that name is visible only to objects of that class. +you can create closures in python, so independent enclosing scope. e.g. #605122 – Adolfoadolph

@Leon oh, i think i see now -- you appreciate the nested lexical scoping of perl, which you miss in python. i guess that is a matter of taste :) i was trying to point out some analogous things in python which have similar effects/results. – Adolfoadolph

Answer 7

I consider the 'use strict' in Perl more like a pragma as you hinted at: it changes the behavior of the compiler.

Perl language philosophy is different from python philosophy. As in, you are given more than enough rope to hang yourself repeatedly, in Perl.

Larry Wall is big into linguistics, so we have from Perl what is referred to as the TIMTOWTDI (say tim-toe-dee) principle vs. Zen of python:

There should be one-- and preferably only one --obvious way to do it.

you could very easily use pylint and PyChecker to come up with your own flavor of use strict for python (or something analogous to perl -cw *scriptname*) but because of the different philosophies in the language design, you will not encounter this in practice widely.

Based on your comment to the first poster, you are familiar with python's import this. There are a lot of things in there which illuminate why you do not see an equivalent of use strict in Python. If you meditate on the koan found in the Zen of Python, you may find enlightenment for yourself. :)

Adolfoadolph

I was hoping I wouldn't see an answer telling me to "import this" without giving a real explanation. How would a compiler that checks identifier scoping violate the Pythonic Theses? How does runtime lookup make static lexical analysis sufficiently useless? We know the philosophies are different. – Subscribe

f you have a less-advnaced editor which will not help you uncover ill-named variables (in both perl or python, or whatever) or prevent them in the first place, then there may be a positive benefit to static lexical analysis. pylint and PyChecker would seem to adequately cover this space? – Adolfoadolph

Yes, so my question is whether there is a need for a compiler that automatically invokes this kind of static analysis if you give it a directive, and why/why not. Also, which editors are you referring to? Surely you don't auto-complete every variable name no matter which editor you're using. – Subscribe

@cdleary: emacs - and yes, i do auto-complete on every variable (except the first time of course), and I love it! – Adolfoadolph

@popcnt: Interesting -- I feel like that would actually slow down my typing, having to verify and/or select the autocomplete every time. Does it slow you down at all? – Subscribe

Answer 8

I've found that I only really care about detecting references to undeclared vars. Eclipse has pylint integration via PyDev and, although pylint is far from perfect, it does a reasonable job at that.

It does kind of go against Python's dynamic nature, and I do have to add #IGNOREs occasionally, when my code gets clever about something. But I find that happens infrequently enough that I'm happy with it.

But I could see the utility of some pylint-like functionality becoming available in the form of a command-line flag. Kind of like Python 2.6's -3 switch, which identifies points of incompatibility between Python 2.x and 3.x code.

Abaft

Yeah, this is along the lines I was thinking as well -- wrap the regular interpreter with an additional flag to turn on "use strict" compilation. I've never seen anything like this, but Perl thinkers tend to consider it necessary/desirable. Python thinkers tend not to care. Why don't we care? – Subscribe

Answer 9

It is very difficult to write large programs without 'use strict' in Perl. Without 'use strict', if you use a variable again, and misspell it by leaving a letter out, the program still runs. And without test cases to check your results, you can never find such errors. It can be very time-consuming to find why you are getting wrong results due to this reason.

Some of my Perl programs consist of 5,000 lines to 10,000 lines of code broken into dozens of modules. One cannot really do production programming without 'use strict'. I would never allow production code to be installed in the factory with languages that do not enforce "variable declarations".

This is why Perl 5.12.x now has the 'use strict' as the default behavior. You can turn them off.

PHP has given me quite a few problems because of no variable declaration enforcement. So you need to limit yourself to small programs with this language.

Just an opinion ...

abcParsing

Mercenary

This is why Perl 5.12.x now has the 'use strict' as the default behavior. You can turn them off. wut?? – Lobar

Answer 10

Perl is a unrestrained language as they said :) . So you can use the variable before announced ; Eg : If you use a var name "is_array" but type in "is_arrby" ,the compiler won't report error without "use strict" . So when coding long program in perl ,better use "use strict" statement. Of course, less than 50 lines for running one time script, there is no need :)

Burtonburty

I'm confused. Are we talking about Python or Perl here? – Carin

@ccjmne,Hah,yes. I think perl gives you a lot of freedom to do anything to build the program if there is no syntax errors; In python, a little bit restriction , but is enough ...:) – Burtonburty