Variable-length lookbehind-assertion alternatives for regular expressions

Question

Is there an implementation of regular expressions in Python/PHP/JavaScript that supports variable-length lookbehind-assertion?

/(?<!foo.*)bar/

How can I write a regular expression that has the same meaning, but uses no lookbehind-assertion?

Is there a chance that this type of assertion will be implemented some day?

Things are much better that I thought.

Update:

(1) There are regular expressions implementation that support variable-length lookbehind-assertion already.

Python module regex (not standard re, but additional regex module) supports such assertions (and has many other cool features).

>>> import regex
>>> m = regex.search('(?<!foo.*)bar', 'f00bar')
>>> print m.group()
bar
>>> m = regex.search('(?<!foo.*)bar', 'foobar')
>>> print m
None

It was a really big surprise for me that there is something in regular expressions that Perl can't do and Python can. Probably, there is "enhanced regular expression" implementation for Perl also?

(Thanks and +1 to MRAB).

(2) There is a cool feature \K in modern regular expressions.

This symbols means that when you make a substitution (and from my point of view the most interesting use case of assertions is the substitution), all characters that were found before \K must not be changed.

s/unchanged-part\Kchanged-part/new-part/x

That is almost like a look-behind assertion, but not so flexible of course.

More about \K:

• Perl Regular Expression \K Trick
• PCRE Regex Spotlight: \K

As far as I understand, you can't use \K twice in the same regular expression. And you can't say till which point you want to "kill" the characters that you've found. That is always till the beginning of the line.

(Thanks and +1 to ikegami).

My additional questions:

• Is it possible to say what point must be the final point of \K effect?
• What about enhanced regular expressions implementations for Perl/Ruby/JavaScript/PHP? Something like regex for Python.

Dib

To know how to properly write an alternative that doesn't use a lookbehind assertion, we're going to need a little more context. What is this actually for? – Maceio

@minitech: there is no additional context. This is a general question – Dib

No, it requires additional context. The best way to solve your problem currently is to use indexOf to find 'foo' and then repeat to find all 'bar' after it. – Maceio

@minitech: I can remove this simple example; I provided it just for illustration purposes; the question is: "how (generally) can I avoid look-behind-negative-assertions and what (generally) I could use instead?". Why don't you like the answer from ikegami? I think that the answer is almost perfect. I was not aware of this \K trick and I find it really killing – Dib

I do, but it's not the most efficient solution if your problem is actually so simple. To recognize the validity of variable-width lookbehinds, I really need an example that can't be done using simple string searching (which works in engines without \K, too). – Maceio

I doubt Python's version is bug free. To implement variable width look behind correctly, you basically need to have two identical copies of the regex engine, except that one works backwards. It's simply not worth the cost. – Crematory

@ikegami: you mean (?:(?!foo).)* would be more (or at least not less) effective? – Dib

Also, not everyone agree as to what the following should capture: 'foo bar baz moo' =~ /(?<=foo.*(ba.).*)moo/. I'd say bar. Some might say baz. If you say baz, variable-width lookbehind becomes very inefficient and (?{ ... }) won't work sensibly. – Crematory

What do you mean by "effective"? – Crematory

@ikegami: why bar? .* is greedy here, right? – Dib

It's not a question of greediness, it's a question of at which end do you start matching. If you start from the right, the rightmost .* is encountered first and gobbles up as much as it can, and the leftmost gobbles up what's left. If you start at the left, there's all sorts of problems, and it doesn't make as much sense conceptually, but that seems to be what some people (e.g. you) expect. – Crematory

How can you compare efficiency of code that works and code that doesn't work in the general case? But yeah, in this case, they'll surely be equally efficient because Python surely does exactly the same thing internally. – Crematory

@ikegami: what code doesn't work? In the regex module work both variants. – Dib

No, it cannot work both right and efficiently without having two implementations of the regex engine, and I bet it doesn't. – Crematory

@ikegami: "it's a question of at which end do you start matching", ok, I've understood. I think that is just a question of definition. – Dib

@Igor Chubin, yes, but there's no agreement as to a definition, and there are issues with both definitions. It would be irresponsible to implement variable-width lookbehind at this point. – Crematory

Answer 1

Most of the time, you can avoid variable length lookbehinds by using \K.

s/(?<=foo.*)bar/moo/s;

would be

s/foo.*\Kbar/moo/s;

Anything up to the last \K encountered is not considered part of the match (e.g. for the purposes of replacement, $&, etc)

Negative lookbehinds are a little trickier.

s/(?<!foo.*)bar/moo/s;

would be

s/^(?:(?!foo).)*\Kbar/moo/s;

because (?:(?!STRING).)* is to STRING as [^CHAR]* is to CHAR.

---

If you're just matching, you might not even need the \K.

/foo.*bar/s

/^(?:(?!foo).)*bar/s

Crematory

This trick with \K is really cool, but is it possible to specify several \K in one regular expression? Probably, not – Dib

No (or not usefully), but you can use captures: s/foo.*\Kbar/moo/s; === s/(foo.*)bar/${1}moo/s;. – Crematory

captures are obvious but it is not interesting :) \K is much better :) – Dib

But you can only have one. I was pointing out what you could do if needed more than one as you asked. (Captures also work before 5.10 when \K was introduced.) – Crematory

ikegami, of course I'm aware of captures, but there are many situations where they can't help; although \K is also not a silver bullet, it is really a cool thing. And this trick (?:(?!foo).)* is brilliant also. – Dib

Wow. You made my day. I had no idea about \K. Thanks!! – Rubbico

This answer has been added to the Stack Overflow Regular Expression FAQ, under "Lookarounds". – Morisco

This is great, thanks a lot. But please add a note about what \K actually is. It's not exactly easy to Google. – Florescence

Perl's regular expressions are documented in perlre – Crematory

Answer 2

For Python there's a regex implementation which supports variable-length lookbehinds:

http://pypi.python.org/pypi/regex

It's designed to be backwards-compatible with the standard re module.

Pantaloon

This answer has been added to the Stack Overflow Regular Expression FAQ, under "Lookarounds". – Morisco

Works swimmingly on Python 3.4.1. It also seems to be a little faster than re. – Jarlen

Answer 3

You can reverse the string AND the pattern and use variable length lookahead

(rab(?!\w*oof)\w*)

matches in bold:

raboof rab7790oof raboo rabof rab rabo raboooof rabo

Original solution as far as I know by:

Jeff 'japhy' Pinyan

Argos

Benjamin, thank you for the answer, but are you sure that it is possible reverse any pattern? – Dib

I've never run into a situation where this didn't work. Creating the pattern takes more time tough then a "normal" pattern. – Argos

This answer has been added to the Stack Overflow Regular Expression FAQ, under "Lookarounds". – Morisco

Answer 4

The regexp you show will find any instance of bar which is not preceded by foo.

A simple alternative would be to first match foo against the string, and find the index of the first occurrence. Then search for bar, and see if you can find an occurrence which comes before that index.

If you want to find instances of bar which are not directly preceded by foo, I could also provide a regexp for that (without using lookbehind), but it will be very ugly. Basically, invert the sense of /foo/ -- i.e. /[^f]oo|[^o]o|[^o]|$/.

Fayola

Alex, thank you for the answer, but in general all is not so simple as you write. I provided just a small example of a regular expression with an assertion. Of course the re could be much complexer, and the assertion could be deep inside of it. In this case you couldn't just simple check a string for some substring. – Dib

Alex, when you need "instances of bar which are not directly preceded by foo", you can just use normal lookbehind assertion (?<!foo)bar. That works. But the trick is that between foo and bar can be other characters. – Dib

Answer 5

foo.*|(bar)

If foo is in the string first, then the regex will match, but there will be no groups.

Otherwise, it will find bar and assign it to a group.

So you can use this regex and look for your results in the groups found:

>>> import re
>>> m = re.search('foo.*|(bar)', 'f00bar')
>>> if m: print(m.group(1))
bar
>>> m = re.search('foo.*|(bar)', 'foobar')
>>> if m: print(m.group(1))
None
>>> m = re.search('foo.*|(bar)', 'fobas')
>>> if m: print(m.group(1))
>>> 

Source.

Portillo