Regex for quoted string with escaping quotes

Question

How do I get the substring " It's big \"problem " using a regular expression?

s = ' function(){  return " It\'s big \"problem  ";  }';     

Miserly

How do you find "It's" in a string that only contains "Is"? I'd fix it for you, but I don't know which single-quote/escape conventions apply in the language you're using. – Febrific

Duplicate of: PHP: Regex to ignore escaped quotes within quotes – Hochstetler

Actually, looking at the dates, I see that the other question is a duplicate of this one. Either way, be sure to check out my answer. – Hochstetler

@ridgerunner: I'm voting to close this as you suggested. It's true other question is more recent, but it's also much better (thanks mostly to your answer). – Janot

Answer 1

/"(?:[^"\\]|\\.)*"/

Works in The Regex Coach and PCRE Workbench.

Example of test in JavaScript:

    var s = ' function(){ return " Is big \\"problem\\", \\no? "; }';
    var m = s.match(/"(?:[^"\\]|\\.)*"/);
    if (m != null)
        alert(m);

Lilith

Makes sense. Plain english: Two quotes surrounding zero or more of "any character that's not a quote or a backslash" or "a backslash followed by any character". I can't believe I didn't think to do that... – Candiscandle

I'll answer myself. =) (?:...) is a passive or non-capturing group. It means that it cannot be backreferenced later. – Izak

after searching a lot and test a lot this is the real and only solution I found to this common problem. Thanks! – Costumer

thanks for this. i wanted to match single quotes as well so i ended up adapting it to this: /(["'])(?:[^\1\\]|\\.)*?\1/ – Lester

With var s = ' my \\"new\\" string and \"this should be matched\"';, this approach will lead to unexpected results. – Bookrack

@WiktorStribiżew Your string doesn't conform to the description: a string including a part between double quotes, that can contain escaped double quotes. Not sure what you expect... – Lilith

For those that are interested, placing "\\." first yields better performance. I assume it's because doing this first makes the extra lookup for backslash in "[^"\\]" redundant. Looking at the other answers such as Darrell's below gives more performant regex (and that's the one included in many Linux distros according to the answer.) So for performance go with \"(\\.|[^\"])*\". Timing it in Python 3.7 gave 1.375 millis vs 1.55 millis. – Balduin

@nr5 I don't know Swift. Perhaps you need to double all backslashes, if it doesn't have special syntax for regexes. We usually to do this in C, Java, and so on because REs are just strings. (Assuming you talk about a syntax error, not a runtime error, it isn't clear, you don't even give the error message...) – Lilith

Translation: Match quote, match single character except quote or backslash OR match 2 characters if the first is a backslash, match previous group zero or more times, match quote. – Shoulder

Hey! What if i want to match the second occurence of a string with quotes? Like, "Test1" "Test2", match "Test2" only. – Yasukoyataghan

It doesn't seem like it handles the following string: '"\\"'.match(/"(?:[^"\]|\\.)*"/); null – Dislike

@SamGoto It works. In JavaScript, the string literal '"\\"' evaluates to the string value "\", with the second double quote being escaped, so there is no valid closing quote for the regex to match, hence the null matches being returned. This is easy to see if you use console.log in the browser console. If you just enter '"\\"' into the browser console by itself, the console will just spit back the string literal as is (eg. '"\\"'), but if you instead console.log it, then it will acutally print the value of the string literal rather than the string literal itself (eg. "\"). – Jacquline

Answer 2

This one comes from nanorc.sample available in many linux distros. It is used for syntax highlighting of C style strings

\"(\\.|[^\"])*\"

Frieder

With var s = ' my \\"new\\" string and \"this should be matched\"';, this approach will lead to unexpected results. – Bookrack

c.nanorc was the first place I went. Couldn't get it to work as part of a C string literal until double-escaping everything like this " \"(\\\\.|[^\\\"])*\" " – Cadman

This works with egrep and re_comp/re_exec functions from libc. – Contingent

Answer 3

As provided by ePharaoh, the answer is

/"([^"\\]*(\\.[^"\\]*)*)"/

To have the above apply to either single quoted or double quoted strings, use

/"([^"\\]*(\\.[^"\\]*)*)"|\'([^\'\\]*(\\.[^\'\\]*)*)\'/

Knavish

This is the only set that worked for me with a single, large 1.5 KB quoted string containing 99 escapes. Every other expression on this page broke in my text editor with an overflow error. Though most here work in the browser, just something to keep in mind. Fiddle: jsfiddle.net/aow20y0L – Frap

See @MarcAndrePoulin's answer below for explanation. – Plotkin

Answer 4

/(["\']).*?(?<!\\)(\\\\)*\1/is

should work with any quoted string

Frye

Nice, but too flexible for the request (will match single quotes...). And can be simplified to /".*?(?<!\)"/ unless I miss something. Oh, and some languages (eg. JavaScript) alas doesn't understand negative lookbehind expressions. – Lilith

@PhiLho, just using a single (?<!\\) would fail on escaped backslashes at the end of the string. True about look-behinds in JavaScript though. – Bleeder

@Lilith Your simplification with this input: "Martha's" would result in this match: "Martha', which is incorrect. The matching group, to determine which type of quote is being used to open it, is important. – Yellowlegs

@Yellowlegs Note 1: there is a double backslash in my answer, somehow SO lost the second one (because of Markdown?). Should have protected in backticks. Note 2: Markus is right… So it is flawed. Unlike my (popular) answer… :-) Note 3: there are no single quotes in my expression, I don't see the problem you mention, and I can't reproduce it. (I say I don't handle single quotes as delimiters, as it wasn't the topic.) – Lilith

@Lilith Huh... weird. Not sure how I misinterpreted it the first time around. You're absolutely correct. I'm not sure how I mistook your original comment. – Yellowlegs

This was great, thank you! I needed to match multiline strings so I added this: (["\'])(.|\r?\n)*?(?<!\\)(\\\\)*\1 – Anguilliform

Answer 5

Most of the solutions provided here use alternative repetition paths i.e. (A|B)*.

You may encounter stack overflows on large inputs since some pattern compiler implements this using recursion.

Java for instance: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6337993

Something like this: "(?:[^"\\]*(?:\\.)?)*", or the one provided by Guy Bedford will reduce the amount of parsing steps avoiding most stack overflows.

Marginate

Answer 6

"(?:\\"|.)*?"

Alternating the \" and the . passes over escaped quotes while the lazy quantifier *? ensures that you don't go past the end of the quoted string. Works with .NET Framework RE classes

Freeholder

But fails with "\\" – Kildare

It will fail with var s = ' my \\"new\\" string and \"this should be matched\"'; – Bookrack

/"(?:(?:\\"|[^"])*)"/g this should fix – Puppet

Answer 7

/"(?:[^"\\]++|\\.)*+"/

Taken straight from man perlre on a Linux system with Perl 5.22.0 installed. As an optimization, this regex uses the 'posessive' form of both + and * to prevent backtracking, for it is known beforehand that a string without a closing quote wouldn't match in any case.

Eellike

Answer 8

This one works perfect on PCRE and does not fall with StackOverflow.

"(.*?[^\\])??((\\\\)+)?+"

Explanation:

1. Every quoted string starts with Char: " ;
2. It may contain any number of any characters: .*? {Lazy match}; ending with non escape character [^\\];
3. Statement (2) is Lazy(!) optional because string can be empty(""). So: (.*?[^\\])??
4. Finally, every quoted string ends with Char("), but it can be preceded with even number of escape sign pairs (\\\\)+; and it is Greedy(!) optional: ((\\\\)+)?+ {Greedy matching}, bacause string can be empty or without ending pairs!

Belldas

It isn't the most efficient pattern of the world, but the idea is interesting. Note that you can shorten it like this: "(.*?[^\\])?(\\\\)*" – Hernadez

Answer 9

An option that has not been touched on before is:

1. Reverse the string.
2. Perform the matching on the reversed string.
3. Re-reverse the matched strings.

This has the added bonus of being able to correctly match escaped open tags.

Lets say you had the following string; String \"this "should" NOT match\" and "this \"should\" match" Here, \"this "should" NOT match\" should not be matched and "should" should be. On top of that this \"should\" match should be matched and \"should\" should not.

First an example.

// The input string.
const myString = 'String \\"this "should" NOT match\\" and "this \\"should\\" match"';

// The RegExp.
const regExp = new RegExp(
    // Match close
    '([\'"])(?!(?:[\\\\]{2})*[\\\\](?![\\\\]))' +
    '((?:' +
        // Match escaped close quote
        '(?:\\1(?=(?:[\\\\]{2})*[\\\\](?![\\\\])))|' +
        // Match everything thats not the close quote
        '(?:(?!\\1).)' +
    '){0,})' +
    // Match open
    '(\\1)(?!(?:[\\\\]{2})*[\\\\](?![\\\\]))',
    'g'
);

// Reverse the matched strings.
matches = myString
    // Reverse the string.
    .split('').reverse().join('')
    // '"hctam "\dluohs"\ siht" dna "\hctam TON "dluohs" siht"\ gnirtS'

    // Match the quoted
    .match(regExp)
    // ['"hctam "\dluohs"\ siht"', '"dluohs"']

    // Reverse the matches
    .map(x => x.split('').reverse().join(''))
    // ['"this \"should\" match"', '"should"']

    // Re order the matches
    .reverse();
    // ['"should"', '"this \"should\" match"']

Okay, now to explain the RegExp. This is the regexp can be easily broken into three pieces. As follows:

# Part 1
(['"])         # Match a closing quotation mark " or '
(?!            # As long as it's not followed by
  (?:[\\]{2})* # A pair of escape characters
  [\\]         # and a single escape
  (?![\\])     # As long as that's not followed by an escape
)
# Part 2
((?:          # Match inside the quotes
(?:           # Match option 1:
  \1          # Match the closing quote
  (?=         # As long as it's followed by
    (?:\\\\)* # A pair of escape characters
    \\        # 
    (?![\\])  # As long as that's not followed by an escape
  )           # and a single escape
)|            # OR
(?:           # Match option 2:
  (?!\1).     # Any character that isn't the closing quote
)
)*)           # Match the group 0 or more times
# Part 3
(\1)           # Match an open quotation mark that is the same as the closing one
(?!            # As long as it's not followed by
  (?:[\\]{2})* # A pair of escape characters
  [\\]         # and a single escape
  (?![\\])     # As long as that's not followed by an escape
)

This is probably a lot clearer in image form: generated using Jex's Regulex

Image on github (JavaScript Regular Expression Visualizer.) Sorry, I don't have a high enough reputation to include images, so, it's just a link for now.

Here is a gist of an example function using this concept that's a little more advanced: https://gist.github.com/scagood/bd99371c072d49a4fee29d193252f5fc#file-matchquotes-js

Poteat

Answer 10

here is one that work with both " and ' and you easily add others at the start.

("|')(?:\\\1|[^\1])*?\1

it uses the backreference (\1) match exactley what is in the first group (" or ').

http://www.regular-expressions.info/backref.html

Underwrite

this is a very good solution, but [^\1] should be replaced with . because there is no such thing as an anti-back-reference, and it doesn't matter anyways. the first condition will always match before anything bad could happen. – Seumas

@SephReed – replacing [^\1] with . would effectively change this regex to ("|').*?\1 and then it would match "foo\" in "foo \" bar". That said, getting [^\1] to actually work is hard. @​mathiashansen – You're better off with the unwieldy and expensive (?!\1). (so the whole regex, with some efficiency cleanup, would be (["'])(?:\\.|(?!\1).)*+\1. The + is optional if your engine doesn't support it. – Polyhydric

Answer 11

One has to remember that regexps aren't a silver bullet for everything string-y. Some stuff are simpler to do with a cursor and linear, manual, seeking. A CFL would do the trick pretty trivially, but there aren't many CFL implementations (afaik).

Karsten

True enough, but this problem is well within the capabilities of regexes, and there are a great many implementations of those. – Janot

Answer 12

If it is searched from the beginning, maybe this can work?

\"((\\\")|[^\\])*\"

Delorenzo

Answer 13

A more extensive version of https://mcmap.net/q/23849/-regex-for-quoted-string-with-escaping-quotes

/"([^"\\]{50,}(\\.[^"\\]*)*)"|\'[^\'\\]{50,}(\\.[^\'\\]*)*\'|“[^”\\]{50,}(\\.[^“\\]*)*”/   

This version also contains

1. Minimum quote length of 50
2. Extra type of quotes (open “ and close ”)

Exarch

Answer 14

I faced a similar problem trying to remove quoted strings that may interfere with parsing of some files.

I ended up with a two-step solution that beats any convoluted regex you can come up with:

 line = line.replace("\\\"","\'"); // Replace escaped quotes with something easier to handle
 line = line.replaceAll("\"([^\"]*)\"","\"x\""); // Simple is beautiful

Easier to read and probably more efficient.

Liuka

Answer 15

If your IDE is IntelliJ Idea, you can forget all these headaches and store your regex into a String variable and as you copy-paste it inside the double-quote it will automatically change to a regex acceptable format.

example in Java:

String s = "\"en_usa\":[^\\,\\}]+";

now you can use this variable in your regexp or anywhere.

Maldon

Answer 16

(?<="|')(?:[^"\\]|\\.)*(?="|')

" It\'s big \"problem " match result: It\'s big \"problem

("|')(?:[^"\\]|\\.)*("|')

" It\'s big \"problem " match result: " It\'s big \"problem "

Drogue

Answer 17

Messed around at regexpal and ended up with this regex: (Don't ask me how it works, I barely understand even tho I wrote it lol)

"(([^"\\]?(\\\\)?)|(\\")+)+"

Elongate