Using Regex to generate Strings rather than match them [closed]

Question

I am writing a Java utility that helps me to generate loads of data for performance testing. It would be really cool to be able to specify a regex for Strings so that my generator spits out things that match this.

Is something out there already baked that I can use to do this? Or is there a library that gets me most of the way there?

Entree

Answer 1

Firstly, with a complex enough regexp, I believe this can be impossible. But you should be able to put something together for simple regexps.

If you take a look at the source code of the class java.util.regex.Pattern, you'll see that it uses an internal representation of Node instances. Each of the different pattern components have their own implementation of a Node subclass. These Nodes are organised into a tree.

By producing a visitor that traverses this tree, you should be able to call an overloaded generator method or some kind of Builder that cobbles something together.

Schmit

I am not sure Xeger is that good. It cannot handle character classes. It fails to recognize a simple [\w]. A look at the last line of their wiki tells us that. – Nance

Note also that these depend on dk.brics.automaton so be prepared to add 3rd party pom dependencies. Most people don't mind that but I wish there was something a bit more compact. – Merganser

There is alternative for xeger and generex. It lacks all these drawbacks and is not obsolete. Please scroll down to my answer. – Hyperaesthesia

"Firstly, with a complex enough regexp, I believe this can be impossible." -- this is not strictly true: any regex that passes against something can also generate a valid input. Explanation: regexes are type-3 on the Chomsky Hierarchy, meaning they can be expressed as FSMs. When stepping through an FSM, each edge is interpreted as a rule for the next character, thus an FSM can be used to parse or generate sequences. If an FSM has a path to the terminal, a valid sequence can be determined. So, it's only "impossible" if there's no path to the terminal (which would be a useless regex). – Paries

Answer 2

It's too late to help the original poster, but it could help a newcomer. Generex is a useful java library that provides many features for using regexes to generate strings (random generation, generating a string based on its index, generating all strings...).

Example :

Generex generex = new Generex("[0-3]([a-c]|[e-g]{1,2})");

// generate the second String in lexicographical order that matches the given Regex.
String secondString = generex.getMatchedString(2);
System.out.println(secondString);// it print '0b'

// Generate all String that matches the given Regex.
List<String> matchedStrs = generex.getAllMatchedStrings();

// Using Generex iterator
Iterator iterator = generex.iterator();
while (iterator.hasNext()) {
    System.out.print(iterator.next() + " ");
}
// it prints 0a 0b 0c 0e 0ee 0e 0e 0f 0fe 0f 0f 0g 0ge 0g 0g 1a 1b 1c 1e
// 1ee 1e 1e 1f 1fe 1f 1f 1g 1ge 1g 1g 2a 2b 2c 2e 2ee 2e 2e 2f 2fe 2f 2f 2g
// 2ge 2g 2g 3a 3b 3c 3e 3ee 3e 3e 3f 3fe 3f 3f 3g 3ge 3g 3g 1ee

// Generate random String
String randomStr = generex.random();
System.out.println(randomStr);// a random value from the previous String list

Disclosure

The project mentioned on this post belongs to the user answering (Mifmif) the question. As per the rules, this need to be brought up.

Wounded

It looks like Generex is your own project. Would you mind mentioning in your post that this is your own project, as per the rules here? – Tracee

Answer 3

Xeger (Java) is capable of doing it as well:

String regex = "[ab]{4,6}c";
Xeger generator = new Xeger(regex);
String result = generator.generate();
assert result.matches(regex);

Grania

Xeger works nicely. BUT make sure that you have the automaton jar on the class path or in your pom/gradle – Spherulite

Answer 4

This question is really old, though the problem was actual for me. I've tried xeger and Generex and they doesn't seem to meet my reguirements. They actually fail to process some of the regex patterns (like a{60000}) or for others (e.g. (A|B|C|D|E|F)) they just don't produce all possible values. Since I didn't find any another appropriate solution - I've created my own library.

https://github.com/curious-odd-man/RgxGen

This library can be used to generate both matching and non-matching string.

There is also artifact on maven central available.

Usage example:

RgxGen rgxGen = new RgxGen(aRegex);                     // Create generator
String s = rgxGen.generate();                           // Generate new random value

Hyperaesthesia

I tried RxGen and it worked much better than Xeger and Generex – Gleet

But your lib doesn't support lookahead and lookbehind, even with brute force regeneration it still can some times produce an invalid string. – Hornbeck

The library has some limitations which are described in a Readme section. – Hyperaesthesia

Works like a charm. I'm using it to generate example XMLs for XSD schema. – Enjoin

Answer 5

I've gone the root of rolling my own library for that (In c# but should be easy to understand for a Java developer).

Rxrdg started as a solution to a problem of creating test data for a real life project. The basic idea is to leverage the existing (regular expression) validation patterns to create random data that conforms to such patterns. This way valid random data is created.

It is not that difficult to write a parser for simple regex patterns. Using an abstract syntax tree to generate strings should be even easier.

Yuk

link does not point to the repository anymore. I would go with openhub.net/p/rxrdg. The solution does not build, however ? – Fouquiertinville

Answer 6

On stackoverflow podcast 11:

Spolsky: Yep. There's a new product also, if you don't want to use the Team System there our friends at Redgate have a product called SQL Data Generator [http://www.red-gate.com/products/sql_data_generator/index.htm]. It's $295, and it just generates some realistic test data. And it does things like actually generate real cities in the city column that actually exist, and then when it generates those it'll get the state right, instead of getting the state wrong, or putting states into German cities and stuff like... you know, it generates pretty realistic looking data. I'm not really sure what all the features are.

This is probably not what you are looking for, but it might be a good starting off point, instead of creating your own.

I can't seem to find anything in google, so I would suggest tackling the problem by parsing a given regular expression into the smallest units of work (\w, [x-x], \d, etc) and writing some basic methods to support those regular expression phrases.

So for \w you would have a method getRandomLetter() which returns any random letter, and you would also have getRandomLetter(char startLetter, char endLetter) which gives you a random letter between the two values.

Tickle

Answer 7

I am on flight and just saw the question: I have written easiest but inefficient and incomplete solution. I hope it may help you to start writing your own parser:

public static void main(String[] args) {

    String line = "[A-Z0-9]{16}";
    String[] tokens = line.split(line);
    char[] pattern = new char[100];
    int i = 0;
    int len = tokens.length;
    String sep1 = "[{";
    StringTokenizer st = new StringTokenizer(line, sep1);

    while (st.hasMoreTokens()) {
        String token = st.nextToken();
        System.out.println(token);

        if (token.contains("]")) {
            char[] endStr = null;

            if (!token.endsWith("]")) {
                String[] subTokens = token.split("]");
                token = subTokens[0];

                if (!subTokens[1].equalsIgnoreCase("*")) {
                    endStr = subTokens[1].toCharArray();
                }
            }

            if (token.startsWith("^")) {
                String subStr = token.substring(1, token.length() - 1);
                char[] subChar = subStr.toCharArray();
                Set set = new HashSet<Character>();

                for (int p = 0; p < subChar.length; p++) {
                    set.add(subChar[p]);
                }

                int asci = 1;

                while (true) {
                    char newChar = (char) (subChar[0] + (asci++));

                    if (!set.contains(newChar)) {
                        pattern[i++] = newChar;
                        break;
                    }
                }
                if (endStr != null) {
                    for (int r = 0; r < endStr.length; r++) {
                        pattern[i++] = endStr[r];
                    }
                }

            } else {
                pattern[i++] = token.charAt(0);
            }
        } else if (token.contains("}")) {
            char[] endStr = null;

            if (!token.endsWith("}")) {
                String[] subTokens = token.split("}");
                token = subTokens[0];

                if (!subTokens[1].equalsIgnoreCase("*")) {
                    endStr = subTokens[1].toCharArray();
                }
            }

            int length = Integer.parseInt((new StringTokenizer(token, (",}"))).nextToken());
            char element = pattern[i - 1];

            for (int j = 0; j < length - 1; j++) {
                pattern[i++] = element;
            }

            if (endStr != null) {
                for (int r = 0; r < endStr.length; r++) {
                    pattern[i++] = endStr[r];
                }
            }
        } else {
            char[] temp = token.toCharArray();

            for (int q = 0; q < temp.length; q++) {
                pattern[i++] = temp[q];
            }
        }
    }

    String result = "";

    for (int j = 0; j < i; j++) {
        result += pattern[j];
    }

    System.out.print(result);
}

Lohse

You may want to indicate what kind of strings are used as pattern input. First of all, it is not all that easy to determine such things from source code. Second, if there are any mistakes or unclarities in the source code, there is no way to see if they are intentional or not. – Byrnes

StringTokenizer is a legacy class that is retained for compatibility reasons although its use is discouraged in new code. It is recommended that anyone seeking this functionality use the split method of String or the java.util.regex package instead. – Dirk

Answer 8

You'll have to write your own parser, like the author of String::Random (Perl) did. In fact, he doesn't use regexes anywhere in that module, it's just what perl-coders are used to.

On the other hand, maybe you can have a look at the source, to get some pointers.

---

EDIT: Damn, blair beat me to the punch by 15 seconds.

Nasia

You only write your own parser if you're young. – Enjoin

Answer 9

I know there's already an accepted answer, but I've been using RedGate's Data Generator (the one mentioned in Craig's answer) and it works REALLY well for everything I've thrown at it. It's quick and that leaves me wanting to use the same regex to generate the real data for things like registration codes that this thing spits out.

It takes a regex like:

[A-Z0-9]{3,3}-[A-Z0-9]{3,3}

and it generates tons of unique codes like:

LLK-32U

Is this some big secret algorithm that RedGate figured out and we're all out of luck or is it something that us mere mortals actually could do?

Bum

Answer 10

It's far from supporting a full PCRE regexp, but I wrote the following Ruby method to take a regexp-like string and produce a variation on it. (For language-based CAPTCHA.)

# q = "(How (much|many)|What) is (the (value|result) of)? :num1 :op :num2?"
# values = { :num1=>42, :op=>"plus", :num2=>17 }
# 4.times{ puts q.variation( values ) }
# => What is 42 plus 17?
# => How many is the result of 42 plus 17?
# => What is the result of 42 plus 17?
# => How much is the value of 42 plus 17?
class String
  def variation( values={} )
    out = self.dup
    while out.gsub!( /\(([^())?]+)\)(\?)?/ ){
      ( $2 && ( rand > 0.5 ) ) ? '' : $1.split( '|' ).random
    }; end
    out.gsub!( /:(#{values.keys.join('|')})\b/ ){ values[$1.intern] }
    out.gsub!( /\s{2,}/, ' ' )
    out
  end
end

class Array
  def random
    self[ rand( self.length ) ]
  end
end

Mcripley

Answer 11

This question is very old, but I stumbled across it on my own search, so I will include a couple links for others who might be searching for the same functionality in other languages.

• There is a Node.js library here: https://github.com/fent/randexp.js
• There is a PHP library here: https://github.com/icomefromthenet/ReverseRegex
• The PHP faker package includes a "regexify" method that accomplishes this: https://packagist.org/packages/fzaninotto/faker

Hat

Answer 12

If you want to generate "critical" strings, you may want to consider:

EGRET http://elarson.pythonanywhere.com/ that generates "evil" strings covering your regular expressions

MUTREX http://cs.unibg.it/mutrex/ that generates fault-detecting strings by regex mutation

Both are academic tools (I am one of the authors of the latter) and work reasonably well.

Trefler