Random selection from CSV file in Jmeter

Question

I have a very large CSV file (8000+ items) of URLs that I'm reading with a CSV Data Set Config element. It is populating the path of an HTTP Request sampler and iterating through with a while controller.

This is fine except what I want is have each user (thread) to pick a random URL from the CSV URL list. What I don't want is each thread using CSV items sequentially.

I was able to achieve this with a Random Order Controller with multiple HTTP Request samplers , however 8000+ HTTP Samplers really bogged down jmeter to an unusable state. So this is why I put the HTTP Sampler URLs in the CSV file. It doesn't appear that I can use the Random Order Controller with the CSV file data however. So how can I achieve random CSV data item selection per thread?

Pharmacy

Answer 1

There is another way to achieve this:

• create a separate thread group
• depending on what you want to achieve:
  • add a (random) loop count -> this will set a start offset for the thread group that does the work
  • add a loop count or forever and a timer and let it loop while the other thread group is running. This thread group will read a 'pseudo' random line

It's not really random, the file is still read sequentially, but your work thread makes jumps in the file. It worked for me ;-)

Agent

Answer 2

The new Random CSV Data Set Config from BlazeMeter plugin should perfectly fit your needs.

Tatterdemalion

The OP specifically needs a solution for a 'very large file' but this plugin will not work well in that case because it can only work by reading the file into memory when the test initialises (the plugin's own docs confirm this issue). It's a good way to go for convenience but it's not scalable. – Kovacev

Answer 3

There's no random selection function when reading csv data. The reason is you would need to read the whole file into memory first to do this and that's a bad idea with a load test tool (any load test tool).

Other commercial tools solve this problem by automatically re-processing the data. In JMeter you can achieve the same manually by simply sorting the data using an arbitrary field. If you sort by, say Surname, then the result is effectively random distribution.

Note. If you ensure the default All Threads is set for the CSV Data Set Config then the data will be unique in the scope of the JMeter process.

Kovacev

I would suggest this answer suggested by Oliver Lloyd. There is always a way to randomize the csv dataset file by using a JSR223 sampler or by writing a small program which runs before jmeter and randomizes the rows in the csv file. – Dianadiandra

Answer 4

For my case:

• single column
• small dataset
• Non-changing CSV

I just discard using CSV and refer to https://mcmap.net/q/667773/-jmeter-set-variable-to-random-option and use a Bean Preprocessor instead, something like this:

String[] query = new String[]{"csv_element1", "csv_element2", "csv_element3"};
Random random = new Random();
int i = random.nextInt(query.length);
vars.put("randomOption",query[i]);

Performance seems ok, if you got the same issue can try this out.

Feverous

Precisely what I needed, simple enough and works well for a small data set. Can easily be modified to work for multiple columns by defining another array of values (of the same size) and using the same random index to get data from that array and put into a new variable. – Dianetics

Answer 5

I am not sure if this will work, but I will anyways suggest it.

Why not divide your URLs in 100 different CSV files. Then in each thread you generate the random number and use that number to identify CSV file to read using __CSVRead function.

CSVRead">http://jmeter.apache.org/usermanual/functions.html#_CSVRead

Now the only part I am not sure if the __CSVRead function reopens the file every time or shares the same file handle across the threads.

You may want to try it. Please share your findings.

Vtol

Answer 6

As other answers have stated, the reason you're not able to select a line at random is because you would have to read the whole file into memory which is inefficient.

Rather than trying to get JMeter to handle this on the fly, why not just randomise the file order itself before you start the test?

A scripting language such as perl makes short work of this:

 cat unrandom.csv | perl -MList::Util=shuffle -e 'print shuffle<STDIN>' > random.csv

Jesus

Answer 7

A much straight forward solution. In CSV file, add another column (say B) apply =RAND() function in the first cell of column B (say B1). This will create random float number. Drag the cell (say B1) corner to apply for all the corresponding URLs Sort column B. your URL will be sorted randomly. Delete column B.

Finery

Answer 8

Here is my workaround. I realize this is an old thread but posting anyway since jMeter still does not have a native way of choosing random values from parameter file.

You convert your file into json list. If you have a CSV file, all you need to do is surround each line with double-quotes and surround the whole list by square brackets. For example, a CSV like this would become a json list like following.

param1,
param2,
param3

["param1",
"param2",
"param3"]

After doing this, read this file using an HTTP Request sampler like so:

Now add a JSON extractor post-processor to this HTTP request, and parse a random json value! The JSON Path expressions field should contain $.* and the Match No. should be 0.

Lietuva

Answer 9

One possible solution that may work that has not been discussed yet is distributing the test across multiple agents running JMeter. (Such is the basic concept behind cloud-based JMeter load testing.) You can split the large CSV file into multiple parts, then distribute each part to a parallel-running JMeter instance. The Random CSV Data Set Config would have less of an issue with the smaller file sizes of a split CSV file. At least one popular cloud-based platform allows you to split a CSV file amongst load generator instances – with a single click – equally each running an identical copy of your JMeter load test.

Dwanadwane