What object to hold a large amount of text in?

Question

I am planning a Seaside app to hold text, a single instance which may be up to, say, 5Mb. What kind of object is best for this?

I would also like to do some iterations over this text.

Thanks, Vince

Edit: Thanks for your replies thus far. The file is a CSV file that takes ~40 minutes to generate from a legacy finance system, so it must be pre-generated and stored. Each line is a customer record and I need to pull each one out and use the values as and when the customer logs in. Customer access is not predictable and interfacing with the legacy system to generate each line on the fly is a very last resort.

Cunning

Is there any reason that would rule String out? – Jacintha

Be more specific what is in the text what did you plan to do with? Why is it 5Mb? – Bauxite

Answer 1

Given that the file takes that long to generate and that you need more-or-less random access to the file later on, I would opt for parsing the file and keeping the structured data in memory afterwards.

There is a CSV Parser project on Squeaksource that you can use. It will create a structured object tree of the CSV records that you can use.

Poodle

We recently used this package and have made some fixes to it. However, we are not able to upload them because we are not able to contact the project admin. If you experience problems, let me know and I can send you are latest version already. – Poodle

Hello Johan. I gave it a go and removed the deprecated selectors and updated the syntax, but I have come to a halt with a UTF8 encoding problem; "Error: Invalid utf8 input detected". I would like to try your changes, and I will post it as a separate question/answer so that others may find it useful. Vince – Cunning

You probably only need to use a different stream because your file will not be in utf8 encoding. Check out the TextConverter class. I'm currently looking to publish the CSV package in a different repo. – Poodle

Hi. Yes thanks, I had already downloaded. Thank you again for the good work Johan. Vince – Cunning

Answer 2

Use an external Text file and some instance of a specific class as representation of that file. Use the oop of the object as the name of the file.

Brunildabruning

Beware though that on Squeak asOOp return a non-unique 12-bit object hash. – Ent

Answer 3

Just use a collection of customers, and fill it from the CSV, as Johan said. Depending on your accessing needs you can use a Dictionary or an OrderedCollection to hold it.

Magnuson

Answer 4

5 megs is nothing. Don't worry about that.

If you can't reify those CSV records into objects (after parsing and instantiating them), then a Collection of Strings or even Streams would be just fine.

If you need keyed lookup then a Dictionary or LookupTable would do the job.

I had 100 megs of text data in memory (1 millon rows), and even persisted in the image (image save) with no problems.

Regards.

Streamy