ESP32: Best way to store data frequently?

Question

I'm developing a C++ application in the ESP32-DevKitC board where I sense acceleration from an accelerometer. The application goal is to store the accelerometer data until storage is full and then send all the data through WiFi and start all again. The micro also goes to deep-sleep mode when is possible.

I'm currently using the ESP32 NVS library which is very well documented and pretty easy to use. The negative side of this is that the library uses Flash memory, therefore a lot of writings will end up degrading the drive.

I know that Espressif also offers some other storage libraries (FAT, SPIFFS, etc.) but, as far as I know (correct me if I'm wrong), they all use Flash drive.

Is there any other possibility of doing what I want to but without using the Flash storage?

Aclarations

• Using Flash memory is not the problem itself, but degrading it.
• Storage has to be non volatile or at least not being erased when the micro goes to deep-sleep mode.
• I'm not using any Arduino library.

Pyonephritis

Are you buffering data until a complete page can be wriiten? – Hasseman

@Hasseman I think I'm not. Each time my accelerometer overflows in its FIFO buffer, I read the buffer and use nvs_set_blob() with the buffer data. Each time I do this, I create a new pair key:value. More precisely, each key contains 300 bytes. – Pyonephritis

A couple questions: (1) how much data do you collect before you send it out via WiFi? (2) is lost data a concern (is that why you're storing it in NVS)? – Towers

@Towers (1) I've to take care of power consumption, so I should use WiFi only when the storage is full. According to Espressif, Flash storage has 4MB. (2) Yes, I can't lose data. – Pyonephritis

OK. Well romkey's answer is excellent. The TL;DR on it is if you can't afford to lose data (and it's a non-trivial amount of data), then the NVS system is the only option on the module. Of course you're free to add other forms of external storage. Since you said nothing to the contrary, I'm assuming that the NVS system performance is adequate for your purposes. – Towers

Answer 1

That's a great question that I wish more people would ask.

ESP32s use NOR flash storage, which is usually rated for between 10,000 to 100,000 write cycles (100,000 seems to be the standard these days). Flash can't write single bytes; instead of writes a "page" of bytes, which I believe is 256 bytes. So each 256 byte page is rated for at least 100,000 cycles. When a device is rated for 100,000 cycles it's likely to be usable for at least 10 times that, but the manufacturer is not going to make any promises beyond the 100,000.

SPIFFS (and LittleFS, now used on the ESP8266 Arduino Core) perform "wear leveling", to minimize the number of times a particular page is written. So if you modify the same section of a file repeatedly, it will automatically be written to different pages of flash. FAT is not designed to work well with flash storage; I would avoid it.

Whether SPIFFS with wear leveling will be adequate for your needs depends on your needed lifetime of the device versus how much data you'll be writing and how frequently.

NVS may perform some level of wear levelling, to an extent I'm unsure about. Here, in a forum post with 2 ESP employees, they both confirm that NVS does do some form of wear levelling. NVS is best used to persist things like configuration information that doesn't change frequently. It's not a great choice for storing information that's updated often.

You mentioned that the data just needs to survive deep sleep. If that's the case, your best option (if it's large enough) is to use the ESP32's RTC static RAM. This chunk of memory will survive restarts and deep sleep mode, but will lose its state if power is interrupted. It's real RAM so you won't wear it out by writing to it frequently, and it doesn't cost a lot of energy to write to. The catch is there's only 8KB of it.

If the 8KB of RTC RAM isn't enough and you're writing too much data too frequently to trust that SPIFFS will be okay, your best bet would be an SD card. The ESP32 can talk to an SD card adapter. SD cards use NAND flash, which has a much greater lifespan than NOR and can be safely overwritten many more times (which is why these kinds of cards are usable for filesystems in devices like Raspberry Pis).

Writing to flash also takes much more energy than writing to regular RAM. If your device is going to be battery powered, the RTC RAM is also a better choice than SPIFFS or an SD card from a power savings perspective.

Finally, if you use the RTC RAM I'd recommend starting to write it over wifi before it's full, as bringing up wifi and transmitting the data could easily take long enough that you might run out of space for some samples. Using it as a ring buffer and starting the transmit process when you hit a high water mark rather than when the buffer is full would probably be your best bet.

Tharpe

Very clear and useful, thank you @romkey! I will definitely try RTC RAM. Fortunatily, looks very simple to use. – Pyonephritis

I'm no flash expert, but doesn't NOR flash allow writing single words to an already erased block (in contrast to erasing, which is possible only block-wise)? You could erase the whole flash memory and then slowly write words to it (each word/byte only once) until the memory is full and then erase everything again. I gues the erase process is what degrades the chip and that would be reduced to the absolute minimum. – Ultimogeniture

The above comment is exactly what SPIFFS does. Unfortunately, this may no longer be the case because some crappy PUYA-branded flash chips don't exhibit this behavior, and broke compatibility. It has since been deprecated as a result, so you may need to implement it yourself. As another note, there are also I2C and SPI external FRAM, MRAM, and EERAM chips available. The first two are nonvolatile RAM with millions+ of rewrite cycles per byte, and the latter is a RAM that has the option of storing to EEPROM on power loss (if you keep it powered it won't do that). It only needs a tiny capacitor. – Orontes

In the end I didn't, but I almost downvoted. All this answer says is "if you only need to survive deep sleep then whack RTC_DATA_ATTR in front of your variable and you don't have to worry about wear leveling", but first it spends several paragraphs talking about wear leveling and how you don't have a solution for that. – Microscopium

Is it possible to write json values as strings into RTC RAM? – Barnett

@cjacky475 Sure. You can store any sequence of bytes that is smaller than the available RTC storage. You need to serialize the JSON as a string and store that, assuming it fits. Different JSON libraries will provide their own ways to serialize and deserialize objects. – Tharpe

@romkey, thank you for the answer. I'm going to store access token retrieved from the server, and it's going to be around 200 symbols. Do I need to create a char array and do a loop through the access token string and write each letter to that char array? Because as I recently seen, espressif documentation states: "any string constants used in this way must be declared as arrays and marked with RTC_RODATA_ATTR". If I am mistaken with my idea, would you mind sharing a simple example? Thank you. – Barnett

@cjacky475 Please post your question as a new Stack Overflow question; it's difficult to answer things like this in comments. – Tharpe

How about a FRAM module? It's able to do the work for small amounts of data, and yes is not volatile – Nutgall

Answer 2

I know i'm late with this answer but you can buy ESP32 modules with external RAM even with 4-8mb. External ram is really fast ( at least much faster than the flash, it uses SPI interface to communicate ) and you can fit a lot of sensor readings in there.

I'm using an ESP32_WROVER_E module with 8mb external ram ( 4mb is usable with normal function calls ) and 16mb flash.

Here is a link of the module that i'm using at TME's site.

German