Limit the growth of ETS storage

Question

I'm considering using Erlang's ETS as a cache for user searches in a new Elixir project. Based on user input, the system will do lookups using an expensive third-party API.

In order to avoid making duplicate calls for the same user input, I intend to put a cache layer in front of the external API, and ETS seems like a good option for this. However, since there is no limit to the variations of user input, I'm concerned that the storage space required for the ETS table will grow without bound.

In my reading about ETS, I haven't seen anyone else discuss concern about the size of tables in ETS. Is that because this would be an abnormal use case for ETS?

At first blush, my preference would be to limit the number of entries in the ETS table, and reject (i.e. delete) the oldest entries once the limit is reached…

Is there a common strategy for dealing with unbounded number of entries in ETS?

Luu

the first page of results from google for "erlang ets cache" gives a lot of useful hits. Did you have a look? – Lavabo

Answer 1

I use ETS tables in production like a 'smart invalidated cache' with a redis API (also it have master-master replication like a SQL WAL log).

The biggest sizes is ~ 200-300Mb and they have more than 1million items. There are no any problems for last 2 years. I know about limits ERL_MAX_ETS_TABLES but havn't any information about sizes.

I have special 'smart indexes' for this tables. ETS select/match/etc is slow because this methods passing all the elements in the table.

Olette

Answer 2

use the ets:tab2list(TableId) function to convert the ETS table to a common list. After doing that, you are able to check the size of the list with the, well known BIF length(List). Last but not least, you are now able to set a buffer (just check the size of the list with pattern matching, if, or case expression

Kinaesthesia