HBase ASYNC WAL when I'm able to recover data from Kafka

Question

I want to optimize my usage of HBase for faster writes. I have a task which reads from a Kafka topic then write to HBase based on that. Since Kafka will have a log of everything to be written, it's an easy source to recover from. I'm reading "HBase High Perormance Cookbook" and there's this note:

Note that this brings an interesting thought about when to use WAL and when not to. By default, WAL writes are on, and the data are always written to, WAL. But if you are sure the data can be rewritten or a small loss won't be impacting the overall outcome of the processing, you disable the write to WAL. WAL provides an easy and definitive recovery. This is the fundamental reason why, by default, it's always enabled. In scenarios where data loss is not expectable, you should leave it in the default settings; otherwise, change it to use memstore. Alternatively, you can plan for a DR (disaster recovery)

How do I configure this recovery to be automatic? I see 2 options:

1. I write to HBase without WAL (only to memstore) and am somehow notified that writes were lost and not committed to disk. Then I go back in the Kafka log and replay. or
2. I write to HBase without WAL (only to memstore) and every so often get notified from HBase what Kafka offset can be committed.

How do I do either of these?

Vaasa

Have you considered switching to multiwal instead? hbase.apache.org/book.html#_multiwal. Because as stated in the same HBase reference: "The only situation where [disabling WAL] is recommended is during a bulk load. etc. etc. etc. " – Midnight

I understood that with async WAL I gain performance and lose persistency since the write is stored just in mem and then the put function returns. – Vaasa

Are you considering a premature optimization, or there is an actual bottle neck in your writes to the WAL? – Mirthless

I do see slowness when writing, mainly during peaks, I don't know if this is due to WAL. I was thinking that such a configuration change could help since writes would be quickly committed during the peak and then written to WAL asynchronously afterwards. And if anything goes wrong I can recover from Kafka. – Vaasa

Async writes to the WAL might have more operational overhead than you would expect, because you still need to be able to recover from Kafka when necessary (then why using the WAL at all?). If you experience some performance issues, you have to profile the cluster - not just assuming the WAL is the root cause. There are many moving parts and parameters related to writing performance. Standard monitoring (hbase metrics, jvm, os) should be enough. – Mirthless

@Mirthless There are numerous metrics, can you recommend what to look at to determine if such a change will help? – Vaasa

From HBase try to monitor memstore flush, memstore size, WAL size, compaction, and possible splits. From JVM monitor GCs. From OS check IO, especially disk writes. All these are crucial factors in heavy writes. Also, make sure the table has compression enabled. – Mirthless