How to import 1M records in batch with ElasticSearch rails?

I want to run a bulk import from my MySQL table to ES - for my model Wine - in my production server. There is 1.5M records.

My model - code for ES gem:

include Elasticsearch::Model
include Elasticsearch::Model::Callbacks

def as_indexed_json(options={}) 
  as_json(only: [:id, :searchable]) 
end 

mapping do
  indexes :id, index: :not_analyzed
  indexes :searchable
end

In development, I ran successfully:

bundle exec rake environment elasticsearch:import:model CLASS='Wine' BATCH='100'

but I have 1000 records only...

Can I run a similar command in prod without issues? Is there another way?

I noticed that I need to push the model updated with the code above, else it won't work. The problem is if a user wants to update an object before the bulk import and after my model is changed, there will be an ES issue (DocumentNotFound) - logical. Is it possible to use a callback to create the ES index if not already created, rather than getting an ES exception?

What the right way to do that? Does "elasticsearch:import:model" works in background?

Recommended topics

Hot tags