Multi-language indexes with Laravel Scout and Algolia

Question

How should I manage the multi-language indexes (For example: page / page_translations models should become page_en / page_fr indexes). I am using "Dimsav\Translatable" package.

Page model: id, status_id, created_at, updated_at

PageTranslation model: id, page_id, locale, title, slug, body

Algolia offers support for this (https://www.algolia.com/doc/guides/search/multilingual-search/) but I am not sure how to achieve this with Laravel Scout.

The only solution that comes in my mind is to index both language rows (from the translations model) in the same index storing the locale and applying a condition on search.

Algolia

objectID=1, title='English title', locale_id='1'

objectID=2, title='Franch title', locale_id='2'

$pages = App\PageTranslation::search('Star Trek')->where('locale_id', 1)->get();

Or maybe a better approach? Maybe to index page / page_translations separately and search in both?

I would like to achieve something like:

pages_en index : objectID=1, title='English title', etc.

pages_fr index : objectID=2, title='Franch title', etc.

$pages = App\Page::search('Star Trek')->where('locale', 'en')->get();

Peach

Hey Dorin Niscu, I'm trying to wrap my head around that. I think it's a great usecase, thank you for asking :) – Itching

Well, yes ..seems to be challenging for doing it in the current version. I don't want to reinvent the wheel. I was thinking to iterate through languages on model changes. – Peach

Answer 1

I thought about it a lot and I think the best way would be to use 1 index per model and take advandate of the callback you can pass to ::search()

Indexing data

First you need to use toSearchableArray() to prepare the data. I would unset every unnecessary attributes (like dates) then nest content under its ISO.

{
  objectID: 1,
  en: {
    title: "Title in english",
    body: "trucated body in english"
  },
  fr: {
    title: "Titre en français",
    body: "contenu tronqué en français"
  }
}

Please note that Algolia has a limit of 10KB per records. The best way to handle this is to truncate your biggest attributes. Don't worry, it doesn't impact relevance. If you miss the second half of your article, usually all the relevant content is already in the first haft.

Setup Algolia config in dashboard

Then head to your dashboard and add fr and en to the searchableAttributes.

Search

You can restrict searchableAttributes at query time with a callback passed to the search

$lang = 'en';
Model::search($query, function ($algolia, $query, $options) use ($lang) {
    $options = array_merge($options, [
        'restrictSearchableAttributes' => [$lang],
    ]);

    return $algolia->search($query, $options);
});

I created a trait to achieve something similar. Maybe you can do something similar, in order to have a easy-to-use syntax like:

Model::searchLang($lang, $query);

After all the thinking, I really think it's the least hacky way to use Laravel Scout with your constraints.

Please let me know what you think :)

Itching

Hi, @Julien. This solution could work as alternative of storing translations as separate rows but I would like to store each language in different index. This way I will not truncate the content. I was thinking to create a fork from Laravel Scout and maybe use something like abstract public function update($models, $language); and $array = $model->toSearchableArray($language); and generate the searchable array depending on language or default locale. github.com/laravel/scout/blob/3.0/src/Engines/AlgoliaEngine.php – Peach

Of course, the $language param could become $params or $options for multiple purposes. – Peach

If you want, you can extend Scout instead of modifying it. I started this lib to extend Laravel Scout capabilities. Maybe you'll find something interesting: github.com/julienbourdeau/laravel-scout-advanced-algolia-driver I need to write the doc for it. – Itching

A better approach to my solution storing each translation on different row could be using filter facets (en / fr etc.) but Laravel Scout does not support it out of the box, only numeric filters. We can extend the Algolia engine with it. Maybe the main stack too. – Peach

Having multiple indexes per locale in some case could be the only possibility. With lots of locales or lots of searcheable attributes, the 10Kb limit per records could be reached fast – Nonah