Symfony2 FOSElasticaBundle update index for all entities related to the entity updated

Question

I'm using FOSElasticaBundle and Doctrine in my project, and my code works for the selective index update using the Doctrine lifecycle events. The issue I come up against is if I an update a related entity separately.

For example a person may be related to a company through a manytomany relationship. If I update the company name through company entity directly, then indexes for the person related to the company will be out of date and still relate to the company's old name.

I'm a bit lost as to how to handle this, does anyone have any suggestions? Do I have to rely on a scheduled index update and cope with inaccurate index data in the mean time, or is there a way I can call an update for entities related to the entity that has been updated.

I am relying on JMSSerializer groups to establish the mappings. I appreciate this might not be the best way to do things in the longterm.

Firedamp

Answer 1

I think I've found the solution on this page https://groups.google.com/forum/#!topic/elastica-php-client/WTONX-zBTI4 Thanks Cassiano

Basically you need to extend the FOS\ElasticaBundle\Doctrine\ORM\Listener so you can look for related entities and then update their index as well.

class CompanyListener extends BaseListener
{

    /** @var \Symfony\Component\DependencyInjection\ContainerInterface */
    private $container;

    public function setContainer(\Symfony\Component\DependencyInjection\ContainerInterface $container) {
        $this->container = $container;
    }

    protected function initialiseJob() {
        $this->objectPersisterJob = $this->container->get('fos_elastica.object_persister.application.job');
        $this->em = $this->container->get('doctrine')->getEntityManager(); //maybe move this to postUpdate function so it can be used for all
    }

    /**
     * @param \Doctrine\ORM\Event\LifecycleEventArgs $eventArgs
     */
    public function postUpdate(LifecycleEventArgs $eventArgs)
    {
        /** @var $entity Story */
        $entity = $eventArgs->getEntity();

        if ($entity instanceof $this->objectClass) {
            if ($this->isObjectIndexable($entity)) {
                $this->objectPersister->replaceOne($entity);
                $this->initialiseJob();
                foreach ($entity->getJobOpenings() as $job) {
                    $this->objectPersisterJob->replaceOne($job);
                }
            } else {
                $this->scheduleForRemoval($entity, $eventArgs->getEntityManager());
                $this->removeIfScheduled($entity);
            }
        }
    }

    public function preRemove(\Doctrine\Common\EventArgs $eventArgs)
    {
        $entity = $eventArgs->getEntity();

        if ($entity instanceof $this->objectClass) {

            $this->scheduleForDeletion($entity);
            $this->initialiseJob();
            foreach ($entity->getJobOpenings() as $job) {
                $this->objectPersisterJob->replaceOne($job);
            }
        }
    }


}

and your services defined as below

fos_elastica.listener.application.company:
    class: 'xxx\RMSBundle\EventListener\CompanyListener'
    arguments:
        - '@fos_elastica.object_persister.application.company'
        - 'xxx\RMSBundle\Entity\Company'
        - ['postPersist', 'postUpdate', 'postRemove', 'preRemove']
        - id
    calls:
        - [ setContainer, [ '@service_container' ] ]
    tags:
        - { name: 'doctrine.event_subscriber' }

this will then update indexes for both :-)

Firedamp

can you please add your config.yml elastica configuration along with your above code? i am following your solution and missing something. somehow it is not working for me. – Audieaudience

Answer 2

I've had the same problem. It seems my installation (Symfony 2.5.4 and FOSElastica 3.0.4) differs quite a bit from yours though. Therefore, there were some problems to get the code working. I'm posting my solution, because it may be useful for other developers out there.

The Listener isn't in FOS\ElasticaBundle\Doctrine\ORM\, but in FOS\ElasticaBundle\Doctrine. So you'll have to use that one. Also I had to use Doctrine\Common\EventArgs instead of Doctrine\ORM\Event\LifecycleEventArgs, 'cause otherwise my own postUpdate-method wasn't compatible with the one in the BaseListener.

In my app, a course (seminar) can have a lot of sessions, but in this project, elastica will only be using those sessions. The app needs to know some details of the course that is related to the session of course. So, here's my code:

In config.yml my elastica bundle config looks like this:

fos_elastica:
    clients:
        default: { host: localhost, port: 9200 }
    indexes:
        courses:
            index_name: courses
            types:
                session:
                    mappings:
                        id: ~
                        name: ~
                        course:
                            type: "nested"
                            properties:
                                id: ~
                                name: ~

A little further, still in config.yml

services:
     # some other services here

     fos_elastica.listener.courses.course:
         class: XXX\CourseBundle\EventListener\ElasticaCourseListener
         arguments:
             - @fos_elastica.object_persister.courses.course
             - ['postPersist', 'postUpdate', 'preRemove']
             - @fos_elastica.indexable
         calls:
             - [ setContainer, ['@service_container', @fos_elastica.object_persister.courses.session ] ]
         tags:
             - { name: 'doctrine.event_subscriber' }

My own listener (XXX\CourseBundle\EventListener\ElasticaCourseListener) then looks like this:

<?php

namespace XXX\CourseBundle\EventListener;

use Doctrine\Common\EventArgs;
use FOS\ElasticaBundle\Doctrine\Listener as BaseListener;
use FOS\ElasticaBundle\Persister\ObjectPersister;
use Symfony\Component\DependencyInjection\ContainerInterface;
use XXX\CourseBundle\Entity\Course;

class ElasticaCourseListener extends BaseListener
{
    private $container;
    private $objectPersisterSession;

    public function setContainer(ContainerInterface $container, ObjectPersister $objectPersisterSession)
    {
        $this->container = $container;
        $this->objectPersisterSession = $objectPersisterSession;
    }

    public function postUpdate(EventArgs $args)
    {
        $entity = $args->getEntity();

        if ($entity instanceof Course) {
            $this->scheduledForUpdate[] = $entity;
            foreach ($entity->getSessions() as $session) {
                $this->objectPersisterSession->replaceOne($session);
            }
        }
    }
}

Now, when I update a course, it will be updated as a nested object in ElasticSearch ;-)

Reorganization

@Reorganization I've just updated FOSElastica and used your code as guide. Thanks!! – Firedamp

in FOSElastica 3.0.0.alpha6 I've had to update the service definition with a fourth argument - { indexName: 'courses', typeName: 'course'} to avoid php notices and errors in testing – Firedamp

This - unfortunately - only works with FOSElasticaBundle 3.0 versions. The 3.1 versions seems to use a different listener architecture. – Hindward

I noticed the BC breaking changes in 3.1. I'll update my answer as soon I'm on another ElasticSearch-project. – Reorganization

I have "friendsofsymfony/elastica-bundle": "~3.0" and: [Symfony\Component\Debug\Exception\ContextErrorException] Runtime Notice: Declaration of Artel\SiteBundle\EventListener\ElasticaCourseListener::postUpdate() should be compatible with FOS\ElasticaBundle\Doctrine\Listener::postUpdate(Doctrine\Common\Persistence\Event\LifecycleEventArgs $eventArgs) – Boondoggle

Answer 3

I'm using FosElastica 3.1.0 and I have tried the solution provided by Julien Rm without success :-(

After many days of research, I finally found the solution here

$persister = $this->get('fos_elastica.object_persister.jaiuneidee.post');
$persister->insertOne($post);

Hope this help !

Remnant

Answer 4

with all comments and my research, I made a generic Gist for auto index child objects with fosElastica:

https://gist.github.com/Nightbr/ddb586394d95877dde8ed7445c51d973

In fact, I override the default Listener from FOSElastica and I add the function updateRelations($entity). We will search all relations linked to the $entity and if there are indexed in ES (the ES type exists) it will update the related documents.

If anyone want to look at it and make any improvement it would be great! ^^

Thanks in advance

Jounce

Thanks for this. looks great. I tend to use scheduledForInsertion as scheduledForUpdate doesn't handle updating a value to null very well, at least not in the version I'm using. I think it might also be a fraction faster. – Firedamp

I'm using ES 2.3 with a fork of FOSElastica: github.com/fazland/FOSElasticaBundle – Jounce

Answer 5

Sorry, i can not comment under your answer but something is missing in the solution. You have to override preRemove too.

public function preRemove(\Doctrine\Common\EventArgs $eventArgs)
{
    $entity = $eventArgs->getEntity();



    if ($entity instanceof $this->objectClass) {

        $this->scheduleForDeletion($entity);
        $this->initialiseJob();
        foreach ($entity->getJobOpenings() as $job) {
                $this->objectPersisterJob->replaceOne($job);
            }
    }
}

Parch

Answer 6

With the BC Break #729 of FosElastica 3.1.0, things have changed and the code above wasn't working :

BC BREAK: Removed Doctrine\Listener#getSubscribedEvents. The container configuration now configures tags with the methods to call to avoid loading this class on every request where doctrine is active. #729

For those who are trying to make it work with FOSElastica 3.1.X here is how I did manage to make a nested objected to be indexed into his parent into Elastic Search when persisting/updating/removing a nested entity :

Define the service listener :

fos_elastica.listener.entity.nested:
    class: XX\CoreBundle\EventListener\EventSubscriber\ElasticaNestedListener
    arguments:
        - @fos_elastica.object_persister.app.entityname
        - @fos_elastica.indexable
        - {"indexName" : "app", "typeName": "entityname"}
    tags:
        - { name: 'doctrine.event_subscriber' }

Create the listener :

<?php
class ElasticaNestedListener implements EventSubscriber
{ // some indentations missing!

public function getSubscribedEvents()
{
    return array(
        'postPersist',
        'preRemove',
        'postUpdate',
        'preFlush',
        'postFlush',
    );
}

/**
 * Object persister.
 *
 * @var ObjectPersisterInterface
 */
protected $objectPersister;

/**
 * Configuration for the listener.
 *
 * @var array
 */
private $config;

/**
 * Objects scheduled for insertion.
 *
 * @var array
 */
public $scheduledForInsertion = array();

/**
 * Objects scheduled to be updated or removed.
 *
 * @var array
 */
public $scheduledForUpdate = array();

/**
 * IDs of objects scheduled for removal.
 *
 * @var array
 */
public $scheduledForDeletion = array();

/**
 * PropertyAccessor instance.
 *
 * @var PropertyAccessorInterface
 */
protected $propertyAccessor;

/**
 * @var IndexableInterface
 */
private $indexable;

/**
 * Constructor.
 *
 * @param ObjectPersisterInterface $objectPersister
 * @param IndexableInterface       $indexable
 * @param array                    $config
 * @param LoggerInterface          $logger
 */
public function __construct(
    ObjectPersisterInterface $objectPersister,
    IndexableInterface $indexable,
    array $config = array(),
    LoggerInterface $logger = null
) {
    $this->config = array_merge(array(
            'identifier' => 'id',
        ), $config);
    $this->indexable = $indexable;
    $this->objectPersister = $objectPersister;
    $this->propertyAccessor = PropertyAccess::createPropertyAccessor();

    if ($logger && $this->objectPersister instanceof ObjectPersister) {
        $this->objectPersister->setLogger($logger);
    }
}



/**
 * Looks for objects being updated that should be indexed or removed from the index.
 *
 * @param LifecycleEventArgs $eventArgs
 */
public function postUpdate(LifecycleEventArgs $eventArgs)
{
    $entity = $eventArgs->getObject();

    if ($entity instanceof EntityName) {

        $question = $entity->getParent();
        if ($this->objectPersister->handlesObject($question)) {
            if ($this->isObjectIndexable($question)) {
                $this->scheduledForUpdate[] = $question;
            } else {
                // Delete if no longer indexable
                $this->scheduleForDeletion($question);
            }
        }
    }


}


public function postPersist(LifecycleEventArgs $eventArgs)
{
    $entity = $eventArgs->getObject();

    if ($entity instanceof EntityName) {
        $question = $entity->getParent();
        if ($this->objectPersister->handlesObject($question)) {
            if ($this->isObjectIndexable($question)) {
                $this->scheduledForUpdate[] = $question;
            } else {
                // Delete if no longer indexable
                $this->scheduleForDeletion($question);
            }
        }
    }


}


/**
 * Delete objects preRemove instead of postRemove so that we have access to the id.  Because this is called
 * preRemove, first check that the entity is managed by Doctrine.
 *
 * @param LifecycleEventArgs $eventArgs
 */
public function preRemove(LifecycleEventArgs $eventArgs)
{
    $entity = $eventArgs->getObject();

    if ($this->objectPersister->handlesObject($entity)) {
        $this->scheduleForDeletion($entity);
    }
}

/**
 * Persist scheduled objects to ElasticSearch
 * After persisting, clear the scheduled queue to prevent multiple data updates when using multiple flush calls.
 */
private function persistScheduled()
{
    if (count($this->scheduledForInsertion)) {
        $this->objectPersister->insertMany($this->scheduledForInsertion);
        $this->scheduledForInsertion = array();
    }
    if (count($this->scheduledForUpdate)) {
        $this->objectPersister->replaceMany($this->scheduledForUpdate);
        $this->scheduledForUpdate = array();
    }
    if (count($this->scheduledForDeletion)) {
        $this->objectPersister->deleteManyByIdentifiers($this->scheduledForDeletion);
        $this->scheduledForDeletion = array();
    }
}

/**
 * Iterate through scheduled actions before flushing to emulate 2.x behavior.
 * Note that the ElasticSearch index will fall out of sync with the source
 * data in the event of a crash during flush.
 *
 * This method is only called in legacy configurations of the listener.
 *
 * @deprecated This method should only be called in applications that depend
 *             on the behaviour that entities are indexed regardless of if a
 *             flush is successful.
 */
public function preFlush()
{
    $this->persistScheduled();
}

/**
 * Iterating through scheduled actions *after* flushing ensures that the
 * ElasticSearch index will be affected only if the query is successful.
 */
public function postFlush()
{
    $this->persistScheduled();
}

/**
 * Record the specified identifier to delete. Do not need to entire object.
 *
 * @param object $object
 */
private function scheduleForDeletion($object)
{
    if ($identifierValue = $this->propertyAccessor->getValue($object, $this->config['identifier'])) {
        $this->scheduledForDeletion[] = $identifierValue;
    }
}

/**
 * Checks if the object is indexable or not.
 *
 * @param object $object
 *
 * @return bool
 */
private function isObjectIndexable($object)
{
    return $this->indexable->isObjectIndexable(
        $this->config['indexName'],
        $this->config['typeName'],
        $object
    );
}
}

EntityName could be a Comment and getParent() could be the Article who owns this comment ...

Hope this help !

Gynaecology

Julien: One question! For the entity Article what function getParent() has to return null? Merci beaucoup! – Vibes

Answer 7

I'm using Symphony 3 and FOSElasticaBundle 3.2 and I did things a bit differently. After reviewing the code given in the other answers, which helped a lot, I've decided not to extend the default listener. Instead I let it do its thing and I just added my own listener.

I have some Categories (1) which can have multiple (many-to-many) Subjects (2) which can have multiple (one-to-many) Posts (3). The Posts are the entities being saved in Elasticsearch with infos on their respective Subject and its own Categories.

Like so:

fos_elastica:
  #...
  indexes:
    my_index:
      #...
      types:
        post: # (3)
          mappings:
            field_one: ~
            # ... Other fields
            subject: # (2)
              type: "object"
              properties:
                subject_field_one: ~
                # ... Other fields
                categories: # (1)
                  type: "nested"
                  properties:
                    category_field_one: ~
                    # ... Other fields

The service definition (app/config/services.yml)

services:
  # ...
  app.update_elastica_post.listener:
    class: AppBundle\EventListener\UpdateElasticaPostListener
    arguments: ['@service_container']
    tags:
      - { name: doctrine.event_listener, event: postUpdate }

And the listener AppBundle\EventListener\UpdateElasticaPostListener.php

namespace AppBundle\EventListener;

use Doctrine\ORM\Event\LifecycleEventArgs;
use Symfony\Component\DependencyInjection\ContainerInterface;

use AppBundle\Entity\Category;
use AppBundle\Entity\Subject;

class UpdateElasticaPostListener
{
    private $container;
    private $objectPersisterPost;

    public function __construct(ContainerInterface $container)
    {
        $this->container = $container;
        $this->objectPersisterPost = null;
    }

    /**
     * @param \Doctrine\ORM\Event\LifecycleEventArgs $eventArgs
     */
    public function postUpdate(LifecycleEventArgs $eventArgs)
    {
        $this->checkAndUpdate($eventArgs);
    }

    protected function checkAndUpdate(LifecycleEventArgs $eventArgs)
    {
        $entity = $eventArgs->getEntity();

        if ($entity instanceof Category) {
            foreach ($entity->getSubjects() as $subject) {
                $this->updateSubjectPosts($subject);
            }
        } elseif ($entity instanceof Subject) {
            $this->updateSubjectPosts($entity);
        }
    }

    protected function updateSubjectPosts(Subject $subject)
    {
        $this->initPostPersister();
        foreach ($subject->getPosts() as $post) {
            $this->objectPersisterPost->replaceOne($post);
        }
    }

    protected function initPostPersister()
    {
        if (null === $this->objectPersisterPost) {
            // fos_elastica.object_persister.<index_name>.<type_name>
            $this->objectPersisterPost = $this->container->get('fos_elastica.object_persister.my_index.post');
        }
    }
}

And that's it! I didn't try it for the remove event and now that I think about it, maybe this solution wouldn't be the best one for it... but maybe it is...

Thanks a lot to @Ben Stinton and @maercky above.

I hope it helps! (this is my first answer around here so I hope I didn't screw up)

Atcliffe