Apache Mahout Performance Issues
Asked Answered
M

2

8

I have been working with Mahout in the past few days trying to create a recommendation engine. The project I'm working on has the following data:

  • 12M users
  • 2M items
  • 18M user-item boolean recommendations
  • I am now experimenting with 1/3 of the full set we have (i.e. 6M out of 18M recommendations). At any configuration I tried, Mahout was providing quite disappointing results. Some recommendations took 1.5 seconds while other took over a minute. I think a reasonable time for a recommendation should be around the 100ms timeframe.

    Why does Mahout work so slow?
    I'm running the application on a Tomcat with the following JVM arguments (even though adding them didn't make much of a difference):

    -Xms4096M -Xmx4096M -da -dsa -XX:NewRatio=9 -XX:+UseParallelGC -XX:+UseParallelOldGC
    

    Below are code snippets for my experiments:

    User similarity 1:

    DataModel model = new FileDataModel(new File(dataFile));
    UserSimilarity similarity = new CachingUserSimilarity(new LogLikelihoodSimilarity(model), model);
    UserNeighborhood neighborhood = new NearestNUserNeighborhood(10, Double.NEGATIVE_INFINITY, similarity, model, 0.5);
    recommender = new GenericBooleanPrefUserBasedRecommender(model, neighborhood, similarity);
    

    User similarity 2:

    DataModel model = new FileDataModel(new File(dataFile));
    UserSimilarity similarity = new CachingUserSimilarity(new LogLikelihoodSimilarity(model), model);
    UserNeighborhood neighborhood = new CachingUserNeighborhood(new NearestNUserNeighborhood(10, similarity, model), model);
    recommender = new GenericBooleanPrefUserBasedRecommender(model, neighborhood, similarity);
    

    Item similarity 1:

    DataModel dataModel = new FileDataModel(new File(dataFile));
    ItemSimilarity itemSimilarity = new LogLikelihoodSimilarity(dataModel);
    recommender = new GenericItemBasedRecommender(dataModel, itemSimilarity);
    
    Melcher answered 23/11, 2011 at 10:11 Comment(0)
    M
    4

    With the gracious help of the Mahout community via its mailing list, we have found a solution to my problem. All of the code related to the solution was committed into Mahout 0.6. More details can be found in the corresponding JIRA ticket.

    Using VisualVM I found that the performance bottleneck was in the computation of item-item similarities. This was addressed by @Sean using a very simple but effective fix (see the SVN commit for more details)

    Additionally, we have discussed how to improve the SamplingCandidateItemsStrategy to allow finer control over the sampling rate.

    Finally, I did some testing with my application with the aforementioned fixes. All the recommendations took less than 1.5 seconds with the overwhelming majority taking less than 500ms. Mahout could easily handle 100 recommendations per second (I did not try to stress it more than that).

    Melcher answered 8/12, 2011 at 7:56 Comment(0)
    S
    2

    Small suggestion: your last snippet should use GenericBooleanPrefItemBasedRecommender.

    For your data set, the item-based algorithm should be best.

    This sounds a little slow, and minutes is way too long. The culprit is lumpy data; time can scale with the number of ratings a user has provided.

    Look at SamplingCandidateItemsStrategy. This will let you limit the amount of work done in this regard by sampling in the face of particularly dense data. You can plug this in to GenericBooleanPrefItemBasedRecommender instead of using the default. I think this will give you a lever to increase speed and also make response time more predictable.

    Stepsister answered 23/11, 2011 at 11:5 Comment(16)
    Thnx Sean. I tried your suggestions with the following code pastebin.com/XiuJvRha . But performance is still not good. Even with the 6M set (1/3rd of the real set), recommendations still take between 3-15 secs. What do you make out of it?Melcher
    Ok - I have tested it a bit more and I have noticed that for users that had made 1-2 recommendations are quick, about 400ms, but for users who have made 10 or 20 recommendations it takes much more. One user with 28 recommendations took over a minute to complete.Melcher
    You'll want to adjust the values in SamplingCandidateItemsStrategy. Try (10,5) for example. This all still sounds quite slow, though it looks pretty good. There's some degree of warm-up as the caches fill with precomputed similarity; I don't know if that's a factor?Stepsister
    It works great for most of the users but there are still users where a query for takes a lot of time. It seems that what common for these users is that they made at least 20-30 recommendations. And that the resulted 'RecommendedItem's values are high. I assume that Mahout puts a lot of effort because there are many options to choose from. Is there are other tweak I can do to prevent it from handing for a whole minute? Maybe somehow lower the sampling rate?Melcher
    Yes, that's what I'm suggesting -- lower numbers mean lower sampling rate. Do you have access to a profiler? It all still seems quite slower than I'd imagine. I wonder if you can in this way gain direct insight into the slowdown. That would allow for more targeted advice.Stepsister
    I have lowered the sampler and added CachingRecommender just in case. Response time is around 700ms which is not bad. I will try to play around with the parameters and use a profiler to diagnose the bottleneck. Thanks for your time..Melcher
    I found out the app hangs for users with many existing user-item connections on GenericItemBasedRecommender:131 (Mahout0.5). It seems that possibleItemIDs is very big and that 'TopItems.getTopItems' iterate over all that set. Debug info: possibleItemIDs FastIDSet keys (id=69) numEntries 3390 numSlotsUsed 3430 My thought was to have the recommender only handle a limited number of recent choices a user had made. However, looking at the implementation of FastIDSet it seems that the values cannot be sorted by the time they were added. How do you suggest to tacle this?Melcher
    This is exactly what CandidateItemStrategy helps you with. It will let you sample from those 3390 entries instead of picking them all. You could write your own implementation that picks whatever you like. It sounds like you are not using CandidateItemStrategy quite right if you're observing it actually come back with 3,390 entries; the settings I suggest would allow no more than 10 or so. (And again I think you want to use GenericBooleanPrefItemBasedRecommender.)Stepsister
    I am using GenericBooleanPrefItemBasedRecommender which extends GenericItemBasedRecommender and not overriding the recommend method. See the code here. In SamplingCandidateItemsStrategy.doGetCandidateItems, maxPrefsPerItemConsidered evaluates to 32 because of the high num of users we have. For each previously recommended item it samples Math.min(prefs.length(), maxPrefsPerItemConsidered) users, for each it loads all the item they recommended (SamplingCandidateItemsStrategy:77). This can easily get to hundrands or very well over 3000 items.Melcher
    Got it -- it sounds like you just want to clamp down sampling further. I had recommended a max of perhaps 10, but you could go lower.Stepsister
    I lowered the SamplingCandidateItemsStrategy values to (1,1). Recommendations for users with up to 15 past recommendations are up to 1-1.5 seconds. Beyond that it takes already 3-8 seconds. Additionally, if a user had previously recommended an extremely popular item (we have one with 400k recommendations) and at least a few more items, the recommendation will be significantly slower. It seems to me that unless a very different approach is possible here, Mahout just does not perform well with this data for some reason. @SeanOwen What do you think ?Melcher
    Perhaps it would help if I pre-computed item similarities? I also tried replacing the LogLikelihoodSimilarity with TanimotoCoefficientSimilarity but it didn't have any meaningful affect.Melcher
    You can definitely pre-compute similarity -- GenericItemSimilarity will help you there. It takes a lot of memory. But yes that should also go straight to your bottle-neck.Stepsister
    The app already uses about 3G of memory, I cannot fit much more. What would one do if I had twice or three times more data? In the documentation it says a single machine can take up to 100M user-item entries (I only have 18M)Melcher
    Perhaps we can continue at [email protected]? This is getting long for comments, and would be useful there. 100m is achievable but not perhaps on "just" 3GB of RAM. I meant that as a guide for what could fit on one beefy machine. At this scale you'd stil have to sample a lot to run fast enough. One thing you should definitely do is increase the heap portion allocated to long-lived object. By default it's only like 75% of the heap, and so 25% of the heap is wasted for this app. 95% probably works. (NewRatio=19). "-d32" for a 32-bit VM can actually save memory too.Stepsister
    Thanks @Sean, I will write to the list and post here the solution once we have it.Melcher

    © 2022 - 2024 — McMap. All rights reserved.