sunspot_rails unable to index due to "404 Not Found" error

Question

I'm trying to install Sunspot in a small Rails app, exactly following the gem setup instructions, but every time I run into RSolr::Error::Http: RSolr::Error::Http - 404 Not Found errors when I try to index data. I can reproduce this with a fresh app; here are the exact steps I follow:

Create a fresh Rails 4.2.5 app:

$ rails new test_sunspot
$ cd test_sunspot/
$ spring stop # spring can cause `generate` commands to hang
$ rails g model Thing title:string
$ rake db:migrate
$ rails c
  > Thing.create!(title: "Cats")
  > Thing.create!(title: "Pizza")
  > exit

Add a Sunspot index to the model:

class Thing < ActiveRecord::Base
  searchable do
    text :title
  end
end

Add Sunspot to Gemfile:

...
gem 'sunspot_rails', '2.2.2'
gem 'sunspot_solr',  '2.2.2'  
...

Install, start, and reindex Sunspot:

$ bundle install
$ rails g sunspot_rails:install # default sunspot.yml is not changed
$ ps aux | grep solr # confirm that no Solr services are running
$ bundle exec rake sunspot:solr:start # generates solr/ dir; no errors
$ bundle exec rake sunspot:solr:reindex

This reindex command yields the following output. When I go into the Rails console and tried to create a new Thing object, it triggers the same error (because Sunspot attempts to update the index):

Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile
rake aborted!
RSolr::Error::Http: RSolr::Error::Http - 404 Not Found
Error:     Not Found

URI: http://localhost:8982/solr/development/update?wt=ruby
Request Headers: {"Content-Type"=>"text/xml"}
Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete><query>type:Thing</query></delete>"

Backtrace: /Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:284:in `adapt_response'
/Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:190:in `execute'
/Users/topher/.rvm/gems/ruby-2.2.0/gems/rsolr-1.0.13/lib/rsolr/client.rb:176:in `send_and_receive'
# ...lots of backtrace omitted...
/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_rails-2.2.2/lib/sunspot/rails/tasks.rb:19:in `block (2 levels) in <top (required)>'
/Users/topher/.rvm/gems/ruby-2.2.0/bin/ruby_executable_hooks:15:in `eval'
/Users/topher/.rvm/gems/ruby-2.2.0/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => sunspot:solr:reindex => sunspot:reindex
(See full trace by running task with --trace)

The output of ps aux | grep solr (after starting Solr): Note that the PID mentioned in solr/pids/development/sunspot-solr-development.pid is 62449, which matches the third line:

topher          62617   0.0  0.0  2432772    520 s002  R+    3:00PM   0:00.00 grep solr
topher          62484   0.0  1.3  3274624 105756   ??  S     2:57PM   0:03.65 /usr/bin/java -server -Xss256k -Xms512m -Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+ParallelRefProcEnabled -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_solr-2.2.2/solr/server/logs/solr_gc.log -DSTOP.PORT=7982 -DSTOP.KEY=solrrocks -Djetty.port=8982 -Dsolr.solr.home=/Users/topher/Sites/john_kole/test_sunspot/solr -Dsolr.install.dir=/Users/topher/.rvm/gems/ruby-2.2.0/gems/sunspot_solr-2.2.2/solr -Duser.timezone=UTC -Djava.net.preferIPv4Stack=true -jar start.jar
topher          62449   0.0  0.0  2444632   1304   ??  Ss    2:57PM   0:00.04 bash ./solr start -f -p 8982 -s /Users/topher/Sites/john_kole/test_sunspot/solr

Other details:

• I'm on Mac OSX Yosemite
• I've uninstalled and reinstalled the relevant gems, tried downgrading to Sunspot 1.3.0, and even ran gem pristine --all, with no change in outcome
• EDIT: I have read through the other similar tickets on Sunspot/Solr 404 Not Found errors. It looks like the solutions in those cases amounted to "resetting" the Solr config and aren't relevant to a fresh (dev environment) project: no prior Solr instances are running before I run rake sunspot:solr:start; this is a completely fresh Rails project so the solr/ directory was just newly generated anyway; this is in development, not production; and adding solr_home: solr to sunspot.yml or updating path: /solr/default have no effect on the 404 outcome.

Questions:

• Any idea why this is happening?
• Could this breakage be due to something mis-installed in my OSX environment? What should I look at?
• Can you get Sunspot working properly on a fresh install of current Rails, following these same steps?

Thanks in advance!

Danzig

spring stop sometimes does not kill every spring process. maybe run a ps aux | grep -i spring to find out if something is still around – Fusil

Thanks, I can confirm that ps aux | grep -i spring doesn't show any running processes though. – Danzig

Answer 1

Remove the sunspot_solr gem and the install solr on your machine by the following commands

brew install solr

if it doesn't work run brew update first to fetch the latest solr links

brew will install all missing dependencies automatically.

To start solr:

solr start

Appropriation

This would allow me to move past the problem assuming that I can wire up Sunspot to the global Solr install and assuming I don't have to worry about conflicts with other Solr-enabled projects, but it gets around the central question: why isn't Sunspot-Solr working, and what could I do differently to get it working? Any insights on that would be super appreciated. At any rate I'm currently exploring a Vagrant solution that does roughly what you recommend here. – Danzig

So there are many things that might cause this problem. Solr requires java to run so if you don't have java configured correctly on your local system, the server might not work. I passed exactly through the steps in your question and it worked and I was able to reindex without any errors. However I always prefer to install the server on the system to simulate how the server acts on production env. – Appropriation

Understood, it definitely sounds like there's something wonky with my environment. I'll go the Vagrant route for now. – Danzig

The server works fine. It throws a 404, as it lacks some permissions. I managed to reproduce all error messages to 100% and provide one way to solve the issue. Another could be to fix the permissions on the server location (gem) in the filesystem via groups. – Fusil

Answer 2

TL;DR: You have done everything right, but the user installing the gems is not the same as the one running solr (aka permissions issue)

I tried to reproduce your case and succeeded, which felt very wrong. The issue is that installing the gem sunspot_solr ships an embedded Java webserver Jetty. I managed to trace the cause of the problem to the fact that this webserver is not run as the person who installed it.

This can be verified by installing gems per:

$ touch Gemfile # to mark it as edited
$ bundle install --path vendor/bundle # to install as user who runs the server later

instead of the usual

$ gem install ... bundle install

as another (possibly root) user.
Now you can succeed performing all your steps as mentioned in your post.

Fusil

So frustrating... before posting this on SO I had replicated these steps around 10 times to make sure I couldn't wiggle my way around the problem. Now ten days later, I follow the exact same steps and the errors don't occur as they did before (noticing other issues, though). So it's possible that something in my environment shifted in the past week and "solved" this problem (in the meantime, we've switched to using ElasticSearch instead); at any rate I can't verify that your advice helps, but it's the closest thing I have to an answer so here's the bounty. Thanks! – Danzig

TopherHunt, I am sorry for your persisting troubles. Have you switched to ES due to these issues? In case you still need to get solr working, @RenegadeAndy's 4 posts on it could help. It looks like he succeeded to set up solr as standalone + sunspot in production. – Fusil

Thanks @benjamin, I suspect I could get around this issue by now but yes we've moved to ElasticSearch (loss of confidence in the tool and all that). elasticsearch-rails' documentation is horribly disorganized by comparison to Sunspot, but so far it's Just Worked so we'll stick with that for now. – Danzig