Any ideas why Thinking Sphinx Rake tasks are not running?

Question

I'm finding that Thinking Sphinx sometimes errors out when I try to run its Rake tasks. Sometimes the tasks work fine, and sometimes I get errors like the one below.

I'm running the tasks as a normal user, not root. Not using sudo.

In the example below, searchd is still running after the Rake task fails.

I've also seen similar errors with ts:rebuild.

Has anyone else seen this?

I'm using Sphinx 0.9.9-release (r2117)

I'm using Thinking Sphinx 1.4.4 instead of the latest version because I'm on Rails 2.3.11.

---

$ rake ts:stop RAILS_ENV=production
(in /var/www/blog_app/releases/20110425053509)
rake aborted!
execution expired

(See full trace by running task with --trace)
[ethan@apphost current]$ rake ts:stop RAILS_ENV=production --trace
(in /var/www/blog_app/releases/20110425053509)
** Invoke ts:stop (first_time)
** Invoke thinking_sphinx:stop (first_time)
** Invoke thinking_sphinx:app_env (first_time)
** Execute thinking_sphinx:app_env
** Invoke environment (first_time)
** Execute environment
** Execute thinking_sphinx:stop
rake aborted!
execution expired
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/1.8/timeout.rb:60
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/thinking-sphinx-1.4.4/lib/thinking_sphinx/tasks.rb:58
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `execute'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:597:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:607:in `invoke_prerequisites'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `each'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:604:in `invoke_prerequisites'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:596:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/1.8/monitor.rb:242:in `synchronize'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:590:in `invoke_with_call_chain'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:583:in `invoke'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2051:in `invoke_task'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `each'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2029:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2023:in `top_level'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2001:in `run'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:2068:in `standard_exception_handling'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:1998:in `run'
/opt/ruby-enterprise-1.8.7-2010.01/lib/ruby/gems/1.8/gems/rake-0.8.7/bin/rake:31
/opt/ruby/bin/rake:19:in `load'
/opt/ruby/bin/rake:19

Hamza

Were you running rake ts:stop from a capistrano deploy script? – Marathi

Answer 1

I believe this error is caused when you have a search daemon already running.

Try

ps | grep searchd

if that gives you something like this...

73470 ttys002    0:00.03 searchd --pidfile --config /Users/dwickwire/Projects/production/config/development.sphinx.conf
81907 ttys002    0:00.00 grep searchd

then kill the searchd

sudo kill -9 process_id

process_id in my case would be 73470

Then try rake ts:rebuild again.

Groundsill

I would like to add, the '9' should only be included after a attempt without it has not been successful. Thats just a good rule generally speaking. – Groundsill

This is not a solution. The purpose of rake ts:stop is to stop searchd. Saying that "you should manually kill searchd before running ts:stop" doesn't make sense. – Siesta

Alex, rake ts:stop was already tried in this example, the issue here can be caused when two daemons are running and the stop doesn't know which one to stop. – Groundsill

OK, good point. But in the example you show, there is only one searchd daemon running. I am having the same problem as Ethan, but in my case, there is also only one searchd daemon running, and ts:stop can't stop it. Of course, I could keep manually killing it rather than using ts:stop, but that's not solving the actual problem. – Siesta

I usually do in the following two steps: sudo killall -9 search # will kill sphinx RAILS_ENV=production rake ts:rebuild – Amaryllis

I usually only pull out my nine when the baddies are around. – Groundsill

Answer 2

I tried command 'ps | grep searchd' to get process ID, but I didn't get it. Then I tried command 'pgrep searchd', it gives me process ID. I killed that process and ran command 'rake ts:rebuild'. It worked.

I blogged about the entire experience here (disclaimer: I'm the author).

Priorate

Answer 3

In thinking_sphinx 1.4.4 there is a 5 second timeout on rake ts:stop

If rake ts:stop takes close to or longer than 5 seconds, ruby will throw an exception.

I'm using the thinking_sphinx 1.3.18 in the meantime. (there is no timeout on 1.3.18).

Also I have submitted a patch to the github repository and the timeout will be configurable with a sphinx.yml configuration variable stop_timeout.

NOTE: rake ts:rebuild == rake ts:stop; rake ts:index; rake ts:start

Marathi

stop_timeout variable is not working, as i tried setting it as 1 sec and rake not aborted. – Dhammapada

@mirza yes that setting appears to not work anymore. – Marathi

Any other solution , tried timeout: 1 too which did not work, I am stuck. I am using thinking-sphinx 2.1 – Dhammapada

Answer 4

I guess after deploy by capistrano you didn't run rake thinking_sphinx:configure
it would generate sphinx config file in config/production.sphinx.conf

try add pidfile to config/sphinx.yml

production:
  ...
  pid_file: "/tmp/searchd.pid"

it will make sure to find pid of searchd to kill

and added following code to config/deploy.rb

namespace :my_tasks do
  task :generate_sphinx_config, :roles => [:web] do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} bundle exec rake thinking_sphinx:configure"
  end
end

....

after :deploy, "my_tasks:generate_sphinx_config"

to generate sphinx config file in each release folder

Eames

Answer 5

The Problem

This problem occurs if your searchd crashed or you killed it manually. On Windows it doesn't overwrite your searchd.<environment>.pid, so still the old PID is saved and ThinkingSphinx doesn't check the real running processes in your taskmanager. So all these commands (rake ts:restart, rake ts:rebuild, ...) won't work, rake will always abort and you get the error rake aborted, ... searchd already running

The solution

Delete your path-to/your-app/log/searchd.<environment>.pid.

<environment> may be development or production, depending on your current stage.

Hope this answer will help the googler's.

Oppidan