How do I run all rake tasks?

Question

Have just installed whenever gem https://github.com/javan/whenever to run my rake tasks, which are nokogiri / feedzilla dependent scraping tasks.

eg my tasks are called grab_bbc, grab_guardian etc

My question - as I update my site, I keep add more tasks to scheduler.rake.

What should I write in my config/schedule.rb to make all rake tasks run, no matter what they are called?

Would something like this work?

    every 12.hours do
        rake:task.each do |task|
            runner task
        end 
    end

Am new to Cron, using RoR 4.

Anamorphic

Answer 1

namespace :sc do
  desc 'All'
  task all: [:create_categories, :create_subcategories]

  desc 'Create categories'
  task create_categories: :environment do
    # your code
  end

  desc 'Create subcategories'
  task create_subcategories: :environment do
    # your code
  end
end

in console write $ rake sc:all

Lonely

This is the correct answer - both on topic, and succinct – Danita

Answer 2

Make sure you have a unique namespace with all the tasks in it, like:

namespace :scrapers do

  desc "Scraper Number 1" 
  task :scrape_me do
    # Your code here
  end

  desc "Scraper Number 2"
  task :scrape_it do
    # Your code here
  end

end

You could then run all tasks of that namespace with a task outside of that namespace:

task :run_all_scrapers do
  Rake.application.tasks.each do |task|
    task.invoke if task.name.starts_with?("scrapers:")
  end
end

That said, I'm pretty sure that this is not how you should run a set of scrapers. If for any reason the if part should return true you might unintenionally run tasks like rake db:drop

Either "manually" maintaining schedule.rb or a master task seems like a better option to me.

Hagar

what is the reason for a namespace? How does it help? – Anamorphic

It is useful to structure tasks into logical groups. rake db:migrate runs the migrate task from the db namespace. In the above example this helps to easily identify which rake tasks to to run. – Hagar

You can also use something like Rake.application.in_namespace(:scrapers){|namespace| namespace.tasks.each(&:invoke)}. See https://mcmap.net/q/234768/-default-task-for-namespace-in-rake – Cannabis

Answer 3

write separate rake tasks for each scraping tasks. then write a aggregated task to run all those scraping rake tasks.

desc "scrape nytimes"
task :scrape_nytimes do
  # scraping method
end

desc "scrape guardian"
task :scrape_guardian do
  # scraping method
end

desc "perform all scraping"
task :scrape do
  Rake::Task[:scrape_nytimes].execute 
  Rake::Task[:scrape_guardian].execute 
end

then call the rake task as

rake scrape

Misbehavior

Thanks Khan. How do I run those tasks automatically every X hours, or every time I view the development site, so that my development site updates without my writing "rake scrape" manually in Terminal? – Anamorphic

@Jonathan_W, you can simply add a cron job to schedule this rake task to run every x hour hour. if you want to scrape when loading dev site then my suggestion is to write the scraping task as a seperate function and call this function from your web apps controller. also call the same function from rake task. – Misbehavior

Answer 4

The aggregated task can be concise:

namespace :scrape do
  desc "scrape nytimes"
  task :nytimes do
    # scraping method
  end

  desc "scrape guardian"
  task :guardian do
    # scraping method
  end
end

desc "perform all scraping"
task scrape: ['scrape:nytimes', 'scrape:guardian']

Namespaces are also a good practice.

Lien

Answer 5

Use namespace and in_namespace to run all tasks dynamically.

I prefer this method because it keeps things clean and precludes you from having to remember to update your "parent" task if any of our namespace tasks change.

Note, the example was borrowed from Dmitry Shvetsov's excellent answer.

namespace :scrape do
  desc "scrape nytimes"
  task :nytimes do
    # scraping method
  end

  desc "scrape guardian"
  task :guardian do
    # scraping method
  end
end

desc "perform all scraping"
task :scrape do
  Rake.application.in_namespace( :scrape ){ |namespace| namespace.tasks.each( &:invoke ) }
end

Cannabis