DataSync

The DataSync module was written to schedule and execute jobs and import data in a much more scalable and reliable way than with cron.php and hook_cron. It allows you to schedule and run module-definable types of jobs on multiple servers in a centralized way. It uses database transactions in order to ensure that no data is corrupted while running multiple jobs at the same time. DataSync keeps track of the status of jobs and what phase they are in, so that if a job fails it can intelligently decide how to proceed. Almost all of DataSync's default behavior is overridable when you create your own job types.

DataSync has run over 180,000 jobs on our production servers already.

Warning: A bug introduced in CTools 1.2 causes the DataSync scheduler ajax forms to break. Either update your CTools to the 6.x-1.3 release or apply this patch.

Some of what DataSync for Drupal 6 does:

• Runs scheduled jobs at any interval or as fast as possible
• Provides an interface for you to schedule and timeout jobs on certain intervals
• Provides an API for you to define those jobs
• Handles job errors and timeouts gracefully
• Allows you to run many jobs in parallel using database transactions
• Allows you to specify which jobs, which phases of jobs, and how many parallel processes for each phase of jobs you want to run on each server

I am looking for people who are interested in testing this module on their own setups. Please read the README.txt and INSTALL.txt and then contact me if you need help.

Originally contributed by Sony Music.

NOTE: This project is not related to the DataSync Suite produced by DataSync Corp, which consolidates all the software applications you need for your business into one platform, accessible via your web browser.