Best practice forum (Archived) - Cron Script Freezing on production server

Skip to main content

Skip Create an account

Best practice forum (Archived)

Cron Script Freezing on production server

Hi,

I Checked the cron script as the automated backups have not been completed. it seems as though the cron script is freezing at this point:

Server Time: Tue, 10 Jan 2012 14:11:21 +1100

Starting activity modules

Processing module function assignment_cron ...done.

Processing module function facetoface_cron ...

No reminders need to be sent.

done.

Processing module function forum_cron ...Starting digest processing...

Cleaned old digest records

done.

Finished activity modules

Starting blocks

Processing cron function for helpdesk....done.

Processing cron function for search....Global searching is not enabled. Nothing performed by search.

done.

Processing cron function for totara_stats....done.

Finished blocks

Starting admin reports

Finished admin reports

Updating languages cache

Removing expired enrolments ...none found

Starting main gradebook job ...

done.

Starting processing the event queue...

done.

Starting the completion cron...

Running completion_criteria_date->cron()

Running completion_criteria_activity->cron()

Running completion_criteria_duration->cron()

Running completion_criteria_grade->cron()

Running completion_criteria_course->cron()

Aggregating completions

done

Starting the report builder cron...

Running 'feedback_questions' pre-processor on group 'finance feedback' (1 items).

Processing 1 responses for feedback 3

Ignoring 1 unchanged responses.

Processing 3 scheduled reports

Saved search not found or search is not public

Permalink | Reply

Hi Ira,

I can confirm this is a bug - I think that this is being caused by a scheduled report which is using a 'saved search' that no longer exists or is not accessible.

Obviously it shouldn't stop the whole cron from running and we'll get that fixed. I'll see if I can come up with a workaround for you in the meantime.

Simon

Permalink | Show parent | Reply

Hi,

should I just remove the scheduled report? or the saved search?

-Ira

Permalink | Show parent | Reply

The reason for the bug was that there was a scheduled report based on a saved search that had already been deleted, so deleting the scheduled report (or even just editing it and setting the data field to 'All data') should fix it.

However, we managed to squeeze a patch into today's release (1.1.8) which prevents this issue from crashing the cron - we'll get a proper fix (which will prompt and delete scheduled reports when the saved search is deleted) in the next one.

Simon

Permalink | Show parent | Reply

Hi,

I have not been able to upgrade our live production server yet(which is still v1.1.3), under conrtrol of IT services at the moment.

The cron watcher settings are saying cron execution status: crashed.

How do i fix the cron script as it has crashed?

Does this mean it is no longer running? When i try to run it manually, nothing happens and i get a blank screen...

-Ira

Permalink | Show parent | Reply

The content of this forum post has been removed and can no longer be accessed.

Permalink | Show parent | Reply

Hi,

should the cron be set to terminate automatically after exceeding the time limit?

Also I have the debuggins setting currently on " Developer" but I have not seen any errors displayed on the site, including in the cron script screen...

Will the Totara error logs list anything valuable about cron script failure or should i be chekcing the Linuz server logs?

thanks,

-Ira

Permalink | Show parent | Reply

The content of this forum post has been removed and can no longer be accessed.

Permalink | Show parent | Reply

Hi Ira,

Have you had any luck?

A couple of other things to check:

Is the cron watcher script being run via your servers cron (this is necessary for a crashed cron to get cleared). It needs to have a separte crontab entry to the main cron (calling admin/cron_watcher.php instead of admin/cron.php).

Is the cron watcher script running as root? This is necessary or it may not be able to clear a crashed cron (it may not have the correct permissions).

Simon

Permalink | Show parent | Reply

Hi,

I don't have full access to the live production server, so I'm waiting for the IT guys to get back to me with the answers to our questions.

Although as part of the install I instructed that:

cron watcher script is to be run via web servers cron (crontab)
cron watcher has a seperate crontab entry to the main cron script

I'm pretty certain that is how it is set up at the moment, just not sure if it is running as root.

-Ira

Permalink | Show parent | Reply

The totara cron and cron_watcher tasks are running every 5 mins as root.

Jan 16 12:05:01: (root) CMD (wget -O /dev/null --no-check-certificate https://vplearninghub.vicpolice.internal/admin/cron_watcher.php >/dev/null 2>&1)

Jan 16 12:05:01: (root) CMD (wget -O /dev/null --no-check-certificate https://vplearninghub.vicpolice.internal/admin/cron.php >/dev/null 2>&1)

Jan 16 12:10:01: (root) CMD (wget -O /dev/null --no-check-certificate https://vplearninghub.vicpolice.internal/admin/cron.php >/dev/null 2>&1)

Jan 16 12:10:01: (root) CMD (wget -O /dev/null --no-check-certificate https://vplearninghub.vicpolice.internal/admin/cron_watcher.php >/dev/null 2>&1)

The cron.pid file doesn't seem to be removed once the cron script crashes...

Even though I have removed all user scheduled reports, I created a new report which uses a new feedback activity group. After doing this, it seems to have crashed the cron again sad

Is this related to the same bug?

-Ira

Permalink | Show parent | Reply

Hi Ira,

What is actually happening here is that the cron_watcher is being called through wget which will run it via HTTP as a web page with the apache user's permissions, so it isn't actually running as root and therefore cannot clean up the pid file. The crontab entry for the cron_watcher job should run the script directly e.g.

*/5 * * * * php [path to totara installation]/admin/cron_watcher.php > /dev/null 2>&1

The main cron job should run fine using wget, it's only cron_watcher which needs higher-level permissions.

cron_watcher emails also go to the first admin user created on the site, which is usually the 'admin' user with user id 2, so you should check the email address assigned to that user profile if you want to receive cron_watcher emails.

Permalink | Show parent | Reply

Hi,

Thanks for that info, i'll try and get the crontab updated on our prod server.

Does v1.1.9 of Totara fix this issue without changing the crontab settings?

Just wondering if the cron_watcher and cron script should be running on the same time interval (every 5 minutes) or be different?

-Ira

Permalink | Show parent | Reply

Hi Ira,

No upgrading to 1.1.9 won't help, the watcher script always needs to be run this way, or it won't have permissions to remove the pid file.

In the install file we suggest running the cron watcher a bit more often - every 5 minutes rather than every 15 minutes for the cron, but it should be fine if you run them both every 5 minutes.

Simon

Permalink | Show parent | Reply

Hi,

where is the "cron.pid" usuall located in the moodle-data folder? I tried to locate it and only found /var/run/crond.pid - is this the same file?

thanks,

-Ira

Permalink | Show parent | Reply

The content of this forum post has been removed and can no longer be accessed.

Permalink | Show parent | Reply

The content of this forum post has been removed and can no longer be accessed.

Permalink | Show parent | Reply

I'm pleased to report a fix for this issue was included in the 1.1.9 release yesterday (26th Jan 2012), if anyone else is having the same problem.

Permalink | Show parent | Reply