Best practice forum (Archived)

Cron backup - Fatal error: Allowed memory size exhausted in /var/www/html/totaralms/lib/adodb/adodb.inc.php on line 3008

Hi,

I received an email stating that the cron execution was automatically terminated (took longer than maximum execution time).

After looking at the cron logs, I saw the cron log stopped at this error:

VP Learning Hub

            Executing backup

Fatal error: Allowed memory size of 201326592 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/totaralms/lib/adodb/adodb.inc.php on line 3008

This backup is for the Sitewide course (front page), which in the course backup logs, doesn't pass this stage:

calculating users

There are 20,000 users synced to our site and the default role for the front page is guest.

I checked the memory limit set in the php.ini file, which is currently set as: 128mb - too low considering 201326592 bytes = 192mb...

Our site also has the Server > Performance > Extra PHP memory limit set to 512M.

Is this extra memory limit utilised for the scheduled backup function of the cron?

I also noticed that in the /admin/cron.php script, line 499 appears to have a hard coded memory limit:

@raise_memory_limit('192M');

Should the memory limit be increased? If so where is the optimal place to increase it?

-Ira

Permalink | Reply

Hi Ira,

The optimal place to raise the memory limit would be in the php.ini file on the "memory_limit = xxxM" line. This should over ride all the other raise memory limits in the code so there shouldn't be any out of memory crashes any where else in the program. Let me know if you have any problems after doing this.

Cheers
David

Permalink | Show parent | Reply

Hi,

We increased the memory_limit to 256M in the server php.ini file, but I am still receiving the same Fatal Errors:

Front page - 20,000 users

Include Course Users

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 84 bytes) in /var/www/html/totaralms/lib/dmllib.php on line 735

Course - 4600 grades x 10 quizzes

Writing gradebook info

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 71 bytes) in /var/www/html/totaralms/lib/adodb/adodb.inc.php on line 3008

Course - 4800 grades x 3 quizzes

Writing gradebook info

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 78 bytes) in /var/www/html/totaralms/lib/dmllib.php on line 735

How far can I increase the php memory limit on our server?

Should I just increase it to 512M to handle the large number of grades recorded?

Our Prod system is split, 1 Web Server with 8GB or RAM and 1 Database Server with 32GB of RAM.

Thanks,

-Ira

Permalink | Show parent | Reply

Hi Ira,

Looking at your first post you were 71 bytes short before you raised it to 256, and your still failing by the same amount. I would try bumping it up to 512, and if you are still getting the same error try something that should DEFINITLY cover it, say a couple of gig. If you are still getting that same error with the limit that high I would say something else is the problem, let me know and i'll look into it.

Hope that helps
David

Permalink | Show parent | Reply

Hey Ira,

It seems when backing up there is a zip created, and it tries to hold the entier zip in your ram. So depending on the size of your database (sounds fairly big) even a couple of gigs might not be enough. See this moodle thread here http://moodle.org/mod/forum/discuss.php?d=37006

Some highlights

Try running the cron from command line, and use an external zip instead of moodle zip
http://moodle.org/mod/forum/discuss.php?d=37006#p171547

using get_recordset_sql instead of get_recordset
http://moodle.org/mod/forum/discuss.php?d=37006#p170957

If none of that works this should help with diagnostic
http://moodle.org/mod/forum/discuss.php?d=37006#p170884

Hope upping the memory limit a bit works and this post is superfluous.

Cheers
David

Permalink | Show parent | Reply

Hi again,

Just following up my follow up. Using an external zip and running from shell/cmd is the best of the options above, the recordset change should already be implemented in totara code.
To change the zip settings go to
/admin/settings.php?section=systempaths

and follow the instructions at
http://docs.moodle.org/19/en/System_paths#Path_to_zip

Cheers
David

Permalink | Show parent | Reply

We have already set the path to use the linux system zip.

I believe it is a sizing issue as the course has so many users, grades and records contained in it...

I've requsted an increase for the memory_limit to 2GB to see if this overcomes it, then will reinvestigate.

-Ira

Permalink | Show parent | Reply

Hi,

IT guys have increased the memory limit to 1GB and one of the courses is still not backing up past the 'gradebook' stage.

I checked the course quizzes there are:

Section	Name	Attempts
2	Quiz: What is mental illness?	Attempts: 12501
3	Quiz: Depression	Attempts: 10096
4	Quiz: Bipolar Disorder	Attempts: 12439
5	Quiz: Suicide	Attempts: 9927
6	Quiz: Psychotic Disorder quiz	Attempts: 12750
7	Quiz: Communicating with Mentally Ill People	Attempts: 9273
8	Quiz: Planning a Response	Attempts: 9651
9	Quiz: Section 10 Legislation	Attempts: 13141
10	Quiz: Zoe	Attempts: 12850
11	Quiz: Excited Delirium	Attempts: 10024

There are currently 8683 users participating in the course...

Permalink | Show parent | Reply

Hey Ira,

I asked around and got 2 opinions on this.

1) Its obviously a very big site, up the memory again (not sure how high to go, maybe try 2-4 gig)

2) try a site backup instead of a course backup, should be the same as the instructions here http://docs.moodle.org/23/en/Site_backup

I'm liking option 2 myself, but it looks like a manual job that the site admin would have to do every so often. However it would let you remove the course backup from the cron.

Cheers,
David

Permalink | Show parent | Reply

Hi,

we did an audit of the course files and data - there is aproximately 5GB of moddata (mainly certificates) plus other course file and data...

would this mean that the memory limit would need to exceed this amount (>5GB) in order to backup successfully via the automation method?

-Ira

Permalink | Show parent | Reply

Hey Ira,

That is a lot more than I would have thought. Yes, the back up holds absolutely everything in memory while it creates the backup zip file. You would probably want it somewhat higher than that so you don't have to up it again when you add more content.

Cheers
David

Permalink | Show parent | Reply

Hi,

Is the backup process handled across the web server or database server?

I am worried about overloading the servers by increasing the memory limit beyond the amount of RAM available...