So I've had a slice with slicehost for about a year and a half now. I love all the things I can do with my very own relatively cheap server on the Internet. I started out with a 256 slice, and then I talked my wife into upgrading to a 512. I kept putting more things on my server because it was so convenient, and fun to have a 100% static IP on the internet. I've started running my own mail server, and slowly I've been moving all my personal sites to Django. I even host a website on my slice for a small fee.
After a little while, I was curious as to why all my Django apps seemed to run so slowly. I don't use my own website very often, so I didn't notice the slowness creeping in. Running top revealed that I was using all of the ram on my 512 slice, and I had about half a gig in swap. I won't be able to talk my wife into moving to yet another price double on my slice, so I decided to try to combat this problem with smarts rather than money.
Django apps seemed to be the most telling when it came to slowness. Sometimes imap is slow, but it's really only a problem if it's in a folder that hasn't been cleaned out and is getting rather large. I've either taken down or replaced anything that isn't Django. One thing that kills my memory usage is the fact that I'm running satchmo, which usually has at least a 180MB footprint according to the satchmo developers.
My first instinct was to turn caching on. I have several low traffic sites that I never bothered to configure caching for. I already had memcached running because I had enabled caching for one Django site I run. I added caching to the other sites, and all was well. There was some crazy memcached error for this site, so I turned on locmem caching. The problem with locmem is that the cache is only available to one thread, so each thread caches. That isn't ideal for combating slowness when the underlying cause is a memory problem.
The next thing on my list to try to reduce memory usage is to try fastcgi instead of mod_python. I'm also willing to try out mod_wsgi. I'm also considering switching from apache to lighttpd.
...so this has been a draft for a while. I did switch to mod_wsgi in daemon mode. That seemed to help. Ultimately, I ended up writing a cron job to restart apache periodically. This helps with the apparent Django slowness, but became necessary as Apache was doing other strange things. Unfortunately I don't have as much time as I'd like to be the System Administrator for my own server. Many of these performance problems are a non-issue any time I've deployed a Django app on a work-related production server, because those servers are usually beefy enough. I will continue experimenting from time to time with my Django setup, and post anything I find.
Are you running with DEBUG set to True? That would explain the results that you're seeing.
Don't know if you tried this, but I've found that tweaking prefork MPM values in apache2.conf solved similar issues for me. I recommend you to take a look at http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory-usage-down
I also run several small trafic websites, and these values seem to work quite well for me:
prefork MPM
StartServers: number of server processes to start
MinSpareServers: minimum number of server processes which are kept spare
MaxSpareServers: maximum number of server processes which are kept spare
MaxClients: maximum number of server processes allowed to start
MaxRequestsPerChild: maximum number of requests a server process serves
They won't work for every case but it's a matter of testing what works best for you.
I user to run a commercial site, a CMS application with thousands of active users, and was running into memory issues all the time so consequently got fairly adept at tuning Apache.
The biggest performance tip that affects memory that I turned to in the end was to deinstall Apache in favour of something far more lightweight: lighttpd.
One one box of mine I have traffic that might look more like yours - a number of small Python web apps. I've one lighttpd instance occupying 23470k and then a number of python application children (a number I can control). The web framework I use, QP, allows me the ability to control how many children are forked; they do not die at some arbitrary number of requests but are replaced if they happen to die.
Myself I use SCGI to communicate with the python apps; one can also simply use lighttpd as a proxy if your primary need is merely to push out your apps to different domains as port 80.
For those who do not need Apache's breadth for anything else, lighttpd may be an option if all else fails.
I'm currently moving my small personal web apps off the server which I do commercial work on - over to a VPS similar to SliceHost. Initially I picked up a 2GB RAM "slice" for $34 but I think I'll be able to reduce that greatly. In the process I'm being forced to adopt a new OS (debian... all my other gear runs FreeBSD) so I wanted some flexibility until I finish sorting things out.
QP on this slice churns out over 6000 requests per second running Python 2.6 for the prototypical "hello world" app. My other machines are envious!
You can give a try to fapws3 - wsgi web server with binding to fast C libev library.
Or create web proxy with nginx or lighthttpd.
I created a 256MB slicehost account and used apache for everything. Pretty soon I was running out of memory. I switched to nginx and all my memory problems were solved. I use fast cgi for drupal/php sites and django apps and it's great.
I actually still have apache running for an http svn server, but it doesn't seem to take much memory all, maybe because the svn server isn't being used much at all.
debug=True in django was also something that was slowing me down at first.
I set up nginx using the tutorials on slicehost. I didn't compile nginx from source though, I just used the one that came with Ubuntu Intrepid.
BTW. there is something wrong with your layout. The "Required", "Kept Private", and "Used for Gravatar" are shown on one line, instead of where I think they should be showing up...
No, I'm not running with DEBUG on, but that's the first thing I checked for.
I did at one point change the Prefork MPM values in my
httpd.conf, and it did help, but it wasn't quite enough.I've definitely been considering switching to lighttpd, but I haven't made the time to make the switch. Also, I've just made myself so accustomed to apache since I started web programming when I was a kid that it almost hurts a little inside to switch away from it. I just need to pick a day off, and make the switch.
I'll definitely look into the fapws3 server. I haven't really considered switching to nginx because I believe that simply reconfiguring apache or switching to lighttpd should do the trick. I've heard more and more good things about nginx, so I should probably look into it.
As far as the placement for the "Required. Kept Private. Used for Gravatar" line, it is referring to the Email field. I haven't finished the CSS for this site by any measure, and I plan to make the placement of things more logical.
Thanks for all the suggestions and comments!
Try serving pages that don't change as static files, thus reducing the number of django instance you start in the first place.
This is an intresting django app. to do that.
http://superjared.com/projects/static-generator/
Jeff: I'm an idiot... Something about the periods in that made me think that one sentence was meant for each field in the form...
Definitely check out nginx. I can't remember exactly why I went with it over lighttpd but I did research it at the time.
Another issue you're running in to is that Slicehost is 64 bit only.
64bit = more RAM usage. You could try switching to a different VPS solution that allows 32 bit setups.
I run a fairly active Django site on a Linode 360 with no problems whatsoever. I run nginx as a reverse proxy to apache2 running mod_wsgi (nginx also serves any static files). Even running postfix and PostgreSQL, I'm very rarely over 100MB.
~JW
Using 32-bit programs is possible in a 64-bit kernel. Maybe if I compile Python for x86 instead of x86-64 I could get the same result.
It seems like my memory problem is mostly present because of all the Python processes that spawn as various URLs are accessed during the lifetime of the apache instance.
I believe that Python byte code itself is architecture independent. I could be wrong, but I was able to run a pyc created on a 64-bit machine on a 32-bit machine and vice versa. The file sizes of each
.pycwere identical for the same Python source file.The only other thing that might help is compiling apache itself as a 32-bit executable. Apache by itself isn't that big. I think that I could definitely get a respectable memory footprint just by changing the configuration. If I want to cut down the memory footprint of the webserver, I can use lighttpd or nginx, which is easier than recompiling or switching VPS hosts.
Besides, 64-bit is just sweet.
How would turning on caching reduce memory usage? Wouldn't you then be storing more in memory? Maybe you should try no caching at all and see how the memory usage is. Or, if you need cache, use memcache at a fixed size.
Also check to see if python is compiled as a shared library, or statically. If it's static, then each apache child has a fully python interpreter in memory. If it's shared, then I believe all child processes use the same single instance. I'm no expert on this, but you should be able to ask google about compiling python as a shared object.
As mentioned, there are a ton of Apache parameters you can tune. Once of the first things to do though is make sure Apache isn't loading (or has compiled in) any modules you know you don't need. This can really help with memory usage, since any memory used us multiplied by the number of child processes. Of course, the basic Django tuning is also recommended: turn off keep-alive and use lightweight processes for static files.
Messing with the prefork settings is definitely the way to go (as already mentioned) because you can set them in a way that effectively does what you're doing: restarting apache every while.
Just make sure your MaxClients and MaxRequestsPerChild are low. Those apache processes will degrade quickly.
This is a great article on the subject http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html
Check if your python executable is dynamicly linked to libpython (Ubuntu and Debian bin is statically linking, Fedora is dynamic). Might help.
Heavens almighty, by all that's good and slightly crunchy, turn off KeepAlives in Apache if you haven't done so. KeepAlives will KILL your performance, I promise you.
Other than that, you almost certainly want to have to types of contents serving configuration.
Static contents (pictures, CSS, JS...) should be served with the most minimal Web server you can manage. An Apache with as few modules loaded as possible is an option! Otherwise a lightweight Web server (Lighttpd, Nginx) is a good idea. KeepAlives are no problem for serving this kind of content, they even help.
Dynamic contents -- your Django instances -- can be managed with an external application, with a number of concurrent instances left independent from the number of Apache instances. FastCGI and mod_wsgi in aemon mode work well for this purpose. If you want to use Apache with mod_wsgi in embedded mode called via a reverse proxy, for instance, limit the number of Apache children that can run simultaneously. But whatever you do, no KeepAlives!
In my experience, for a small site, very few instances of Django will suffice, because its threaded FastCGI runner works very well. I've pounded a FastCGI installation of Django that had a max of two simultaneous instances with 50 threads each with 150 simultaneous connections, under a virtualhost with 256MB of RAM, and the memory didn't even come close to exhaustion.
Hope this helps!
-- S.