Kernel Samepage Merging <http://www.linux-kvm.org/page/KSM> is a feature of recent (>= 2.6.32) Linux kernels that allows processes to share pages of memory with the same content. This is accomplished by a kernel task that scans specific memory areas and compares and merges, if possible, them periodically.
Born as an enhancement for KVM (the main vanilla Linux virtualization technology), it can be used for all of those processes having lot of common data, such as uWSGI processes with their language interpreters and standard libraries.
If you are lucky, using KSM could exponentially reduce the memory usage of your uWSGI instances. Especially in massive Emperor deployments enabling KSM in each vassal may result in massive memory savings.
KSM in uWSGI was the idea of Giacomo Bagnoli of Asidev s.r.l. http://www.asidev.com/en/company.html .Many thanks to him.
To enable the KSM kernel daemon, simply set /sys/kernel/mm/ksm/run to 1, like so:
echo 1 > /sys/kernel/mm/ksm/run
Remember to do this on machine startup, as the KSM daemon does not run by default.
Note that KSM is an opt-in feature that has to be explicitly requested by processes, so just enabling KSM will not be a savior for everything on your machine.
If you have compiled uWSGI on a kernel with KSM support, you will be able to use the ksm option.
This option will instruct uWSGI to register process memory mappings to the KSM daemon after each request or master cycle.
Don’t worry – if no page mapping has changed from the last scan, no expensive syscalls are used. (Each mapping requires a madvise call.)
Checking for process mappings requires parsing the /proc/self/maps file after each request.
In some setups this may hurt performance. But don’t worry – you can tune the frequency of the uWSGI page scanner by passing an argument to the ksm option.
# Scan for process mappings every 10 requests (or 10 master cycles) ./uwsgi -s :3031 -M -p 8 -w myapp --ksm=10
The /sys/kernel/mm/ksm/pages_shared and /sys/kernel/mm/ksm/pages_sharing files contain statistics regarding KSM’s efficiency.
Higher values means lesser memory consumption for your uWSGI instances.
A simple bash script like this is useful for keeping an eye on KSM’s efficiency.
#!/bin/bash export LC_ALL=C if [ -e /sys/kernel/mm/ksm/pages_sharing ]; then pages_sharing=`cat /sys/kernel/mm/ksm/pages_sharing`; page_size=`getconf PAGESIZE`; saved=$(echo "scale=0;$pages_sharing * $page_size"|bc); echo "PUTVAL <%= cn %>/ksm/gauge-saved interval=60 N:$saved" fi
In your collectd configuration, add something like this.
LoadPlugin exec <Plugin exec> Exec "nobody" "/usr/local/bin/ksm_stats.sh" </Plugin>