After a bit of trail and error the solutions was found. There seems to be a bug in the Linux Kernel 3.2.x. Now after upgrading to Kernel 3.3.x the problem has disappeared! Below are the steps to update the kernel in Ubuntu 12.04.
Please note I do not take any responsibility if this breaks things. This worked for me and I am merely trying to help others.
At the time of writing this article the latest stable kernel is 3.3.6
The kernel can be found at kernel.ubuntu.com or you can just ignore this and copy and past the commands below to the terminal depending on you installation
For Ubuntu (i386 / 32-bit) run these commands
cd /tmp && wget -O linux-headers-3.3.6-030300_3.3.6_all.deb http://goo.gl/zNlMy
sudo dpkg -i linux-headers-3.3.6-030300_3.3.6_all.deb
cd /tmp && wget -O linux-headers-3.3.6-generic_i386.deb http://goo.gl/TdBex
sudo dpkg -i linux-headers-3.3.6-generic_i386.deb
cd /tmp && wget -O linux-image-3.3.6-generic_i386.deb http://goo.gl/osZhw
sudo dpkg -i linux-image-3.3.6-generic_i386.deb
For Ubuntu (amd64 / 64-bit) run these commands
cd /tmp && wget -O linux-headers-3.3.6-030300_3.3.6_all.deb http://goo.gl/zNlMy
sudo dpkg -i linux-headers-3.3.6-030300_3.3.6_all.deb
cd /tmp && wget -O linux-headers-3.3.6-generic_amd64.deb http://goo.gl/Z9Ztt
sudo dpkg -i linux-headers-3.3.6-generic_amd64.deb
cd /tmp && wget -O linux-image-3.3.6-generic_amd64.deb http://goo.gl/jji3o
sudo dpkg -i linux-image-3.3.6-generic_amd64.deb
Just reboot your system and done.
I you have any questions/problems please leave a comment and I will try to help