I have written a program in Cuda which creates several streams with kernels in it's runtime. The problem is, after a certain time the whole System freezes. I'm even not able to change into the terminal.
What I've done so far:
At first I thought everything crashs because I was using the same GPU for computation as for the display-output. I reinstalled Ubuntu and made sure that the Nvidia driver(375) was not affecting the Xserver configuarations anymore.
From this moment on I am using the iGPU for display-output. Unfortunately nothing changed. Same proplems.
For making sure the GPU is used for computation only In booted in textmode and run the program from the console. In the console the program terminates with the following output:
Out of memory: Kill process (...) score 938 or sacrifice child
Killed process (...) total-vm:28861512kB, anon-rss:7495780Kb
The error code above leaded me to the Cuda debugging tool Memcheck. After using this for my Software I was indeed able to fix some errors in the kernel code. And after this the problem has changed a little bit. Just a little bit. If I run the software now the System starts to hang extrem but I am able to switch into the terminal. If I run this in textmode now it terminates with the same error. What surprises me a bit is the fact that nvidia-smi shows a GPU Memory usage of 86Mb of 4Gb until the whole system freezes.
Ubuntu 14.04 64bit
CPU: Intel i5-4440 | Intel Hashwell for display output
GPU: Nvidia GTX980 EVGA 4Gb
Cuda Version: 7.5
I hope somebody can help me, I ran out of ideas.