Recent Driver Cuda-safe Project List

[Jason G]

Hi All,

Sticky DownClocks ?

    It's come to my attention that there are cuda enabled Boinc projects around that still cause 'sticky downclocks' on newer technology drivers, through terminating processes mid kernel execution (non-threadsafe behaviour, see tech stuff below). during snooze, exit &/or error out scenarios. 

I'll start a list here indicating what is 'safe' for Cuda 270.xx+ drivers, & what isn't:

Avoid killing Cuda or OpenCL applications via task manager unless you really need to, and expect a reboot will be needed for most newer cards (with newer drivers) if you do, even with applications that use a threadsafe strategy:

Project Name    Stock App ThreadsafeExit?          3rdParty App ThreadSafeExit?
-------------   -------------------------          ----------------------------
Collatz          No                                 N/A
Einstein@Home    No (reportedly being updated)      ?
GPUGrid          No (fall/Autumn update)            ?
PrimeGrid        No                                 ?
Seti@Home        No (version updates in progress)   Yes (Later Lunatics builds)

Last updated: 4th September, 2011

Technical Background:  Cuda & OpenCL Compute languages, and the underlying infrastructure they depend on including drivers & Hardware, have moved to heavy use of  'asynchronous execution' constructs.  The use of the Windows api TerminateProcess() function, as used in outdated BoincApi for Snooze/Exit or by killing via task manager or similar, immediately halts execution & frees the resources.  Too bad if the Graphics hardware is in the middle of writing to a memory area (asynchronously), and the buffer is freed from under it... These errors tend to force drivers into a failsafe mode only a rebbot can remedy at this time.  It's possible future drivers may be hardened somewhat to some forms of abuse, but it seems unlikely this particular application induced situation can be easily prevented, by other means than making the applications behave in a more threadsafe fashion..

Jason

Update: 18th November 2011.
How to determine if your Cuda enabled Boinc project science application might be using non-threadsafe exit code, that can induce 'sticky downlclocks' on any kind of snooze/completion/exit
- Ensure a task for the project is processing 'normally'
- Suspend all inactive tasks (so new ones won't start)
- Monitor GPU clock rate in something like GPU-Z, nVidia Inspector or similar
- repeatedly snooze then resume the running task, a threadsafe snooze shutdown should always return to full normal clocks when resumed.

Update: March 2nd 2012
Unfortunately the message about sticky downclocks caused by non-threadsafe application exit behaviour hasn't apparently been getting out enough for users.  The Cuda 4.1 Toolkit release notes text file contains  a fairly concise description of the issues at hand, suitable for developers, relating directly to later drivers & proper application termination.

* The CUDA driver creates worker threads on all platforms, and this can cause issues at process cleanup in some multithreaded applications on all supported operating systems.
On Linux, for example, if an application spawns multiple host pthreads, calls into CUDART, and then exits all user-spawned threads with pthread_exit(), the process may never terminate. Driver threads will not automatically exit once the user's threads have gone down.
The proper solution is to either:
(1) call cudaDeviceReset()* on all used devices before termination of host threads, or,
(2) trigger process termination directly (i.e, with exit()) rather than relying on the process to die after only user-spawned threads have been individually exited.

*note that the Cuda 3.2 equivalent of cudaDeviceReset() is cudaThreadExit()

[arkayn]

Collatz                 No                                      N/A

[Jason G]

Collatz                 No                                      N/A

Thanks, Updated.

[Edit:] added GPU Grid as well.

[Jason G]

Updated first post with:

Update: 18th November 2011.
How to determine if your Cuda enabled Boinc project science application might be using non-threadsafe exit code, that can induce 'sticky downlclocks' on any kind of snooze/completion/exit
- Ensure a task for the project is processing 'normally'
- Suspend all inactive tasks (so new ones won't start)
- Monitor GPU clock rate in something like GPU-Z, nVidia Inspector or similar
- repeatedly snooze then resume the running task, a threadsafe snooze shutdown should always return to full normal clocks when resumed.

[Richard Haselgrove]

I asked GPUGrid specifically about this in September.

This reply 5 October:

Probably. We have to do it a bit differently, but yes.

Then again, 7 November:

We will probably postpone this new application in 2012 to focus on upgrading the server and the new AMD alpha application.

Edit - their attitude was "our app is compiled using CUDA3.1 - so nobody needs to use CUDA 4.x drivers yet". Thus, as we've seen elsewhere, ignoring that part of the userbase who might wish to update drivers to suit other projects, or even non-BOINC uses of their GPUs.

Edit2  - to be fair, that last comment came from skgiven, who is a moderator/tester - not from GDF, who is a developer/scientist. So it's not necessarily indicative of the developers' attitudes.

[Jason G]

Oh well, no 560's (without ti) or upcoming Keplers for them I guess.  I understand the attitude, I really do.  The old one works with the flawed code design, so why break it ? My answer to that of course is because it is flawed, so drives away users.  My only real fear is they'll leave it too late.

Jason

[Richard Haselgrove]

Well, let's face it - the majority of users probably never suffer downclocking, or don't notice it if they do (the machine will boot normally next time they turn it on).

At GPUGrid, the tasks run for hours, and the scientists like them back within 24 hours (exch job is cascaded from a previous one, so they like to keep the chain moving). So I doubt many people are switching from project to project.

At PrimeGrid, the other place where the subject has come up recently, even the longer class of tasks only runs for 20 minutes on Juan's 560, so again there should never be any need for preemption.

[Jason G]

Here's what it comes down to in the medium-longer term:

The deeper I've looked into the range of problem situations, the more it seems Microsoft, nVidia and all the other members of the Khronos Consortium really got things right for the 'bigger picture' (this time), and are seriously looking to the future.  Unfortunately, IMNSHO,  the GPGPU community has subsisted for so long on sloppy design & bodgy bandaid workarounds that the addition of true heterogeneous processing capability & secure virtualised environment is going to bring many tears along the way.

[Richard Haselgrove]

...  the ... community has subsisted for so long on sloppy design & bodgy bandaid workarounds that the addition of true heterogeneous processing capability & secure virtualised environment is going to bring many tears along the way.

So unlike our own dear BOINC

[Jason G]

So unlike our own dear BOINC

  Now Now, stick to the topic please 

[Claggy]

I've done my first post over at PrimeGrid: Message 43381

Claggy

[Jason G]

I've done my first post over at PrimeGrid: Message 43381

Claggy

  Cheers, will be interesting to see if there's any reaction.  Chances that future drivers will revert to not using any DMA engine I'd say are zero, so they'll pretty much have to fix it eventually.  Hopefully it doesn't get lost in the Ati stuff there, which is a different set of circumstances leading to another whole messy scenario we won't go into in the Cuda safety thread 

[Claggy]

I've posted over at Collatz: Cuda threadsafe exit code

Claggy

[Jason G]

If anything, it's going to be an interesting study on user Vs developer goals, and inertia in technological development.  My prediction is it's going to take nVidia Kepler release to kickstart revision, whereupon neither Legacy Cuda nor OpenCL apps will be functioning nicely due to mandatory new driver models.  I'll watch it a bit then drop the observations to nVidia, just to see if they have a better strategy in mind.

Jason

[Claggy]

Anything to add to Slicker's reply Jason?

Claggy

The past is a source of knowledge, and the future is a source of hope. Love of the past implies faith in the future.

- Stephen Ambrose, in Fast Company