Nice article, I enjoyed reading it. That race condition seems very nasty. I don't know anything about GPU memory orderings and schedulers, but I can't think of how it shouldn't be possible to hit it in practice. In the paper you say:
Simply delaying the check is not safe, as kernel i+1 may contain unchecked operations that are safe if and only if the preceding kernel i completed successfully.
Doesn't that constitute a dependency and therefore you should only launch two such kernels serially?
The kernels are executed in the order they are enqueued. The issue is that it would be inefficient to wait for the first kernel to finish (and read its failure state) before enqueueing the second one.
Ah, I think understand now, thanks! Would writing first to something other than global_failure solve this? Such that you could distinguish between a failure of the previous and current kernel.
Yes, I think there needs to be some kind of little dance where one thread in a block reads global_failure and then disseminates its value to the other threads, so they all have a consistent view of it.
1
u/8d8n4mbo28026ulk 1d ago
Nice article, I enjoyed reading it. That race condition seems very nasty. I don't know anything about GPU memory orderings and schedulers, but I can't think of how it shouldn't be possible to hit it in practice. In the paper you say:
Doesn't that constitute a dependency and therefore you should only launch two such kernels serially?
Cheers.