Those threads I'm spinning up are in charge of sending information to other machines without making the result wait.
Yikes. I hope not. That's what IOCP are for. In .NET, use the built-in async methods and you'll get that without much need to think about it, but you should never be blocking a thread waiting for a network or file I/O result. That's just crazy.
You definitely don't need a thread to buffer and batch file writing.
I don't understand how this comment is responsive to the parent comment at all. Node's async model may have problems, but batch file writing is certainly not one of them. The parent comment is making the clear suggestion that you don't need to do 1000 individual inserts, so I don't understand what you're suggesting.
Node isn't capable of doing something similar. You can somewhat simulate it with a timer, but since it is single-threaded you [cores] instance of Node, each with its own log separate queue and timer.
Wait, what? You have a set of queues and a set of timers, which are checked every threshold interval (like every second or so). You store your log entries in queues and write them with async batch writes when they pass a size/time threshold.
This is even a more efficient solution because I'm not forced to use multiple threads to do this. What is this solution missing for you?
If you use cluster to create processes they can all communicate with the parent process via message passing. The parent process now maintains 1 queue and 1 timer and is solely responsible for batch writes.
5
u/ruinercollector Apr 27 '16
Yikes. I hope not. That's what IOCP are for. In .NET, use the built-in async methods and you'll get that without much need to think about it, but you should never be blocking a thread waiting for a network or file I/O result. That's just crazy.
You definitely don't need a thread to buffer and batch file writing.