r/DistributedComputing • • May 21 '26

How do you handle live data sync delays when filtering heavy sports/betting menus?

6 Upvotes

Hey everyone,

I'm currently working on a sports betting (Toto) platform and ran into a frustrating issue. Whenever users try to compress or filter the match menus, we notice temporary drops or lags in the live odds data.

After digging into it, the root cause is a timing mismatch between our cache refresh cycle and the API layer sync. Essentially, parsing massive amounts of match data into lightweight views for the front-end is putting a heavy toll on the system.

To handle this, we recently deployed a lumix solution for in-memory mapping right in front of the data layer. We also optimized our queries to better distribute the real-time call load. It helped a lot, but we are still tweaking the settings to get it perfect.

For those who have built similar real-time platforms, what synchronization cycle or interval do you prefer to keep data 100% consistent without crashing the server?

Would love to hear your thoughts or any alternative architecture tips!


r/DistributedComputing • • Mar 08 '25

Distributed Systems jobs

6 Upvotes

Hello lads,

I am currently working in a en EDA related job. I love systems(operating systems and distributed systems). If I want to switch to a distributed systems job, what skill do I need? I study the low level parts of distributed systems and code them in C. I haven't read DDIA because it feels so high level and follows more of a data-centric approach. What do you think makes a great engineer who can design large scale distributed systems?


r/DistributedComputing • • Jan 24 '25

Every System is a Log: Avoiding coordination in distributed applications

Thumbnail
restate.dev
6 Upvotes

r/DistributedComputing • • Oct 25 '24

Learning P2P

6 Upvotes

I'm interested in learning P2P networks, but I noticed that there are not a fair amount of books out there. I would like to get recommendations about this topic.

Thanks!


r/DistributedComputing • • Oct 02 '24

Best resources for understanding consensus

6 Upvotes

Looking for a resource to understand various consensus concepts and algorithms (paxos/raft etc).

Finding it difficult to understand these concepts, looking for your favorite articles/resources!


r/DistributedComputing • • May 25 '23

Understanding the Dapr Workflow engine & authoring workflows as code

Thumbnail
diagrid.io
6 Upvotes

r/DistributedComputing • • Apr 03 '23

What's the next big thing in distributed computing?

6 Upvotes

With container orchestration systems like Kubernetes now widely employed in cloud computing, I am wondering what could be the next big thing in distributed computing. Will there be some ground-breaking product or technology like ChatGPT in the AI field? What are the possible candidates? FaaS? Sky computing?


r/DistributedComputing • • May 12 '22

Remember that time (1997?) SETI had us all download a DC program to help them sift all of the terabytes of data they had, thinking it would speed the search for ET life?

6 Upvotes

That was so cool. I felt like proof was only weeks away. Wouldn’t this be much more effective now that we all have supercomputers (comparatively) in our pockets?


r/DistributedComputing • • Jul 07 '21

Looking for some guidance on making my own distributed computer cheap for ML/Scientific computing purposes

6 Upvotes

Hello,

I am very new to distributed computing and I wanted to make one that can train neural networks. I wanted to know if you all had any tips. I saw maybe there was potential to do so with the raspberry pi (multiple raspis in a beowulf cluster) but I also see a lot of people saying otherwise, and some people say the oodroid is better.

I have no idea what I am doing so here is what I am asking:

1.) Is there a cheap way I can build one of these computers? I don't have an exact budget but I would like to avoid spending a lot. I would prefer smaller boards approx the size of the raspberry pi, for the sake of keeping the overall size as small as possible

2.) What resources should I look at to get a good idea of learning distributed computing and the stuff that goes along with it? I have a BS in Computer Engineering, so I know the basics about computers but not specifically distributed computers. I know that there aren't guides that will spell out exactly what to do (I found one with raspi and tensorflow but that's about it for viable solutions)

EDIT: Also I heard hierarchical computing might be a good idea???

Thank you for the help!


r/DistributedComputing • • Jun 12 '21

Paper on serviceq - a probabilistic load balancing and queuing system

6 Upvotes

Made the paper on ServiceQ publically available - https://github.com/gptankit/serviceq-paper. The paper aims to describe the probabilistic approach followed in the load balancer. Couple of points to note regarding the implementation:

  • ServiceQ considers both historical error feedback and current state of cluster nodes before deciding to forward a request.

  • ServiceQ queues the request if it cannot find any active node to forward which are then deferred forwarded when the cluster is available next.

Comments/suggestions are welcome.


r/DistributedComputing • • Feb 17 '21

Raft for distributed workloads

6 Upvotes

I’ve been reading and researching about distributed systems in Go. After reading about Raft consensus I’ve been finding it hard to find examples that go beyond data replication.

Can Raft be used to distribute computational work to followers? Or is this the wrong protocol for that?

Example say I have a terabyte of json files that I want to run through some function. Could the leader split the work among the followers and have them return the result?

I’m new to this space and am still learning so any help would be appreciated.


r/DistributedComputing • • Feb 09 '21

The Shift to Event-Driven: Evolution, Impact, and Competitive Advantages

Thumbnail
ably.io
6 Upvotes

r/DistributedComputing • • Sep 06 '20

Covid-19: Could your smartphone speed up the search for treatments?

Thumbnail
newscentre.vodafone.co.uk
6 Upvotes

r/DistributedComputing • • Apr 16 '20

Researchers using the Dreamlab platform explaining their work on hyperfoods

Thumbnail
youtu.be
6 Upvotes

r/DistributedComputing • • Apr 16 '14

Distributed computing for a good cause - Compute Against Alzheimer's Disease

Thumbnail
computeagainstalzheimers.org
6 Upvotes

r/DistributedComputing • • Mar 01 '12

My kid's computer can do more than play minecraft? I can't wait to explain the screensaver from Boinc Rosetta@home when they wake.

Thumbnail
boinc.bakerlab.org
6 Upvotes

r/DistributedComputing • • Sep 11 '26

What actually happens when you kill -9 a process mid-write to your WAL

5 Upvotes

I've been building a distributed KV store from scratch in Java (Raft, NIO, custom wire protocol, no external runtime deps in the core) mostly to understand the internals instead of just using etcd or Raft-as-a-library. The part that took the longest to get right wasn't consensus — it was crash recovery of the write-ahead log.

The naive plan is: write a record, write its length, and on restart replay records until you hit garbage. That breaks the moment a crash lands mid-write. You can end up with a record whose length field says "128 bytes" but only 40 bytes actually made it to disk before the power/process died — the tail is torn, not just missing. Trusting the length field alone means you either replay corrupted garbage as if it were a real record, or you silently drop a suffix of valid records after it because you bailed out at the first sign of trouble.

What ended up working: every record carries a CRC32 alongside its length, and replay treats a checksum mismatch as "this is the torn tail, stop here and truncate" rather than "this is corruption, panic." That distinction — expected torn tail vs. actual corruption — is really a statement about where in the write path a crash was allowed to land, which took a few iterations to reason about correctly. I ended up testing this with a deterministic-simulation harness that injects process kills at controlled points in the write path so the interleavings are reproducible instead of "run it in a loop and hope it flakes."

Repo (MIT, Java 21, hand-rolled Raft + NIO, still evolving): https://github.com/Utkarsh-patel26/Kronos

Curious how others have handled this — do you also lean on a checksum-per-record scheme, or something like a fixed-size trailer/footer that's cheaper to validate on replay?


r/DistributedComputing • • Aug 27 '26

ZMQ Arena: A deterministic benchmark harness for ZeroMQ/ZMTP implementations (Latency, Concurrency, and Throughput Analysis)

5 Upvotes

Context: The ZeroMQ ecosystem currently lacks a unified, deterministic benchmarking harness to strictly evaluate latency, concurrency overhead, and throughput across different ZMTP implementations (C++ libzmq, pure-Rust async ports, Python bindings, etc.). Relying on isolated micro-benchmarks introduces systemic bias, failing to accurately account for kernel I/O paths, reactor thread contention, and zero-copy vs. copy-based buffer management under high-density distributed loads.

Decision: I developed ZMQ Arena (Source) to standardize this evaluation. This harness enforces fair-play measurement across runtimes and language boundaries, focusing entirely on execution costs, context switching, and raw throughput limits without application-layer interference.

Technical Scope & Metrics:

  • Latency Analysis: P50/P90/P99 latency distribution across both loopback and physical network boundaries.

  • Concurrency: Stress-testing async reactor overhead (e.g., epoll / io_uring polling mechanisms) versus OS thread pool scaling.

  • Memory Allocation: Evaluating the impact of memory footprints and ABI boundaries when switching between native runtimes and FFI wrappers.

Trade-offs & Known Consequences: Measuring loopback versus physical network I/O presents an inherent trade-off in syscall amortization. Loopback execution can collapse the kernel path, occasionally skewing batching mechanics natively utilized by interfaces like io_uring or IOCP. ZMQ Arena exposes these bottlenecks explicitly rather than abstracting them away, allowing engineers to analyze exactly where the overhead is introduced in the stack.

This harness is built for engineers dealing with critical systems, distributed messaging, and high-frequency data pipelines where deterministic performance is a hard requirement.

If you are dealing with similar infrastructure, review the methodology, replicate the benchmarks, or submit PRs for unrepresented bindings/runtimes.


r/DistributedComputing • • Aug 11 '26

The consistency gap between your database and object storage

Thumbnail
blog.gaborkoos.com
5 Upvotes

A practical overview of designing file uploads and object storage for production: partial failures, database consistency, large uploads, presigned URLs, CDN caching, cleanup, and recovery.


r/DistributedComputing • • Aug 10 '26

Lettuce distributive computing testing

5 Upvotes

I’m probably the last tech savy person in this group, but with help, I was able to set it up. It’s currently configured to need 3 separate computers to verify one work unit. I was looking for someone (more specifically with an Nvidia gpu) to validate my work units. Below are links to the site project, Git, and discord. I figure a vast majority of us are science nerds and we can knock out this testing to hopefully, at some point, get this on the whitelist. Lettuce is basically an easier to set up BOINC. This will get us involved closer with researchers. Anyway, come validate my work units. (Note: there are also CPU only projects.)

Git/download: https://github.com/jring-o/lettuce-compute

Project: https://compute.scios.tech/leafs

Discord testing: https://discord.com/channels/211637812968161280/1517249245596614686


r/DistributedComputing • • Jun 18 '26

Ranja: Enabling Smart Caches for Distributed Database Serving Layers

Thumbnail researchgate.net
5 Upvotes

r/DistributedComputing • • Mar 04 '26

Meet S2C - Cloud-native, quorum-free replicated state machine.

Thumbnail
github.com
5 Upvotes

r/DistributedComputing • • Jan 03 '26

Event Driven Architecture where to learn

5 Upvotes

I am searching resources to learn event driven architecture, microservices communicating using a broker, acting as publisher and consumer, raising events. Thank you


r/DistributedComputing • • Jul 20 '25

Traditional IO vs mmap vs Direct IO: How Disk Access Really Works

Thumbnail nihcas.hashnode.dev
5 Upvotes