r/DistributedComputing • • Aug 07 '26

The Authentication Stack: 7 Concepts Every Software Engineer Should Understand

Thumbnail
youtu.be
3 Upvotes

r/DistributedComputing • • Jul 09 '26

Transitioning from Crud Software Developer to Linux engineer / Infrastructure Engineer to be future ready.

Thumbnail reddit.com
3 Upvotes

r/DistributedComputing • • May 27 '26

mosaik - A Rust runtime for building self-organizing, leaderless distributed systems.

Thumbnail reddit.com
3 Upvotes

r/DistributedComputing • • Apr 07 '26

Rebalancing Traffic In Leaderless Distributed Architecture

3 Upvotes

I am trying to create in-memory distributed store similar to cassandra. I am doing it in go. I have concept of storage_node with get_by_key and put_key_value. When a new node starts it starts gossip with seed node and then gossip with rest of the nodes in cluster. This allows it to find all other nodes. Any node in the cluster can handle traffic. When a node receives request it identifies the owner node and redirects the request to that node. At present, when node is added to the cluster it immediately take the ownership of the data it is responsible for. It serves read and write traffic. Writes can be handled but reads return null/none because the key is stored in previous owner node.

How can I solve this challenge.? Ideally I am looking for replication strategies. such that when new node is added to the cluster it first replicates the data and then starts to serve the traffic. In the hind-sight it looks easy but I am thinking how to handle mutation/inserts when the data is being replicated?

More Detailed thoughts are here: https://github.com/goyal-aman/distributed_storage_nodes/?tab=readme-ov-file#new-node-with-data-replication


r/DistributedComputing • • Mar 18 '26

Treating cache entries as in-flight computations instead of just values

Thumbnail
infoq.com
3 Upvotes

r/DistributedComputing • • Mar 06 '26

HRW/CR = Perfect LB + strong consistency, good idea?

3 Upvotes

Hello, I have this idea in my mind since a while and want to get some feedback if its any good and worth investing time into it:

The goal was to find a strong consistent system that utilizes nodes optimal. The base is to combine chain replication with highest random weight. In CR you need to store the chain configuration somewhere. Why not skip that and use HRW on a per key base? That would give you the chain configuration in the order that should be used for every key.

The next advantage would be that you end up with a system that does perfect load balancing (if the hashing is good enough).

Challenges I saw would be a key based replication factor, but for now I would say its fixed/not supported. Another point would be: how to handle node failure and the needed key moves? Here I was thinking that you use some spare nodes. E.g. you have a replication factor of 2, so you choose 5 nodes in total (the idea here is that not all keys need to be moved on failure).

As CR is the core, you win all of its benefits (e.g. N-1 nodes can fail). I have the feeling that approach is simpler compared to CRAQ.

Any thoughts on that?


r/DistributedComputing • • Jan 04 '26

C++ code generator that helps build distributed systems

3 Upvotes

Hi. I'm working on a C++ code generator that helps build distributed systems. It's implemented as a 3-tier system. The back and middle tiers only run on Linux. The front tier is portable. It's geared more towards network services than webservices.

It's free to use -- there are no trial periods or paid plans. I'm willing to spend 16 hours/week for six months on a project if we use my software as part of the project.


r/DistributedComputing • • Jul 04 '25

SWIM Vis: A fun little interactive playground for simulating and visualizing how the SWIM Protocol functions

Thumbnail ryanolee.github.io
3 Upvotes

Did a talk recently on the SWIM protocol and as part of it wanted to create some interactive visuals, could not find any visual simulators for it so decided to make my own one. Thought some of you you might appreciate it! Might be useful as a learning tool also, have tried to make it as true as I could to the original paper.


r/DistributedComputing • • Jun 15 '25

MiniClust: a lightweight multiuser batch computing system

3 Upvotes

MiniClust : https://github.com/openmole/miniclust

MiniClust is a lightweight multiuser batch computing system, composed of workers coordinated via a central vanilla minio server. It allows distribution bash commands on a set of machines.

One or several workers pull jobs described in JSON files from the Minio server, and coordinate by writing files on the server.

The functionalities of MiniClust:

  • A vanilla minio server as a coordination point

  • User and worker accounts are minio accounts

  • Stateless workers

  • Optional caching of files on workers

  • Optional caching of archive extraction on workers

  • Workers just need outbound http access to participate

  • Workers can come and leave at any time

  • Workers are dead simple to deploy

  • Fair scheduling based on history at the worker level

  • Resources request for each job


r/DistributedComputing • • May 14 '25

FlowG - Distributed Systems without Raft (part 2)

Thumbnail
david-delassus.medium.com
3 Upvotes

r/DistributedComputing • • Apr 18 '25

BOINC on Android - current status and experience

3 Upvotes

On my Samsung Galsxy S25, with the Snapdragon 8 Elite chip, I've found that only 3 projects currently work:

 

Asteroids@Home

 

Einstein@Home

 

World Community Grid

 

Also, the annoying battery percentage issue is present for the first couple of minutes after I've added the projects, but then after disabling "pause when screen is on, setting the minimum battery percentage setting to the lowest 10%, and Android has asked me to disabled battery optimization for the app, after a couple of more minutes, the app starts working on Works Units.

 

So now, for me at least, on this device, BOINC on Android works fine for me.

 

Just remember to enable "battery protection" or 80% charging limit, if your phone supports this, and in BOINC, not to run while om battery, and you're good to go.

 

Anybody who've still got issues with BOINC on Android, please comment below 

 

P.s. There's an Android Adreno GPU option you can enable in your profile project settings on the Einstein@Home website, but are there actually works units available for the GPU, or is it not working?

 


r/DistributedComputing • • Mar 21 '25

Folding on Apple SIlicon Macs

3 Upvotes

Just got an M4 mac mini, and here’s what I’ve found testing folding on MacOS:

You can actually download the mobile dreamlab app, and run this on your Mac. Usually your mobile device must be plugged in, so I don’t know how it would work on a macbook. Also, the app still heavily underutilizes the CPU, only utilizing around 10%/1 core, but it’s still better than nothing. And it being available on Mac means there’s no excuse not to release it on chromebooks, windows, and linux too.

Then for folding@home, it works fine, and you can move a slider to adjust CPU utilization, but there is no advanced view and options like there is on Windows, which I miss, but that’s probably a Mac thing and design. And it works best setting the slider to match the amount of performance cores you have, which is 4 for me.

As for BOINC, 11 projects work, and they either have Apple Silicon ARM support, Intel x86 tasks are being translated using Rosetta 2, both, aor there are currently no tasks available, where only Einstein@Home has tasks for the GPU cores. The projects are Amicable Numbers, asteroids@Home, Dodo@Home (not on the project list, and no tasks at the moment), Einstein@Home, LODA, Moo! Wrapper, NFS@Home, NumberFields@Home, PrimeGrid, Ramanujan Machine (currently not getting any tasks), and World Community Grid (also currently no tasks).

 Also, in the Mac Folding@Home browser client, it says 10 CPU cores but 0 GPU cores, and that's cause the Apple Silicon hardware doesn't support something called "FP64" which is necessary for most project to utilize the GPU cores.

And if your M4 Mac mini for instance is making too much fan noise at 100% utilization, you can enable "low power mode" at night, to get rid of it, sacrificing about half of the performance, but still.
Lastly, for BOINC, I recommend running Asteroids@Home, NFS@Home, World Community Grid, and Einstein@Home all the time. That way you never run out of Work Units, and these have the shortest Work Units on average.

Please Comment if you want more in depth info about Folding on Mac, in terms of tweaking advanced settings for these projects, getting better utilization, performance, or whatever, and I'll try to answer as best I can :)


r/DistributedComputing • • Mar 10 '25

My notes on Paxos

3 Upvotes

I am jotting down my understanding of Paxos through an anology here - https://www.algocat.tech/articles/post8


r/DistributedComputing • • Dec 11 '24

Are there general limitative results for Byzantine fault tolerance (BFT) and crash tolerance (CFT) outside of consensus algorithms?

3 Upvotes

Given that there are distributed algorithms other than consensus algorithms (e.g., mutual exclusion algorithms, resource allocation algorithms, etc.), do any general limitative BFT and CFT results exist for non-consensus algorithms?

For example, we know that for consensus algorithms, a consensus algorithm can only tolerate up to n/3 Byzantine faulty nodes or n/2 crash faulty nodes.

But are there any such general results for other distributed algorithms?


r/DistributedComputing • • Aug 17 '24

What do you think of my framework? I call it nyssr.net.

3 Upvotes

Hi!

I have written a framework for distributed computing, which is free for non-commercial use. I would like to classify the framework, how it is correctly described and which frameworks it competes with. I would also be interested to know what you think of it. And what is still missing, what should be addressed next.

The framework is called nyssr.net and is written in JAVA. nyssr.net is a network of interconnected JAVA nodes using TCP channels to facilitate message exchange. Messages are routed through these channels, avoiding the need to establish new connections dynamically.

Each node is built around a nimble and quick-loading micro-kernel. This micro-kernel loads additional functionalities in the form of plugins during startup. Remarkably, even essential features like TCP or the transport layer are loaded as plugins, alongside various services and applications.

A range of services now exists based on these characteristics:

  • Nanoservices (services that process only one type of message)

  • Microservices (services with a messaging API)

  • Namespaces with one or more threads, as well as target registries and nanoservice registries

  • Distributed microservice registries (redundant and synchronized)

  • Distributed file stores

  • Automatic software update for some or all nodes

  • Consoles for each node, with access to all connected nodes

  • Reboot mechanisms, immediate, scheduled, and for parts of or the entire network

  • A job machine for the execution of asynchronous, freely communicating jobs

  • Distributed applications, to be started and stopped via message

  • An integrated session management with users, roles, and permissions.

  • RemoteSkin for the web, truly distributed web applications

  • RemoteSkin for Swing, truly distributed Swing applications

and much more

You can find the framework at sillysky.net.

Many greetings,
Michael Hoppe
[email protected]


r/DistributedComputing • • Apr 05 '24

Model Parallelism using Pytorch and sockets

3 Upvotes

Hey! Can somebody share a sample implementation of model parallelism using pytorch and sockets, i have a project presentation coming up and its a tight schedule :/
Thanks in advance.


r/DistributedComputing • • Mar 29 '24

New BOINC 8.0.0 is ready for testing

Thumbnail
self.boinc4science
3 Upvotes

r/DistributedComputing • • Mar 28 '24

Anyone cognizant of the distributed computing DreamLab app/platform?

3 Upvotes

It's a great way to contribute to projects focused on cancer research, long COVID, & Cyclone weather modeling. The app is available for iOS & Android. You can also run it on Windows 11 & Linux distros with emulators. There's also an unofficial subreddit, which has been running for a couple of years now.


r/DistributedComputing • • Mar 06 '24

What do you think about Oxia: a new high-performant metadata store?

3 Upvotes

Oxia is a new metadata store and coordination system similar to Zookeeper or Etcd. But in comparison with the others, it can store 100s of GBs of data and can handle millions of reads and writes per second.

Oxia GitHub repository: https://github.com/streamnative/oxia

It's developed by StreamNative - the company that works mostly on the Apache Pulsar.

Disclaimer: I'm NOT a StreamNative employee, but recently became an Apache Pulsar committer.

Oxia is licensed under Apache License 2.0. From public discussions, I know that StreamNavtive has plans to donate it to the Apache Software Foundation, but they haven't shared a concrete date yet.

The Java client has been just open-sourced. Here is a blog post: https://streamnative.io/blog/the-oxia-java-client-library-is-now-open-source

Oxia is already integrated with Pulsar but isn't a default option yet. Folks from StreamNative claim that they used it in the cloud for a few months without any problems.

I didn't see any independent benchmarks yet, therefore I can't validate the performance and stability claims. Probably this post may change it and attract engineers who are interested in trying Oxia for their projects and publicly share the results.

I think it may be interesting for engineers who work on building distributed systems.
I would greatly appreciate hearing about your thoughts on the project!


r/DistributedComputing • • Mar 05 '24

GitHub - taubyte/tau: Easily Build Distributed Cloud Computing Platforms with features like Serverless WebAssembly Functions, Frontend Hosting, Object Storage, K/V Database, and Pub-Sub Messaging.

Thumbnail
github.com
3 Upvotes

r/DistributedComputing • • Jan 30 '24

How does CUDOS empower Cloud Computing Technology?

3 Upvotes

CUDOS empowers cloud computing through decentralized, cost-efficient, and secure computing. It democratizes access by incentivizing individuals and organizations to contribute spare computing resources. This decentralized model enhances scalability, flexibility, and global accessibility, reducing costs and reliance on centralized servers. By incorporating blockchain, CUDOS ensures security and transparency, mitigating the risk of data breaches. The incentive structure encourages a diverse network of resource providers, fostering a collaborative ecosystem for innovation. CUDOS revolutionizes cloud computing by addressing inefficiencies and offering a resilient, distributed, and economical alternative to traditional cloud services. Embrace the future with r/Cudos_Official! https://cudos.org


r/DistributedComputing • • Jan 19 '24

Dask Demo Day: Apache Beam on Dask, expressions for Dask Array, and 1BRC for Dask vs Spark

3 Upvotes

Today's talks:

- Apache Beam DaskRunner
- Array expressions
- One billion row challenge in Dask vs. Spark

Recording available on youtube: https://www.youtube.com/watch?v=wkQzVNQdgW0

Each month folks from the Dask community give short demos that show off ongoing and/or lesser-known work. Hopefully this helps elevate some of the great work people do.

If you're interested in presenting, comment on this github issue with a brief (a couple sentences) description: https://github.com/dask/community/issues/307


r/DistributedComputing • • Jan 17 '24

Distributed system engineer

3 Upvotes

I'm a mobile developer with 2.3 years experience. Now I want to become a distributed system engineer but I don't know where to start and what to learn. I know java, first should I learn to build rest api (with spring boot, hibernate). Should I find a new job for backend developer role Can anyone please guide me...


r/DistributedComputing • • Dec 08 '23

Need insights to build a distributed key value store from scratch.

3 Upvotes

I am a grad student pursuing MS in CS from RIT. I am interested to pursue a career in distributed systems and would like to build a distributed key value store from scratch. I have taken the distributed systems cluster in the program and have read many research papers on systems like Raft, Dynamo, Paxos, etc. I am also familiar with the concepts of consistent hashing and vector clocks. But when I decided to build a distributed key value store from scratch, I am completely blank. Any guidance would be much appreciated. Thank you.