r/DistributedComputing • • Jun 05 '26

What problems could we solve with millions of computers contributing 5–7 seconds of CPU time?

31 Upvotes

I'm interested in large-scale distributed computing projects. Imagine millions of users voluntarily donate spare CPU power, but only in very small chunks—around 5–7 seconds of computation per task before requesting a new one.

The ideal problem would:

  • Be highly parallelizable

  • Require little communication between nodes

  • Allow work to be split into millions or billions of independent tasks

  • Be CPU-heavy rather than bandwidth-heavy

  • Be verifiable (so malicious or faulty clients can be detected)

Besides the usual examples like protein folding, cryptocurrency mining, or SETI-style searches, what are some interesting real-world problems that fit this model?


r/DistributedComputing • • Apr 29 '11

Distributed Computers. We should convert the rest of Reddit with it's 8 million unique visitors.

22 Upvotes

We as a group should settle on one distributed computing project. Folding@Home vs. BOINC (And if BOINC what projects?)

We should then settle on a time to submit a link to a subreddit with a larger user base and upvote the hell out of it. This could help the target project greatly. signing on 1/20th of 1% of all visitors would be an 4000 user increase to the target project.

I find Folding@Home to be more difficult for the average user to install when compared with BOINC but that's just me. Opinions?

Update: Glad to see this got some upvotes. I've been computing since the old SETI client and I'd love to see distributed computing grow further. My schedule is pretty busy for the next month but I'll try to organize enough people to insure we hit the front page. I'll send out messages about it some time this month (probably later in the month) to anyone who comments or messages me indicating their interest.

I mostly compute for World Community Grid but I'd be open to promote whatever everyone feels offers the most benefit for mankind.


r/DistributedComputing • • Apr 12 '20

Great results for some Dreamlab projects using BlueStacks. Project needs way more contributers.

Post image
19 Upvotes

r/DistributedComputing • • May 10 '24

"Parallel-Committees": A Novelle Secure and High-Performance Distributed Database Architecture

13 Upvotes

In my PhD thesis, I proposed a novel fault-tolerant, self-configurable, scalable, secure, decentralized, and high-performance distributed database replication architecture, named “Parallel Committees”.

I utilized an innovative sharding technique to enable the use of Byzantine Fault Tolerance (BFT) consensus mechanisms in very large-scale networks.

With this innovative full sharding approach supporting both processing sharding and storage sharding, as more processors and replicas join the network, the system computing power and storage capacity increase unlimitedly, while a classic BFT consensus is utilized.

My approach also allows an unlimited number of clients to join the system simultaneously without reducing system performance and transactional throughput.

I introduced several innovative techniques: for distributing nodes between shards, processing transactions across shards, improving security and scalability of the system, proactively circulating committee members, and forming new committees automatically.

I introduced an innovative and novel approach to distributing nodes between shards, using a public key generation process, called “KeyChallenge”, that simultaneously mitigates Sybil attacks and serves as a proof-of-work. The “KeyChallenge” idea is published in the peer-reviewed conference proceedings of ACM ICCTA 2024, Vienna, Austria.

In this regard, I proved that it is not straightforward for an attacker to generate a public key so that all characters of the key match the ranges set by the system.I explained how to automatically form new committees based on the rate of candidate processor nodes.

The purpose of this technique is to optimally use all network capacity so that inactive surplus processors in the queue of a committee that were not active are employed in the new committee and play an effective role in increasing the throughput and the efficiency of the system.

This technique leads to the maximum utilization of processor nodes and the capacity of computation and storage of the network to increase both processing sharding and storage sharding as much as possible.

In the proposed architecture, members of each committee are proactively and alternately replaced with backup processors. This technique of proactively circulating committee members has three main results:

  • (a) preventing a committee from being occupied by a group of processor nodes for a long time period, in particular, Byzantine and faulty processors,

  • (b) preventing committees from growing too much, which could lead to scalability issues and latency in processing the clients’ requests,

  • (c) due to the proactive circulation of committee members, over a given time-frame, there exists a probability that several faulty nodes are excluded from the committee and placed in the committee queue. Consequently, during this time-frame, the faulty nodes in the committee queue do not impact the consensus process.

This procedure can improve and enhance the fault tolerance threshold of the consensus mechanism.I also elucidated strategies to thwart the malicious action of “Key-Withholding”, where previously generated public keys are prevented from future shard access. The approach involves periodically altering the acceptable ranges for each character of the public key. The proposed architecture effectively reduces the number of undesirable cross-shard transactions that are more complex and costly to process than intra-shard transactions.

I compared the proposed idea with other sharding-based data replication systems and mentioned the main differences, which are detailed in Section 4.7 of my dissertation.

The proposed architecture not only opens the door to a new world for further research in this field but also represents a significant step forward in enhancing distributed databases and data replication systems.

The proposed idea has been published in the peer-reviewed conference proceedings of IEEE BCCA 2023.

Additionally, I provided an explanation for the decision not to employ a blockchain structure in the proposed architecture, an issue that is discussed in great detail in Chapter 5 of my dissertation.

The complete version of my dissertation is accessible via the following link: https://www.researchgate.net/publication/379148513_Novel_Fault-Tolerant_Self-Configurable_Scalable_Secure_Decentralized_and_High-Performance_Distributed_Database_Replication_Architecture_Using_Innovative_Sharding_to_Enable_the_Use_of_BFT_Consensus_Mec

I compared my proposed database architecture with various distributed databases and data replication systems in Section 4.7 of my dissertation. This comparison included Apache Cassandra, Amazon DynamoDB, Google Bigtable, Google Spanner, and ScyllaDB. I strongly recommend reviewing that section for better clarity and understanding.

The main problem is as follows:

Classic consensus mechanisms such as Paxos or PBFT provide strong and strict consistency in distributed databases. However, due to their low scalability, they are not commonly used. Instead, methods such as eventual consistency are employed, which, while not providing strong consistency, offer much higher performance compared to classic consensus mechanisms. The primary reason for the low scalability of classic consensus mechanisms is their high time complexity and message complexity.

I recommend watching the following video explaining this matter:
https://www.college-de-france.fr/fr/agenda/colloque/taking-stock-of-distributed-computing/living-without-consensus

My proposed architecture enables the use of classic consensus mechanisms such as Paxos, PBFT, etc., in very large and high-scale networks, while providing very high transactional throughput. This ensures both strict consistency and high performance in a highly scalable network. This is achievable through an innovative approach of parallelization and sharding in my proposed architecture.

If needed, I can provide more detailed explanations of the problem and the proposed solution.

I would greatly appreciate feedback and comments on the distributed database architecture proposed in my PhD dissertation. Your insights and opinions are invaluable, so please feel free to share them without hesitation.


r/DistributedComputing • • Oct 08 '24

Is a master in distributed systems/computing worth it?

12 Upvotes

I've been thinking about getting a masters in distributed systems/computing. As that's a role I'll like to settle in for the long term. But taking that two years career break to go for masters is not really making sense to me! What do you all think? How do think i can get into this type of role? Any advice is welcome

A little context: I recently transitioned from native Android dev to DevOps/cloud


r/DistributedComputing • • Mar 15 '22

How to build experience with large scale distributed systems

12 Upvotes

I work at a financial company and the projects that I had been part of doesn't have much scope for distributed systems engineering. How does one grow this experience so as to interview for other companies and perform well? I have gone through multiple books and blogs including Martin Klepmann DIA book and his youtube videos, but without real work, theory doesn't seem to be sufficient.


r/DistributedComputing • • Feb 13 '11

I'm surprised that Reddit doesn't have a larger BOINC team.

12 Upvotes

How many of you are currently on the reddit team?

It would seem like a lot of this would be right up our alley. If we could mobilize, we could have quite a formidable team.

Sorry for being somewhat hypocritical, but I am obligated to stay with another team for the time being (I am the founder of multiple individual project's teams, so I can't just up and leave, and wouldn't want to). I would have joined the reddit team a few years back if it didn't seem so dead.

What is surprising, is that the team I am on consists of users from a website with FAR less traffic (~60,000 ranking on Alexa compared to Reddit's ~150 ranking), but our RAC and world ranking is top 100 and top 150 respectively. The majority of our credit comes from only about 10 users, but still, this team has more than double the users that reddit's team has.

How could we recruit redditors? It seems like most BOINC posts I have found using the search function have lukewarm responses. We could easily be top-100 in the world if it was done right.

Also, I realize I am posting in a dead subreddit...Oh well.


r/DistributedComputing • • Aug 24 '25

Guidance on transitioning to Distributed Computing field – conferences, research areas, future scope

11 Upvotes

I'm a software developer with 5+ years of experience, and I’m now looking to explore a bit deeper area than my current work, especially in distributed computing field.

I would like to get suggestions on attending top conferences, learning recent advancements and hot research topics in the field. I would also like to get guidance on expanding my knowledge in the area(books, courses, open source, research papers, etc.) and picking and researching on most relevant problem in this space.

I'm trying to understand both the research side (to maybe publish or contribute) and the practical side (startups, open-source). Any suggestions, experiences, or resources would mean a lot.
Thanks.


r/DistributedComputing • • Feb 16 '24

How to get into distributed computing?

11 Upvotes

I mean where do I get a distributed system to play with? Why should I aim for a distributed system in the first place?

I am fairly interested In trying some hpc adjacent things on a distributed setup but not sure how to go about it.


r/DistributedComputing • • Nov 25 '21

Tiny Mile, a company putting delivery robots on the streets of Toronto, is hiring engineers to work remotely

11 Upvotes

We're If you're interested in robotics, distributed systems and cloud computing, apply here: https://tiny-mile.breezy.hr/p/ed934cfad0ee

TLDR is: we're looking for senior software engineers to work mainly on our back-end which consists of microservices written in Go. We're based in Toronto, but the position is fully remote!


r/DistributedComputing • • May 06 '21

How to get started in Distributed Systems as a Software Engineer?

11 Upvotes

I am a Software Engineer based in India. A little bit about myself, I am well versed in Data Structures and Algorithms. I am familiar with Operating Systems, but don't have in-depth knowledge. The same goes for other subjects like Networks, Databases, etc. I found Distributed Systems to be really interesting. I saw a lot of posts regarding how to get started and most of them suggest reading papers. But I want to get my hands dirty writing code, but most of the blog posts don't cover this.
What language should I pick? How should I go about writing Distributed Systems code as a beginner?


r/DistributedComputing • • Apr 08 '26

Created Distributed Leaderless Hash Tables in go

10 Upvotes

I was fascinated by cassandra. It has so many cool features and virtually scales infinitely. Most importantly it is leaderless. I got so curious about this that I spend last few weeks learning about its working but still I didn't understand nuances of it. Thats when I decided best way to learn it to make it. I spent 2 long weekends and 2 workings days trying to build it ( I took two PTO). Things I learned along the way, I feel like a different person now as a engineer and feel so confident. I implemented

  • Consistent Hashing

  • Leaderless coordination w/ Gossip Protocol

  • Live data replication during node bootstrapping (or Splitting nodes/shards. This took so much more than than any other thing)

  • Dual writes, key level versioning.

There is so much more that I understood that I don't know. Particularly, I learned about new concepts like LSM Trees which can enable point-in-time snapshots for database, Merkle trees which enable transferring minimum about of data to sync nodes. Most importantly, this time I took slightly different approach of learning, I documented first and then implemented. I took my time to jot down what I am thinking, why, what challenges I am thinking, and my plans to tackle them. Once I had a clear picture in mind then I took it upon my self to start the implementation. This approach actually helped me a lot. I could start something today and then continue it next day by reading exactly what was going in my mind earlier. This was more useful as I looked back through the notes and realised few places where I needed more clarity.

At this point, there is so much more that I need to learn. Currently implementation of point-in-time snapshot is not ideal, there are not ways to merge the nodes (opposite of adding new node to handle high traffic load). No persistent storage, no quorum (tuneable consistency levels, I am most excited about this after persistent storage).

Code can be found here, my thoughts during building are here. Current features are here. Features I am excited about and will implement in future are here, things I want to implement if get enough time are here. I am happy with current stage and going forward i'll take things slow and add new things (no promises though) if you are interested you can send in a pr for some of the features you are interested.

Cheers. Thanks to this community and similar other communities which helped me get few answers when I had them


r/DistributedComputing • • Aug 15 '24

Simple Distributed Computing framework based on Redis

10 Upvotes

I gave a presentation on using Redis as middleware for distributed processing at EuroTcl/OpenACS 2024. I think this is a simple but powerful technique, combining communication between multiple client and server instances with caching. Clients and servers connect over the net to a Redis instance, using Redis lists as queues of requests and responses. For the talk I did a demonstration using servers and Redis in different countries, but normally these would all be on a local network, with much less latency.

The implementation is in Tcl, but the same approach could be implemented in any language with a Redis interface. The video is at https://learn.wu.ac.at/eurotcl2024/lecturecasts/729149172?m=delivery and the slides are at https://openacs.org/conf2024/info/download/file/DisTcl.pdf . The code for the demonstration can be found at https://cmacleod.me.uk/tcl/mand/ . Some background info can be found at https://wiki.tcl-lang.org/page/DisTcl and https://colin-macleod.blogspot.com/2020/11/fundis-functional-distributed-processing.html .


r/DistributedComputing • • May 25 '21

Can someone help me understand how the actor model frameworks like Akka and Erlang can achieve concurrency at high throughput compared to traditional lock, semaphore based concurrency models?

10 Upvotes

r/DistributedComputing • • Apr 14 '21

Encore - A Go backend framework with superpowers

Thumbnail
github.com
10 Upvotes

r/DistributedComputing • • Jan 18 '21

Ballista: Distributed compute platform implemented in Rust using Apache Arrow

Thumbnail
github.com
10 Upvotes

r/DistributedComputing • • Mar 17 '20

Which project helps most with coronavirus?

10 Upvotes

I know what this question could be answered quickly on a bigger subreddit like BOINC or FAH but I want an unbiased opinion. Right now, it looks like there are two projects that are contributing to fighting coronavirus: F@H and Rosetta@home on BOINC. Does one of these projects have a clear winner over the other in terms of specifically fighting the coronavirus, or is it less clear/ down to opinion? Thanks for the responses.


r/DistributedComputing • • Sep 17 '19

Distributed computing project to check convergence of the Collatz problem

10 Upvotes

Hi all,

I start new distributed computing project to check convergence of the Collatz problem. So far, the convergence of all numbers below 87 × 260 has been verified (this is approximately 266.44). I want to raise this upper bound by at least 260. This small achievement takes roughly about 3 153 600 core-hours. And here I need your help. Source codes and pre-compiled Linux client are available to download.

The project page is here: http://pcbarina2.fit.vutbr.cz/~ibarina/


r/DistributedComputing • • May 20 '25

distributed.net & RC5-72

9 Upvotes

I just rejoined the distributed.net effort to crack the RC5-72 encryption challenge. It's been going on for over 22 years now, and I was there in the beginning when I first started working on it in 2002. Fast forward to today and my current hardware now completes workloads 627 times faster than it did back in 2002. Sure it's an old project, but I've been involved with it for 1/2 of my lifetime and the nostalgia of working on it again is fun. Have you ever worked on this project?


r/DistributedComputing • • Dec 13 '24

My rosetta@home account has vanished into thin air

9 Upvotes

Has anyone else also experienced this?

It's just gone: https://boinc.bakerlab.org/rosetta/view_profile.php?userid=2415202

Logging in tells me that no user with my email address exists. My client can't connect because of an invalid account key; telling me to remove and add the project again (which doesn't work cause I cant log in).

Does rosetta@home have a support contact?


r/DistributedComputing • • Jan 01 '24

Could A Database Lock Be Used To Elect A Transaction Coordinator?

9 Upvotes

I’m reading about consensus algorithms in “Designing Data Intensive Applications” and I had a sort of naive thought, so I want to know why it is wrong.

The author discusses the two phase commit protocol and the problem with two phase commit as a motivation for distributed consensus. What I got is basically that the leader may fail and leave all ready nodes in permanent limbo. But choosing a new leader would require consensus among the nodes.

So I have a rather naive solution. Why not just have a database somewhere that encodes the commit log? Then the leader would be whoever acquires either a lock or a valid token to append to that table, and the token would be something that you’d have to renew after a certain period of time. Whatever node controls the database would be delegated the task of deciding which requesting node actually gets the token.

So I imagine if this were so simple, that’s what people would do and this idea must be horribly stupid and naive — but I’m curious if someone is patient enough to explain why this wouldn’t work.


r/DistributedComputing • • Jul 27 '23

BlazingMQ: High-performance open source message queuing system

Thumbnail
github.com
9 Upvotes

r/DistributedComputing • • Jul 18 '23

Why Adaptive Rate Limiting is a Game-Changer

Thumbnail
medium.com
9 Upvotes

r/DistributedComputing • • Apr 03 '20

Contributing to COVID-19 Therapy Research with Folding@Home

Thumbnail
medium.com
9 Upvotes