r/DistributedComputing • • Sep 17 '22

The Rise of Web3: confidential computing

Thumbnail
youtu.be
0 Upvotes

r/DistributedComputing • • Apr 14 '20

Design a Distributed File Backup and Search System

0 Upvotes

There are 'N' servers in a VPN. Each server can store a finite number of files in the pattern backup_filename_yyyy_mm_dd.extension .Files to be backed up are received on a specific Email ID as attachment. Design a system for receiving and then backing up these incoming files on the 'N' servers and a fast search + retrieval system to find + retrieve the latest available backup for a given file name. For a given file, you should store the last 2 backups only and remove previous backups. Your solution should have the following questions answered:

Q1. Architecture diagram of the system. Note: Consider CAP theorem and no SPOF while designing your system. No file backup should ever be lost and the system should support 99.99% availability.

Q2. Given a specific file (type) that needs to be backed up, come up with a consistent way to select a server out of N servers to store and retrieve the file. What will happen if I add or remove servers?

Q3. How can we guarantee a backup will never be lost? What is the best way to transfer large files from one to another server?

Q4. How can we really fasten up search such that results are instantly available?

Q5. Given that we are ok with a slow file retrieval time, what is the best way to save storage on your servers when storing these files on these servers?

Q6. What changes will you make in your system if file names are not unique?

Q7. How will you find max, min, median, mean of file size for all files in all 'N' servers?

Q8. For question 7 above, what can you do to make these numbers instantly available?

Q9. In a few lines, tell your simple disaster recovery and failover strategy for your system.

Q10. How can you secure your system and files?


r/DistributedComputing • • Feb 16 '21

High performance computing

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Sep 15 '20

What’s the true potential of decentralized cloud technology? Learn about recent developments, emerging use cases, and interoperability as the next leap forward. Insights from Protocol Labs (Filecoin), Bluzelle, Crust Network. Free online event on Sept 17th.

Thumbnail
parity.link
0 Upvotes

r/DistributedComputing • • May 22 '24

help

0 Upvotes

i was studying distributed systems and i came across this question online, can you guys help me with it?
note: the question is part of a Homework Assignment given to students in 2009 at university of California San Diego and here is the link to it : https://cseweb.ucsd.edu/classes/fa09/cse91/resources/cse91hw2.pdf

Suppose you want your ATM to give you $100. and ATM has two separate processing steps:
 it must record a debit for $100 and it must give you the cash.

By the two generals problem, it cannot do both at the same time. It can do these steps in either order and a crash can occur any time.

– Suppose it gives you the cash first. What can go wrong?
 – Suppose it does the debit first. What can go wrong? How the problem might eventually be fixed.
– Based on your analysis, which option would banks choose?


r/DistributedComputing • • Sep 16 '24

Desktop airbnb

0 Upvotes

Hey gang,

I’m looking into ways for my machine to generate revenue whilst idle. Just checking if people would be interested in borrowing my machine for their network/startup they are making.

Apologies for my terrible terminology I’m still learning the lingo


r/DistributedComputing • • Jun 09 '26

can a P2P mesh make your laptops one giant machine?

0 Upvotes

Hey everyone,

I'm working on the architecture for a new open‑source distributed compute framework called Project Cerberus, and I want to get reality‑checked by this community before I commit to the core stack.

The Concept:
A zero‑config, P2P mesh network that aggregates heterogeneous consumer hardware (Macs, Windows rigs, headless Linux boxes, and even phones) on a local subnet into a single virtual hyper‑computer.

I know Exo (exo-explore) has basically perfected local AI model sharding (pipeline parallelism for LLMs over Wi‑Fi/Thunderbolt). Cerberus isn't trying to just be another AI inference engine. I want to build a decentralized virtual OS layer that pools everything:

  1. General Compute: Compiling user workloads to WebAssembly (Wasmtime) and distributing the execution slices across node threads.

  2. Storage Pooling: A FUSE‑based user‑space filesystem that erasure‑codes and strips data across the unused NVMe/SSD space of all connected devices.

  3. Audio/Peripheral Virtualization: Using PipeWire/CoreAudio to route synced audio across multiple device speakers (sub‑millisecond NTP sync) or pooling local mics.

Proposed Tech Stack:
Control Plane: Go (Golang) + libp2p (mDNS for zero‑config discovery, Gossipsub for topology mapping, QUIC for transport).
Compute Plane: Wasmtime for architecture‑agnostic sandboxed execution + MLX/tinygrad for the AI tensor sharding.
UI/Gateway: Tauri (Rust/Next.js) for a lightweight dashboard, exposing an OpenAI‑compatible REST API wrapper for devs to point localhost to.

My questions for the distributed systems vets here:

  1. Network Latency: For the WASM general compute sharding, is QUIC transport fast enough to handle the context handoffs over standard Wi‑Fi 6, or will the network latency obliterate any compute gains?

  2. Consensus: I want to avoid a master‑worker setup. If the coordinator node drops (e.g., someone closes their laptop lid), what's the cleanest way to run a fast raft/bully election in Go without dropping the current FUSE read/writes?

  3. Are there any existing crates/libraries for the distributed audio sync that I should look at before trying to build a custom WebRTC/RTP bridge?

Would love any feedback, warnings, or pointers to obscure repos doing something similar. Thanks!


r/DistributedComputing • • Aug 17 '24

What do you think of my framework? I call it nyssr.net.

3 Upvotes

Hi!

I have written a framework for distributed computing, which is free for non-commercial use. I would like to classify the framework, how it is correctly described and which frameworks it competes with. I would also be interested to know what you think of it. And what is still missing, what should be addressed next.

The framework is called nyssr.net and is written in JAVA. nyssr.net is a network of interconnected JAVA nodes using TCP channels to facilitate message exchange. Messages are routed through these channels, avoiding the need to establish new connections dynamically.

Each node is built around a nimble and quick-loading micro-kernel. This micro-kernel loads additional functionalities in the form of plugins during startup. Remarkably, even essential features like TCP or the transport layer are loaded as plugins, alongside various services and applications.

A range of services now exists based on these characteristics:

  • Nanoservices (services that process only one type of message)

  • Microservices (services with a messaging API)

  • Namespaces with one or more threads, as well as target registries and nanoservice registries

  • Distributed microservice registries (redundant and synchronized)

  • Distributed file stores

  • Automatic software update for some or all nodes

  • Consoles for each node, with access to all connected nodes

  • Reboot mechanisms, immediate, scheduled, and for parts of or the entire network

  • A job machine for the execution of asynchronous, freely communicating jobs

  • Distributed applications, to be started and stopped via message

  • An integrated session management with users, roles, and permissions.

  • RemoteSkin for the web, truly distributed web applications

  • RemoteSkin for Swing, truly distributed Swing applications

and much more

You can find the framework at sillysky.net.

Many greetings,
Michael Hoppe
[email protected]


r/DistributedComputing • • Jul 26 '22

Support Team Ukraine / FB Restart!

0 Upvotes

Formula Boinc had a nice restart restart this weekendwith the new face of the FB.UBT Timbo. (with Seb's support)

The Formula Boinc Sprint #11 went off without a hitch this weekend.
That is great news for FB and Seb fans. After 3 canceled Sprints
Amicable Numbers was picked and both the FB and the Amicable Numbers
servers responded well, it was a close sprint till the 3rd day when a
big dump from TSBT took 1st running away.
TSBT also won all 3 days points, to round out a good win for them.
2nd and 3rd place was hotly contested between
Dutch Power Cows and Team Ukraine.
Dutch Power Cows took 2nd the first day, with Team Ukraine
taking 2nd the 2nd day and Dutch Power Cows squeaking by TU
to lock in 2nd place overall.
Team Ukraine scored a solid 3 place overall finish while still
2nd overall in the Sprints for 2022!

Sprint
07/21/2022 21:00 (UTC) - 07/24/2022 20:59 (UTC)
Amicable Numbers
Last update : 07/24/2022 21:00 (UTC)
Pics - https://ibb.co/album/9ZM3gX

Overall Standing
https://formula-boinc.org/index_v2.py?year=2022&lang=en&team=&thx=

Overall Standing
https://formula-boinc.org/index_v2.py?year=2022&lang=en&team=&thx=

Forum
https://formula-boinc.org/forum/viewforum.php?f=22

Nice job Tim and everyone in the FB!
Stay Cool.
SystemViper.
XtremeSystems
https://xs4s.org/index.php

Don't forget BoincGames.com
A new competition for BOINC teams
Homepage
https://www.boincgames.com/index.php
Overall Standing


r/DistributedComputing • • Aug 15 '22

Earn MayoCoin by running MilkyWay@Home on BOINC. Join Team MayoCoin!

Thumbnail
self.mayocoin
1 Upvotes

r/DistributedComputing • • Oct 20 '20

Cloud computing versus Cloud storage

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Dec 25 '21

The Cloud Revolution Draft

1 Upvotes

Hey I’ve been looking for a fun way to learn cloud computing and distributed systems. I loved the way eliyahu goldratt and gene Kim used novels to teach. I’ve been working on some creativity exercises myself and would love any feedback I can get. All comments are appreciated. Please feel free to correct my definitions if you have the time.

December 22nd, 2021

Introduction

So what is incident management thought Paul from managed services at Cloud Computing incorporated? It is December 22nd, and the team is left wondering how they will tackle problems as they occur. Paul wonders to himself often how he could develop better services if only he had more time to develop instead of dealing with problems.

Paul had always thought innovation would drive most companies to success as a developer. However, Paul did not realize where exactly innovation had yet to come. Innovation was thriving in areas such as development, yet service providers were beginning to have a harder time dealing with issues at hand.Problems were scaling fast and coming more often. How on earth would a small managed services team like Cloud Computing Incorporated handle this work? There seems to be only one way, and that is incident management.

Incidents are the key identifier that helps us understand where unplanned worm comes from. Services are planned, and the requests that come with them tend to be planned. However, incidents are like the evil cousin of service requests. Incidents like to come out of the blue, and if there were a family dinner, the incident's goal would be to ruin it once all the food is out.

Incidents are the key indicator of how much technical debt a company is dealing with inside their cloud organization. IT used to deal with mainframes, and with the innovation of cloud computing incorporated, they have been able to transform how IT works in the industry, acting as a major disruption. Instead of hosting on a mainframe, Cloud Computing Incorporated, though, why not handle all of the mainframes for the customer? Why should the customer even have to deal with handling their technology? Isn't that what a provider is for?

Cloud computing incorporated changed the game, and Paul loves going to work every day because of it. The questions that this new field of computing brings us and distributed systems especially are humbling. There are so many unknowns for us to uncover every day, and Paul is glad to be a part of this big unknown.

December 23rd, 2021

A Brief History on Incident Management

And so Paul entered work another day today after walking a long brutal windy winter storm. Paul had always thought IT would have been different growing up. He had never imagined himself in the position now. How could he? Technology is changing so often and so rapidly that the only way to keep up is by reading all day, and even then, you probably would not have all the answers. However, much some of us hope.

Paul had always envisioned himself working at the mainframe side of some big company. Only big companies could afford the hardware to efficiently sustain an IT organization. However, Paul was thrown off to see how wrong he was. The cloud computing disruption and the idea of micro-services led to more mobile and more cost-effective applications. Developers can now deploy applications more readily than ever, and it only takes a few clicks.

Paul had spent so much of his life learning to code only to realize that much of it could be clicked away now. However, that does not take away from a solid programming project though. Paul thinks that is one of the best parts of developing. People often assume developers use a lot of math and science to formalize processes and procedures in autonomous ways.

However, being a developer is much like being an artist. It takes a certain kind of person to get up each morning, ready to tackle a certain set of problems that most likely have no solution. Most solutions are only the most optimal solution we have available, which does not imply it is a good solution. Paul loves how young the field of computing is. It is easy to see veterans of the workforce because of how many "IT revolutions" there were in the past 20 years, let alone 40+. Some people spend their whole lives working on a single problem, and it is a developer's job to systemize that and then some.

Paul has always been inspired by the novel "The Goal" and has loved Gene Kim's teachings. He understands that the revolution of technology makes observing work even more important. Currently, at cloud services incorporated, Paul is dealing with many unplanned work items that hurt him and his practice. Developers spend too long identifying problems rather than identifying strategies to work on these problems.

The revolution of technology has made it easier and harder to manage operations in many ways. The world is readily available to handle problems and search knowledge bases at the tips of their fingers. Nations worldwide are starting to see cloud computing as a solution than a cost. The IT world is changing rapidly again, and Paul can feel it. He is just looking for where.

For a long time, research and development have served as the area of a company's resources, though each needs to carry these attributes. Especially with how fast ideas and theories are changing in distributed systems. Paul had recently read Lamport's paper on time clocks and was surprised that such a novel paper in distributed computing was only released in 19080. It took the world almost 200 years to move from Newtonian physics, and here computer scientists were fighting over how to solve distributed models of computation which is just a fancy form of the theory of relativity for computing.

Problems are inevitable. As my manager always says, "Everyone has a plan until you get punched in the face" though he is not literal, he is still 100% spot on. Problems are inevitable. You cannot implement a perfect procedure on day 0. It is almost impossible unless you happen to be replicating it, and even then, the problems of the cloned system can transfer over. Therefore, the service providers at cloud services incorporated have been investigating what makes a good incident management process. How can someone identify a problem as fast as possible and handle it appropriately? How can someone figure out whom to contact and at what time?

Too many companies today overlook this idea thinking that the time lost calling multiple people on the phone or scrolling through a wiki is negligible. However, they are wrongly mistaken. When one person takes a long time to solve a problem, many people are likely taking a long time to handle the problem. That scales large and fast. The goal of information technology is to provide the best information to our customers and us at the lowest costs optimizing the money made by the company.

The model for cloud services incorporated is one of the best models Paul has seen. It is efficient at smaller sizes and is scaling large and optimally. It is a beauty to see the company grow as it is. However, Paul notices much technical debt starting to accumulate in areas of the company he would not like to see. As the company scales larger and larger, more teams are causing the cloud services company to use multiple systems for a service management tool. Rather than one company managing an organization of projects, each product owner essentially manages their project on their own choice of a service management platform. While this works at an individual scale, this scales terribly across a company.

This debt is not only causing a significant amount of information loss, but teams are more disconnected than ever. Paul had read Sunstein's book talking about how technology is polarizing the country. However, Paul is starting to notice how much technology is driving away areas of communication in a company. Especially when it seems like people are talking more than ever since the pandemic, it almost seems that technology is making it easier than before but harder than ever.

December 24th, 2021

What is an Incident?

Incidents are parts of unplanned work that happen every day, whether we notice it or not. An incident management process is designed to reconcile that problem.

It is December 24th, 2021, Christmas Eve. Paul has had a great weekend and is happy with the way things have been going. He has had his fun and honestly loves to take time and relax. Paul is a fan of silence. He is a thinker. Paul loves to solve problems and learn more. He thinks that education is not a goal but an ongoing process. He had recently left college and often wondered how much more he could learn and was happy to uncover how much more he could still learn.

He only has a few hours to himself on the weekend nights like this and is looking to learn more. He has been dealing with incidents at work long enough and is tired of being blocked by unplanned work. He and his team have been working on an IT service management process that entails how the practice will function in the eyes of planned work, unplanned work, and business work. Though our board does not show it, it is still working to be done. Many shadow operations are going on with the practice, and though it is not bad as a small team, it can be so much better if we just identified ways technology could connect us rather than optimize one process by separating another, such as communications.

By definition, an incident requires a service to be running. Once a service is up and running, some companies may ask Cloud services incorporated to handle this service for them. Cloud services are the company, and great people take the offer and handle the service. However, how much of that service is being provided and when? That is typically agreed on, and with today’s technology, you can get a solid 99% most of the time. Now when a service does go down, we mark that down as an incident for when it is down.

Paul loves technology incidents because it is funny to see the abstract form falling. Essentially what incidents in IT are the equivalent of a robot falling off a ladder. So when a robot is hosting your website and shuts down randomly, the cloud services team marks that down as an incident. This incident reporting procedure is awesome for the company and the client because technology increases communications. Though it is bad to lose service for any time, Cloud Services Incorporated engineers take pride in providing rapid and readily available solutions to handle growing customer needs.

Paul believes that incidents are an awesome way to further practice development. Each incident is a sneak peek at what the system is trying to tell you. Suppose you go to the doctors and try to figure out if WebMD was right about your cough being cancer or if you have a cold. A doctor may perform multiple tests to examine multiple responses from you. Similarly, a system can output symptoms and diagnostics of some cool stuff. Each incident gives us the ability to see a little further into the future.


r/DistributedComputing • • Aug 15 '20

is cloud computiing safe?

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Jul 11 '21

Distributed computing system

Thumbnail itbloggy.com
0 Upvotes

r/DistributedComputing • • Jan 12 '23

Gridcoin was going up while Bitcoin was going down, Why?

3 Upvotes

Anyone have any ideal's

I've been watching GRC since november. Almost sold a bundle in early december for a special grow project i started. Lucky I didn't. I couldn't believe irt started popping in mid december doing the exact opposite of the whole crypto market, now it's still riding high.

I know the volume is low, today it says 60k but there has always been speculation on bot action on the exchanges. It will be interesting to see it it will take a ride with BTC or or what.

Anyone heard any real news on GRC, i'll do some reading over the next week and see if I can find anything. Glad been crunching it for a while now.

https://www.coingecko.com/en/coins/gridcoin-research

All the best in 2023 : Let's go Gridcoin!

SystemViper

XtremeSystems

XS Medical Cannabis Grower Project


r/DistributedComputing • • Mar 26 '26

Reduced p99 latency by 74% in Go - learned something surprising

Thumbnail
reddit.com
0 Upvotes

r/DistributedComputing • • Mar 06 '26

[Bounty] Maintaining Consensus at 10M Nodes: Can you find the flaw in this 55.6% Byzantine-stable architecture? (5 Gold)

0 Upvotes

The Engineering Challenge: Most distributed consensus models (Paxos, Raft, etc.) struggle with high node counts due to quadratic communication overhead. I’ve been stress-testing a decentralized federated learning protocol, the Sovereign Mohawk Protocol, and recently completed a 10M node simulation.

The Result: The network maintained convergence stability with a 55.6% malicious (Byzantine) actor fraction, utilizing a communication reduction of roughly 1,462,857x compared to standard all-to-all broadcast methods.

The Architecture (Theorem 1): The stability is derived from a dAuth Weighted BFT mechanism. Instead of a flat quorum, it uses:

  • Weighted Consensus: Influence is a function of "Node Health" and "Contribution History," governed by a strictly defined Decay Function to prevent long-term centralization.

  • Dissensus Preservation: A unique "Outlier Protection" layer that prevents a 51% majority from pruning valid but rare data paths (vital for Federated Learning).

  • Byzantine Throttling: The SGP-001 Privacy Layer identifies and throttles nodes exhibiting high-entropy "noise" patterns characteristic of Sybil attacks.

The Evidence:

The 15 Gold Bounty: I am awarding 5 Gold each to the first three people who can identify a structural or theoretical flaw in this distributed model:

  1. Partition Tolerance: How does the model handle a "Split Brain" scenario if the SGP-001 throttling creates an accidental network partition?

  2. Convergence Math: Find an inconsistency in the Theorem 1 stability claims regarding the 55.6% threshold.

  3. Liveness vs. Safety: Provide a scenario where the "Dissensus Preservation" layer causes a permanent stall in consensus (Liveness failure).

Is this a scalable solution for global-scale DePIN/AI, or is there a "hidden cliff" I haven't hit yet? Tear the logic apart.


r/DistributedComputing • • 4d ago

Kiri: a tiny, fast container runtime for macOS, Windows and Linux with free tunneling like ngrok(509 stars)

Thumbnail
reddit.com
0 Upvotes

r/DistributedComputing • • May 29 '19

Distributed computing in mobiles

0 Upvotes

Hey all , i wanted to make a network of mobile phones overwhich i could save my files(distributed all over it), and could access it from any of the device. Its just like what Richard (from Silicon Valley) was building, saying the new internet.

How can we make that ?


r/DistributedComputing • • Mar 19 '19

Soon at Cloudfest Cloud computing conference...

Thumbnail
youtube.com
0 Upvotes

r/DistributedComputing • • May 23 '13

Noob question: distributed compiting within a lo-fi video game

0 Upvotes

I'll start by saying that I have virtually no experience with distributed computing, other than having run seti@home for a while. This is a feasibility question for people with the proper experience.

I've been thinking it would be interesting to implement some small-scale distributed computing that could run in the background of a lo-fi or turn-based or point-and-click style game -- something where there is a fair amount of processor downtime. Obviously something where maxing out the fps is not an issue.

Let's say, for example, we download 1MB of unprocessed data for each level. Whenever there is downtime, we process some data in another thread and save the result as raw data in a buffer of some sort -- perhaps an array of "integers" (or integer-length byte strings; not saying that the results are literally integers all the time, but you could interpret any 16 or 32 bit chunk as such). Then, whenever the game needs a random number (to determine attack damage in pokemon for instance), one of these "integers" is pulled from the buffer and interpreted as a percentage (x/255), thereby giving you a semi-random number within any range.

At the end of the level, at the end of the level, we should have a fully processed batch of bytes to upload. After all, the processed data will remain in order and not be manipulated after its actual processing.

Is this feasible? Is it worthwhile? Is there anything like this in existence or any good platform to use? I'd love to have some vague idea of its potentiality before reading a whole book on the subject. I'm open to more discussion on the subject if anyone has thoughts/questions/ideas. Would love to make it an open source project or contribute to one if anyone feels like starting their own.


r/DistributedComputing • • Nov 23 '22

[video] System Design Interview - Consistent Hashing

Thumbnail
youtu.be
0 Upvotes

r/DistributedComputing • • Apr 02 '20

How to use 15 servers, any ideas?

0 Upvotes

We’ve got about fifteen servers at our disposal. We don’t know how to use them. Since we don’t have a public static IP address we have already discarded the webserver. Instead, we turned to grid computing projects. They would be interesting. They’d cost us money, though. we aren’t sure yet. Would you have any ideas to give us? Thank you


r/DistributedComputing • • Apr 23 '22

Formula Boinc Sprint#4 - Team Ukraine takes 2nd in first day

0 Upvotes

Show support for Ukraine - Sprint#4 - Team Ukraine takes 2nd in first day

Sprint#4 - Team Ukraine takes 2nd in first day
Posted on XS also - https://xs4s.org/index.php?topic=30190. ... 1#msg96331

https://ibb.co/album/pBVrBz

Team Ukraine came from 3rd to take 2nd over P3D just missing TSBT moving into day 2.

Team Ukraine is going strong in the FB with an
- Overall Position of 5th
- Marathon Standing of 6th
- 2nd place in the Sprints @ the start of the 4th sprint yesterday - Einstein

https://ibb.co/album/pBVrBz

04/21/2022 21:00 (UTC) - 04/24/2022 20:59 (UTC)
Einstein@home

Thanks for your support
SV

Anyone Can Join Team Ukraine for the sprint

Anyone who feels froggy, could easily join Team Ukraine by dropping your
current and signing up to Ukraine on the Einstein@home site
You can switch teams anytime, then after the sprint you can go back to your MAIN team

Link here - https://einsteinathome.org/community/teams/172302

Day one Finals
https://formula-boinc.org/sprint_v2.py? ... n&sprint=4

Overall Standings
https://formula-boinc.org/index_v2.py?y ... team=&thx=

Thanks to all that helped and to all those are on other projects