r/DistributedComputing • u/dylan_dev • Oct 17 '11
I don't want to particpate in Folding@Home since the client isn't open source
Security via obscurity is the farthest from secure and I think the community is missing out on innovative clients.
r/DistributedComputing • u/dylan_dev • Oct 17 '11
Security via obscurity is the farthest from secure and I think the community is missing out on innovative clients.
r/DistributedComputing • u/DisastrousNinja911 • Jun 19 '26
A bit of background about me.
“I have about 4 years of experience primarily working on traditional software development building enterprise apps. While the career path is stable now ( don't know if AI takes over), I find myself deeply interested in systems architecture, low-level optimization, and parallel computing. However, giving up a full-time income to go back to school feels like a massive investment. Is a Master's degree the right vehicle for this transition, or are there better ways to break into the industry? I'd love to hear from anyone working in HPC, climate modeling, quantitative finance, or scientific computing."
All I want is to future proof my career.
r/DistributedComputing • u/No_Peach_8990 • Jun 09 '26
Over the last 2 years I've been building Atomic, a Spark‑inspired distributed data processing framework written entirely in stable Rust. The goal is to keep the parts of Spark that are great (lazy DAGs, shuffles, distributed execution) but re‑imagine them with modern infra and language design.
A few things that make Atomic different:
Rust core: Strong typing, predictable performance, and memory safety by default. You get a real systems‑level engine, not a JVM box you bolt on next to your stack.
Natural‑language workflows: On top of the engine, Atomic is designed to be driven by natural‑language workflows – letting you describe what you want done and compile that into a typed DAG, instead of hand‑wiring every pipeline.
Multi‑language support: Rust is the "ground truth" but the plan is first‑class bindings for Python and JavaScript, so you can drive Atomic from the languages your data and app teams already use.
No closure serialization: Instead of shipping arbitrary closures across the wire, tasks are registered at compile time via a #[task] macro and dispatched by ID. Driver and workers run the same binary, so the dispatch table is identical on every node — no serialization failures, no version skew surprises.
This is my attempt to build modern infrastructure tooling for data processing. It initially started as a learning project and now I feel it has evolved into something significant to be delivered. I'd love to hear your feedback:
r/DistributedComputing • u/mpetryshyn1 • Mar 19 '26
So, are we due for a 'vibe DevOps' or am I dreaming? Tools can spit out frontend and backend code in minutes, which still blows my mind. But deployments fall apart once you go past prototypes or simple CRUD - everything gets manual and ugly. I see people shipping fast, then stuck doing manual DevOps, or rewriting the whole app just to make it deploy on AWS/Azure/Render/DigitalOcean. Imagine a web app or VS Code extension where you point it at your repo or drop a zip and it actually understands your code and requirements. It would wire up CI/CD, containers, scaling, infra setup using your own cloud accounts, not lock you into platform tricks. Seems like it could bridge the gap between vibe coding and real production apps, but maybe I'm missing something obvious. How are you handling deployments today? scripts, Terraform, stuff like that? Curious what people actually use and what fails.
r/DistributedComputing • u/rootseat • May 08 '22
Been reading up on distributed systems theory a bit lately and had a practical question.
Reasoning about even a single mutex in a multi-threaded application is non-trivial. A bug might not show up in the debugger until the 1e10th test. Getting users to adopt even the simplest software is, more often than not, a reason for jaded complaints than interest; builders of those systems joke about how messy implementations are. And distributed applications seem to just scale up the same problems as much as they do the solutions: multi-level coherence, extra readability/debugging overhead, etc.
All that out of the way:
How does a developer get the confidence that the distributed system can be shipped without problems? Is it kind of a swiss cheese defense of static analysis, trust in the namebrands of middleware, etc.? Or is it that distributed applications aren't built from the ground up, but rather build around exported middleware promises from the get-go?
How does one get an initial userbase to have confidence in your application, and adopt it, the extreme case in being distributed applications that require nuanced privacy/protection policies?
r/DistributedComputing • u/give_me_fucking_name • Feb 21 '22
I am taking Distributed Computing as a course in my CS program. I was wondering of there is any good playlist I can follow ?
r/DistributedComputing • u/ggvh • Jun 01 '21
r/DistributedComputing • u/EgaoNoGenki-II • May 10 '13
I used to use "Folding@Home" and "World Community Grid" a long time ago, but software changes, sometimes can't run on an updated OS, and many of other life's variables, so I'm not sure that they're still the best.
But if you know of any grid computing project that'll let me donate my laptop's idle-time to an awesome cause, what would it be, why do you think it's awesome, and what's the link to download its client?
Thanks in advance.
r/DistributedComputing • u/G-M • Feb 19 '10
r/DistributedComputing • u/G-M • Jan 15 '10
Join! It's winter for most of us so there is no excuse. Ask here for help, no matter how simple.
Currently placed ~1300th with over 2.3 million points
r/DistributedComputing • u/Realistic-Face1315 • Mar 05 '26
Hi everyone,
I’m a student who’s recently become really interested in distributed computing and large-scale systems. I’d like to eventually understand how systems like distributed storage, fault-tolerant services, and large-scale infrastructure work.
Right now my programming experience is mostly in general software development, and I’m comfortable with basic programming concepts. However, I don’t have a clear roadmap for getting into distributed systems.
Some things I’m wondering:
• What fundamental topics should I learn first? (e.g., networking, operating systems, concurrency, etc.)
• Are there specific books, papers, or courses you would recommend for beginners?
• Are there small projects that help in understanding distributed systems practically?
• Is it better to first build strong foundations in systems programming before diving into distributed computing?
My goal is to eventually build and understand systems like distributed storage or decentralized infrastructure, but I want to make sure I’m learning things in the right order.
Any guidance or resources would be greatly appreciated.
Thanks!
r/DistributedComputing • u/Vw-Bee5498 • Dec 01 '24
Hi folks, I know that Zookeeper has been dropped from Kafka, but I wonder if it's been used in other applications or use cases? Or is it obsolete already? Thanks in advance.
r/DistributedComputing • u/jordepic • Mar 30 '22
I graduated college this past December and don't start work at Google until June, as a result I've been studying systems design and reading the resources everyone here likes (DDIA, systems design primer, grokking).
I created a study guide, which as of now is around 30 pages, and wanted to share it with all of you. I find that the primer, which itself is supposed to be a study guide, is not nearly in depth enough, and I hope that this is much better than that while still not being overwhelming.
As of now, I only have comments on, but if there are any topics to add, issues with what I wrote, or any changes you want, feel free to make them.
Link: Tinyurl.com/2p95wz9s
Have a great day!
r/DistributedComputing • u/A27_97 • Nov 27 '21
Hi I’m interested in Distributed Systems research, more inclined to applications in Machine Learning, Robotics. Do you guys have recommendations for folks in this field to follow (in general) and for ex on twitter
r/DistributedComputing • u/satoshigekkouga2309 • Sep 27 '21
Hello! Are there any papers that you'd recommend for understanding how distributed message queues work? It'd be nice if the paper provides some pseudocode as well. Cheers!
r/DistributedComputing • u/mgalarny • Apr 13 '21
r/DistributedComputing • u/binaryfor • Oct 26 '20
r/DistributedComputing • u/ra-yokai • Jul 11 '20
Hi. I have no knowledge about distributed systems but I've recently joined a team that uses DynamoDB and a new scary (to me) world unfolded in front of me. My teammates keep telling me that Dynamo is an eventually consistent data store but I'm not confident I really know what that means. I have been jumping from resource to resource trying to really understand what makes Dynamo different from, say, Postgres but I can't say for sure that my understanding is correct.
I never had to scale a relational (is this the correct term?) database before as well, so this might be something I should try to do.
In very simple terms, and knowing that things are more complex than that, would it be correct to say the following:
Speaking about consistency (in the context of databases) only makes sense when he have replicas;
Writes always go through a master node that then replicates the data to the other nodes;
Postgres/MySQL keeps its strong consistency because it's master node writes to the other nodes before deciding that the write succeed (problems: higher latency and non partition tolerant) - it's an all or nothing behaviour;
Dynamo's master node sends a response back without waiting for the other nodes and replication happens later with the aid of an algorithm like Paxos or Raft.
It might be out of scope, but any resource recommendations, specially with exercises (I can't learn properly without creating something myself, but I tend to overcomplicate when I create my own exercises - creating good exercises is a fantastic skill that I miss) would be very much appreciated.
Thank you very much for you patience and help.
r/DistributedComputing • u/ExperimentalMeatBag • Jul 06 '19
I want to build a system where I can keep plugging android phones into a single system, which can utilise their shared processing power, RAM and storage space, and act as a linux server of sorts, even if for just mathematical data crunching.
I know it is possible, Ubispark is doing something similar, can someone give me a roadmap of things I would have to learn to accomplish this ? I am working as programmer/sw architect from last 10 years.
r/DistributedComputing • u/lukepighetti • Jun 11 '18
Hey all,
I noticed that Kaggle exists for machine learning, and boy does it look like fun. I'm personally less interested in machine learning and more interested in distributed computing. Is there anything like Kaggle for distributed computing?
r/DistributedComputing • u/[deleted] • Nov 16 '11
r/DistributedComputing • u/nilukush • Sep 18 '26
Netflix has reworked its Conductor workflow orchestration engine to address growing scale across the company. Conductor now supports roughly 200,000 workflow definitions across 150 applications and executes about 420 million workflows per month. The redesign increases the supported workflow size from approximately 2,500 to 30,000 tasks while reducing p99 workflow evaluation latency by about 40%.
r/DistributedComputing • u/wufuheng • Jul 12 '26
I recently published a technical book, Distributed AI Systems, which summarizes my experiences in AI over the past 10 years, from research and training to optimization, inference, and cloud deployment. I started writing it in the second half of last year, and it took almost a year to complete, with many revisions made later due to the rapid pace of development in the industry. But it's finally published. The book on Amazon is titled Distributed AI Systems: A practical guide to building scalable training, inference, and serving systems for production AI.
Book is here: 🔗 https://www.amazon.com/dp/1807301710/
The publisher asked me to find some people to review my work. Do you know of any such people here? If so, please reply to me. Thank you.
