r/Clojure • • 6d ago

Two apps one process

https://andersmurphy.com/2026/11/09/two-apps-one-process.html
27 Upvotes

30 comments sorted by

7

u/andersmurphy 6d ago

I'm curious what people think about this part: If a deploy is indistinguishable from network blip did the app really go down?

6

u/bananasdoom 6d ago

> Disclaimer: all my writing and code is handcrafted by me. By the end of this you may wish it wasn't.

Your writing is charming.

Technically; On a principled level I think I agree this is a good approach, but as the other commenter pointed out, too much has been built on the assumption of machine level “dumb” isolation. Feels like one of those endeavours where for the want of shared process you’ll end up rebuilding the world.

2

u/andersmurphy 6d ago

Thank you! Yes there's definitely always a risk of rebuilding the world. But, simple isolation is easy enough, i.e apps can't crash other apps, apps can't accidentally access another apps database (unless you load your own sqlite driver and search the filesystem). So it's good enough for deploying apps you own/trust.

I didn't really go into it in the post as I was saving it for a more in depth write but the game loop based render/write batching gives you a lot of headroom and also makes detecting if the system is at capacity (tick overrun). I cover it a little in this post:

Discrete time based systems are simple to benchmark

I do need to add sampling for more fine grained profiling, but again the game loop model makes that relatively trivial.

2

u/yatadger 6d ago

>> Disclaimer: all my writing and code is handcrafted by me. By the end of this you may wish it wasn't.

> Your writing is charming.

Appreciate the ones who handcraft their craft

4

u/Krackor 6d ago

To address the question more directly, taking your app offline is done under the presumption that the next instance of the app will come up, and do so in a network blip's amount of time. In practice that is not always true, so some health check on the incoming app is prudent before rerouting traffic to it. What happens if some configuration server you depend on during start up is down for an extended period of time? Will you notice before shutting down the outgoing instance? You may need to keep the outgoing instance up until the startup dependencies are available again.

Philosophically, a short network blip is indistinguishable from a short downtime so it may not take away a meaningful 9 of uptime. But there are logistical factors to consider besides the happy path uptime math.

4

u/andersmurphy 6d ago

Oh definitely. This is very much on the spectrum of your app has no external dependencies that can crash it. The database is embedded sqlite etc. There is no configuration server. External feature flag server etc.

The only time it should crash is if backups are not working.

1

u/aHackFromJOS 2d ago

“A deploy is indistinguishable from a network blip” is definitely true if you have an invincible, autarkic app!

1

u/andersmurphy 2h ago

Thanks for teaching me a new word. Autarkic is definitely the goal.

3

u/Krackor 6d ago

Run N copies of the uberapp and blue green deploy. It depends on your use case but imo actual downtime between deployments would be unacceptable for most things I build for a paycheck. And if an app can't tolerate running concurrently, correctly, with another copy of itself or the prior outgoing version, is it a real app in the first place?

1

u/andersmurphy 6d ago

What things have you built where it was unacceptable?

In my experience most blue green deploys do lead to some level of degradation and/or anomalies. The added complexity often makes the system more fragile as there are more interdependent components.

A warm standby with a manual failover (avoid split brain) has always felt like the more reliable approach to me.

2

u/Krackor 5d ago

Microservices handling activity from public users. If a request chain is inflight I don't want to drop it on the floor by shutting down the outgoing instances until the request chain has had time to complete. I also don't necessarily trust the entire system to have good process durability and retry discipline so it's valuable for downstream services to be as reliable and prompt as possible. 

There are ways to design around these problems with a different architecture but one doesn't always have control over the holistic system design, and the design choices needed to make the "stop the world" deployment strategy work wouldn't necessarily make sense in a large enough system with a large enough team.

3

u/Liistrad 6d ago

Network blip is one of the effects, but there's more, especially around state like user sessions or caches that can be ephemeral. REPL sessions also come to mind.

I noticed your deploy process was uberjar + scp. Have you considered/tried not even doing uberjar? I've been toying with the idea of very long lived clojure processes that reload their code and it sorta sounds possible? Deps can now be reloaded at runtime, and source has been reloadable for a while.

https://github.com/filipesilva/invoker#deploy is an example of I'm talking about. Maybe it is fine to just push the source and reload the app.

2

u/andersmurphy 6d ago

Thanks for the link I'd check it out!

Yeah, I've been somewhat hand-wavy. The apps use datastar and have no client side state (other than animations/user input). All app state is in sqlite, so there's nothing ephemeral on the server that isn't transparent. I.e if a user sees a UI change it's been fsync'ed to disk. So there's nothin to lose on restart.

I do occasionally use the repl to patch a running jar. But, those changes don't persist when you restart.

I've considered not using uberjars and having source files on the server I think Biff does this. My problem is that reloaded workflows often break in weird wonderful ways.

I am considering exploring storing app code in the database and loading it from there at runtime.

3

u/jacobobryant 5d ago

Yeah, with biff you can do `clj -M:run deploy --soft` to eval files without restarting the process or any stateful components, and then omitting `--soft` does a process restart. it's kinda nifty. Took a bunch of work to make sure as much stuff as possible was using late binding.

2

u/whoisarepo 5d ago

That’s what I wondered, what about data assurance? For my style programs this is very useful but for multi tenant…it may work but needs a framework to handle what SoC did naturally, create boundaries between error types. It seems harder to diagnose now which app failed what and why without some telemetry service layer…which is all fine tbph but it’s the tip of the iceberg of a good idea…

3

u/beders 6d ago

Does anyone remember JAR hell and it’s „solution“ OSGi?
We certainly don’t want to repeat that.

I think the approach above is fine if you have full control over your apps.

3

u/andersmurphy 6d ago

Haha, yeah it was partly inspired by OSGi and servelts. This is definitely only for when you have full control over your apps. There's no sandboxing here other than avoiding making it easy to accidentally mix app data (handlers only get their database passed in etc).

2

u/fk00 6d ago

What happens if one app crashes. What happens if it keeps crashing, or, worse, goes into endless loop, consuming all available resources? I'm not talking JVM is particularly good at working in a containerized environment, but there are practices there you can follow.

Saving resources is a good thing, but when monolith goes down, you loose all apps.

2

u/andersmurphy 6d ago

Apps don't crash other apps. The system is relatively draconian in terms of what/where they can run. Which makes monitoring trivial and opens up interesting options like canceling work by misbehaving apps to protect the wider system.

For example you could, if you wanted to set a max render time for a given render-fn (how the apps implement views) and either log or cancel. Same for the transaction thunks scheduled on the db.

For the record I'm not arguing against having separate boxes for different apps. This is more about when you have a single machine, with highly variable app resource usage/traffic. If all apps were easy to size correctly then a bigger box with separate processes and a reverse proxy is fine.

1

u/fk00 6d ago

So what happens when this monolith exhausts RAM?

3

u/andersmurphy 6d ago edited 6d ago

What happens when any system exhausts its resources? The process dies and restarts and the loop continues.

The same applies to a multiple node set up too. Except it's more complex.

0

u/fk00 6d ago

So you bring down all apps because of one bug. Good luck selling this to your customers.

4

u/didibus 6d ago

Just rollback, debug, restart. Honestly many services even if deployed on their own machine or container often break or have downtime, and cloud providers go down, etc. You have to think about your use case.

2

u/fk00 6d ago

Right, so when one app suffers from the bug, you rollback the entire set. And imagine your customer can suffer from inconvenience of missing access to one feature, but you suddenly revert the entire monolith. Not a great approach.

1

u/didibus 5d ago edited 5d ago

It looked to me from the post you can selectively rollback just one app. There's a project with versions per-app, so you can rollback the version of the app with issues and redeploy, rolling back only the app with issues and not the others.

It's not always going to be what you want, but I think it's an interesting approach when you control all apps and want to really minimize infra cost and have something simple to manage.

2

u/whoisarepo 5d ago

I like this! I use babashka sub processes for my clojuredart and clojure script projects, if this technique translates well then I can simplify some things, good work! I wonder how it handles data corruption from interruption between multiple writers, feels like the orchestrator eats the bulk of abstraction to make it feel magical

2

u/andersmurphy 5d ago

Not sure what you mean about corruption. Each app has it's own sqlite and batches to them separately. All apps implement CQS, so writes are async so the can be batched. You only see the changes in the next tick.

2

u/whoisarepo 4d ago

This makes a bit more sense, like the mechanics of a LAN Game server. It’s just…weird lol, but cool. And makes me wonder why haven’t more polylithic projects attempted unifying this way as it’s not clj specific

2

u/geokon 5d ago edited 5d ago

I always thought this would be the natural way to do things. Having to dance around the OS abstractions seems like an unnecessary straightjacket.

I don't work in the webspace, but is this effectively working through loopback having VM threads communicate that way?

You could probably reach into each others namespaces and pull at atoms, but this probably will lead to people taking shortcuts and making coupled code. Maybe that's what you've solved - and present a structured way to do interprocess communication (sorry, I couldn't quite figure it out from the Github)

Hopefully in the end you're gunna be copying memory buffers all over the place for no good reason (which is what the OS, loopback, and other isolating abstractions end up forcing you to do)

1

u/andersmurphy 5d ago

The apps are separate projects that can be run and developed on their own, so you have an element of decoupling there.

There's no inter app communication. The framework handles the scheduling. It's basically a simple game loop that batches writes and renders.

Not sandboxing though, so if you really wanted to you could have one app reach into another app, but that would be a bad idea so I've tried to at least make it hard to do.