r/javahelp • • Mar 25 '20

Advice for a java high performance project

Hi everyone, I'm looking for some tips on designing a performance constrainted system. It should handle 1k+ request per second and have response time under half second. The specification explicitely says it should be spring based and will listen on an existing kafka stream. I thought of using a webflux listener and pair it with a reactive mongo persistence layer, since it has to keep track of aggregated counters such as message per hour and so on. One thing I'm doubtful is, will concurrent updates be an issue with reactive mongo driver? What else can be done aside from boosting cpu and ram to improve performance on such a component? I've been developing for quite some years but a performance constrainted project is a first timer for me. Any help is appreciated, cheers!

20 Upvotes

29 comments sorted by

8

u/BestUsernameLeft Mar 25 '20
  1. Use SQL. Seriously. MongoDB has its place, but the SQL world is still the most complete and feature-rich ecosystem. Grab you a PostgreSQL or MariaDB and your favorite ORM tooling, and fuhgeddaboutit because it just works.

  2. Yes, reactive/webflux is the way to go here.

  3. Optimize for performance, but only after you get the thing working right. However, measure your throughput every sprint -- part of "done" for a sprint needs to be "meets performance requirements". It's okay if things get a little out of hand at some point -- but if you're partway through and the trend doesn't look good, it's time to take stock and assess the situation.

  4. Don't forget about the JVM! Tweak your garbage collector, memory, etc.

  5. You don't mention anything about the runtime environment (physical machine, kubernetes, ??) but obviously that's a factor as well. Prefer not to deploy onto that old laptop some contractor left behind a couple years ago :).

7

u/surfinThruLyfe Mar 25 '20

Tweaking your GC is the 20th step in architecting a performant Java enterprise system.

1

u/realqmaster Mar 25 '20

Thanks! About Mongo, the data model doesn't really need it in my opinion, I went there since I heard relational SQL drivers are still somewhat not standardized, but I also heard pgsql has some option.

2

u/slowfly1st Mar 25 '20

Considering performance, IMHO there shouldn't be a difference between mongodb and any sql server - both will have the same issues when you have 1k requests that want the same write lock ...

SQL Drivers are specified in the JDBC specs, and should be standardized to a large extent. The only thing I once saw was that a firebird driver that didn't implement a certain timeout and always defaulted to 60 seconds.

1

u/realqmaster Mar 25 '20

I was referring to reactive drivers in this case. I assumed jdbc is blocking?

1

u/slowfly1st Mar 25 '20

Aye, yes, you're right. JDBC itself is blocking, there's things in the pipeline (e.g. Oracle has an ADBA driver afaik), but this will probably take some time...

1

u/BestUsernameLeft Mar 25 '20

Yes, there are still differences between SQL drivers. If that's a real concern (and, actually, even if it isn't) check out http://www.jooq.org/.

3

u/slowfly1st Mar 25 '20
  1. Lichess.org runs on reactive mongo db. 13'400 players in game. It probably has its place, yes...

1

u/BestUsernameLeft Mar 25 '20

MongoDB might be a good solution for OP if they need a document-style database with a flexible schema and ad-hoc queries. Most problems are still a good fit for SQL. As always, choose the right tool for the problem at hand.

3

u/slowfly1st Mar 25 '20

Half a second 95% of the time or 99.9% of the time? How much uptime?

There's probably not a one size fits all solution, my best advice would be to measure. (Site Reliability Engineering - Monitoring)

At first glance, 1k requests per second doesn't sound like that much, but it also depends on what you actually do with the data.

1

u/realqmaster Mar 25 '20

It's probably 99% rather than 95. The data has to be evaluated on criteria based on some current aggregated stats about messages themselves, say how many messages of that type ever arrived.

2

u/slowfly1st Mar 25 '20

This has to be specified. If you assume 99% but 99.99% is expected, you'll have a hard time. For our critical systems, we have gradual SLA's, say 98% in 0.5s and 99% in 2s. Sometimes high expected up times (which means deployments w/o down times), sometimes with maintenance hours.

If I understand you correctly, it's more like a monitoring system? Can you elaborate?

1

u/realqmaster Mar 25 '20

Yes this is a valid question I'll ask the committer. This system should monitor user events and send messages if certain conditions are met. The SLA is because the system sourcing these events waits up to half second to acknowledge them, and some of the response from this system can prevent the authorization.

1

u/Taster001 Mar 25 '20

If you'll need some help with boosting your CPU and RAM, feel free to ask me.

1

u/DeliveryNinja JPain Mar 25 '20

I build similar systems with high throughput requirements. I see people are recommending not to use MongoDB but I haven't found that to be my constraint. We are currently building on a Spring/Akka stack and it works very well for scaling and performance. Of course the biggest blocker to you reaching your required throughput is going to be your data and your business logic.

1

u/predrag_b Mar 25 '20

go with cassandra instead of mongodb

0

u/MoreCowbellMofo Mar 25 '20

If you can scale across many nodes your application just needs to be stateless I.e don’t maintain state in the system. Any state that needs to persist should be held in your db.

Pick your version of java. Java 14 has some performance improvements as I understand. As others have said set the relevant memory settings -Xmx, and garbage collector settings.

Report metrics to an external system rather than using an agent running alongside your system.

Use indexes to select data from the db.

As the other poster said, first get things right, then measure response times between your application and the external dependencies to identify the bottlenecks. Then you can optimise.

I’ve worked with mongo. Never experienced any major problems with it so long as you have good indexes in place (as with other database systems)

Import single items rather than whole packages as it will keep the memory footprint smaller and so it will run faster.

If you need faster access to the db you can have a single database you write to and replicas that are read only. Another strategy used in enterprise set ups.

A cheap(ish) way to improve performance is to cache data in the app on a read or write operation. There are commercial solutions for this like Redis.

Batching data up to be processed can help if the requests come through in sudden bursts (spikey). E.g trading systems. I actually recall working somewhere and being told something along the lines of, “batching requests up can be faster than making 100 individual requests”, say. So you can have adaptive processing when it offers an improvement.

1

u/realqmaster Mar 25 '20

Thanks for all the tips. One thing about caching: since I must evaluate on live stats updated by the messages themselves, wouldn't cache make it harder to achieve certainty that the data is properly refreshed?

1

u/slowfly1st Mar 25 '20

You can do "write-through" caches with active expiration... But if you want to make that scale (running several instances), you'll end up with a clustered cache (yikes), and honestly, I'm not sure if you can guarantee live data like this.

And you add lots of complexity (update what cache on what write). And you end up with requests that read from cache, and requests, that wait for an updated cache.

Anyhow, I wouldn't look at caches until you can't keep up with the SLA - what we did in one of our projects is to remove most of the caches, because people thought they were "a good thing" back then, but had low cache hit ratios and didn't really improve response times. ~5ms of 1s is 0.5% of the overall response time, that's wasn't worth the complexity...

1

u/realqmaster Mar 25 '20

Thanks, will keep that in mind!

1

u/MoreCowbellMofo Mar 27 '20 edited Mar 27 '20

Just reading up about something for work and came across this Quasar library - by parallel universe. It is used by a blockchain company, Corda, (which I'm currently learning about) and it explains how using its version of threads (called Fibers) you can get performance an order of magnitude faster than typical with threads. Code is cleaner, less 'callback hell' etc. https://www.youtube.com/watch?v=9GlTc59XctE. It was a feature for Java 9 - I'm trying to find out what state its currently in but it looks really useful.

We use continuations for the following reasons:

It allows us to write code that is free of callbacks, that looks like ordinary sequential code.

A suspended continuation takes far less memory than a suspended thread. It can be as low as a few hundred bytes. In contrast a suspended Java thread stack can easily be 1mb in size.

It frees the developer from thinking (much) about persistence and serialisation.

1

u/MoreCowbellMofo Mar 25 '20

You can read in a table of data at startup if you need a live representation of something and update the cache + db for writes as new data enters. If you don't want to do a (costly) read at startup, you can store unknown data in the cache. Typically this process involves having a (Hash)Map that stores objects by their ID. As you read a value, you check its presence in the Map, if its not available you do a full read from the DB, store the value in your Map, then return it to the user. The only time this fails is if you scale across multiple nodes... then you could consider returning old/bad data, as a trade off for performance. It depends on your application requirements. For a message-board app its fine. For an application that needs accurate information (trading) it is bad. You can have endpoints that will wipe the cache's if they become stale and cause them to refresh also if you want some control without restarting the application.

1

u/daCrimsonSnasher Mar 25 '20

I would recommend foregoing a cache to begin with. It can always be added with minimum fuss later on. You're right - cache updates take a non-trivial amount of work, especially since you have to get live stats, it doesn't make much sense either unless the live stats have a substantial delay.

1

u/BestUsernameLeft Mar 25 '20

Yes.

There's an old saying: "There are two hard things in computer programming: naming things, and cache invalidation." Both of these things are deceptively and treacherously simple, superficially.

4

u/Kambz22 Intermediate Brewer Mar 25 '20

Import single items rather than whole packages as it will keep the memory footprint smaller and so it will run faster.

Please correct me if I'm wrong, but I think this is incorrect. The compiler will use the full class name at compile time. All imports do is say which ones will be referenced. You can import *; and have no hit on performance.

The reason it is bad practice to say import one.*; is because if one has a class called CoolClass and package two has a class called CoolClass, it could get confusing on what's referenced.

I haven't worked with other OOP languages in a while, so this may be unique to Java.

1

u/MoreCowbellMofo Mar 25 '20

You are indeed correct. I failed to check that when someone told me it a few yrs ago. * imports only affect compile time not run time performance and the main reason for importing one item rather than multiple is to avoid namespace conflicts. Other than that its considered cleaner/less ambiguous what the intent of the code is if you avoid * imports.

1

u/BestUsernameLeft Mar 25 '20

It's been a few years since I've done anything with Mongo, is the tooling around it better these days?

Regarding caching -- if you mean "on disk" caching I agree. In-memory caching is begging for trouble unless you can ignore what happens when the app crashes.

I haven't done batching in forever but that's a good point.

-1

u/[deleted] Mar 25 '20

Use Rust or C++.

2

u/realqmaster Mar 25 '20

Not applicable, it's with Java by specification.