r/java • u/prisonbird • Dec 19 '22
Java streams are black magic !
damn, they are really changed my life.
i was trying to run some custom logic on every line on some big CSV files. tried python dataframes, multiprocessing in python, pyspark. none of them did it in an acceptable time.
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
more people should know about streams.
stream operations are more intuitive than anything i tried. i can make a lot of analysis on our data in record times thanks to streams.
i dont know how java does this.
10
u/whyNadorp Dec 19 '22
python df are so popular but as usual with python they’ve a messy interface which lives in its own world and can’t be understood without reading abstruse docs and trial and error. good luck maintaining pandas projects.
3
u/zappini Dec 20 '22
I had to fiddle with some data pipeline stuff. Python, numpy, whatever. I was shocked at how awful that language, stack, ecosystem are.
I'm not crazy about (JDBC's) ResultSet and RowSet. But holy cow DataFrame and whatnot make them look like paragons of architectural beatitude. Really, the Python APIs are laughably bad.
I don't understand why RowSet-based (or successor) hasn't emerged as an alternative.
3
u/whyNadorp Dec 20 '22
another very bad thing for me is you can deploy code with syntax errors and realize there’s an error only when that part of the code is used in production. there’re tools that check the code and compile it, but then what’s the point of python if you have to compile it.
also there’s no standard way of dealing with dependencies. there’s pipenv, poetry, etc. try to up a little your python version and you can end up having nasty problems.
3
u/zappini Dec 20 '22
no standard way of dealing with dependencies
IKR? And then switching between projects done by diff pythonistas, who all have their own preferred package manager, borks your system.
All python projects should just start as dockerfiles, or equiv.
1
u/prisonbird Dec 19 '22
100% agree. you have to learn a lot of things, master a lot of things to get any meaningful job done in python world.
on the other hand, i am not a java developer. i learned java in school and just used it as as hobbyist in high school but i was able to develop an app for my needs in couple days.
and with quarkus i will be able to make it a web service.
for example: quarkus comes with its own dockerfile for build and deploy. this seems small but i think i spend more than 20 hours in last year alone for dockerfiles in our python projects.
6
296
u/audioen Dec 19 '22
Java is probably just 10 times faster than Python. Nothing to do with streams per se.
52
u/TurbulentSocks Dec 19 '22 edited Dec 19 '22
Python can be performant - provided you aren't actually using python for anything. That means using libraries that delegate out to C/C++ (and maybe Rust lately), and being careful never actually to do any intensive computation in the python.
That's often non-trivial to get right, and the learning curve for things like PySpark (and using it in the aforementioned way) is much higher, and can be harder to maintain. But that doesn't mean Python can't be used for performant applications.
13
u/penguuuuuuuuu Dec 19 '22 edited Dec 19 '22
Man, the amount of super shallow fanboy style replies to your comment is a bit sad. I expected better of r/java.
6
u/prisonbird Dec 19 '22
i think what you are trying to say is: if you write your application on python and you just need a little part of it to be performant you can just write that part using the help of c++ without leaving python ecosystem/tooling etc. which is an amazing idea in my humble opinion.
if i knew how to do it i might have gone down that road
6
u/TurbulentSocks Dec 19 '22
Many of the popular python libraries do exactly this without telling you. Optimising python code often means making sure you don't 'leave' those libraries at any point (passing the data back and forth to the lower-level code and python can be slow).
1
u/stefanos-ak Dec 19 '22
It would be much easier and more performant in Java. (Also less languages, which is always a plus).
This is because of runtime optimizations that the JVM does, in "hot" code spots (hence the name jvm hotspot).
The difference with C/C++ optimizations is when they run. Compile time vs runtime. At runtime the JVM has some advantages, because more information is available for how and what exactly to optimize.
Java code can reach some ridiculous speeds... The best part is that it's also very easy to write, compared to C.
The only downside is what's called "cold start". It basically behaves MUCH worse than anything else, before the hotspot optimizer kicks in. But in cases like yours, it should kick-in pretty fast, maybe even after the first loop.
2
u/slaymaker1907 Dec 19 '22
Have you ever tried to use a JVM library with a native dependency? It’s a giant mess compared to python which has special functionality in pip to make using native code easier.
Also, C and C++ and do often do use profiling, it’s just kind of messy and often not worth the trouble.
1
u/prisonbird Dec 19 '22
but i think you would need a lot less native dependencies when using java
1
u/slaymaker1907 Dec 19 '22
That’s true, but there are absolutely cases where native code is useful/requires. For example, using GPU accelerated libs or even something as simple as using SQLite.
3
u/impune_pl Dec 19 '22
There is a project under way, aiming to fix that problem in Java https://openjdk.org/projects/panama/
2
u/slaymaker1907 Dec 19 '22
While important, I don’t think Panama is looking at things from the tooling side. That would really be on the Maven/Gradle side of things.
2
u/KarnuRarnu Dec 20 '22
I didn't, its par for the course. You cannot have a balanced discussion here, where you suggest that maybe another language has some slight advantages in certain scenarios. It must always be java java java. (and imo this is symptomatic and probably why the language is behind in quite a few areas)
63
u/wildjokers Dec 19 '22
But that doesn't mean Python can't be used for performant applications.
So you are saying python is performant by calling out to c/c++ libraries. Doesn't that mean python is in fact not performant?
41
17
u/TurbulentSocks Dec 19 '22
I'm saying it depends on how you use python (e.g. as an orchestra versus as an engine). You're welcome to derive a semantic 'win' against my post, but I mentioned it to provide context and inform, not to score points.
-6
u/popasmuerf Dec 19 '22
...it's not a matter of mere semantics when you just offered that under-performing Python can be performant by writing out your Python tooling in C++ while trying to imply that Python is performant.
9
u/BlakeH1984 Dec 19 '22
I think you're missing the point majorly. What he means is it's not a competition. Sometimes people truly do just want to share information. It's a shame most of us are too busy playing the dopamine roulette to engage.
9
u/TurbulentSocks Dec 19 '22
One of the reasons python is so popular is that for many domains, the tooling is already written. Python might be slower than Java, but the performance of e.g. some complicated ML model is going to be 'good enough' when calling C++ backed libraries in python, and those libraries often don't even exist in Java.
4
u/UnGauchoCualquiera Dec 20 '22
Adding to your point. Python is mostly plumbing that allows to declaratively program what you want the high performance libs to do while they do all the magic in the background.
2
u/barbaneigro Dec 20 '22 edited Dec 20 '22
You are comparing oranges, bananas and apples, dear.
Python, C and C++ are all languages meant to do different tasks.
It ain't fair, at all.
Python, as said, is usually meant to be used as a glue to all sort of routines.
Of course there is more that can be done with it, but in that use-case that all it does.
2
u/Puzzled-Bananas Dec 19 '22
Python the language can be interpreted very efficiently, CPython the interpreter is slow. You can run Python on GraalVM, too.
2
1
-2
u/slaymaker1907 Dec 19 '22
Partially, but it helps that python has very performant ways to interact with native code.
1
u/_1aM Dec 26 '22
Really python and native code? 😂 Dude Java was the reason we are able to use Remote controllers today.
0
Dec 19 '22
[deleted]
-12
u/popasmuerf Dec 19 '22
Not the same thing and you know it. Writing your tooling in C++ in order to make Python "performant" isn't at all the same thing as using BASH to script out some order of execution.
3
u/Sensi1093 Dec 19 '22
Also worth to mention that PySpark is just an orchestrator for Spark (which is implemented mainly in Java) itself.
It was likely used in a wrong way when doing this comparison
12
u/roberp81 Dec 19 '22
you are telling us why python is not performant lol
why use pyspark if spark is made on Java
5
u/2Insaiyan Dec 19 '22
A lot of people writing Spark transformations are data scientists, not developers. Python is already the standard in that field
-5
u/roberp81 Dec 19 '22
if python is not for developers, why a developer will use it?
is like a chef cooking fastfood at macdonals.
4
3
-4
u/popasmuerf Dec 19 '22
Or....you could just dispense with trying to optimize a slow as fsck scripting language via finding/writing-your-own C++ modules and just go with ultra-peformant, and scalable Java.
27
u/CubsThisYear Dec 19 '22
Java is usually closer to 100 times faster than Python, but it depends on the workload
2
u/HawkInevitable5733 Dec 20 '22
I would agree, but also add if a task is Time critical, much can be achieved via Garbage Collection tuning. The default is great in most cases, but it can be helpful for certain profiles, if done scientifically
13
u/agentoutlier Dec 19 '22
Also given that Spark runs on the JVM I have to imagine the streaming to a python slows it down as well where as JVM code would not need that.
Also I don't know spark but I was under the impression its Streams were non blocking (aka reactive) so using regular Java Streams may just be an accidental boost because of the above and not actually recommended (java streams are not inherently non blocking since termination is a common thing).
1
Dec 19 '22
Also java has actual parallel processing. Python global interpreter lock dosent really achieve true parallelism in most cases
5
u/danielaveryj Dec 19 '22 edited Dec 19 '22
I agree with the other comments here- the performance difference is probably mostly python vs java, and not java streams specifically. Java streams are cool though. If you inline the abstractions, the logic is pretty much just what you'd write by hand with loops, if-else, etc, but factored in a way that is amenable to parallelization (by splitting the source into chunks, and iterating each chunk on a different thread).
Just to throw another lib into the pool of "might be interesting to look at", based on what you've mentioned in this thread about joining & grouping: https://github.com/davery22/vinyl (disclaimer: author). I don't recommend use in production yet though.
1
3
u/DuneBug Dec 19 '22
Java should run faster but if python was that slow something was probably not setup correctly. Of course, if you have to read 10 pages of documentation vs adding .parallel, that's a good reason to use Java too.
Unfortunately for a lot of enterprise stuff we don't get to use parallel streams as the threads are typically occupied elsewhere. :(
1
u/prisonbird Dec 19 '22
are threads something expensive in enterprise world?
i fire bunch of dedicated machines on hetzner.com and run my computations there. compared to clouds they are pretty cheap
1
u/DuneBug Dec 19 '22
They're only expensive when your app is about to crash from the load. So maybe 1% of the time. Which unfortunately is what you'd really like to avoid.
TBH we could do some optimizations to make it work but it's almost never worth it in dev/testing/debug time to shave time off a request... Unless someone asks for it.
Obviously depends on what the service you're providing is.
2
u/prisonbird Dec 19 '22
ohh i understand. considering how much computing power enterprises buying i expected them to throw computing power at every problem , do everything in memory etc
1
u/DuneBug Dec 19 '22
The place I work, we do performance testing simulating our max requests / minute and if the software buckles under load we try to optimize it before throwing hardware at it.
I think the wisdom is... You look at what the app is doing and most of the crap we do isn't computationally expensive, so if something is dying it's probably the result of some bottleneck. And ideally there are some experienced people around to help out with that.
I think part of the reason we bother is if you do have a bottleneck you're more likely to fail even if you do add more hardware. Like I had a service that used a distributed cache, and it got bogged down with all the nodes talking to each other. When we reduced the number of nodes, performance improved, which helped find the cause.
Tldr; you're not wrong. Virtual boxes are cheap.
2
u/Worth_Trust_3825 Dec 19 '22
Unfortunately for a lot of enterprise stuff we don't get to use parallel streams as the threads are typically occupied elsewhere. :(
I wouldn't say preoccupied elsewhere, but rather operations are blocking, and not fit for the common thread pool that the regular parallel stream would operate in.
10
u/barrycarter Dec 19 '22
Hmmm, brief googling suggests streams are like functional languages or like the ruby convention of foo.x1.x2.x3.x4 which is similar.
I suspect your time difference is due to something else, most likely the difference between loading a file in memory and simply processing it a little bit at a time, though loading the file into memory should be ultimately faster.
It's a lot to ask, but can you mockup an example of where you're seeing Java streams seriously outperform other languages, so I can call shenanigans?
4
u/prisonbird Dec 19 '22
i read the file using Files.lines method in nio. it gives me a stream. then in that stream i use couple filter() and couple map() functions to mutate the data (remove fields, calculate some fields etc) and write output to another file. my flow is not that complicated but my files are sometimes big (10-15GB).
3
u/senseven Dec 19 '22
Still three minutes is a "long" time. Your SSD can read 500mb/s, a rather slow NVME 3GB/s, 3 minutes feels wrong for simple processing. Do you write the result file to another drive? Are all your cores working during the processing?
1
u/prisonbird Dec 19 '22
thank you for your comment.
i read from a drive and write results back to same drive. i am planning to try my operations on a ram backed drive (ramdisk in linux)
your last question got me curious some of my cores working during processing not all of them. so i checked my stream with stream.isParallel() it returned false. i thought it was working in parallel. still, i dont see a single core getting overloaded instead i got 4 of my 16 cores doing some work (at 50% load in windows task manager).
i did some google searches and it looks like i can't make a stream sourced from a file parallel. is this the case? or is there something i can do ?1
u/senseven Dec 19 '22
Reading is fine if its done by one fast reader, but if you read/write to the same harddisk can get the speed down to a crawl since two io "actions" block each other.
The idea is not to split the file reading (which means chunking and makes often no sense). Instead the first thread parses line 1, second line 2 and so on. There where the parallelisation makes sense, because it uses CPU not IO. Parallel() will work, but it needs go get ParallelStreams(). Maybe you do it at the wrong point of the file parsing.
Third, parallel Processing may not activate all your cores and local setup. What is shown by Runtime.getRuntime().availableProcessors()? Maybe your Java is odd, windows or tool is blocking full access of your Java app to all cores.
1
u/danielaveryj Dec 20 '22
The file parsing example you linked to is copying the entire file into memory before doing any compute. For OP's 10-15 GB files, that seems like a lot to buffer up-front. (To be fair, I'm not sure if your link was intended to be a "good" or "bad" example.)
As for parallelism, the File.lines() method that OP mentioned using documents that it is designed with parallelism in mind. (Looking a bit closer, the stream (if parallel) actually memory-maps the file when the underlying spliterator is first split. Chunks use separate indexes into the mapped byte buffer.)
I am curious why you believe that chunking often makes no sense? Your proposal (I'll call it "striping") seems like it would incur more cache misses, as each thread would only be looking at a fraction of the contiguous memory it pulls into cache, rather than all of it.
1
u/senseven Dec 20 '22 edited Dec 20 '22
You will probably not reach io max if you chunk due to different concurrent threads annoying the same IO controller. Its easier to have one thread to IO max, distribute the workload to threads, as we do with FileChannels and a LMAX Queue. Post processing over a group of disconnected chunks can be a nightmare, usually its not worth the work.
Op probably reached io max on the reading device and trashed the speed by writing to the same device. You can spend month optimizing code fill all cpus, streams might not even be the right solution. Experimenting is part of the parallel processing voyage.
1
u/Xenofonuz Dec 19 '22
Hmmm do you really need to do several filters and maps? You can usually just open it up with brackets instead.
filter(x->x.something) .filter(x->x.something2) .filter(x->x.something3)
Could be
filter(x-> {
//Extra code
return true/false
})
2
u/prisonbird Dec 19 '22
i read somewhere that using multiple small operations are faster than using one big operation inside stream operations. jvm can optimize small operations better or something.
also it is a lot easies to read if i do it in multiple operations and easier to find what i am looking for in case i want to debug/change something2
u/Xenofonuz Dec 19 '22
Ok interesting, I'm pretty sure it's the other way around.
Can't argue with the readability part, although I find it easier to debug the ones with a method body instead of a oneliner :)
18
u/participantuser Dec 19 '22
I mean, is there any reason why you’d expect Python to perform as quickly as Java, especially in a multi threaded task (OP mentions multiprocessing)? Different languages have different strengths, and I don’t think Python is noted for high performance for multithreaded tasks (the GIL is a Python specific example of a conscious decision to prioritize other things).
18
u/rv5742 Dec 19 '22
My bet is that it's probably java.NIO reading the large file in a sensible fashion and feeding the data to the stream in good size chunks, allowing for a good balance between IO and processing.
But to be fair, the streams API does make this kind of thing really easy to do.
2
1
u/GuyWithLag Dec 19 '22
loading the file into memory should be ultimately faster.
As all things, it depends(tm). Java is _really_ good at handling short-term garbage; if you keep your on-heap dataset small, it's likely faster than loading everything at once into memory.
2
u/stormcrowsx Dec 20 '22 edited Dec 20 '22
My guess on the difference in speed is likely that in the case of Python the whole dataset is getting loaded into memory either for input and/or output and causing swapping.
Java streams on the other hand are going to naturally only store what’s needed in memory.
Total guess but the difference in speed is more likely a testament to good Java api design than it is language run speed. Be impossible to say for sure without hooking a profiler up to Python and seeing what it’s wasting it’s time doing.
Kudos for reaching out and trying a different language on your problem. Now you know what’s possible. But I’d recommend you spend a little time in a profiler for your Python code now, there’s a fantastic lesson to learn about Python in here. I suspect either your going to learn just how fast the jvm is or that there’s a pitfall in Python apis that requires coding a little differently for big dataset use cases.
1
u/prisonbird Dec 20 '22
i dont know how to debug pyspark. since it goes back and forth to jvm and some of the job is done by jvm some is done by python etc.
1
u/stormcrowsx Dec 20 '22
Standard Python apps can just be started with ‘python -m cProfile myscript.py’
PySpark would be harder to profile but to start you could import cProfile and use it within your Python code to prove whether the slowdown is in Python or happening in the jvm.
14
u/developer_how_do_i Dec 19 '22
Yes, the best part with streams and lambda support is expressiveness, which you wouldn't have got with earlier jdk version
4
u/WagwanKenobi Dec 19 '22
Out of curiosity (as I'm still learning streams), what expressivity does streams enable that isn't present with non-streams Java? To me it just seems like streams is syntactic sugar + lazy evaluation + magic parallel() method.
4
u/emaphis Dec 19 '22
I think that it's supposed to be implied that streams are as expressive as custom Java looping constructs.
2
3
u/humoroushaxor Dec 19 '22
Isn't "expressiveness" just syntax sugar done well? Removing imperative bloat allows a clearer expression of intent.
1
u/8igg7e5 Dec 19 '22
...which you wouldn't have got with earlier jdk version
Though I can't resist pointing out that 'earlier JDK version' would mean before 2014... if we really want to row all the way to the horizon hunting for major releases, there was generics in 5.0 and collections in 1.2.
I agree though that some problems are much more expressively written as streams. And if they're using even newer JDK editions than 8, then there are even more features added that make
Stream(and friends) even better -Stream.dropWhile,Stream.mapMultiand numerousCollectorsmethods... And a few language features that can help too (var, local types, records, switch expressions)
9
u/thephotoman Dec 19 '22
i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
This doesn't surprise anybody. Python is very much a language for I/O bound stuff and prototyping. However, when it comes to performance, it isn't even good. While PyPy helps, it lags a bit in terms of features.
4
u/Sensi1093 Dec 19 '22
In PySpark, Python acts only in a „Stream Definition“ role. The actual stream is executed using spark, which is implemented in Java.
Think of SQL. The Python portion of PySpark is the SQL Expression String, the Java (Spark) portion of PySpark is the actual database engine.
I’m surprised Java Streams outperform pyspark and it’s likely because it was used in a way it is not intended to be used.
-7
u/BlobbyMcBlobber Dec 19 '22
I can't believe anyone who claims "python is only good for prototyping" actually developed anything significant with python, because it can and does drive countless real world products.
17
u/thephotoman Dec 19 '22
It drives countless real world projects that tend to be I/O bound.
But also, doing anything big with it turns into a mess very quickly. Dynamic typing can introduce significant type errors that just don't happen with statically typed languages. It winds up with your test suite growing exponentially compared to your codebase.
2
u/BlobbyMcBlobber Dec 19 '22
Dynamic typing can introduce significant type errors that just don't happen with statically typed languages.
This is what I thought too, back when java was my main language. But out of all the shortcomings of python, dynamic typing turned out to be not so problematic, and it never really caused major issues in my work. You pick up a couple of habits to manage this and you don't have to think too often about it again.
1
u/UnGauchoCualquiera Dec 20 '22
At that point and everything else being equal why not simply use a statically typed language?
I've worked in large scale node codebases that worked just fine but almost inevitably as the codebase and number of developers grew we always ended up enforcing typehints in jsdocs or just plain typescript.
1
u/BlobbyMcBlobber Dec 21 '22
I choose a language based on how close it is to the problem domain, rather than strongly typed or not. I look at features, frameworks and what kind of work is usually done with this language.
In the end of the day we're trying to solve a problem, not prove that anything could be done with a certain language.
Dealing with absolutes is just a waste of time.
1
u/UnGauchoCualquiera Dec 21 '22 edited Dec 21 '22
That's a pretty absolutist take :)
I'm talking about any language here, not singling out Java.
Statically typed languages offer things dynamic or duck typed languages do not. They enforce interfaces between developers and add safety to the project by doing compile time checks instead of runtime.
That's why the original commenter said python is used mostly for quick prototyping. Has nothing to do with the quality of the language but specifically because it lacks type safety which ends up being a big draw as the project grows.
Here's Python creator word take on types and why most large companies end up using a static type checker in python as part of their testing suite anyway. Exactly the same thing a compiler would enforce out of the box.
In the end of the day we're trying to solve a problem, not prove that anything could be done with a certain language.
Pretty much my point python is just a tool with limited domain applicability. Unless you need quick prototyping or depend on sci libraries I don't see what you gain with python over using any static language.
1
u/BlobbyMcBlobber Dec 22 '22 edited Dec 22 '22
You can create interfaces in python with abstract classes, and enforce types in certain ways. However with good clean practices you'd rarely need anything like this. For instance if your function can return different types depending on state, it's a code smell and it won't pass code review. Not anything that is technically possible is being used for the worst effect. At least not where I work.
Like I said before, out of all of python's issues, it seems funny making typing into an issue, and I get the impression that people claiming python is only good for prototyping haven't had the pleasure of working on good production python code.
Factually there are more than a few python products out there with support from excellent frameworks and claiming otherwise is just burying your head in the sand, which I don't understand.
1
u/Worth_Trust_3825 Dec 19 '22
Before people get fed up, and rewrite the entire thing in either C# or java.
4
u/BlobbyMcBlobber Dec 19 '22
If this happens, then the wrong tool was chosen for the job. Python has its advantages and java has its own. They complement each other. The trick is using them for the right thing and not imagine they are some sports teams with a rivalry.
2
u/Worth_Trust_3825 Dec 19 '22
Yes. Python's advantage is prototyping. Or when you have some maths phd statistician that does not want to learn anything.
3
u/BlobbyMcBlobber Dec 20 '22
That sounds hostile and somewhat entitled.
I like Java, and I like Python, and I don't understand why anyone should be so defensive about a programming language.
But you do you. If you insist on believing Python is only good for prototyping and no real work can be done with it, go ahead. You're just doing yourself a professional disservice.
0
u/Worth_Trust_3825 Dec 20 '22
In your previous comment you mentioned that each language has its own strengths, yet now you're moving the goalpost to claim that python in particular can do more than its main purpose. That sounds like you're the one looking for nails for your hammer.
Each tool has its limits. At some point you will need to lose that hammer, and use a screwdriver. I've been there multiple times, and each time we opted to get rid of python.
2
u/BlobbyMcBlobber Dec 21 '22 edited Dec 21 '22
claim that python in particular can do more than its main purpose
You're the one who claims python's main purpose is prototyping, i.e. it's not capable of being used in production, which is just objectively wrong, and sounds elitist. If you only used python for prototyping, that's on you, not the language.
Each tool has its limits. At some point you will need to lose that hammer, and use a screwdriver.
This just shows you didn't realize beforehand what's a hammer and what's a screwdriver until you tried them for the wrong job. Tools are good for certain tasks and it's up to you to use them well. Nobody is going to blame a hammer for failing to be a fork, except this one person on reddit apparently, but you can go around believing you're perfect and trying to eat with hammer didn't work out because hammers suck.
1
Dec 19 '22
The selling point of Python is it's readability, the maths/science community can focus more on the stuff that matters to them, the language gets out of the way; besides, nowadays the maths PhD statistician most likely learned Python from highschool onwards and the C++ backed analysis packages are taught in most maths colleges alongside R and MatLab, at least here in Serbia.
7
u/hilbertglm Dec 19 '22
They shouldn't be black magic. The source code is right there. Check it out. It helped me add functional programming to my object-oriented programming skills.
They are cool, and the ability to add parallelism without the effort of coding to the executors is very productive.
1
u/lbalazscs Dec 19 '22
Have you tried the tablesaw library? It's data frames in Java, with stream support. https://jtablesaw.github.io/tablesaw/
1
1
u/data_addict Dec 20 '22
then i discovered java streams. i have a workflow that takes 70 minutes with pyspark. it takes 3.5 minutes with java streams.
Ah got it.. you don't know how to use spark. 👍
2
u/prisonbird Dec 20 '22
might be. but i spent 1/10th of the time i spent on pyspark on java and it got the job done
1
u/data_addict Dec 20 '22
Yeah sure that's cool, use whatever tech you want. But you didn't say you spent 70 minutes trying to setup a spark job you said 70 minutes running it.
Again, use whatever tech you want but to but don't blame spark for being poor performance. Just say the line: "I suck at spark" or "I didn't have time to learn it" or "I couldnt afford it".
There's people who are in high school or college and see the names of these technologies the first time and then form an opinion based off what you say. They'll base their opinion off of what you say. It's much better to be honest and say what gave you trouble. Good, non-toxic engineers are honest about what gives them trouble.
2
u/prisonbird Dec 20 '22
i am trying to explain my story the best i can.
let me rephrase it again : i spent about a week to implement my flow in pyspark. and it takes 70 minutes to run. yes i can debug it, make it faster etc. but as far as i see that requires some expertise. and because i dont have that expertise i dont know if it will work for me in the end.
on the other hand, i was able to make a proof of concept in java streams in an hour. tested various parts of my flow. it seemed like it could work for me. so i spend couple days to implement my entire flow in java streams. and now i can do the same job a lot lot faster.
i never intended to mean anything like "pyspark bad, python bad , java good". each tool has its own advantages. pyspark just didnt worked for my workload. but i learned about it now and in the future i might use it for something else i see fit.
1
u/data_addict Dec 20 '22
No I mean that's fine and I know I was roasting you a little in my original comment - I hope you didn't take too much offence to it. I get what you're saying though.
If you find yourself working with spark in the future for large datasets feel free to reach out. It's a pretty kick ass application that can really chew through data. Not to belabor the point but in the way it should normally behave with a small-meduim amount of hardware, working with 10GB of data should take a matter of 10s of seconds to a minute. I've written production applications that operate on 50TB+ of data in less than 5 min. So anyways that's why I felt the need to comment. Streams are pretty cool so I wasn't talking shit on that either.. just wanted to make it clear spark is pretty cool too 😅
1
u/DanielDimov Dec 20 '22
What exactly is "multiprocesing in python" ?!?
1
u/prisonbird Dec 20 '22
https://docs.python.org/3/library/multiprocessing.html
i tried to implement something like java streams, but it uses all my memory.
1
u/Puzzleheaded_Load779 Dec 31 '22
Hi, having worked with spark ecosystem for sometimes and building/extending stuff around it, here is the reason for why it is faster:
pyspark is mainly a wrapper around spark scala code => java code
pyspark exists only because of the multitude of ml, ai, etc... libs that emerged during the last severals years
How does it work internally ?
as long as any operation are being done using pyspark
sql queries
pyspark dataframes
no "big" performance gap and memory usage will happen between a "java/scala" spark implementation.
BUT as soon as you start resorting to mutate pyspark objects to let says the infamous "pandas" dataframe objects, there is a cost and a pretty big one which leads in general to bad performance and (absurdly) high resource usage ram,cpu,io...
As for performance, still, there are ways to not lose too much by using arrow implementations baked into spark but thats another story :)
So all in all, using java instead of pyspark means that the deserialization of
java objects to python objects
python objects to java objects
doesn't happen... leading to overall a performance boost
1
u/prisonbird Dec 31 '22
ohh i see. so can i just simply use spark on scala/java and have similar performance to java streams?
1
u/Puzzleheaded_Load779 Dec 31 '22
No. Making operations on data types other than those of spark will always involve a cost. It is just that the cost of using java/scala object has a cost much less when compared to python. If you are seeking pure performance, use exclusively Spark sql dsl or; Spark dataframe for programmatically operations...
1
Jan 07 '23
C# has been doing this with LINQ for nearly 20 years.
1
u/prisonbird Jan 07 '23
how?
1
Jan 08 '23
LINQ is a set if libraries primarily wrapping
IEnumerable<T>that provides projections over the enumerable collection. You provide lambdas or delegates to the projections, just like java Streams. This has been part of C# since 2006.
1
Jan 13 '23
Im struggling a lot with Streams, watched PluralSight video playlist of José but still nothing. Can u pls recommend a youtube video, even more that 30 mins is fine.
Just not getting how those intermediate lazy operations and flat maps
1
u/prisonbird Jan 13 '23
i just read the documentation. what are you trying to achieve?
1
Jan 13 '23
I know the basic, like filter and forEach.. not able to solve the actual use cases and complex ones like flat maps
1
1
u/wildjokers Dec 19 '22
more people should know about streams.
All java developers know about stream. Or are you meaning non-java developers?
0
2
u/shagieIsMe Dec 19 '22
With regular processing of data, one is tempted to write:
for (Data datum : someCollection) {
// lots of lines doing stuff
}One of the things that happens there is that the small chunks of processing can't get JIT'ed easily. The entirety of it is one big chunk of "too big" for HotSpot or other JITs to optimize.
However, when you write it as:
someCollection().stream()
.map(d -> new Foo(d, d.id, d.data * 2))
.filter(f -> f.id > 1000)
.mapToInt(f -> f.data)
.sum()(realize I'm just making up stuff there)
Each one of those functions gets complied in Java to a separate function and the JIT can easily work with optimizing small functions.
The way that you work with the data, having each part of a stream being a small function makes it easier for Java to optimize the small parts and get some fairly hefty performance gains when crunching data.
There are other things about the functions in streams and side effects that can't be done and thus allow for more aggressive optimizations.
public static void main(String[] args) {
List<String> objects = new ArrayList<>();
objects.add("foo");
objects.add("bar");
int count = 0;
var list = objects.stream()
.peek(o -> count += 1)
.toList();
System.out.println("" + count + " " + list.size());
}That code doesn't compile - java: local variables referenced from a lambda expression must be final or effectively final
And so, this means that the JIT can know things about the nature of the function that it is compiling because its a lambda.
If you change it to:
final AtomicInteger count = new AtomicInteger(0);
var list = objects.stream()
.peek(o -> count.incrementAndGet())
.toList();It works (and is still silly) but you'll note the addition of final in there.
1
1
u/Puzzled-Bananas Dec 19 '22
Can you share how you formulated your Spark queries? Do you mean Pandas DataFrames?
1
u/Worth_Trust_3825 Dec 19 '22
I suspect that you also changed your method of reading and interpretting the data (entire thing vs one row at a time). My go to example of this is scryfall's public magic the gathering card database (available at https://scryfall.com/docs/api/bulk-data). It's a hundred megabytes in JSON, but it's one of the few go to datasets (at least for me) when trying to explain that it might be a bad idea to load entire file into memory and operate on it, and that you should look for ways to operate on file in parts (ex. in this case, per json object). If you're feeling adventurous, you can try to operate on the big boy 1gb set.
1
u/baubleglue Dec 20 '22
https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.newAPIHadoopFile.html - you can stream CSV with Spark or Pyspark (only don't use pure python for your custom logic).
1
u/a-person-called-Eric Dec 20 '22
looks like someone found a new hammer
be careful not to start whacking every nail with it
1
u/gooeydumpling Dec 20 '22
For ML libraries = python, to see it it works, always python and jupyter
For production? I’d go with MLeap and scala (of course, java). For speed, Java blows Python out of the water, or at least in my experience. Please don’t bite my head off
1
u/FinTechno Dec 20 '22
Take a look C++ and the rest is history
1
u/DanielDimov Dec 20 '22
C++ can be faster, but it's not guaranteed. You should think about the performance and put a significant effort to achive this better performance.
1
u/RajSingh9999 Dec 20 '22
Dont know if I should agree with "stream operations are more intuitive than anything", especially when compared with python etc.
1
u/Cultural-Ad3775 Dec 20 '22
I would not call Java's implementation of this concept really special. OTOH it is cool, particularly if you can mark stream processing as parallel, which is a nice abstract way to say "Hey, run these in separate threads" or in the future it could lean on Loom style lightweight threads, etc. without even needing any code changes. Other languages do basically allow for the same sort of thing however, still Java/JVM have quite strong abilities to optimize and multi-task, meaning it can be leveraged very effectively (in theory, I wouldn't assume any specific parallelized stream will necessarily be super optimum, you have to test).
-2
u/Careful-Necessary-59 Dec 19 '22
Just curious, how big is your data? I still doubt PySpark is slower than Java streams.
2
u/prisonbird Dec 19 '22
around 10 gb. pyspark is only fast when you use built in functions and if you dont use too many of them.
for example : i have to calculate a field using a lot of conditions(imagine a very complicated IF/Else block). when i used a lot of whens in pyspark (df.withColumn(xx, when(yy).otherwise(when))) kind of thing) it straight up bugs out. giving me some sql error.
6
u/RandomName8 Dec 19 '22
I've worked a lot with spark (though via scala, not python), what you describe sounds off the mark with my experience, specially given that you are using spark's sql instead of rdd api. Spark's sql is internally compiled to optimized bytecode before running, making it possibly faster than manual writing if-else due to exploiting relational data properties. In any case the difference shouldn't be 20:1. That said it is hard to master spark and its performance.
3
u/prisonbird Dec 19 '22
there is two things here:
when i use multiple when/otherwise spark could not generate an sql. gave me a lot of errors and i wasnt able to debug it/find the error after a point. so i just used an udf (user defined function)
i think there is a difference between pyspark and spark. i wrote some scala scripts and used them with "spark-shell -i myscript.scala" and there was notable time difference for same jobs. not huge like 20:1 but , 3 minutes vs 5 minutes. i think pyspark adds another layer to the data and it also takes some time
3
u/RandomName8 Dec 19 '22
udfs with pyspark. That's you answer. There's a massive overhead with python udfs. That can perfectly explain the 20:1 overhead
2
u/prisonbird Dec 19 '22
yeah, but i had to use udf. and i didn't liked this a bit. its like "hey we have this feature but you literally would be shooting yourself in the foot".
anyways. i am still planning to use spark for joining, grouping kind of things
2
u/cockoala Dec 19 '22
Yeah... This is a little fishy. If streams were so much faster than Spark there would be no need for Spark. I think OP doesn't understand how to write performant Spark jobs. In another comment he said he gave up on debugging his chained filter so my guess is he wasn't able to write in spark and tried it in Java instead.
2
u/prisonbird Dec 19 '22
yes that is my point. i was able to do some amazing things with java streams despite knowing nothing about them. that is why i called it "magic" :)
i might go into a rabbit hole and come out as an spark expert and implement an amazing solution on pyspark but i dont have the time and i dont want to :/
4
Dec 19 '22
Spark is needed for data that cannot possibly fit on a single machine, you'd still need it if its performance is atrocious (which it is when used on data that, like OP's, fits easily on commodity hardware) - "Scalability! But at what COST" paper illustrates how the overhead of map-reduce and stream processing products is often disregarded/dismissed
1
u/Careful-Necessary-59 Dec 20 '22
The overhead should not be as huge 60 minutes difference…
1
Dec 20 '22
Never said it should, I was just pointing out that expecting Spark to be faster than plain Java for a small dataset doesn't make much sense - that's not the problem that it was built to solve
0
0
Dec 20 '22
There is also an option to run parallel streams which may run even faster depending on the workflow.
0
Dec 21 '22
[deleted]
1
u/prisonbird Dec 21 '22
tried, reading a file and writing it back without doing anything in python takes a lot more time than my entire flow in java
80
u/[deleted] Dec 19 '22
I agree with other commenters. While I love streams, unless you were using parallel streams then the stream API should have no impact on performance.