r/javahelp • u/[deleted] • May 23 '20
Recommendations on String manipulation
A company I work with is replacing an old COBOL process with Java.
Pretty much the business rules are about reading some long lines from a text file, do some snippin and reordering before writing to a new file text.
That part is covered, but I'm doing some heavy use of String.substring. I haven't seen a real input file, but they are supposed to hold near 2 million lines each.
Because of that, I would like to make the new process as memory efficient as possible. (When I first learned Java they always warned us about the perils of String)
The process will run on a server (don't know the exact specs) if that's useful.
Thanks in advance.
3
u/xyphanite Quantum Brewer May 23 '20
You can likely make some estimates about run-time and memory usage based on what you're working with.
This tutorial https://www.baeldung.com/java-read-lines-large-file has some good methods, as you may get a memory exception if you read in the entire file at once or try to write. You may have to get creative when parsing it if you need to re-order content based on previous searches. I think that statements stands true for writing as well. Hope this helps.
3
u/xyphanite Quantum Brewer May 23 '20
Also, if you need speed and have multiple cores (I'm sure you do), and you can do something similar to MapReduce, which splits the process, does some searching, for example, and then combines it in the end (the reduce step).
1
1
May 23 '20
I'll take a look at the tutorial.
About the file, each line is independent, so no fancy stuff right there.
Thanks.
2
u/garblz May 23 '20 edited May 23 '20
I'm doing some heavy use of String.substring
Oooh, this will be interesting! So, depending on how many substrings of each particular String you're going to use, you're in for a treat.
Okay, so if you have one large String and a thousand substrings of it, you'd probably want all of the substrings to basically be indexes/offsets pointing back to the big one, to save space. And this is exactly how substring in Java works prior to JDK7.
But... if you're working with Java 7+, substring behaviour has changed. Now each call to substring produces it's own character array, and is not dependent on the source String. Which was probably introduced due to memory leaks - say you initialize String which is large, then get a tiny substring of it. Then the substring, as long as it lives, prevents garbage collection of the large one.
So it really depends on what your analysis oh how the substrings will distribute, how large the source substrings are, and if you're using to use Java prior to version 7. If you find you want the first solution, and can't use Java 6, then you probably need to implement your own Substring class (use char arrays as suggested by /u/Philboyd_Studge)
EDIT: Just remember not to store the Strings you read from the file in any globally accessible variables so they can be GCed as soon as you're done with them, and have fun!
8
u/Philboyd_Studge May 23 '20
Creative use of regex might do the trick, but for pure speed and memory usage, using char arrays would probably be the best.