r/javahelp • u/alpacaemojipls • Dec 19 '19
Efficient way to read through a text file in java?
I am writing an adjacency array based implementation of Dijkstra for an assignment. The input is a text file containing nodes and edges, so my program needs to read throuth the lines and convert the text into ints and doubles which have to be stored into arrays.
My first implementation I used a scanner and the split method, then converted the strings into ints and doubles. It worked but the performance was very bad.
The second try was using a buffered reader and accumulating the chars into a stringbuilder. This worked way better, for a file size of 400MB it did the job in around 13 seconds. However for a file of 2.7 GB my computer crashes (the prof said it should work ok with 8GB of RAM, which is the amount I have). I've experimented with different buffer sizes but it still didn't work.
Does anyone have any suggestions for the most efficient way to do this? Maybe an ideal buffer size for my implementation or a completely new way. Thanks in advance
2
u/ethomp18 Dec 19 '19
Depending on how your input data is structured would determine the most efficient method for reading the file. How is the data structured, this should guide the best way to process this data and trying to find that balance between speed and size.
I would approach it by reading the file line by line with a BufferredReader to a String and then figure out what you need to do with each line.
2
u/AryanPandey Dec 19 '19 edited Dec 19 '19
Well what I can say one of the biggest reason for crashing could be (as the post doesn't mentioned ) java.lang.OutOfMemoryError (I am just assuming stuff , maybe a whole lot of other error would be there.) It would be great if you can share your source code, well as it's your homework, I think it will be better if you can at least share the error message.
For java.lang.OutOfMemoryError
Primary Reason it is happening would be:-
(It's also an assumption.)
1)Spikes in usage/data volume. The application was designed to handle a certain amount of users or a certain amount of data. When the number of users or the volume of data suddenly spikes and crosses that expected threshold, the operation which functioned normally before the spike ceases to operate and triggers the java.lang.OutOfMemoryError: Java heap space error.
2)Memory leaks. A particular type of programming error will lead your application to constantly consume more memory. Every time the leaking functionality of the application is used it leaves some objects behind into the Java heap space. Over time the leaked objects consume all of the available Java heap space and trigger the already familiar java.lang.OutOfMemoryError: Java heap space error. Taken from here
If that's the case , I would like to you know get knowledge of this error , can you can also play with it, using totalMemory() and freeMemory methods. Check this out
Well after understanding the error and it's working , you might design a solution for itself, by I think if in you project it's possible to read the buffer [not the whole file at once!] do the process, on that buffer , and then remove it (Removing will automatically be done by garbage collector , you just have to make sure you are not storing it while getting new buffer.) Thanks , I hope it will work in your case ,if I have done some right assumptions.
2
u/Saiboo Dec 19 '19 edited Dec 19 '19
If you want to tweak the input read performance I suggest to leave out building the graph, and instead to simply read the values in. Here is a test I wrote:
This class BigFileWriter creates a file of size 2.8 GB. The content is
A single line describing the number of lines, here: 60 million
60 million lines of two integer values
60 million lines of two double values
This class InputReadPerformance reads the lines in that file. On my machine it takes about 50 seconds.
I used a BufferedReader and a StringTokenizer, see the references here, here and here.
Some possible optimization tweaks:
Use
Integer.parseInt()intead ofInteger.valueOf(). The first one creates an int (primitive data type), whereas the latter creates an Integer object, see here.In your graph implementation: if you're using ArrayList for the adjacency lists, then you could initialize the size of the ArrayLists provided that you know the approximate number of neighbors of a node. This prevents resizing the ArrayList.
Switch to primitive data types, i.e. you would have to implement your graph without using the data structures in Java since they use generics / objects. Primitive data types consume less memory than their corresponding wrapper classes which can result in a speedup.
2
u/evils_twin Dec 19 '19
Can you split the file up and process them concurrently in multiple threads?
0
Dec 19 '19
[deleted]
2
u/alpacaemojipls Dec 19 '19 edited Dec 19 '19
This is the relevant class, this is an input example. Sorry for the vagueness, for a large file the program never terminates. I don't get an error message, but eclipse stops responding and I end up having to restart my computer so I'm not sure what exactly is not working. At first I got the lang.OutOfMemoryError but I allocated more memory and didn't get it anymore.
Edit: I tried another approach, this time it worked but it took around 5 minutes for the 2.7 GB file, which is still too slow.
2
u/mentholmeow Dec 19 '19
In your new approach, in the last while-loop, you’re instantiating a StringTokenizer which is an expensive operation.
Since the lines follow a pattern, you could try using String[] values = line.split(‘ ‘); int x = Integer.parseint(valuas[0]); ...
2
u/alpacaemojipls Dec 19 '19
I tried it but for some reason the execution time was higher
2
u/mentholmeow Dec 19 '19
Integer.parseInt() might be slower than Integer.valueOf(). Try switching back to valueOf() and serif it improves the running time.
3
u/yangmungi Dec 19 '19 edited Dec 19 '19
One of the responses in https://stackoverflow.com/questions/14037404/java-read-large-text-file-with-70million-line-of-text say that a 1MB file took ~130ms to read, which would imply a 2.7GB file should take around 350 seconds (or about 6+ minutes).
Generally speaking, a sequential file read, depending on disk speed, can have a throughput of anywhere from 10MB/s to 100MB/s, so if your disk is on the slower end you're going for about 270 seconds (5+ minutes) just to read the file.
If your disk is doing other things (like if you have swap on disk), it can crawl to a KB/s as your RAM usage creeps up.
Edit: you did mention the 400 MB file took 13 seconds, which would mean your disk throughput is around 31 MB/s, and ideally a 2.7 GB file should take about 90 seconds. So maybe it's not disk throughput.
Regarding your code for reading the file, the class instantiation can be a potential slowdown, since it will require GC, but otherwise it seems fine for its purposes. You can try leveraging https://docs.oracle.com/javase/8/docs/api/java/io/StreamTokenizer.html over
StringTokenizer, maybe that'll help
1
u/alpacaemojipls Dec 19 '19
Thank you to everyone who commented, I learned a lot today, for reference the main problem was that I used Double and Integer arrays instead of double and int arrays in my Graph implementation. A lot of other tips were useful for a slightly better performance