Smart questions
Smart answers
Smart people
Join Tek-Tips Forums
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Member Login




Remember Me
Forgot Password?
Join Us!

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips now!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

Join Tek-Tips
*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.
Jobs from Indeed

Link To This Forum!

Partner Button
Add Stickiness To Your Site By Linking To This Professionally Managed Technical Forum.
Just copy and paste the
code below into your site.

Best method to read very large text files in Java (>1.5GB)

bitwise (Programmer) (OP)
5 Dec 11 15:53
I have a very large ASCII text file. It's an Apache webserver log. It contains 4,783,012 lines and is 1.66GB. I need to look at each line of the file independently. This makes using "java.nio" a bit more difficult because I'll have to parse out a line from a buffer by looping through the buffer looking for '\r' or '\n' to determine a line in the log.

Of course, I could do the standard...

CODE

FileReader fr = new FileReader("C:\access.log");
BufferedReader br = new BufferedReader(fr);

String strLine;            
while ((strLine = br.readLine()) != null) {
     //PARSE LINE...
}

However, this takes 30 - 40 seconds just to do a line count. Is there a more efficient method I should look at to doing this?

Thanks!    
          
TonHu (Programmer)
5 Dec 11 16:14

Quote:

Is there a more efficient method I should look at to doing this?
Not do it in Java, but use existing tools to parse a webserver logfile. Several available after a Google search
bitwise (Programmer) (OP)
5 Dec 11 16:36
The question is not just about how to parse log files. There are tons of tools and scripts available to do that. That was more of an example to illustrate my question.
Diancecht (Programmer)
12 Dec 11 3:15
You can read data blocks instead of lines to reduce read operations. Take a look at this

Cheers,
Dian

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Back To Forum

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close