×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Data mining in text logs: associating sentences with each other

Data mining in text logs: associating sentences with each other

Data mining in text logs: associating sentences with each other

(OP)
Hello everyone.

Can you advise me please on this project I'm trying to do.
I've got about 45Gigs of text logs. I've done some search and extracted regular expressions for certain text sentences related to errors that I'm mostly interested in. Now I'd like to be able to do some of the following:

1. be able to predict a possibility of occurrence of some sentences in their relation to other sentences (eg: occurrence of error string 1 is likely with probability P to be located with error string 2 in the range of N lines).

2. at least to be able to cluster roughly error strings by their occurrence together with some range of lines.

Could you advise me please what tools and methods to use best? Thank you in advance!

RE: Data mining in text logs: associating sentences with each other

For analysis #1, you are looking at time series data. You are interested in events before or after other events. This is regression (but within time series). Analysis #2 is probably best approached as time series as well, although it probably can be done using other data mining techniques. Note that time series analysis tools are generally not included in most data mining packages. Look for keywords like Box-Jenkins techniques.

==================================
adaptive uber info galaxies (bigger, better, faster than agile big data clouds)


RE: Data mining in text logs: associating sentences with each other

(OP)
Hi John, thanks a lot for your answer.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close