×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Unix Script for realtime log monitoring and performing tasks

Unix Script for realtime log monitoring and performing tasks

Unix Script for realtime log monitoring and performing tasks

(OP)
Hi Guys,

Need help on how can i write a unix script which does below

1. Realtime monitoring of huge realtime updating log
2. get a sepecific keyword on log - Do below
save a particular string it will be a number in that log line eg. new mas:400 recieved : we keep 400
search the next line of logs which will have mas:400 and look for more keywords
search that keyword in a database and forget 400 if the keyword is not in DB
finally we get mas:400 stored:#1234
we forget 400 now and proceed with next
3. the log will keep getting new numbers and we need to make sure we check all of them and notify if there is a error

I am thinking of reading logs , making a qeueu and holding them until its done

Please share your ideas on how we can best achive the above task

RE: Unix Script for realtime log monitoring and performing tasks

Look at Logstash for pulling from a big log. Or even multiple big logs. Send it to something like Elasticsearch to be able to access and analyse it. Maybe add something like RabbitMQ in there to queue the log data if it's coming in fast or from multiple servers. Also look at Splunk maybe. There are many other tools out there besides these.

There are a lot of open source log management and analysis tools out there. Some can handle insane amounts of log data.

And if you have a budget, there are commercial tools too.

RE: Unix Script for realtime log monitoring and performing tasks

(OP)
thanks for the response guys - i was thinking more of writing on our own

RE: Unix Script for realtime log monitoring and performing tasks

I wrote a similar script in Perl to filter HP-UX syslogs. Basically it does something similar to a 'tail -f' on the log file, matches substrings and invokes various commands or logic based on what is matched.

It's too specific to our requirements to share here, but yours sounds equally specific so you would need to implement it from scratch really. But if you search for 'perl' and 'tail' you'll find some examples for how to do that and build it up from there.

Annihilannic
tgmlify - code syntax highlighting for your tek-tips posts

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close