In our organization, we must now back up our datafiles to a network attached storage device. In order to save space, we have to compress the datafiles. Does anyone know of a faster method than compress? I looked at bzip2, pack, gzip, but they don't seem any better. Is there any other methods to try (like copy local, then compress then copy back?) Any hints to improve performance? Thanks.
Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations derfloh on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
Backups and Compress on Network Attached Storage
- Thread starter dbtoo2001
- Start date
Well indeed they aren't faster but realize a better compression factor.
If you haven't CPU problem you could split the file in more chuncks and then compress them in parallel.
Furthermore you can compress on the fly the DB files so you will send less data on the net
If you haven't CPU problem you could split the file in more chuncks and then compress them in parallel.
Furthermore you can compress on the fly the DB files so you will send less data on the net
are you compressing on the local device over the network, or compressing on the remote device over the network?
if it is the former and you are worried about time taken, then you _could_ copy the file locally.
you could compress the file through a stream locally if it is a large file, especially if you're using IDE disks (IDE drives use CPU time), although 2 different SCSI disks on 2 different controllers would not necessarily suffer from file size.
if it is the latter, then it won't really matter which compression method you use, because the whole file will be transferred over the network uncompressed (from server), and compressed locally (on the remote machine)
what kind of network attached storage device are you talking about? tape, disk, cd?
what kind of network? 10, 100, 1000MB?
what kind of server? x86, sparc, risc, Solaris, Windows, Linux?
Jon
if it is the former and you are worried about time taken, then you _could_ copy the file locally.
you could compress the file through a stream locally if it is a large file, especially if you're using IDE disks (IDE drives use CPU time), although 2 different SCSI disks on 2 different controllers would not necessarily suffer from file size.
if it is the latter, then it won't really matter which compression method you use, because the whole file will be transferred over the network uncompressed (from server), and compressed locally (on the remote machine)
what kind of network attached storage device are you talking about? tape, disk, cd?
what kind of network? 10, 100, 1000MB?
what kind of server? x86, sparc, risc, Solaris, Windows, Linux?
Jon
- Thread starter
- #4
We are hooked into a Network Appliance, not sure of their model number, but it isn't there fastest. We are on solaris 9, 100m to the network appliance, but I'm told that network traffic doesn't seem to be the problem. Even when I compress on the fly, like during a hot backup, it is very slow. And it is the latter scenario. Thanks.
so the network appliance does the compressing?
what do you see on your server screen when you do an:
iostat 5
(it will update every 5 seconds, the first line is an average since boot)
it should show you if you are CPU or resource bound as the process is happening.
what do you see on your server screen when you do an:
iostat 5
(it will update every 5 seconds, the first line is an average since boot)
it should show you if you are CPU or resource bound as the process is happening.
- Thread starter
- #6
I believe I misunderstood the scenario's, we do not do anything directly on the newtork appliance, i.e., we cannot issue any commands on that machine. So we read from there, compress the file and put it back - which should be an in memory process, in that we don't put the file locally before compressing it. Compressing on the fly, with a hot backup, actually takes about 4 hours longer. Thanks.
woa, how big is the database?
what do you see on your server screen when you do an:
iostat 5
(it will update every 5 seconds, the first line is an average since boot)
it should show you if you are CPU or resource bound as the process is happening.
i'm assuming that your machine is just _VERY_ resource bound
do you have IDE or SCSI devices (i'm asuming that you're using sparc solaris)?
what do you see on your server screen when you do an:
iostat 5
(it will update every 5 seconds, the first line is an average since boot)
it should show you if you are CPU or resource bound as the process is happening.
i'm assuming that your machine is just _VERY_ resource bound
do you have IDE or SCSI devices (i'm asuming that you're using sparc solaris)?
- Thread starter
- #8
Database is 120gig. Largest tablespaces are 20g. Most of them are 6 - 8 gig. I do the backup at midnight Sunday's and I can't tell you what the iostat is during that timeframe, I can do a cron job and see, maybe this weekend. I am told by unix support that the machine is only 50% cpu, but heavy network, which fits the problem, but doesn't address how to get the compression to run faster. Thanks.
As sem said, you should compress on the fly using a pipe.
In this way you should send a 20% of data (if you there aren't images in your DB).
If the bottleneck resides on the net, as your system engineer said, you can use gzip with the "-9" parameter in order to achieve a better compression factor and less data on the net.
Command line should be something like:
dd if=local_filesystem/datafile bs=64k |gzip -9 > network_filesystem/datafile.gz
In this way you should send a 20% of data (if you there aren't images in your DB).
If the bottleneck resides on the net, as your system engineer said, you can use gzip with the "-9" parameter in order to achieve a better compression factor and less data on the net.
Command line should be something like:
dd if=local_filesystem/datafile bs=64k |gzip -9 > network_filesystem/datafile.gz
if you have the money, you might be better off creating a second network in this machine, add a second network card and move the network storage device onto that network, if nothing else shares the storage device.
i'm assuming that the network traffic is for this machine only and wouldn't be sorted out by getting a switch ...
if the network is the bottleneck you should be able to compress to the local file system to speed up compression, but it would increase the overall time for the transaction (compress to local, then transfer over the network afterwards)
i'm assuming that the network traffic is for this machine only and wouldn't be sorted out by getting a switch ...
if the network is the bottleneck you should be able to compress to the local file system to speed up compression, but it would increase the overall time for the transaction (compress to local, then transfer over the network afterwards)
- Thread starter
- #12
- Status
Similar threads
- Locked
- Question
- Locked
- Question
- Locked
- Question