Tek-Tips is the largest IT community on the Internet today!
Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!
-
Congratulations Rhinorhino on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!
speed
- Thread starter robertd18
- Start date
I'm going to have to guess that only possible answer is "it depends". What are you doing and how are you doing it? ----------------------------------------------------------------------------------
...but I'm just a C man trying to see the light
...but I'm just a C man trying to see the light
- Thread starter
- #3
rosenk
Programmer
- Jan 16, 2003
- 403
Random selection from a database can be significantly faster than random selection from a flat file (unless you go through the process of creating your own indexing system for the flat file.) This becomes more true the more data you have. (If you only have a few records you might find that a flat file is faster, in fact.)
Making updates to a database will certainly be faster than doing so with a flat file.
Making updates to a database will certainly be faster than doing so with a flat file.
- Thread starter
- #5
I slightly different perspective.....my own humble opinion....
The choice between text file data-store and a db server data-store is a question of:
- data set complexity
- the need, or lack thereof, for multiple concurrent
connections. Will multiple users need to update the
data set at the same time?
- data set size
- expected traffic
(I'm sure there are other things to pay attention to.... this could probably be a book.)
- data set complexity
If you are serving alot of complex data that is truely relational in structure and you will need to make complex queries on that data set, then a db server is required. If the data set is 2 deminsional or close to it (like a spreadsheet), then a db server may be overkill.
- Connections....
If you need multiple concurrent connections to update the data set, you almost have to use a db server. You can do some file locking with the text file based approach, but, each connection must wait for any previous connection to finish and unlock the file.
However, if your users are querying the data set and not updating it, no locking is necessary. Since file access is a fundamental behavior for an OS, most OSes do it quite well. Each 'connection' just reads the file. (Not really a 'connection'... except in the sense that your application kind-of becomes a server for the text file.)
- data set size
If the data set has less than 1 or 2 thousand records and the records are fairly short, then a text file based approach can be very fast and simple to maintain with the advantage that there is no db expertise required.
- expected traffic/load
If this is to support a web app, traffic load becomes a question. But, heavy traffic can over whelm the text file approach.
In the last year or so, I have taken to writing my simple 'db' type web apps using text files and the DBI:
BD::CSV module(s). The web interface is usually a search interface (no updating). The DBI:BD::CSV approach provides a very light/simple application and with an easy upgrade path to using one of our db servers if the complexity or load of the situation increases significantly. These apps are generally written for internal company clients who can use Excel or QuattroPro or other spreadsheet app (or a text editor) to view/manage their comma delimited data sets. Additionally, the normal report generation with SQL via DBI:BD is straight forward and totally portable.
' just another perspective.....
I think many people are seduced by the 'glamour' of a db server and quickly overlook the utility in the text file approach consequently building way to much complexity into some applications/systems (....using the proverbial sledge hammer to drive in the proverbial tack). 'hope this helps
If you are new to Tek-Tips, please use descriptive titles, check the FAQs, and beware the evil typo.
The choice between text file data-store and a db server data-store is a question of:
- data set complexity
- the need, or lack thereof, for multiple concurrent
connections. Will multiple users need to update the
data set at the same time?
- data set size
- expected traffic
(I'm sure there are other things to pay attention to.... this could probably be a book.)
- data set complexity
If you are serving alot of complex data that is truely relational in structure and you will need to make complex queries on that data set, then a db server is required. If the data set is 2 deminsional or close to it (like a spreadsheet), then a db server may be overkill.
- Connections....
If you need multiple concurrent connections to update the data set, you almost have to use a db server. You can do some file locking with the text file based approach, but, each connection must wait for any previous connection to finish and unlock the file.
However, if your users are querying the data set and not updating it, no locking is necessary. Since file access is a fundamental behavior for an OS, most OSes do it quite well. Each 'connection' just reads the file. (Not really a 'connection'... except in the sense that your application kind-of becomes a server for the text file.)
- data set size
If the data set has less than 1 or 2 thousand records and the records are fairly short, then a text file based approach can be very fast and simple to maintain with the advantage that there is no db expertise required.
- expected traffic/load
If this is to support a web app, traffic load becomes a question. But, heavy traffic can over whelm the text file approach.
In the last year or so, I have taken to writing my simple 'db' type web apps using text files and the DBI:
BD::CSV module(s). The web interface is usually a search interface (no updating). The DBI:BD::CSV approach provides a very light/simple application and with an easy upgrade path to using one of our db servers if the complexity or load of the situation increases significantly. These apps are generally written for internal company clients who can use Excel or QuattroPro or other spreadsheet app (or a text editor) to view/manage their comma delimited data sets. Additionally, the normal report generation with SQL via DBI:BD is straight forward and totally portable.' just another perspective.....
I think many people are seduced by the 'glamour' of a db server and quickly overlook the utility in the text file approach consequently building way to much complexity into some applications/systems (....using the proverbial sledge hammer to drive in the proverbial tack). 'hope this helps
If you are new to Tek-Tips, please use descriptive titles, check the FAQs, and beware the evil typo.
Faster to write or faster to execute? Generally, flat files are faster to write code for, but harder to port to SQL if you haven't included an upgrade path like goBoating suggested (which at that point is going to be the same complexity to write anyway). I'd go with a relational database if you ever expect this program to be upgraded in any way. MySQL is a fine choice. As for PHP vs Perl, I've read many articles saying that Perl is faster, but maybe I was biased in my choice of reading. Perl is probably faster to write the code though (more compact). Perl is also more functional IMHO. And if you're writing for the web, then mod-perl makes it especially fast to execute. Sincerely,
Tom Anderson
Order amid Chaos, Inc.
Tom Anderson
Order amid Chaos, Inc.
- Status