using data from text file faster

[RSGB]

Hi guys,

I have taken over a program (which graphically shows some analysis of data) which is rather slow and I was wondering if anyone has any tips on making it quicker.

On running the program, it reads in a text file using line input (the file contains a header line which explains the structure of each subsequent record) into a string array - myRecord(i)

That part isn't slow but the next is.

Each analysis routine when run reads through each record in the myRecord(i) string and extracts the relevant bits of data using Mid.

I've heard that converting to a byte array would help - does anyone know how I would go about this or if there are other better ways?

Thanx in advance,

RSGB

 

[MichaelRed]

Hmmmmmmmmmmmmmmmmmmmm ... mmmmmmmmmmmm ... mmmmmmmmmmm

some interesting things here?

It is almost ALWAYS better (well at least FASTER) to just 'dump' a whole text file into memory and deal with the parseing stuff after the capture. In general this is done with a single "Get" after tricking the process into deciding the rrecord length is the same as the length of the file.

After that, it is easy to Parse the "lump" of text into seperate records (LINES) using (VB6) Split, or (Earlier vers) basSplit (posted numerous times already).

Since the process of reaking the record into bits/bytes and other logical units of "Fields" is done via the info in the first line, your existing process must know how to use the first line (0th element of the record array) to set up the remainder of the parseing process, so this code must be available and can just be "fed" this record. Llikewise, the remaining "records" are parsed acording to the rules set up in the first record, and are parsed one record at a time, so you can still pass the individual records to this routine from the array of records.

If this sounds difficult, I would accept a small sample of one of your files and the current module and implement the changes for you.

MichaelRed
m.red@att.net

There is never time to do it right but there is always time to do it over

 

[RSGB]

Hi MichaelRed,

Thanks for your reply (and your offer of help!).

Actually to clarify, the first header line was read in using Input then the following record lines using LineInput.

I haven't changed the way the data is loaded (yet) because speed isn't a problem here (users are more tolerant of load time if the program is quick in use, in any case). But I will try the Get method as you suggest.

Anywayz what I've done is change it so instead of reading each record into MyRecord(i) and the analysis routines using Mid on it, it now loads each line into 12 3-dimensional dynamic arrays and each analysis routine just looks at the relevant bits of these.

It means I only deal with long strings once on loading the data and for the largest data set (22000 records with 450 columns of data each) analysis time is typically halved from 10 secs to 5.

Most data sets have around a quarter the number of records so it makes it acceptable

Thanks again,

RSGB