Reading text file to array - why is it slow? 1

[Motor11]

VB-ers,

I just discovered this site this morning and must say that I am excited by the find. This appears to be a great forum.

My question is this: I stumbled upon a seemingly great way to load a text file to an array in the FAQ. The code is:

Get #ff, 1, wholefile
Close #ff

If InStr(wholefile, vbCrLf) > 0 Then
endline = vbCrLf 'ASCII DOS format
Else
endline = vbLf 'ASCII UNIX format
End If

ReDim individ_lines(0)
Do
Cloc = InStr(wholefile, endline)
If Cloc > 0 Then 'Parse the data
ReDim Preserve individ_lines(0 To UBound(individ_lines) + 1)
tmp = Left(wholefile, Cloc - 1)
individ_lines(UBound(individ_lines)) = tmp
Else
Exit Do 'All data has been parsed
End If
wholefile = Right(wholefile, Len(wholefile) - (Len(tmp) + Len(endline)))
Loop


This would seem to be a great and quick way to parse the file. Why is it so much slower than my sloppy method below?

Do While (Not EOF(1))

inchar = ""
readline = ""
Do While (Not EOF(1)) And (inchar <> vbLf)
readline = readline + inchar
inchar = Input(1, #1)
Loop
If Len(readline) > 0 Then
'I parse the line of text here
End If
readline = &quot;&quot;

Loop

As you can see, I parse the file character by character. I would think the second method would be much slower, but it is actually faster. Any thoughts? (I use VB 5.)

Ryan

 

[CajunCenturion]

You might want to try using the Split function.

Dim individ_lines() as String

Get #ff, 1, wholefile
Close #ff

If InStr(wholefile, vbCrLf) > 0 Then
endline = vbCrLf 'ASCII DOS format
Else
endline = vbLf 'ASCII UNIX format
End If

individ_lines = Split (wholefile, endline ) Good Luck
--------------
As a circle of light increases so does the circumference of darkness around it. - Albert Einstein

 

[Motor11]

CajunCenturion

Is split something new for VB 6?

Ryan

(PS: fellow Louisianian?)

 

[vb5prgrmr]

Memory Management and the stack for one reason i.e. how many time does the function reallocate memory (redim). How many times must the UBound function traverse the array looking for the upper bound? And, you end up having the file in memory twice.


BTW

You can also use (which may be faster than your code)

Dim Txt As String, FlNum As Integer, FlName as String

FlName = &quot;C:\Temp.txt&quot;
FlNum = FreeFile

Open FlName For Input As #FlNum

Do While Not Eof(FlNum)

Line Input #1, Txt 'the whole line is read in
'do what you want with the line

Loop

Close #FlNum


Or if you want the whole file in a string variable all at once



Dim Txt As String, FlNum As Integer, FlName as String

FlName = &quot;C:\Temp.txt&quot;
FlNum = FreeFile

Dim Txt As String, FlNum As Integer, FlName As String

FlName = &quot;C:\z\test.txt&quot;
FlNum = FreeFile

Open FlName For Input As #FlNum

Txt = Input(FileLen(FlName), #FlNum)

Close #FlNum

 

[Motor11]

Thanks for the quick responses.

I think you're correct about the memory allocation issue, Vb5prgrmr, the file-at-once method seems fast, but only for relatively small files. Large files slow it down substantially.

The line input command would be perfect for what I need to do, but I can't seem to make it work with UNIX files (ie: EOL = line feed (no carriage return)).

I may be stuck with my character-by-character method.

Ryan

 

[CajunCenturion]

Perhaps Split is new to VB6, with VB5, I would suggest that your write your own split function, something like the following: and replace the call from Split to MySplit

Public Function MySplit(Expression As String, Delimeter As String) As String()

Dim StartPos As Integer
Dim EndPos As Integer
Dim ArrSub As Integer
Dim Tokens() As String

ArrSub = -1
StartPos = 1
EndPos = InStr(StartPos, Expression, Delimeter)
Do While (EndPos > 0)
ArrSub = ArrSub + 1
ReDim Preserve Tokens(ArrSub)
Tokens(ArrSub) = Trim(Mid(Expression, StartPos, (EndPos - StartPos)))
StartPos = EndPos + 1
EndPos = InStr(StartPos, Expression, Delimeter)
Loop
ArrSub = ArrSub + 1
ReDim Preserve Tokens(ArrSub)
Tokens(ArrSub) = Trim(Mid(Expression, StartPos))
MySplit = Tokens

End Function

And yes, fellow Louisianian Good Luck
--------------
As a circle of light increases so does the circumference of darkness around it. - Albert Einstein

 

[JohnYingling]

&quot;wholefile = Right(wholefile, Len(wholefile) - (Len(tmp) + Len&quot;
All that string allocation costs big time. &quot;Traverse&quot; the string.
Do not ReDim for every element. Since a Redim of new elements only costs the size of the empty elements, allocate double every time you run out.
ReDim individ_lines(0)
Dim I as Long
Dim L as Long
Dim K as Long
I = 1
K = -1 ' Next element to store
Do While (I <= Len(WholeFile))
Cloc = InStr(I, wholefile, endline)
' last line need not have terminator
If Cloc = 0 Then Cloc = Len(wholeFile +1)
K = K + 1 ' new element
if K > Ubound(individ_Lines) then
ReDim Preserve individ_lines(2 * K)
end if
individ_lines(K) = Mid(wholefile, I, Cloc - I)
I = Cloc + Len(endLine) ' bump over terminator
Loop
if K > -1 then
ReDim Preserve individ_lines(K) ' Resize to actual
else
...empty file
end if

http://www.VBResizer.com

Forms/Controls Resizing/Tabbing Control

http://www.VBCompare.com

Compare Code (Text)

http://www.VBSortGen.com

Generate Sort Class in VB or VBScript