String Memory question.

[tedsmith]

If I had a large string array MyArray(999999,10) and initially only use a few of the elements scattered through the range I notice it only takes up a much smaller use of available memory than if I put data in every element.

My question is, when I dimension the string, is VB6 reserving a 255 byte space for every possible future element or does it only reserve a pointer to where future elements might be so that elements might end up scattered throughout memory?

The reason for this question is I am looking for the fastest way of adding to and searching about 2000 items in a relatively large array without exhausting the available memory. (Items with 6 digit Serial numbers)

 

[tedsmith]

In addition, I seem to remember that basic had a 'string space' that could be exceeded before the available memory is filled.

Is this still the case and if so how can you measure how much you have left?

 

[dilettante]

If I recall correctly a VB6 String variable contains a pointer. If the String is uninitialized or has been set to vbNullString this pointer is set to the special value 0 (a NULL pointer).

When not NULL, the pointer points to a position 4 bytes offset into a block of memory holding the String value. This position is the first byte of the first character of the value.

The block of memory begins with 4 bytes (a Long basically) holding the Len() value of the String value, then contains two bytes for each character of the String value, then a terminating two-byte Unicode NUL character useful when making API calls expecting C strings. This block is a BSTR.

So if you don't set a value in a String array it takes little space.


Then I think there is a 64K byte data segment that VB6 allocates once and uses as a String Cache. This is where those BSTR blocks get allocated up until the point where you exhaust it. This Cache is managed by the VB6 runtime (allocation, deallocation, garbage collection).

After that "slower" memory is used, perhaps a new separately allocated block of memory for each overflow String. The "slow" part probably comes from the need to use OS calls for memory allocation over and over again.

Larger system RAM sizes, improved disk overlay (virtual memory) techniques, newer CPU instructions for block moves, faster CPUs and hard drives, etc. make the penalty for such "slow" memory much less than in the 486 era VB6 was born into.


The details may be off a little above, but I believe that's roughly how it works.

Ahh, you might find this informative too:

http://oreilly.com/catalog/win32api/chapter/ch06.html

 

[strongm]

> basic had a 'string space' that could be exceeded

http://msdn.microsoft.com/en-us/library/aa716182(v=vs.60).aspx

 

[tedsmith]

Thanks, I was not sure of what changes were made since the original basic.
Seems like a small minefield when handling large arrays.

I am trying to visualise what happens as you increase and reduce the lengths of a lot of existing strings. I guess The RAM must get quite fragmented if you say extend a string so it's end is longer than the memory range it originally occupied so it has to be copied to a new memory position.
Is there some sort of equivalent of a defrag routine for string memory otherwise you would be eventually using up more memory for the same string space?

 

[strongm]

> if you say extend a string so it's end is longer than the memory range it originally occupied so it has to be copied to a new memory position

Strings in VB are always moved when they are changed (in fact VB tends to like copying strings to a new location at the drop of an unexpected hat). That's one of the primary reasons string operations are relatively slow in VB

 

[tedsmith]

If you dimension a string as fixed size is it still moved about when other non fixed strings are changed?
Are fixed stings quicker?

 

[strongm]

>is it still moved

You can check this out for yourself:

[blue]    Dim a As String * 4
    Dim b As String
    
    a = "Mike"
    b = "Left"
    
    Debug.Print StrPtr(a)
    Debug.Print StrPtr(b)
    Debug.Print
    b = "only"
    Debug.Print StrPtr(a)
    Debug.Print StrPtr(b)
    Debug.Print
    a = "only"
    Debug.Print StrPtr(a)
    Debug.Print StrPtr(b)[/blue]

 

[tedsmith]

Hmm
Then is there any advantage in speed in presenting a criteria in direct form rather than first putting the criteria into a string then presenting it?

Eg
openrecordset ("SELECT * From MyTable WHERE MyName='Ted'", dbopensnapshot)
or
Criteria = "SELECT * From MyTable WHERE MyName='Ted'"
openrecordset (Criteria, dbopensnapshot)

 

[dilettante]

The speed difference there is probably neglible if any. When performaing these sorts of operations you probably lose much more performance making the DBMS parse, parse, and reparse the query string.

If the same query is done more than a few times you're better off with some form of prepared query or stored procedure where it only needs to be "compiled" once then reused. Even Jet 4.0 has simple stored procedures and prepared queries and views. using these with parameters means seldom recompiling SQL.

String performance issues can be drastically reduced by employing string builder classes of varying levels of sophistication. This one only optimizes for concatenation in a naive manner but it works well:

StringStore.cls

Option Explicit
'===========
'StringStore
'===========
'
'A trivial "String Builder" Class designed simply to replace String
'concatenation by a faster alternative relying on Join().
'
'Version 1.0
'
'   Initial version.
'

Private Const REDIMBY_DEFAULT As Long = 10
Private mStore() As String
Private mUBound As Long
Private mReDimBy As Long
Private mLastUsed As Long
Private mLength As Long

Public Property Get Length() As Long
    Length = mLength
End Property

Public Property Get ReDimBy() As Long
    ReDimBy = mReDimBy
End Property

Public Property Let ReDimBy(ByVal RHS As Long)
    If 2000 < RHS Or RHS < 1 Then
        Err.Raise &H8004CC00, TypeName(Me), "ReDimBy must be from 1 to 2000"
    End If
    
    mReDimBy = RHS
End Property

'Default member:
Public Property Get Value() As String
Attribute Value.VB_UserMemId = 0
    Value = Join(mStore, vbNullString)
    ReDim mStore(ReDimBy)
    mLastUsed = 0
    mStore(mLastUsed) = Value
End Property

Public Property Let Value(ByRef RHS As String)
    Clear
    Add RHS
End Property

Public Sub Add(ByRef NewString As String)
    mLastUsed = mLastUsed + 1
    If mLastUsed > mUBound Then
        mUBound = mUBound + mReDimBy
        ReDim Preserve mStore(mUBound)
    End If
    mStore(mLastUsed) = NewString
    mLength = mLength + Len(NewString)
End Sub

Public Sub Clear(Optional ByVal NewReDimBy As Long = -1)
    If NewReDimBy <> -1 Then
        ReDimBy = NewReDimBy
    End If
    ReDim mStore(mReDimBy)
    mUBound = mReDimBy
    mLastUsed = -1
    mLength = 0
End Sub

Private Sub Class_Initialize()
    Clear REDIMBY_DEFAULT
End Sub

You can do even better using a large String buffer you update via the Mid$() statement or using Byte array buffers and CopyMemory.

These are very old techniques.