Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations TouchToneTommy on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

STRTOFILE() Large Files not written out

Status
Not open for further replies.

GriffMG

Programmer
Mar 4, 2002
6,304
FR
I use an included table in many of my applications, in there I store binary files that may be
useful at runtime, documents, .dlls, executables, anything I need after the install.

I search the table, find the binary and use STRTOFILE(binaries.binary,m.Filename) to drop
them where required.

Today someone brought to my attention that some of the files were not being created - in this case
large PowerPoint .pptx files.

It turns out that STRTOFILE() was not able to read the binaries.binary field as it was too big
(16MB or so) so I had to store it in a memvar and then write it out

Code:
PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME)

This seems to be good for up to the file limit we are all used to 2GB

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
 
Yes, it's limitation for STRTOFILE() (VFP6-VFP9). But you can use COPY MEMO [memoname] TO [filename].

mJindrova
 
Hi Mike

StrToFile can write a 2GB file, from a memory variable, or possibly from a filetostr() - not tested, but if the source is a binary field
in a table it did not (for me at least) want to take one in excess (I think) of 16MB directly.

I tried

Didn't work
Code:
StrToFile(tablename.fieldname...
StrToFile((tablename.fieldname)...
StrToFile(Evaluate("tablename.fieldname")...

but this works
Code:
PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME)

Not tried copy memo.... yet

** edit **
This works:

Code:
COPY MEMO BINARIES.BINARY TO (M.FILENAME)

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
 
The help topic on System capacities actually says you can't store more than 16,777,184 bytes into a memory variable. Whereas a blob field is documented to enable up to 2GB in it. I know the SPACE() function ends at that limit. I already know the variable length limit is not as hard as the system capacities tells it is, but it's odd you couldn't just write out a Blob ia FILETOSTR and you indeed used a variable, that by documentation should work worse than the blob field.

There is a simpler solution to write out a blob field to a file, though, Griff, much more straight forward: Using the COPY MEMO command. I think that was extended to also work on blob fields, which are just a binary memo field anyway. Let me try... ...yes.

Code:
Create Cursor Files (wBinary Blob)
Append Blank
Append Memo wBinary From (GetFile()) && optionally with OVERWRITE clause, but unneccesary for a blank Blob
* and to write it out
Copy Memo wBinary  To (Putfile())

I would prefer it over using a variable as intermediate storage for the blob field, even though I know the variable length capacity of the documentation is wrong. It's not clear to which extent you can disregard this limit or whether this only is a limit of some functions handling string parameters, or what actually are the limits. It's clear that COPY MEMO is always working.


Edit: Sorry, I just saw you came to that conclusion after posts of mJindrova and Mike. So I think COPY MEMO is the best solution.

Chriss
 
I'm aware of the 16 Mb limit for memory variables. But I just did this:

[tt]x = FILETOSTR("bigtable.dbf")
? LEN(x[/tt])

and it showed 47,000,050, which is 44.82 MB. And when I did this:

[tt]STRTOFILE(x, "bigtable1.dbf")[/tt]

the resulting table had exactly the same data as the original.

I can't explain that.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads
 
Little oddity

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
 
Another little experiment:

Code:
x = REPLICATE("A", 100000)
y = REPLICATE("B", 100000)
DO WHILE .T.
  x = x + y
  ? TRANSFORM(LEN(x), "999,999,999,999")
ENDDO

The length of the variable got up to well over 100 million before I killed it.

The Help gives 16 MB as "Maximum # of characters per character string or memory variable". But can that be correct?

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads
 
Mike,

yes, the limit for variables is a soft limit in some way, I knew that. And indeed Scott Griff used it too in getting STROTOFILE to work with a variable longer than 16MB, but not with a binary field longer than 16 MB (Well, I assume that limit was crossed). The point is Scott Griff already knows STRTOFILE() works with larger strings, too.

For sake of its simplicity and foreseen usage, I'd prefer to use the pair of commands APPEND MEMO and COPY MEMO to store a file and get it out.

I remember we experimented with STRTOFILE already and I find that thread: thread184-1813541. think it has no definitive result, though. I would accept from your's and Scott's Griff's new experience, that STRTOFILE also has limits, somewhere over 16MB, but not clearly definable, if it matters whether the source str is a variable or a memo/blob field. If that matters by Scott's Griff's experience it's not a matter of the length only, but the storage of "too long" strings in a blob/memo vs a variable. The only thing that bugs me is that the blob source didn't work while that's perfectly fine storing anything up to 2GB. So the "defect", if there is one, is with STRTOFILE. Must be with the function handling the string parameter weirdly.

Chriss
 
I think, Scott Griff, STRTOFILE(FILETOSTR('embeddedfile.xyz')) is usual, if not the only way to get an embedded file out of an EXE back to the hard drive and perhaps that's why you got into the habit of using STRTOFILE. If you embed a DBF with a Memo field and hold files in there, you can of course also use COPY MEMO.

It's all because the basis of getting files out of an EXE can't be FOPEN nor COPY FILE, you have to use FILETOSTR(), but since you can simply USE an embedded DBF, that's not your concern, and then you also don't need to use the other half of it, the STRTOFILE() function.

We can conclude that if you embed files to write them out with the STRTOFILE(FILETOSTR()) trick, you are limited in file sizes. A DBF also limits you to 2GB overall, but you could also use more DBFs. I think the limit could be determined exactly, but I'd be lazy to find out if a DBF and COPY MEMO can be the solution that's safe to use. The only overhead is putting files into a DBF is one step more than just adding them as project files and including them in the build.

Chriss
 
I don't know why I did that again, Griff. Sorry.

Chriss
 
Mike,

your test is not that wrong. But to illustrate where the problem was in the thread I referred to with occurs - It's not just a problem of a specific length limit, it's also a problem that occurs with memos and not string variables! That's strange, because whereas variables are documented to be limited and memo length lmit is at 2gb, it's actually memos that cause the errors!

I said I'm lazy to find out the exact limit, well I now did for the case of the OCCURS function:
Code:
Clear
Create Cursor StringLimits (mString M)
Append Blank
x = Space(16777184)
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString)
x = x + ' '
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString)
Notice the "String is too long to fit" error happens with Occurs(' ', mString), not with Occurs(' ',m.x). So it's a problem with memos and not variables.

Astonishing.

And what happened to Griff (Griff?, yes Griff) with StrToFile() from a memo.

The limit 16777185 also is strange as 16MB is 16777216, 31 bytes more. The more interesting point is that the problematic case is happening for memos only, not for string variables.

I'm not astonished from the earlier discussions, that longer strings are possible, I'm astonished that memos cause problems with lengths that are far below the limit of memos. And since the functions can process longer strings in variables, what's causing the error with memos? So, it's not just the functions limit, but a combination of which functon with which type of string (variable vs. memo) you use.

Len(), as you can also see, Mike, is a function that fortunately has no problem also with longer memos, so it is indeed a problem matrix of function, and type (var/memo) of the string.

Chriss
 
Hi

I'm still here, just reading...

Regards

Griff
Keep [Smile]ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0
 
What I wanted to say, Griff, is.

Your case with StrToFile() was just like the case of DanielEvansJr with Occurs() and the error also only happened with memos, not with string variables.

Chriss
 
No issues either way for me. I even bumped it up to 66,777,184 instead of 16,777,184 and still no issues.

Screenshot_2023-03-29_045759_x55rdv.jpg


Stanley
 
Are you using VFP9 or VFPA?
Well, rhetorical question, your screenshot shows the caption is VFP Advanced.

Chriss
 
If you use VFPA, try SPACE(2147483647), that's exactly 2 GB-1 byte and should be no problem for a 64bit process, though it won't be stored into a memo, even with VFPA. As far as I know, the table structure has still the 2GB limits and besides 2GB being the theoretical upper limit for a memo field, the FPT file for such fields also has a header and can only grow to 2GB overall, so the limit is a bit below 2GB. Surprise me, how far VFPA can take it. In theory, you can up things to 4GB, also file sizes, as unsigned 32bit integers go up to 4294967295.

No matter if you think about 2GB or 4GB, though. The C struct storing VFP variables of all types, including string variables, has a 32bit value for length and the length could grow to at least 2147483647. If used unsigned it could even be 4294967295. So that higher limit would even have been possible in VFP9, the reasoning for the system capacity limitation in VFP9 has to be a choice of the developers, and I can imagine memory management nightmares of strings the size of almost 2GB. The different limits of string vs memo is the only new thing I learned and that strings can hold more bytes than memos is the biggest surprise about that.

Chriss
 
Notice:
Large memory support will be in VFPA 10.2.

But MEMO field max size is 2GiB. FPT file size is unlimited (10.1)

mJindrova
 
Hi,

I tested Chen’s new downloads and found them to work. I posted my findings and a screenshot to and very happy to what I see. Here are the screenshot:
33_a3b0vk.jpg


34_upehrb.jpg


Hopefully that answers your questions...

Stanley
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top