×
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Contact US

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

STRTOFILE() Large Files not written out

STRTOFILE() Large Files not written out

STRTOFILE() Large Files not written out

(OP)
I use an included table in many of my applications, in there I store binary files that may be
useful at runtime, documents, .dlls, executables, anything I need after the install.

I search the table, find the binary and use STRTOFILE(binaries.binary,m.Filename) to drop
them where required.

Today someone brought to my attention that some of the files were not being created - in this case
large PowerPoint .pptx files.

It turns out that STRTOFILE() was not able to read the binaries.binary field as it was too big
(16MB or so) so I had to store it in a memvar and then write it out

CODE

PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME) 

This seems to be good for up to the file limit we are all used to 2GB

Regards

Griff
Keep Smileing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0

RE: STRTOFILE() Large Files not written out

Yes, it's limitation for STRTOFILE() (VFP6-VFP9). But you can use COPY MEMO [memoname] TO [filename].

mJindrova

RE: STRTOFILE() Large Files not written out

Griff,

Are you sure that this is a size issue? As a test, I just did a STRTOFILE() on a file that was 44.8 MB, and it worked OK.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

RE: STRTOFILE() Large Files not written out

(OP)
Hi Mike

StrToFile can write a 2GB file, from a memory variable, or possibly from a filetostr() - not tested, but if the source is a binary field
in a table it did not (for me at least) want to take one in excess (I think) of 16MB directly.

I tried

Didn't work

CODE

StrToFile(tablename.fieldname...
StrToFile((tablename.fieldname)...
StrToFile(Evaluate("tablename.fieldname")... 

but this works

CODE

PRIVATE M.FILECONTENTS
M.FILECONTENTS = BINARIES.BINARY
STRTOFILE(M.FILECONTENTS,M.FILENAME) 

Not tried copy memo.... yet

** edit **
This works:

CODE

COPY MEMO BINARIES.BINARY TO (M.FILENAME) 

Regards

Griff
Keep Smileing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0

RE: STRTOFILE() Large Files not written out

The help topic on System capacities actually says you can't store more than 16,777,184 bytes into a memory variable. Whereas a blob field is documented to enable up to 2GB in it. I know the SPACE() function ends at that limit. I already know the variable length limit is not as hard as the system capacities tells it is, but it's odd you couldn't just write out a Blob ia FILETOSTR and you indeed used a variable, that by documentation should work worse than the blob field.

There is a simpler solution to write out a blob field to a file, though, Griff, much more straight forward: Using the COPY MEMO command. I think that was extended to also work on blob fields, which are just a binary memo field anyway. Let me try... ...yes.

CODE

Create Cursor Files (wBinary Blob)
Append Blank
Append Memo wBinary From (GetFile()) && optionally with OVERWRITE clause, but unneccesary for a blank Blob
* and to write it out
Copy Memo wBinary  To (Putfile()) 

I would prefer it over using a variable as intermediate storage for the blob field, even though I know the variable length capacity of the documentation is wrong. It's not clear to which extent you can disregard this limit or whether this only is a limit of some functions handling string parameters, or what actually are the limits. It's clear that COPY MEMO is always working.


Edit: Sorry, I just saw you came to that conclusion after posts of mJindrova and Mike. So I think COPY MEMO is the best solution.

Chriss

RE: STRTOFILE() Large Files not written out

I'm aware of the 16 Mb limit for memory variables. But I just did this:

x = FILETOSTR("bigtable.dbf")
? LEN(x
)

and it showed 47,000,050, which is 44.82 MB. And when I did this:

STRTOFILE(x, "bigtable1.dbf")

the resulting table had exactly the same data as the original.

I can't explain that.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

RE: STRTOFILE() Large Files not written out

(OP)
Little oddity

Regards

Griff
Keep Smileing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0

RE: STRTOFILE() Large Files not written out

Another little experiment:

CODE -->

x = REPLICATE("A", 100000)
y = REPLICATE("B", 100000)
DO WHILE .T.
  x = x + y
  ? TRANSFORM(LEN(x), "999,999,999,999")
ENDDO 

The length of the variable got up to well over 100 million before I killed it.

The Help gives 16 MB as "Maximum # of characters per character string or memory variable". But can that be correct?

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

RE: STRTOFILE() Large Files not written out

Mike,

yes, the limit for variables is a soft limit in some way, I knew that. And indeed Scott Griff used it too in getting STROTOFILE to work with a variable longer than 16MB, but not with a binary field longer than 16 MB (Well, I assume that limit was crossed). The point is Scott Griff already knows STRTOFILE() works with larger strings, too.

For sake of its simplicity and foreseen usage, I'd prefer to use the pair of commands APPEND MEMO and COPY MEMO to store a file and get it out.

I remember we experimented with STRTOFILE already and I find that thread: thread184-1813541: Memo field operations lead to "String is too long to fit" error.. think it has no definitive result, though. I would accept from your's and Scott's Griff's new experience, that STRTOFILE also has limits, somewhere over 16MB, but not clearly definable, if it matters whether the source str is a variable or a memo/blob field. If that matters by Scott's Griff's experience it's not a matter of the length only, but the storage of "too long" strings in a blob/memo vs a variable. The only thing that bugs me is that the blob source didn't work while that's perfectly fine storing anything up to 2GB. So the "defect", if there is one, is with STRTOFILE. Must be with the function handling the string parameter weirdly.

Chriss

RE: STRTOFILE() Large Files not written out

I think, Scott Griff, STRTOFILE(FILETOSTR('embeddedfile.xyz')) is usual, if not the only way to get an embedded file out of an EXE back to the hard drive and perhaps that's why you got into the habit of using STRTOFILE. If you embed a DBF with a Memo field and hold files in there, you can of course also use COPY MEMO.

It's all because the basis of getting files out of an EXE can't be FOPEN nor COPY FILE, you have to use FILETOSTR(), but since you can simply USE an embedded DBF, that's not your concern, and then you also don't need to use the other half of it, the STRTOFILE() function.

We can conclude that if you embed files to write them out with the STRTOFILE(FILETOSTR()) trick, you are limited in file sizes. A DBF also limits you to 2GB overall, but you could also use more DBFs. I think the limit could be determined exactly, but I'd be lazy to find out if a DBF and COPY MEMO can be the solution that's safe to use. The only overhead is putting files into a DBF is one step more than just adding them as project files and including them in the build.

Chriss

RE: STRTOFILE() Large Files not written out

I don't know why I did that again, Griff. Sorry.

Chriss

RE: STRTOFILE() Large Files not written out

Mike,

your test is not that wrong. But to illustrate where the problem was in the thread I referred to with occurs - It's not just a problem of a specific length limit, it's also a problem that occurs with memos and not string variables! That's strange, because whereas variables are documented to be limited and memo length lmit is at 2gb, it's actually memos that cause the errors!

I said I'm lazy to find out the exact limit, well I now did for the case of the OCCURS function:

CODE

Clear
Create Cursor StringLimits (mString M)
Append Blank
x = Space(16777184)
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString)
x = x + ' '
Replace mString with x
? Len(m.x),        Len(mString),  Occurs(' ',m.x), Occurs(' ',mString) 
Notice the "String is too long to fit" error happens with Occurs(' ', mString), not with Occurs(' ',m.x). So it's a problem with memos and not variables.

Astonishing.

And what happened to Griff (Griff?, yes Griff) with StrToFile() from a memo.

The limit 16777185 also is strange as 16MB is 16777216, 31 bytes more. The more interesting point is that the problematic case is happening for memos only, not for string variables.

I'm not astonished from the earlier discussions, that longer strings are possible, I'm astonished that memos cause problems with lengths that are far below the limit of memos. And since the functions can process longer strings in variables, what's causing the error with memos? So, it's not just the functions limit, but a combination of which functon with which type of string (variable vs. memo) you use.

Len(), as you can also see, Mike, is a function that fortunately has no problem also with longer memos, so it is indeed a problem matrix of function, and type (var/memo) of the string.

Chriss

RE: STRTOFILE() Large Files not written out

(OP)
Hi

I'm still here, just reading...

Regards

Griff
Keep Smileing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0

RE: STRTOFILE() Large Files not written out

What I wanted to say, Griff, is.

Your case with StrToFile() was just like the case of DanielEvansJr with Occurs() and the error also only happened with memos, not with string variables.

Chriss

RE: STRTOFILE() Large Files not written out

No issues either way for me. I even bumped it up to 66,777,184 instead of 16,777,184 and still no issues.



Stanley

RE: STRTOFILE() Large Files not written out

Are you using VFP9 or VFPA?
Well, rhetorical question, your screenshot shows the caption is VFP Advanced.

Chriss

RE: STRTOFILE() Large Files not written out

If you use VFPA, try SPACE(2147483647), that's exactly 2 GB-1 byte and should be no problem for a 64bit process, though it won't be stored into a memo, even with VFPA. As far as I know, the table structure has still the 2GB limits and besides 2GB being the theoretical upper limit for a memo field, the FPT file for such fields also has a header and can only grow to 2GB overall, so the limit is a bit below 2GB. Surprise me, how far VFPA can take it. In theory, you can up things to 4GB, also file sizes, as unsigned 32bit integers go up to 4294967295.

No matter if you think about 2GB or 4GB, though. The C struct storing VFP variables of all types, including string variables, has a 32bit value for length and the length could grow to at least 2147483647. If used unsigned it could even be 4294967295. So that higher limit would even have been possible in VFP9, the reasoning for the system capacity limitation in VFP9 has to be a choice of the developers, and I can imagine memory management nightmares of strings the size of almost 2GB. The different limits of string vs memo is the only new thing I learned and that strings can hold more bytes than memos is the biggest surprise about that.

Chriss

RE: STRTOFILE() Large Files not written out

Notice:
Large memory support will be in VFPA 10.2.

But MEMO field max size is 2GiB. FPT file size is unlimited (10.1)

mJindrova

RE: STRTOFILE() Large Files not written out

Hi,

I tested Chen’s new downloads and found them to work. I posted my findings and a screenshot to https://www.foxite.com/archives/0000487428.htm and very happy to what I see. Here are the screenshot:




Hopefully that answers your questions...

Stanley

RE: STRTOFILE() Large Files not written out

Well, Martina already said Chen got rid of the overall 2GB limit of FPT files. And will allow larger strings in 10.2.

You're showing you can now also write and read longer than 2GB files with the file functions FREAD and STRTOFILE. I wondered about the maximum string variable length. You show can grow larger now, too, so I assume that is VFPA 10.2 now.

I talked about a memory management nightmare, that is because of how strings are stored in variables and memory, not only in VFP, with a length and a pointer to allocated memory. VFP allocates the length of the string to store, so an operation that makes a string longer, like lcText=lcText+'.' will lead to allocating a new memory block 1 byte larger, as VFP doesn't reserve more bytes for spare usage. And that leads to a lot of memory allocations and deallocations when operating on large strings. And since 32bit processes only can have 4GB of process memory you can easily clutter the memory and peak the need for garbage collection, if you would allow large memory strings. Well, it's not a memory management nightmare for us, using VFP, it's a nightmare to program variable handling.

That is to say that Chen (hopefully) has done a good job that will make use of ideas like provisionally allocating more memory allowing strings to grow without needing to full copy them in RAM just because they grow by 1 byte. I'm confident it all works, but I'm still saying it's not necessary. At least not to me. Have fun with VFPA, stanlyn.

Chriss

RE: STRTOFILE() Large Files not written out

(OP)
Stanlyn,

Do you have a link to a download for VFPA 64 bit? The originator's site seems to be overloaded...

Regards

Griff
Keep Smileing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

There is no place like G28 X0 Y0 Z0

RE: STRTOFILE() Large Files not written out

Chris,
I'm using VFPA32 10.1



Griff,
How do I contact you privately as I see no way within the forum to message you. I have both the 32 and 64 bit versions, but only installed the 32b version as I have several 32 bit 3rd party sdks I use.

Stanley

RE: STRTOFILE() Large Files not written out

Griff,

Quote (Griff)

Do you have a link to a download for VFPA 64 bit? The originator's site seems to be overloaded...

I think here it's there.

I have not tried it yet, though!

Rajesh

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members! Already a Member? Login


Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close