Can I extract file from General field of Visual FoxPro

Question

Can I extract file from General field of Visual FoxPro

vinod563 (Programmer)

(OP)

15 Nov 19 12:12

I have a DBF with a general field with a Test.esd on it.
I need to save to a file xxx.esd, extract from the general field in Visual Foxpro 9.

I found the below code
LOCAL lcFileString
SELECT "MyTable"
COPY TO tmpTable FIELDS YourGeneralField NEXT 1
lcFileString=FILETOSTR("tmpTable.fpt")
ERASE "tmpTable.fpt"
ERASE "tmpTable.dbf"
lcFileString = RIGHT(lcFileString,LEN(lcFileString) - 599)
=STRTOFILE(lcFileString, "MyExport.bmp")

tested and this works for BMP but not for ESD. where I am missing ??

GriffMG (Programmer) · Answer 1 · 2019-11-15 12:24:04

Hi

Each file type has a different 'header' in the case of a .BMP it's 599 characters, I do not know the value for an .ESD but you can work it out
Find an ESD file, APPEND GENERAL it to your table, then write it out in full to disk, the header will be the difference in size between the two files, your old one and the new one.

Tada

Regards

Griff
Keep Smile ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

Mike Lewis (Programmer) · Answer 2 · 2019-11-15 13:48:09

Can you clarify exactly what your EDF file is? An EDF could be Microsoft's Electronic Software Download format. Or an ExpertScan Survey Document. Or possibly several other things.

The reason I am asking is that you might the file's own application to be installed in order to be able to extract from a General filed.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips and downloads

Olaf Doschke (Programmer) · Answer 3 · 2019-11-15 16:32:00

Maybe we should clarify a general field isn't a Blob field.

It's very specific to MS. The only other place I know it is in Access as OLE Object fields, they are about the same as a VFP general field.

In the case of BMP, it just happens the original BMP file is part of the Gen field content, I faintly remember that is the case and it's prepended by a thumbnail. Aside from that, an FPT file has a header, That altogether might make it possible to find the BMP content after 599 bytes, but that's not a constant and it's even not generally true you can get back the original file or an equivalent. General fields are typically bloating the content of a file. Just imagine how any imaging software reads in any picture, a typical JPG, which is compressed, into the uncompressed format orf RGB pixels in memory. That plus some software-specific overhead makes an OLE image object, and only in case, an imaging software makes use of the OLE idea at all. Most modern imaging software will rather handle a picture as a byte array or such low-level things and co-works with graphics card drivers even circumventing the GDI structures about Graphics and Bitmaps.

The code you posted stripes off FPT header and likely the thumbnail of a BMP in a general field and so gets at the core content. But that's only possible when the OLE object is almost just like the original file. I guess if you store a JPG you could get back a BMP worth a try, just to prove how the content of a general field degrades from what the file was. In the case of images, you could at least go back once you know it's becoming a BMP.

And to make it even clearer: This doesn't work with any file having a file association with a software, it has to be an association with an OLE Server, because what VFP stores is the OLE object data itself, the idea is, once you have that you can skip all the steps from what's stored in the file to how it's represented in memory as an OLE object, VFP and its Olebound control then bind the OLE object to the Ole Server and that shows it.

But this and the nature of these OLE object contents makes such data dependent on a computer, sometimes with non-normal solutions even just this one computer. The most famous way to let General field content work on a newer computer is by reestablishing the association of BMP files with an MS Paint Ole Server. But that also just solves the problem for BMP content, not for any content.

I haven't said this for a long time, and last time I said it this was also already advice previous experts gave for a very long time already: Don't use Gen fields except as temporary usage for Olebound controls. Always stay with the files and keep them.

Not your fault, but you're almost up to yourself knowing how an ESD file structure is defined to maybe find a header within the FPT file your code generates with COPY TO tmpTable FIELDS YourGeneralField NEXT 1. Identify the byte offset where a typical ESD file header begins and try to save from that position to the end as a new file. That's the only chance you have from the DBF file alone.

The other thing is on the computer the DBF/FPT originated from, the general field should work and show whatever Ole object content in an Olebound control. Sometimes even interactive. For example, storing a Word,Document in a general field and displaying that in an Olebound control in VFP together with Word installed can enable this to become an editable Word document within a VFP form. And in that case, you may copy that out into a Word instance and save as word document. Similarly, some images might become zoomable content in an Olebound control. then you might maximize that and get out an image in BMP, GIF, PNG or JPG format as a screenshot.

Since that or similar things typically work, some people will never listen to the advice, but you a) really lose the original file and b) depend not only on VFP the table and OleBound control but also other software and often ole servers you don't even know, no matter if you populate a general field with APPEND GENERAL... DATA... CLASS... clause or FROM Filename. When you just specify a filename what happens is still finding out which Ole Server is associated with that file type and that's used to create the ole object that's finally stored. The same happens when you add an Ole control on a VFP form and choose to create the Ole object from a file.

Bye, Olaf.

Olaf Doschke Software Engineering
https://www.doschke.name

GriffMG (Programmer) · Answer 4 · 2019-11-15 16:50:40

Olaf is mostly right, the original code doesn't even work on my Windows 10 machine (if I append general on windows 10), I have to take the first 1239 characters away from the .fpt to make a working .bmp

But, the resultant file is 100% identical to the original. Same size and same content.

Might be different for other file formats.

Regards

Griff
Keep Smile ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

GriffMG (Programmer) · Answer 5 · 2019-11-15 17:07:41

Interestingly, if you process a .gif in the same way, you need to trim 1348 characters off the beginning - and it leaves a little 'tail' with a reference to the original file (albeit in a different location) on the end.

This doesn't seem to affect the files readability though.

Same thing with .jpg, swipe 1344 chars off the beginning, the balance has a little tail, but the file is otherwise identical to the original.

Going to try a pdf

Exact same thing with a pdf, 1342 off the front, short tail, contents identical - opens perfectly.

Regards

Griff
Keep Smile

ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

Olaf Doschke (Programmer) · Answer 6 · 2019-11-15 23:38:22

Griff, if you only ever test with images you will likely not cover a case an ole object differs so much from the original file, that you don't get back seomthing resembling it and will always depend on an ole server to understand the ole object format instead.

But the advice stands now for about 20 years? I mean I know the advice almost from the time I started with VFP6.

Bye, Olaf.

Olaf Doschke Software Engineering
https://www.doschke.name

GriffMG (Programmer) · Answer 7 · 2019-11-16 08:28:06

Maybe you are right and some file types are corrupted - but I've not found any yet.

Regards

Griff
Keep Smile ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

Olaf Doschke (Programmer) · Answer 8 · 2019-11-16 10:03:30

Corrupted isn't describing it well. If a software should display file contents, show a video, play an audio stream or whatever the file itself encodes, the OLE object representing that has to have essentially the same information in it, it would only add to it. But it doesn't have to be the file nor contain it. So, in general, you don't get the file back and you would need to know the in-memory and in the design of an ole server representation of the original file to get back something.

On the other side, you can be sure a Blob field contains 1:1 the bytes you put in there. But a General field isn't that.

Everybody asking about getting back files from a gen field or "decoding" it starts from the wrong assumption the file is always in there somewhere. There mere definition of the general field tells us this isn't the case. There only is APPEND GENERAL to store it and no inverse command Reading the field as tablename.fieldname will not be the file content you once added by APPEND GENERAL. All that is well known.

Besides all that, even in databases like MSSQL with no such 2gb limitations as FPT files have the norm to store files in field types like image or blob has changed towards filetables and letting MSSQL server databases just have a maintenance structure of a file directory, keeping files as themselves.

What the code works with is, that a fresh table with a single record of only one general field with a single gen field value will have a non bloated fpt file, where you can expect a) the content of the general field starting at a specific offset and then b) knowing a secondary offset of a file type within that general field value where to find that. Adn then there typically will be a tail of the fpt file that's not about the file anymore, but may not harm the file functionality. But the offset 599 may be good for BMP, but only for them, and other file types may not simply mean another offset is all you need to find out.

There is a simple way to see whether file types actually work that simple way always. When you append general with a file you then keep and when you find that file to be embedded in the overall fpt file with AT() and then know the offset and even what number of bytes become a tail:

CODE

#Define ccWORKINGDIR "d:\temp"

Local lcFilename, lcFilecontent, lcWorkingDir, lcFPTFile, lcFPTFilecontent, lnOffset

lcFilename = GetFile()
lcFilecontent = FileToStr(lcFilename )

lcWorkingDir = Addbs(Evl(ccWORKINGDIR,GetEnv("TEMP")))
lcFPTFile = (lcWorkingDir+"tmpgeneral.fpt")

Erase (lcWorkingDir+"tmpgeneral.*")
Create Table (lcWorkingDir+"tmpgeneral") (gFile general)
Append Blank
Append General gFile From (lcFilename) 
Use

lcFPTFilecontent = FileToStr(lcFPTFile)
lnOffset = At(lcFilecontent, lcFPTFilecontent)
If lnOffset=0
   ? "The original file's content isn't found in the general field."
Else
   lnTail = Len(lcFPTFilecontent)-Len(lcFilecontent)-(lnOffset-1)
   ? "The file is found in the fpt file by skipping "+Transform(lnOffset-1)+" bytes. And removing a tail of "+Transform(lnTail )+" bytes."
Endif

I tried several files with this, and in case of xlsx you don't find the original bytes. In the case of TXT, which has no OLE server the gen field still stores that, but you get varying offsets, so there is no general TXT related offset.

I tried with the BMP files in the Home()+"\Graphics\Bitmaps\Offctlbr\Large\Color" folder and you actually don't find the BMP file content 1:1 in the FPT, the code branches into "The original file's content isn't found in the general field."

If I then take the code that works for BMP and pull out the BMP image, the files that are extracted work, but are not the original, they are a few bytes longer, but also differ in themselves, not only at the tail. That you get out something that works as the original BMP just tells me the OLE object is differing from the file and might just have got another palette or anything else differing in the memory representation.

I would not rely on that to deduce you always will find something that is good enough as an extract to work as the original file worked. General fields transform the file content you put into them, they don't store files 1:1 as blob fields do.

Bye, Olaf.

Olaf Doschke Software Engineering
https://www.doschke.name

GriffMG (Programmer) · Answer 9 · 2019-11-16 10:59:32

You are certainly right about xlsx files, the one I tried grew from 7KB to 915!

Regards

Griff
Keep Smile ing

There are 10 kinds of people in the world, those who understand binary and those who don't.

I'm trying to cut down on the use of shrieks (exclamation marks), I'm told they are !good for you.

Olaf Doschke (Programmer) · Answer 10 · 2019-11-16 13:25:31

Docx and xlsx are indeed zip files. And their ole object represetnation is likely not zipped. If you only remove the fpt header and store the rest as new file, that's likely the reason you get a much larger file. The BMP case on the other hand shows, even when the ole object representation changes, it could be so slilghtly that the content saved as a file still works.

In short: You never know. And if you don't make the experment at the stage where you put the files into your dbf and then erase the originals, you're bound to have luck to get back working standalone files.
It'a a bit like expecting to get sugar out of any food you add it as ingredient.

Bye, Olaf.

Olaf Doschke Software Engineering
https://www.doschke.name

Olaf Doschke (Programmer) · Answer 11 · 2019-11-16 13:46:13

In contrast here's almost the same code using blob instead of general:

CODE

#Define ccWORKINGDIR "d:\temp"

Local lcFilename, lcFilecontent, lcWorkingDir, lcFPTFile, lcFPTFilecontent, lnOffset, lwBlob

lcFilename = GetFile()
lcFilecontent = FileToStr(lcFilename)

lcWorkingDir = Addbs(Evl(ccWORKINGDIR,GetEnv("TEMP")))
lcFPTFile = (lcWorkingDir+"tmpblob.fpt")

Erase (lcWorkingDir+"tmpblob.*")
Create Table (lcWorkingDir+"tmpblob") (wFile Blob)
Append Blank
Append Memo wFile From (lcFilename)
* just by the way, reading ou t from wFile is as simple as
lwBlob = wFile
If lwBlob<>lcFilecontent   
   ? "the blob content is not the file content" && you will never see this, unless there is a drive failure.    
Endif
Use

lcFPTFilecontent = FileToStr(lcFPTFile)
lnOffset = At(lcFilecontent, lcFPTFilecontent)
If lnOffset=0
   ? "The original file's content isn't found in the blob field."
Else
   lnTail = Len(lcFPTFilecontent)-Len(lcFilecontent)-(lnOffset-1)
   ? "The file is found in the fpt file by skipping "+Transform(lnOffset-1)+" bytes. And removing a tail of "+Transform(lnTail )+" bytes."
Endif

To me this always gives a 520 byte offset in the fpt file. Besides you don't need that trickery to read from the fpt file directly, you can simply address the blob field and get your content back and it's 1:1 unchanged the file content you put into it beforehand.

There are only differing sizes of tail bytes, and that's because like memos Blobs are stored with a blocksize. Some bytes are not used but still part of the fpt file. Anyway, blob fields are just what you put into them and VFP is able to get exactly the content with original length from them, there is no transformation from file to blob as there is from file to general field content.

Goback to the genfield code and try to read the gFile field the way I read wFile into lwBlob just via assignemnt. You're not allowed to read the genfield content. You can bind it to an olebound control and print it in a report via an olebound report control and that's it.

Bye, Olaf.

Olaf Doschke Software Engineering
https://www.doschke.name

Come Join Us!

Posting Guidelines

Can I extract file from General field of Visual FoxPro