Format of SAVE TO field 1

[foxmuldr2]

Does anybody know the format of the SAVE TO filename.ext fields which are datetime or date?

For example, if you examine the file for numeric, character, logical, they're pretty clear. But it seems the datetime and date fields are encoded.

Best regards,
Rick C. Hodgin

 

[Mike Lewis]

Rick, the entire file is in a binary format. It's not a text file. You could probably hack the format if you need to, but it's not meant to be humanly readable.

The point is that, when you issue a RESTORE FROM, the variables will all be back to where they were before.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[JRB-Bldr]

You could probably hack the format if you need to

But if you did that, then the RESTORE FROM would most likely not work correctly - which may or may not be important to you.

Good Luck,
JRB-Bldr

 

[IlyaRabyy]

Is SAVE TO the only option, colleague?
If it's not - would you like to consider saving your memvars in a DBF file? Or in an ASCII file, i.e.

DISPLAY MEMORY TO FILE (cMyINIFile) NOCONSOLE

Or a combination of the two, i.e.

DISPLAY MEMORY TO FILE (cMyINIFile) NOCONSOLE
SELECT MY_INI_TABLE
APPEND MEMO MY_INI_TABLE.Mem_Dump OVERWRITE

Then, you won't have any trouble reading your memvars, neither determining which one is of date/date-time format.
How's that for a solution, agent Mulder? ;-)

Regards,

Ilya

 

[foxmuldr2]

I am aware of alternate ways of obtaining the data. I have a need to manually parse the SAVE TO file.

I am asking the question:

Does anybody know the encoding format of the date or datetime fields within the SAVE TO file?

Thank you for answering THIS question.

Best regards,
Rick C. Hodgin

 

[foxmuldr2]

I am creating a unique system which does a runtime capture of information allowing for a post-mortem analysis of the program flow. Part of that post-mortem analysis is the examination of memory variables which were of particular values at the time.

I have everything working except date and datetime. I was asking if anybody knows these structures so I don't have to hack at it and figure it out. I seem to remember from years ago (early 2000s) knowing this structure and even using it. But, it's lost now with my aging 42-yr old mind. So ... whatever.

I do appreciate your responses. But yes, I do need THIS SPECIFIC solution.

Best regards,
Rick C. Hodgin

 

[Mike Lewis]

Rick,

I understand that you are not actually proposing to use a MEM file to save and restore variables. Presumably, you have acquired or imported a MEM file form somewhere, and you need to retrieve its contents. Good.

I've been doing some searches in an attempt to find the information you need, but have so far drawn a blank.

However, if you are proposing to programmatically read the file, why not instead write a simple utility that issues a RESTORE FROM, without an ADDITIVE clause? That way, all the values saved in the file (including dates and datetimes) will be availalbe in memory, in their normal format.

You could then go one step further. From within your utility, issue a DISPLAY MEMORY TO FILE (as per Ilya's suggestion, above), to get the variables into a text file. The values will now be available as plain text, which you can parse in the usual way - I'm sure you know how to do that.

Any reason why that wouldn't work?

Mike





__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[foxmuldr2]

Doing so could overwrite the variables I already have in memory.

As I say, I have the code setup to parse the SAVE TO file. Here it is below (most of it, some functions are missing but their implementation should be pretty obvious).

I just need the datetime and date portions.

Best regards,
Rick C. Hodgin

/* C/C++ code */
typedef     unsigned __int64     u64;
typedef     unsigned long        u32;
typedef     unsigned short       u16;
typedef     unsigned char        u8;

typedef     LARGE_INTEGER        i64;
typedef     __int64              s64;
typedef     long                 s32;
typedef     short                s16;
typedef     char                 s8;

typedef     float                f32;
typedef     double               f64;


struct SSaveToVar
{
    s8            name[11];       // First 10 of name, "FOO",00,00,00,00,00,00,00,[trailing NULL]
    u8            type;           // CH,N,I,etc, if lower-case, then variable name is long and is stored in the data area
    u8            unk1[1];        // 00
    u32           length;         // 00,00,00,18 (includes trailing null)
    u8            decimals;       // 00
    u8            unk2[7];        // 00,00,00,00,00,00,00
    u8            val;            // 03
    u8            unk3[6];        // 00,00,00,00,00,00
    // Total 32 bytes
};

// These formats immediately follow SSaveToVar
struct SSaveToVarLongName
{
    u16           length;         // Length of the name (bytes of name immediately follow)
};

struct SSaveToVarArray
{
    u16            rows;          // 02,00
    u16            cols;          // 03,00
};

struct SSaveToVarFloat
{
    f64            fVal;          // 8-byte floating point value
};

struct SSaveToVarLogical
{
    u8            lVal;           // 8-bit indicator, 0=false, 1=true
};

struct SSaveToVarDate
{
    u64            dVal;          // 8-byte date value
};

struct SSaveToVarDatetime
{
    u64            dVal;          // 8-byte datetime value
};


//////////
//
// Called to parse the SAVE TO filename file, to read all variables
// and return them in a list of quote-comma delimited line items,
// such as:
//
//        (C:8)foo = whatever
//        (N:5,2)i = 12.03
//        lnArray[1,1](C:4) = foo
//        lnArray[1,2](N:1) = 0
//
//////
    u32 process_save_to_file(s8* tcInputFile, s8* tcOutputFile)
    {
        u32                    lnLength, lnRow, lnCol;
        STemplate*            builder;
        // Holds loaded content
        s8*                    lcData;
        u32                    lnDataLength;
        SSaveToVar*            stv;
        SSaveToVarArray*    stva;


        //////////
        // Try to open the specified input file
        //////
            iLoadFileContents(tcInputFile, NULL, &lcData, &lnDataLength, NULL);
            if (lcData == NULL)
                return(0);
            // If we get here, we're good, we have our loaded buffer


        //////////
        // Allocate our builder accumulation buffer
        //////
            iAllocateAndInitializeAccumulationTemplate(&builder);


        //////////
        // Iterate through the data
        //////
            stv            = (SSaveToVar*)lcData;
            while ((u32)stv < (u32)lcData + lnDataLength && stv->name[0] != 0x1a/* VFP uses CHR(26) (which is hexadecimal 0x1a) as the terminator for the file (the CTRL+Z character)*/)
            {
                // Find out what type of variable it is
                if (stv->type == 'a' || stv->type == 'A')
                {
                    // It's an array, special processing
                    stva = (SSaveToVarArray*)(stv + 1);        // Array data comes immediately after initial header

                    //////////
                    // Store the name
                    /////
                        iSaveTo_StoreName(builder, stv);


                    //////////
                    // Store the array portion after the name above, so it's like:
                    //    foo[5,5] - array
                    //////
                        iAppendToTemplate(builder, "[", 1, NULL, NULL);
                        iAppendToTemplateInteger(builder, (s32)stva->rows, NULL, NULL);
                        if (stva->cols != 0)
                        {
                            iAppendToTemplate(builder, ",", 1, NULL, NULL);
                            iAppendToTemplateInteger(builder, (s32)stva->cols, NULL, NULL);
                        }
                        iAppendToTemplate(builder, "] - array", 9, NULL, NULL);
                        // Append CR/LF after
                        iAppendToTemplate(builder, "\r\n", 2, NULL, NULL);


                    //////////
                    // Now, move stv forward to the start of the next thing
                    //////
                        stv = (SSaveToVar*)((s8*)(stv+1) + sizeof(SSaveToVarArray));


                    //////////
                    // Repeat for each entry that will follow, for both dimensions of the array
                    //////
                        for (lnRow = 1; lnRow <= (u32)stva->rows; lnRow++)
                        {
                            // Store the array reference
                            if (stva->cols == 0)
                            {
                                // No columns, single-dimension array
                                // Store the name
                                iSaveTo_StoreName(builder, stv);
                                iAppendToTemplate(builder, "[", 1, NULL, NULL);
                                iAppendToTemplateInteger(builder, lnRow, NULL, NULL);
                                iAppendToTemplate(builder, "]", 1, NULL, NULL);

                                // Store the data
                                stv = iSaveTo_StoreData(builder, stv, false);

                                // Append CR/LF after
                                iAppendToTemplate(builder, "\r\n", 2, NULL, NULL);

                            } else {
                                // Two-dimensional array
                                for (lnCol = 1; lnCol <= (u32)stva->cols; lnCol++)
                                {
                                    // Store the name
                                    iSaveTo_StoreName(builder, stv);
                                    iAppendToTemplate(builder, "[", 1, NULL, NULL);
                                    iAppendToTemplateInteger(builder, lnRow, NULL, NULL);
                                    iAppendToTemplate(builder, ",", 1, NULL, NULL);
                                    iAppendToTemplateInteger(builder, lnCol, NULL, NULL);
                                    iAppendToTemplate(builder, "]", 1, NULL, NULL);

                                    // Store the data
                                    stv = iSaveTo_StoreData(builder, stv, false);

                                    // Append CR/LF after
                                    iAppendToTemplate(builder, "\r\n", 2, NULL, NULL);
                                }
                            }
                        }
                        // When we get here, every array has been handld

                } else {
                    // Store the type and data
                    stv = iSaveTo_StoreData(builder, stv, true);

                    // Append CR/LF after
                    iAppendToTemplate(builder, "\r\n", 2, NULL, NULL);

                }
            }


        //////////
        // Copy it back out for the caller
        /////
            lnLength = builder->currentLength;
            utils_write_to_file(tcOutputFile, 0, 0, builder->buffer, lnLength);


        //////////
        // Release our accumulation buffer and source data buffer
        /////
            iFreeAndReleaseAccumulationTemplate(&builder, true);
            free(lcData);


        // All done
        return(lnLength);
    }

    void iSaveTo_StoreName(STemplate* builder, SSaveToVar* stv)
    {
        u32                    length;
        s8*                    updateAfter;
        s8*                    start;
        SSaveToVarLongName*    stvln;


        if (stv->type >= 'a' && stv->type <= 'z')
        {
            // It's a lower-case letter, meaning it has a long variable name
            stvln    = (SSaveToVarLongName*)(stv+1);
            length    = stvln->length;
            start    = (s8*)(stvln+1);

        } else {
            // Regular short variable name
            start    = (s8*)&stv->name;
            length    = strlen((s8*)&stv->name);
        }

        // Store the name
        updateAfter = builder->buffer + builder->currentLength;
        iAppendToTemplate(builder, start, length, NULL, NULL);

        // Fix it up for proper capitalization
        if (length >= 2)
        {
            if (iIsNeedleInHaystack((s8*)cgcKnownFirsts, sizeof(cgcKnownFirsts) - 1, start, 1) && iIsNeedleInHaystack((s8*)cgcKnownSeconds, sizeof(cgcKnownSeconds) - 1, start + 1, 1))
            {
                // It matches the format, so we lower-case the first two characters
                iLowercase(updateAfter, 2);
                // And every character after that
                if (length >= 4)
                    iLowercase(updateAfter + 3, length - 4 + 1/*base-1*/);
            }
        }
    }

    SSaveToVar* iSaveTo_StoreData(STemplate* builder, SSaveToVar* stv, bool tlStoreName)
    {
        s8*                    start;
        SSaveToVarFloat*    stvf;
        SSaveToVarLogical*    stvl;
        SSaveToVarDate*        stvd;
        SSaveToVarDatetime*    stvt;
        s8                    buffer[32];


        //////////
        // Store the type, length and decimals, looks like this:
        // (C:1024)
        // (L:1)
        // (N:18,7)
        //////
            // Store leading parenthesis
            iAppendToTemplate(builder, "(", 1, NULL, NULL);

            // Store the type as upper-case
            sprintf_s(buffer, sizeof(buffer), "%c\000", stv->type);
            if (buffer[0] >= 'a' && buffer[0] <= 'z')
                buffer[0] -= 0x20;        // Convert to upper-case
            iAppendToTemplate(builder, buffer, 1, NULL, NULL);

            // Store colon between type and length data
            iAppendToTemplate(builder, ":", 1, NULL, NULL);

            // Store length data
            iAppendToTemplateInteger(builder, iSwapEndian(stv->length), NULL, NULL);
            if (stv->decimals != 0)
            {
                iAppendToTemplate(builder, ",", 1, NULL, NULL);
                iAppendToTemplateInteger(builder, stv->decimals, NULL, NULL);
            }

            // Store closing parenthesis
            iAppendToTemplate(builder, ")", 1, NULL, NULL);


        //////////
        // Store the name, ends up looking like this
        // (C:1024)foo = 
        //////
            // Store the variable name
            if (tlStoreName)
                iSaveTo_StoreName(builder, stv);
            iAppendToTemplate(builder, " = ", 3, NULL, NULL);


        //////////
        // Get an offset to the actual data
        //////
            start = (s8*)(stv + 1) +
                        (
                            (stv->type >= 'a' && stv->type <= 'z')
                                ? sizeof(SSaveToVarLongName) + ((SSaveToVarLongName*)(stv+1))->length 
                                : 0
                        );


        //////////
        // Store the actual data
        // (C:1024)foo = "whatever"
        // (N:10)foo2 = 5
        // (N:18,7)foo3 = 1.1234567
        //////
            if (stv->type == 'c' || stv->type == 'C' || stv->type == 'h' || stv->type == 'H')
            {
                // Character (h/H is HUGE character data)
                iAppendToTemplateSkipNulls(builder, (s8*)start, iSwapEndian(stv->length), NULL, NULL);
                // Indicate where the next one will go after this entry
                stv = (SSaveToVar*)(start + iSwapEndian(stv->length));

            } else if (stv->type == 'n' || stv->type == 'N') {
                // Numeric
                stvf = (SSaveToVarFloat*)start;
                iAppendToTemplateDouble(builder, stvf->fVal, iSwapEndian(stv->length), stv->decimals, NULL, NULL);
                // Indicate where the next one will go after this entry
                stv = (SSaveToVar*)(start + sizeof(SSaveToVarFloat));

            } else if (stv->type == 'l' || stv->type == 'L') {
                // Logical
                // Indicate where the next one will go after this entry
                stvl = (SSaveToVarLogical*)start;
                if (stvl->lVal == 0)
                {
                    // False
                    iAppendToTemplate(builder, ".F.", 3, NULL, NULL);
                } else {
                    // True
                    iAppendToTemplate(builder, ".T.", 3, NULL, NULL);
                }
                stv = (SSaveToVar*)(start + sizeof(SSaveToVarLogical));

            } else if (stv->type == 'd' || stv->type == 'D') {
                // Date
                stvd = (SSaveToVarDate*)start;
// REMEMBER need to decode the date structure
                // Indicate where the next one will go after this entry
                stv = (SSaveToVar*)(start + sizeof(SSaveToVarDate));

            } else if (stv->type == 't' || stv->type == 'T') {
                // Datetime
                stvt = (SSaveToVarDatetime*)start;
// REMEMBER need to decode the date structure
                // Indicate where the next one will go after this entry
                stv = (SSaveToVar*)(start + sizeof(SSaveToVarDatetime));

            } else {
                // Unknown type
                iAppendToTemplate(builder, "Unknown", 1, NULL, NULL);
                // Indicate where the next one will go after this entry
                stv = (SSaveToVar*)start;
            }


        //////////
        // All done, indicate where the new pointer will be
        //////
            return(stv);
    }

 

[Mike Lewis]

I understand that you written some extensive code that nearly meets your goal, and you are just lacking a couple of details.

But - and I'm sorry if this seems like I'm labouring the point - you could get exactly what you want by using DISPLAY MEMORY TO FILE rather than SAVE TO. It would be vastly simpler.

The disadvantage would be that you would have to do it in VFP. If there is a strong reason for doing it in C++, that's another matter.

I'm not suggesting that you should necessarily re-do your entire postmortem routine in VFP. You would only need three or four lines of VFP code to create the text file.

By the way, it seems that what you are trying to achieve is what I would call an error-handler. I've been using a standard error-handler for many years that writes all the variables to a text file. It's a simple bit of code, and is written entirely in VFP. I'm sure many other people here use something very similar.

Mike





__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[foxmuldr2]

But - and I'm sorry if this seems like I'm labouring the point - you could get exactly what you want by using DISPLAY MEMORY TO FILE rather than SAVE TO. It would be vastly simpler.

Well, I would, but it won't work. The two are the same. I actually do use the list memory and list status in my app as well, but not for these purposes as I need the full memory contents, and not just partial contents with a memory variable name and length.

Here's some proof:

k = replicate("hello, world", 200)
list memory to c:\temp\memory.txt
save to c:\temp\memory.mem

Examine the files. c:\temp\memory.txt has:

K  Pub  C  "Hello, wor..." 2400 bytes
    1 variable defined, 2407 bytes used
16383 variables available

The c:\temp\memory.mem file has the full memory contents (load it in a hex editor and look). It is 2,434 bytes long and contains the fullness of the original memory variable. It is there that I need to be as the apps I'm using this tool on sometimes have long memory variables (16KB and larger), and portions well out into the data change.

I do appreciate your help and suggestions. The application is where it's at because it needs to be where it's at.

Best regards,
Rick C. Hodgin

 

[foxmuldr2]

The two are the same. smile

Should be "The two are NOT the same."

Best regards,
Rick C. Hodgin

 

[Mike Lewis]

OK. Point taken.

I guess it's a question of hacking the MEM file after all. If you haven't cracked it within a couple of day, I might take a shot at it, if I can grab some time.

Mike


__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[foxmuldr2]

I try to imagine you writing your posts with a Scottish accent.

I wrote this code a few months back. I don't have a pressing need for it today. But, in a few weeks I may. Regardless, it would be nice to have it sorted out. I may spend some "oh, wouldn't that be fun on the computer" time hacking at it. It's fairly rewarding.

Best regards,
Rick C. Hodgin

 

[Olaf Doschke]

In a DBF it's 8 bytes, the first four being the julian day number (sys(1)) and the second four bytes the milliseconds since midnight. And a date field simply stores a char representation of a date.

In a (VFP9) mem file this differs, as far as I see. Haven't figured what it is there, I don't recognize a similarity in the hex values. Maybe there it's rather the Unix Timestamp.

Just one word of caution: The mem file structure is not only undocumented, it also has changed with versions of foxpro, so you can't eg restore a mem file saved with VFP6 with VFP9 and vice versa. So it's only viable to make a restore within the same vfp version.

Bye, Olaf.

 

[Olaf Doschke]

Figured it out. A datetime variable is stored as an 8 byte double value in a mem file.

The value is again the julian day number but the fractional part is the time portion, it's not seperately encoded as in a dbf datetime field.
For example CToBin(0hee0471771bbd4241,"8") converts the hexcode to the binary value. In this case it results in about 2456118.9331366, which stands for 07/09/2012 10:23:43 PM, as you can figure out with sys(10) and converting the faractional part to a time by multiplying with 24*60*60 to get the seconds since midnight.

The date also is encoded as such a number. It might be the simplest to add 4 zero bytes and again interpret that as 8 byte double value. Then you get the julian day number again, eg as in CToBin(0h000000001bbd4241,"8"). "1bbd4241" is what you find as a date value for 07/09/2012.

Bye, Olaf.

 

[Mike Lewis]

Good hack, Olaf.

Mike

__________________________________
Mike Lewis (Edinburgh, Scotland)

Visual FoxPro articles, tips, training, consultancy

 

[foxmuldr2]

Olaf, 2 hours 15 minutes between posts. Tell me you didn't spend 2 hours tracking that down.

Nice work though. Much appreciated. I'll post the completed, functional, stand-alone code once I get it implemented (later today, Lord willing).

Best regards,
Rick C. Hodgin

 

[Olaf Doschke]

No, I watched some episodes of doctor who and then got the idea to apply ctobin to the 8 byte hex value and see if I get to the julian day number again.

Bye, Olaf.

 

[foxmuldr2]

Am working on this now.

Do you have the julian day number to date conversion algorithm anywhere?

Best regards,
Rick C. Hodgin

 

[foxmuldr2]

Found this page. The formula works.

http://stason.org/TULARC/society/ca...ormula-for-calculating-the-Julian-day-nu.html

JD = VAL(SYS(1))

a = JD + 32044
b = INT(((4 * a) + 3) / 146097)
c = INT(a - ((b * 146097) / 4))
d = INT(((4 * c) + 3) / 1461)
e = INT(c - ((1461 * d) / 4))
m = INT(((5 * e) + 2) / 153)
  
day    = INT(e - INT(((153 * m) + 2) / 5) + 1)
month  = INT(m + 3 - (12 * INT(m / 10)))
year   = INT((b * 100) + d - 4800 + INT(m / 10))

? DATE(year, month, day)