Memo Bloat 1

[DEDMOD]

I'm trying to store data from a file into memo fields. Some of the lines in the data file contain the header information for a given record and the following lines up to the next header line is raw data (genome data in this case so it's just a string of ATCGs.) Anyway, I'm using code like:

REPLACE genmemo WITH gline ADDITIVE

to add each line to the memo field. And it works but bloats the .fpt table something awful. Just ten records gave me 700,000k (25 records gave me an out of disk space error). Now a 'pack memo' got me down to 826k which is reasonable, but I'd hate to have to pack after every record. Can I create a variable to hold 200,000 characters? (the individual data lines only appear to be 60 bytes long or so which probably has something to do with why the memos are bloating.) Or is there something else I should be adjusting? Dave Dardinger

 

[Scott24x7]

Dave,
Try this instead:

REPLACE GENMEMO WITH ALLTRIM(GENMEMO) + ALLTRIM(GLINE)

If you want a "New line" or "Space" character, just add either:

+ CHR(13) +

Or

+ ' ' +

respectively, in place of the single + in the line above.

Best Regards,
Scott

Please let me know if this has helped

 

[DEDMOD]

Didn't make any difference. (And I didn't expect it would since there weren't any spaces in the memo)

Anyone else have an idea? Dave Dardinger

 

[rgbean]

Dave,
Check into the "SET BLOCKSIZE Command". Adjusting this can make a huge difference, based on the size of the data and the real (OS file system based) blocksize of disk records.

Rick

 

[BlindPete]

I have not experienced this with FPTs. Is there something peculiar about the data or is just text?

You might try building each record's memo in a txt file first then using the REPLACE w/o ADDITIVE clause.

It would be slow and ugly but have you tried putting in a PACK after every reocrd. -Pete

 

[wgcs]

You can store huge strings in memory; There's a discussion at

http://fox.wikis.com/wc.dll?Wiki~OrlandoDevcon2001Gems~VFP

of putting the text of War and Peace as a string variable as an example of VFP's capabilities. Here's an excerpt:

[21:29] {CarlKarsten} while you are looking... heres another from Text:
[21:29] {CarlKarsten} x= FILETOSTR("WarAndPeace.TXT")
[21:29] {CarlKarsten} a 3.2 meg file gets put in x in .14 seconds
[21:30] {EvanDelay} Pretty sweet, huh?

So I'd say just build each memo field's contents in memory, and REPLACE it into the actual memo field once, and this should avoid all bloat.

 

[DEDMOD]

So Rick, what do you suggest? The present setting is the default 64 bytes, and as I said, the lines in the file seem to be 60 bytes. I tried SET BLOCKSIZE TO 1 just after my USE statement in my program. But again it gives me the same results.

I did try doing a PACK MEMO after each record, and it keeps the bloat down, but how much of a hit in performance will I get? Dave Dardinger

 

[wgcs]

From VFP help file:
nBytes

Specifies the block size in which disk space for memo fields is allocated. If nBytes is 0, disk space is allocated in single bytes (blocks of 1 byte). If nBytes is an integer between 1 and 32, disk space is allocated in blocks of nBytes bytes multiplied by 512. If nBytes is greater than 32, disk space is allocated in blocks of nBytes bytes.
----
So, try 0, or 33...64.

 

[DEDMOD]

Much better, wgcs! Not only did constructing the memo first get rid of the bloat, it's infinitely faster too. It was taking a minute or so to do 10 records but now it will do 1000 records (100 meg) in that time or less.

Now all I have to do is parse out the chromosome data (I'm playing with the human genome and unfortunately the size of the data exceeds 2 gig, so I can't keep it all in one table. So I figured it'd be smart to sort it by chromosome where it's mentioned in the header info. Dave Dardinger