INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Log In

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

lowering memory fragmentation

lowering memory fragmentation

(OP)
Hi.

In one of my applications I use an array of records (hundreds of items) with 4 memory streams (a few MB each) in each item.
Most of the times the data from one of the streams is backed up in one of the streams (for Undo) and then is modified and the result is copied into the other 2 and into the source stream (in different forms, not exactly the same) - for each item.
There is no way to know the exact size of each stream before the modification (I'm using 3rd party components to compress the data). So I set the size of the stream to the uncompressed data size, compress and then set the size to the value returned by the 3rd party component.
Sometimes items are inserted and/or removed from the array.
The application is 32 bit and using ~ 1.5 GB RAM on 32 bit OSs and ~ 3 GB RAM on 64 bit OSs (large address aware).
But sometimes is not enough, especially when the user is performing a lot of operations on data, fragmenting the memory.

I use now Delphi XE8 with the default memory manager (FastMM4).
For very good reasons I can't make a 64 bit application so I'm stuck with 32 bit.
My current OS is Windows 8.1 x86.

I searched before asking but couldn't find anything useful.

Once I tried to make a custom TMemoryStream, didn't work.

Any tip on how to lower the memory fragmentation?

Thank you in advance.

RE: lowering memory fragmentation

Fragmentation comes from reallocating blocks of memory.
Try to work with fixed size streams (or try to lower the number of reallocations).

/Daddy

-----------------------------------------------------
Helping people is my job...

RE: lowering memory fragmentation

(OP)
Thank you.

That's a good advice, but unfortunately I can't work with fixed size streams since every item from the records array is unique (different size, different content). And I already lowered the number of reallocations to minimum.

Meantime I tried another approach: to make again a custom memory stream. But this time I didn't use any ancestor and I divided its memory in chunks (usually 8 KB in size).

This is the current code http://files.engineering.com/getfile.aspx?folder=7...

I used this code to test it:

CODE --> Delphi

function CurrentProcessMemory: Cardinal;
var
   MemCounters: TProcessMemoryCounters;
begin
   MemCounters.cb := SizeOf(MemCounters);
   if GetProcessMemoryInfo(GetCurrentProcess,
      @MemCounters,
      SizeOf(MemCounters)) then
      Result := MemCounters.WorkingSetSize
   else
      Result := 0;
end;

const
   Iterations = 500;
   InitialSize = 1048576;
var
   i, j: Integer;
   arrTest1: array of TVirtualMemoryStream;
   pmem: Cardinal;
begin
     pmem := CurrentProcessMemory;
     SetLength(arrTest1, Iterations);
     for i := 0 to Iterations - 1 do
     begin
        arrTest1[i] := TVirtualMemoryStream.Create(InitialSize);
        arrTest1[i].FillMemory(0, InitialSize, 3);
     end;
     i := 0;
     try
        while True do
        begin
           Inc(i);
           for j := 0 to Iterations - 1 do
           begin
              arrTest1[j].Size := 11 * arrTest1[j].Size div 10;
              arrTest1[j].FillMemory(0, arrTest1[j].Size, 3);
           end;
        end;
     except
        pmem := CurrentProcessMemory - pmem;
        for j := 0 to Iterations - 1 do
           arrTest1[j].Free;
        ShowMessage(IntToStr(i) + #13 + FloatToStr(pmem / 1048576));
     end;     
end; 

It gives OutOfMemory error when tries to increase from 1738 MB to 1912 MB (14th operation).
I tested with a default TMemoryStream and the error appears when tries to increase from 1389 MB to 1528 MB (11th operation).

So it appears the memory is less fragmented with TVirtualMemoryStream.
But I'm opened to critics and suggestions...

RE: lowering memory fragmentation

I wouldn't do any reallocs, in your case.
Use one memorystream (allocated at largest size possible) and write a manager that handles your objects.

/Daddy

-----------------------------------------------------
Helping people is my job...

RE: lowering memory fragmentation

It sounds from your program requirements like you need to just allocate one fixed block of memory when your program starts and then leave it until your program exits. Then manage what you do within the program with that memory and record things yourself.

I don't know if I can offer any further assistance, but if it helps, I did write a generic file sorting program once upon a time which handled a lot of memory at one time. One discovery I did make in the process which you will need to think about is that memory you use isn't necessarily all "memory". Depending on how the OS memory manager handles it (I'm pretty sure Microsoft hasn't touched it since XP for the 32 bit side), you might be thrashing the disk a lot for memory being paged in and out, and not end up with what you might think.

In any event, a general rule I got hit with hard is that you can't necessarily guarantee any amount of memory will ever be available on a system at any time your program runs. So you need to be very flexible about how you go about it as well as your expectations.

Hope some of that helps...

RE: lowering memory fragmentation

(OP)
Thank you both for the advice.
But my app has to work with any stream size (not just a few MB) and it should use memory as it needs, not take all the memory.
Yes, I see the advantages of using a fixed size memory area, but unfortunately I can't use this approach sad

For now I'm working on improving the virtual memory stream.
One of the streams from the record array is compressed with LZ4 (C++ dll). The problem is it requires, for compression or decompression, a contiguous memory area; but the virtual memory stream has its memory in noncontiguous 8 KB chunks.
Options:
1. I can use a contiguous memory area as buffer but it will increase memory usage and fragmentation and it will decrease the speed.
2. I could modify the LZ4 sources to work with chunks. But my C++ programing skills are not so great sad

RE: lowering memory fragmentation

>But my app has to work with any stream size (not just a few MB) and it should use memory as it needs, not take all the memory.

Sounds like that generic file sorting program I mentioned. Like I mentioned above, be flexible. Based on the job at hand, I have X byte records, and have Y amount of memory to work with. So I allocate enough memory (once) for Z records when I sort, which won't starve the system and thrash the disk (there are API calls to find this out). If the number of records in the file is greater than Z records, then I have to process the file in sections, put them to temporary files, and then merge the results. Of course if the file records are < Z, then I can just allocate the needed memory, do the sort, write straight to output and be done. More or less, the design takes into account that it may or may not have a certain amount of memory to play with, so it adjusts itself to get success no matter what the memory conditions are and no matter what size of file is presented it.

So, you can definitely work it out to do something similar with whatever your process is. Of course, most programmers don't have to think this way with their programs, since memory is not a factor of consideration 99.999% of the time. So it seems weird. But it's quite workable, when you think it through.

RE: lowering memory fragmentation

(OP)
I'll try to explain more what the app is doing.

The streams are in fact animation frames.
In the main stream each frame is stored using lossless compression (lz4). When the user is changing the frames using the interface, each frame is decompressed, modified and recompressed. Also, in the 2nd and the 3rd stream the image and its transparency are stored using a lossy compression (they are needed for preview and for saving the animation in a file).
The 4th stream is for Undo, it stores the data from the main stream before making the modifications.

This is the app (installer): https://drive.google.com/open?id=0ByKxAD_t9uvLZ19S...

If you want I can show you the source code too.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Tek-Tips Forums free from inappropriate posts.
The Tek-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Resources

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close