methods to avoid heap fragmentation?

[kittyyo]

Hi all!

I am working at an event simulation covering the evaluation of a self-organized TDMA network. With the time my code base got really big. After I started to run the simulation with a lot of actors (800 and more) the RAM consumption went up, though I had been very careful to free all memory allocated.

In a news group I read about heap fragmentation and I think that could be what my program is suffering from. I am only creating small TObject descendants, but I have a lot of dynamic arrays containing quite big records of data which I often call SetLength() on (both increasing and decreasing the size).
So maybe the "big" records in the dynamic arrays "fight" with the small objects such that memory fragmentation occurs?

I have searched for a while on the net but could not find good methods to avoid this with Delphi. (One was to create factories for any kind of object, but since I think that the dynamic arrays are the real problem this is not really applicable.)
Has anyone had the same problem, and which solutions did you apply?

Thank you very much for suggestions!
Anne

 

[whosrdaddy]

In fact there's no easy answer. It is known that windows isn't perfect in memory managment.
IMHO the best way to avoid memory fragmentation is to allocate large chunks at one time, but this isn't always possible. I stopped using dynamic arrays 5 years ago because of the problem you're having. In fact every dynamic array can be rewritten with a TList object. once you acustomed to work with TList objects it is easy to implement factories (interfaces) for it.

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

Thanks for your thoughts. In fact, when I was starting to write that simulator I was not aware of that memory fragmentation problem... and I decided to use dynamic arrays because I thought they would use less memory.

I suppose that the following construction uses most of my memory and also causes most of the fragmentation:

  TStationType = (stMobile, stAircraft, stGroundVehicle,
    stGroundStation, stBroadcast);

  TStationAddress = class(TObject)
  protected
    FStationType: TStationType;
    FNumber: Integer;
  public
    constructor Create(StationType: TStationType; 
      Number: Integer);
    ... [properties and functions]
  end;

  TReservationState = (rsEmpty, rsCancelled, rsBusy);
  
  TReservation = packed record
    State: TReservationState;
    Directed: Boolean;
    Source, Destination: TStationAddress;
    NextInStream: Smallint;
    Priority: 0..15;
    TimeOfRequest: Cardinal;
  end;

  TReservations = array of TReservation;

  TReservationTable = class(TObject)
  protected
    Items: array[0..18127] of TReservations;
    ....
  end;

...and the TReservationTable object is instantiated about 1000 times...
The Items array's members (the TReservation arrays) are (very) often resized, mainly to size zero or one, seldom also to size more than one (but never more than five).

If I would rewrite that (make TReservation an object and
TReservations an object list), would it be much better then? The reason why I'm in doubt is that the resizing is always
in the range of zero to mainly one items.

Thanks a lot for any further suggestions!
Anne

 

[whosrdaddy]

stupid question maybe, but did you try to use a database backend for your data? there are some db engines with very efficient memory managment (and I'm not speaking about M$ here...).

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

No, it's not a stupid question at all! I once programmed it using a database, but that got really slow (updating the 1000 reservation tables in RAM needed parts of a second, selecting and updating the database about 10 seconds) -
that is because updating the reservation table is such a complex task (not to be done with one huge update query).
(BTW, the database I used was MySQL.)

 

[whosrdaddy]

I use TkbmMemTable (memory table) for such tasks, it uses TLists internally. maybe you should give it a try...

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

I didn't know that such a thing exists - sounds really great!

I'll try and post here again if something unbelievable happens (i.e. less memory consumption)

 

[lespaul]

In fact every dynamic array can be rewritten with a TList object.

Could you show a small example of how you do this?

Thanks!

Leslie

Anything worth doing is a lot more difficult than it's worth - Unknown Induhvidual

Essential reading for anyone working with databases: The Fundamentals of Relational Database Design

 

[whosrdaddy]

les,

in fact a Tlist uses an array of pointers internally, so basically it is the same as a dynamic array but you have the Tlist object methods to do the managment of the array (delete, insert,...). My point is that the coding gets easier with tlist objects instead of dynamic arrays.

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

Hi all,

some progress report: I replaced the structure above this way:

  TReservation = packed class(TObject)
  public
    State: TReservationState;
    Directed: Boolean;
    Source, Destination: TStationAddress;
    NextInStream: Smallint;
    Priority: 0..15;
    TimeOfRequest: Cardinal;
  end;

  TReservationTable = class(TObject)
  protected
    Items: array[0..18127] of TObjectList;
    ....
  end;

I only do a TObjectList.Create of one of the reservation table's items when it is needed for the first time, and I keep its Capacity to be the same as Count.

My program now uses significantly more memory.

Now I'm not sure if it's memory fragmentation then... or if I'm just suspecting the wrong structure?

Is there any good method to determine how much of the currently assigned memory is not used (fragmented)?

 

[whosrdaddy]

ok,

did a quick calculation about the memory usage for 1000 TReservationTable objects with 5 TReservation records in the TReservations array. I calculated that the whole thing consumes about 1383Mb!!! are you sure you have enough memory for this? (I mean physical + pagefile)

-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

Someone produced a 2GB RAM machine for me... luckily.
The pagefile also has 2GB.

Yes, it is THAT huge

 

[kittyyo]

\begin{background information}

Er, just for the case that you ask yourself: If I had started the thing more aware about what could happen with the memory consumption of these reservation tables, I would have coded it differently. In fact, I realized after a while that quite many of the TReservation objects are (virtually) the same when looking at different TReservationTables, and I could have been working with one big TReservation storage and have the TReservationTables point to that storage.

But now, this seems much too difficult for me - since I have added control structures to the TReservation records that are not the same for different TReservationTable objects - but maybe, one time I'll have to change it... ;-)

\end{background information}

 

[sggaunt]

I'm inclined to agree about the use of a 'proper' DataBase for all that data.
If thats not pracical? Does all the data have to be in memory, could you stream parts of the data to and from disc as you need it?


Steve: Delphi a feersum engin indeed.

 

[whosrdaddy]

Steve,

this is exactly the behaviour of an enterprise class DB engine (like SQL server or Oracle). IMHO The only advantage of having that amount of data in memory would be calculation speed?

kittyyo,

I don't know the full implementation of your program, but if were in your place, I would write the whole thing in TSQL (stored procedures and so on) and then use delphi to present/manage the calculated data.
if that's really not an option, try to use another memory manager than borland's. here's a very nice one :

http://sourceforge.net/projects/fastmm

Cheers,
Daddy


-----------------------------------------------------
What You See Is What You Get
Never underestimate tha powah of tha google!

 

[kittyyo]

@Steve

The structure of the problem makes it very hard to find a good criterion for dividing "needed" and "not needed" data. I'll search for one though, paging to disc would be a good idea.
Anyway, using a database with RAM tables would also be a good option, I think.

@Daddy

if were in your place, I would write the whole thing in TSQL (stored procedures and so on) and then use delphi to present/manage the calculated data.

You're right, I should have started it that way (make use of the good memory management of databases). I screwed it up quite well ;-)

My problem is that time is running out and I should have results in one week or so... which makes your memory manager tip real great!

I also thought about replacing Win2k on my test machine by WinXP since I read somewhere that the memory management has been improved. (Or maybe, switch to Kylix and benefit from real up-to-date technology ;-))

---

Anyway, thanks to you both for giving me ideas for future projects - things that I should really think about before just starting to code! (A typical programmer's problem maybe.)