Hm, I thought I could prevent that by shooting down all processes that I could on my test machine, so that only the necessary system processes were running and nothing else. But maybe the influences are somewhere else?"
You don't know when and how the system processes will be using memory. The best way to think of Windows memory is that it is going to shift like the sands in the seas. If you can get the memory one execution, don't be surprised to not get it the next execution. No way to prevent it, you have to check what you have when you run your program if you have a memory-intensive process. From what I know, it's really hard to gracefully process on a set of statically defined variables if you don't know whether they can be allocated or not.
"I've never been thinking about that (paging results to disk...) - more details would be great!"
The first question to ask is if you can create your process to run in a segment of processes? i.e.
1) Can you process a set of data and then write it to disk, and then process the next set without issue?
2) Can you page a completed set of data to disk and then recall it if you need to change it and change it easily.
It's a lot more code to consider, but those are initial thoughts to think about in your design. As for my program goes, I could fill memory up with one set of data, process it, page it all to disk, and continue on until my whole data was processed. Then I could pick up my page files and process them until I had one complete correct output or process the page files into another page file if I ran out of allowable page files. Whatever it would take to preserve the data and continue the processing.
It was a lot more complex and not an easy thing to design or code, but was necessary (and a lot of learning went into it too and will continue to go into it I'm sure since it's not perfect for what I want). Like I said, I could have 16MB of data into my program, which would be fine to put into memory, but I could have 16GB too, which would not work. I don't know that, so I have to account for it. So I ended up doing a dynamic allocation of work units.
I'll show a sample of that below. The main thing that might of issue is the calculation of the allocation units, which probably wouldn't hurt to be placed into KB instead of bytes (you'll get an arithmetic overflow if you tweak the memory limit var up). Console mode program, tested under Delphi 3:
program memtest2; uses sysutils;
{ Prototype test program: grab as much memory in system as possible according to various rules.
Released to tek-tips.com as demo. Any use of code is to include a statement "copyright 2005 by Glenn9999@tek-tips.com" }
const
memlist = 4096; { maximum # of work units }
alloclist = 65536; { governs size of each work unit }
upperlimit = 1024; { upper limit of memory in MB }
type
arraytype = array[1..alloclist] of longint;
arrptr = ^arraytype;
var
mem_allocated: longint;
mem_segments: longint;
mem_full: boolean;
seg_size: longint;
i: longint;
proc_seg: longint;
memarray: array[1..memlist] of arrptr;
function alloc_memory: boolean;
begin
{ have we hit our maximum number of segments? }
if mem_segments = memlist then
begin
mem_full := true;
writeln('Maximum segments reached.');
end
else
{ have we hit our arbitrary maximum allocated? }
if mem_allocated >= (upperlimit * 1024 * 1024) then
begin
mem_full := true;
writeln('Maximum amount of allotted memory used.');
end
else
{ OK now to try to allocate memory }
begin
inc(mem_segments);
try
new(memarray[mem_segments]);
inc(mem_allocated, seg_size);
except
on EOutOfMemory do
begin
mem_full := true;
writeln('Out of System Memory.');
dec(mem_segments);
end;
end;
end;
alloc_memory := mem_full;
end;
begin
mem_allocated := 0;
mem_segments := 0;
proc_seg := 0;
mem_full := false;
seg_size := sizeof(arraytype);
{ initial pass - allocate memory, while read. }
while alloc_memory = false do inc(proc_seg);
{ Now that we know memory stats, we can post those to the user }
writeln(mem_allocated, ' bytes memory in ', mem_segments, ' segments.');
writeln('Each segment is ', seg_size, ' bytes.');
writeln;
writeln('Disposing of ', mem_segments, ' segments.');
for i := 1 to mem_segments do
dispose(memarray);
end.
Of course after the appropriate amount of memory is allocated, I do my processing with it, and then dispose of it afterwards. Of course, too, it can be refined, but hopefully it'll give you a good start. If you give mem_list a high enough number and comment out the arbitrary size section you should see it go until you run out of system memory. But like I've noticed in testing too, paging to swap file makes it run like a old dog (and you're paging your work to disk ANYWAY), so it's best to try to find an arbitrary value that will stick to memory in most systems. Unfortunately, there's no way to find out how much real memory there is or whether you're going to swap file or not. (also I'll remove any reference to "segments" in the final version - the main program that uses this memory allocation scheme is in alpha 3)
Hope this helps start you on the way to an idea or two.
