Dynamic Array of records, Blockwrite, problem 1

[majorkup]

Hi there.

I have this dynamic 2d array of records:

TTerrainCell = record
Index : integer;
end;

Terrain : array of array of TTCell;

And a loading procedure:

var
f : file of byte;
begin
assignfile(f,filename);
reset(f);

blockread(f,Header,sizeof(TMapHeader));
setlength(Terrain,header.WIDTH,header.HEIGHT);

blockread(f,Terrain[0],sizeof(TTerrainCell)*header.width*Header.Height,readed);
showmessage(inttostr(terrain[9,0].index));

closefile(f);

Header holds the dimensions of the array: width and height.
The saving procedure is ok, I guess. The size of a saved file is correct.
The header is read correctly, it finds out the dimensions.
The problem is when I try to use the Terrain data i get an access violation error. I'm using Turbo Delphi.

It seems this whole thing has to do with the data type. If i change that Record to a simple Integer it's ok. (array of array of integer)

 

[majorkup]

Sorry, I don't know how to edit this post.

Of course its array of tterraincell.

 

[roo0047]

I think your problem is here:

blockread(f,Terrain[0],sizeof(TTerrainCell)*header.width*Header.Height,readed);

Should be:

blockread(f,Terrain[0,0],sizeof(TTerrainCell)*header.width*Header.Height,readed);

I had some doubt that what you were doing would work, but it does! Here is my code used for testing. I added a TMemo so I could see what was going on. Three buttons are labeled "read", "write", and "fill".

unit Unit5;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ExtCtrls;

type
  TMapHeader = record
    WIDTH : integer;
    HEIGHT : integer;
  end;

  TTerrainCell = record
    Index : integer;
  end;

type
  TForm2 = class(TForm)
    Panel1: TPanel;
    Button1: TButton;
    Button3: TButton;
    Button2: TButton;
    Memo1: TMemo;
    procedure Button1Click(Sender: TObject);
    procedure Button3Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
  private
    { Private declarations }
    MapHeader: TMapHeader;
    Terrain : array of array of TTerrainCell;
    f : file of byte;
    procedure ReadData;
    procedure FillData;
    procedure WriteData;
    procedure WriteMemo;
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

{$R *.dfm}

const
  filename = 'TerrainCell.dat';

procedure TForm2.ReadData;
var readed: integer;
begin
  assignfile(f, filename);
  reset(f);
  blockread(f, MapHeader, sizeof(TMapHeader));
  setlength(Terrain, MapHeader.WIDTH, MapHeader.HEIGHT);
  blockread(f,Terrain[0, 0], sizeof(TTerrainCell)* MapHeader.width * MapHeader.Height, readed);
  //showmessage(inttostr(terrain[9,0].index));
  closefile(f);
end;

procedure TForm2.FillData;
var w, h: integer;
begin
  MapHeader.WIDTH:= 4;
  MapHeader.HEIGHT:= 4;
  setlength(Terrain, MapHeader.WIDTH, MapHeader.HEIGHT);
  for w:= 0 to MapHeader.WIDTH - 1 do
    for h:= 0 to MapHeader.HEIGHT - 1 do
      Terrain[w, h].Index:= random(100)
end;

procedure TForm2.WriteData;
begin
  assignfile(f, filename);
  rewrite(f);
  blockwrite(f, MapHeader, sizeof(TMapHeader));
  setlength(Terrain, MapHeader.WIDTH, MapHeader.HEIGHT);
  blockwrite(f,Terrain[0, 0],sizeof(TTerrainCell)* MapHeader.width * MapHeader.Height);
  closefile(f);
end;

procedure TForm2.WriteMemo;
var w, h: integer;
begin
  Memo1.Clear;
  for w:= 0 to MapHeader.WIDTH - 1 do
    for h:= 0 to MapHeader.HEIGHT - 1 do
      Memo1.Lines.Add('Terrain['+ IntToStr(w) + ',' +
                                  IntToStr(h) + '].Index = ' +
                                  IntToStr(Terrain[w, h].Index))
end;

procedure TForm2.Button1Click(Sender: TObject);
begin
  ReadData;
  WriteMemo
end;

procedure TForm2.Button3Click(Sender: TObject);
begin
  FillData;
  WriteMemo
end;

procedure TForm2.Button2Click(Sender: TObject);
begin
  WriteData;
  WriteMemo
 end;

end.

Roo
Delphi Rules!

 

[roo0047]

OOPS! We have a problem with setlength().
Working on it... will post solution when resolved.

Roo
Delphi Rules!

 

[majorkup]

The problem goes even further when I add another varible to the record, like:

TTerrainCell = record
Index : integer;
Index2 : integer;
end;

 

[roo0047]

Problem is in the readblock. It destroys the array structure. (Set watch on var Terrain and breakpoint on BlockRead in ReadData)

unit Unit5;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ExtCtrls, ComCtrls;

type
  TMapHeader = record
    WIDTH : integer;
    HEIGHT : integer;
  end;

  TTerrainCell = record
    Index : integer;
  end;

type
  TForm2 = class(TForm)
    Panel1: TPanel;
    Button1: TButton;
    Button3: TButton;
    Button2: TButton;
    Memo1: TMemo;
    StatusBar1: TStatusBar;
    procedure Button1Click(Sender: TObject);
    procedure Button3Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
  private
    { Private declarations }
    MapHeader: TMapHeader;
    Terrain : array of array of TTerrainCell;
    f : file of byte;
    procedure ReadData;
    procedure FillData;
    procedure WriteData;
    procedure WriteMemo;
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

{$R *.dfm}

const
  filename = 'TerrainCell.dat';

procedure TForm2.ReadData;
var
  w, readed: integer;
  bufsize: word;
begin
  assignfile(f, filename);
  reset(f);
  fillChar(MapHeader, sizeof(MapHeader), #0);
  Terrain:= nil;
  blockread(f, MapHeader, sizeof(MapHeader));
  setlength(Terrain, MapHeader.WIDTH);
  for w:= Low(Terrain) to High(Terrain) do
    setlength(Terrain[w], MapHeader.HEIGHT);
  bufsize:= sizeof(Terrain) * MapHeader.Width * MapHeader.Height;
  blockread(f, Terrain, bufsize, readed);
  Statusbar1.Panels[1].Text:= '['+inttostr(readed)+'] lines read.';
  //showmessage(inttostr(terrain[9,0].index));
  closefile(f);
end;

procedure TForm2.FillData;
var w, h: integer;
begin
  Terrain:= nil;
  MapHeader.WIDTH:= 2;
  MapHeader.HEIGHT:= 3;
  setlength(Terrain, MapHeader.WIDTH);
  for w:= Low(Terrain) to High(Terrain) do
    setlength(Terrain[w], MapHeader.HEIGHT);
  for w:= Low(Terrain) to High(Terrain) do
    for h:= Low(Terrain[w]) to High(Terrain[w]) do
      Terrain[w, h].Index:= random(32)
end;

procedure TForm2.WriteData;
var n: integer;
begin
  assignfile(f, filename);
  rewrite(f);
  blockwrite(f, MapHeader, sizeof(TMapHeader));
  blockwrite(f, Terrain, sizeof(Terrain) * MapHeader.Width * MapHeader.Height, N);
  Statusbar1.Panels[0].Text:= '['+inttostr(n)+'] lines written.';
  closefile(f);
end;

procedure TForm2.WriteMemo;
var w, h: integer;
begin
  Memo1.Clear;
  for w:= 0 to MapHeader.WIDTH - 1 do
    for h:= 0 to MapHeader.HEIGHT - 1 do
      Memo1.Lines.Add('Terrain['+ IntToStr(w) + ',' +
                                  IntToStr(h) + '].Index = ' +
                                  IntToStr(Terrain[w, h].Index));
  (*
  Memo1.Clear;
  for w:= Low(Terrain) to High(Terrain) do
    for h:= Low(Terrain[w]) to High(Terrain[w]) do
      Memo1.Lines.Add('Terrain['+ IntToStr(w) + ',' +
                                  IntToStr(h) + '].Index = ' +
                                  IntToStr(Terrain[w, h].Index))
  *)
end;

procedure TForm2.Button1Click(Sender: TObject);
begin
  ReadData;
  WriteMemo
end;

procedure TForm2.Button3Click(Sender: TObject);
begin
  FillData;
  WriteMemo
end;

procedure TForm2.Button2Click(Sender: TObject);
begin
  WriteData;
  WriteMemo
 end;

end.

I'm about to start a holiday weekend. Sorry I didn't get it sorted out. If anyone else wants to take a shot at it, be my guest.

Happy 4th of July! (US Independance Day)

Roo
Delphi Rules!

 

[majorkup]

What I discovered is:

The problem occurs only when the map isn't saved to a file before loading. The constructor reads as follow:

(fills the header, and sets the length of the array, then:)

  for x := 0 to Header.width - 1 do
  for y := 0 to Header.height - 1 do begin
    terrain[x,y].Index := random(2);
  end;
  savetofile('2.map');
  loadfromfile('2.map');

There is an error when I hash out the savetofile('2.map'); line. So is it most likely a memory problem? The funny thing is the header is read correctly, only the array is corrupted.

 

[roo0047]

The way I was testing was:
1) define and fill the array;
2) save to file and exit the app
3) re-start the app and then read the array from file.

Yes, same result when reading the array. Header reads ok.

I had this problem once before. My solution was to create an oversized static buffer:

var
  buf: array[0..1023] of byte;

Then:
1) blockread(f, buff, ...)
2) parse the header fields from buf
3) setlength(array, ...)
4) parse the remaining records from buf, field by field

I searched and found nothing about saving dynamic arrays to file. Have you considered saving the header as a seperate file?

I actually had it reading without errors a few times, but the data elements scrambled.

My biggest problem was I was already fried at the end of a very long day.

Roo
Delphi Rules!

 

[Glenn9999]

Okay let me clear this up. Dynamic arrays are coding horrors (I hate seeing 'em worse than I do gotos now), and we're seeing why in this thread. Why is it a coding horror? Because there is a lot happening behind the scenes that no one knows or thinks about on the face of it - in fact it totally goes against the intent and spirit of Pascal (which was a teaching language intended to expose everything).

Let's look at what it's doing. Basically what's being asked by the OP is to create a dynamic array of dynamic arrays.

Behind the scenes, though, what is happening? The system is allocating memory using pointers. Now, those sections of memory can be where ever the system is able to find free memory. So how do they get addressed? By finding the pointer and pointing to its contents.

Now let's take that back to what is being attempted here. We have a dynamic array of dynamic arrays, which means ultimately that you have Header.Width * Header.Height pointers in the system to various spots in memory, NOT a contiguous block of memory to work from. You also have a very inefficient structure, both in terms of memory (Header.Width * Header.Height pointers at 4 bytes per shot plus Header.Width * Header.Height TTerrainCells.), plus the additional processing of having to follow each and every pointer out to the data value. This will be illustrated in the programs below:

This one writes the data file using the dynamic structure.

{$APPTYPE CONSOLE}
program Project1; uses SysUtils;

type
  TTerrainCell = record
    Index : integer;
  end;
  Terrain = array of array of TTerrainCell;
  TMapHeader = record
     Width: integer;
     Height: integer;
  end;
var
  outfile: file;
  Header: TMapHeader;
  MyTerrain: Terrain;
  x, y: integer;
begin
  write('Width: ');
  readln(Header.Width);
  write('Height: ');
  readln(Header.Height);

  assign(outfile, 'TESTDATA.DAT');
  rewrite(outfile, 1);
  // write header
  blockwrite(outfile, Header, Sizeof(Header));

  // create dynamic array dimensions
  SetLength(MyTerrain, Header.Width);
  for x := Low(MyTerrain) to High(MyTerrain) do
    SetLength(MyTerrain[x], Header.Height);

  // now fill with values
  for x := Low(MyTerrain) to High(MyTerrain) do
    for y := Low(MyTerrain[x]) to High(MyTerrain[x]) do
      begin
        MyTerrain[x, y].Index := x + y;
        writeln(x, y:3, MyTerrain[x, y].Index:3);
      end;
  // write to disk
  for x := Low(MyTerrain) to High(MyTerrain) do
    for y := Low(MyTerrain[x]) to High(MyTerrain[x]) do
      blockwrite(outfile, MyTerrain[x, y], sizeof(MyTerrain[x, y]));

  close(outfile);
  readln;
end.

This one reads back the data file:

{$APPTYPE CONSOLE}
program Project2; uses SysUtils;

type
  TTerrainCell =  record
    Index : integer;
  end;
  Terrain = array of array of TTerrainCell;
  TMapHeader = record
     Width: integer;
     Height: integer;
  end;
var
  outfile: file;
  Header: TMapHeader;
  MyTerrain: Terrain;
  x, y: integer;
begin
  assign(outfile, 'TESTDATA.DAT');
  reset(outfile, 1);
  // write header
  blockread(outfile, Header, Sizeof(Header));

  // create dynamic array dimensions
  SetLength(MyTerrain, Header.Width);
  for x := Low(MyTerrain) to High(MyTerrain) do
    SetLength(MyTerrain[x], Header.Height);
  writeln('Array size: ', Header.Width, ', ', Header.Height);
  // read data
  for x := Low(MyTerrain) to High(MyTerrain) do
    for y := Low(Myterrain[x]) to High(MyTerrain[x]) do
      blockread(outfile, MyTerrain[x, y], Sizeof(MyTerrain[x, y]));
  // now write read data
  for x := Low(MyTerrain) to High(MyTerrain) do
    for y := Low(MyTerrain[x]) to High(MyTerrain[x]) do
      writeln(x, ' + ', y, ' = ', MyTerrain[x, y].Index);

  close(outfile);
  readln;
end.

What was the "bug" or "problem"? As you can see, I follow out every created pointer. In what I glanced over in this thread, I don't see that, and see the assumption that every bit of memory in the dynamic array is contiguous, which it is not.

For example, out of roo0047's code:

blockread(f, Terrain, bufsize, readed);

This line reads from file f into Terrain (an address, according to blockread) a continguous set of bufsize bytes.

To the OP: How much memory do your expect your app to use that you are wanting to use dynamic arrays? Is TTerrainCell much bigger in the app that you are showing here? Or are you concerned of disk space? My answer to your problem: Use static memory, or at least more of it than you do here.

----------
Measurement is not management.

 

[roo0047]

Glenn - I was hoping you'd jump in.

Is TTerrainCell much bigger in the app that you are showing here?

majorkup - That is the most important question.
In particular, how is the final output to be used? More specifically, is it for line graphs, excel import, db, or read by another app or system? What is max possible data range?

Your next decision regarding data structure design should take these factors into account.

Glenn - Your change to blockread each element in what was needed. I should have seen that since it had to be that way to write to memeo. That and proc ReadData having var X{w} and no Y{h} should also have been a clue.

Good call and excellent explanation!
&*4u


Roo
Delphi Rules!

 

[roo0047]

I have re-written my GUI version of the example program utilizing glenn's suggestions, as noted in the code. The TMemo has been removed and replaced with a TStringGrid used to display the output as a matrix. (See: WriteMemo)
You should now have no difficulty adding fields to record TTerrainCell. However you'll need to add a new TStringGrid if you wish to display them as I did.

unit Unit03;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ExtCtrls, ComCtrls, Grids;

type
  TTerrainCell = record
    Index : integer;
  end;

  TMapHeader = record
    WIDTH : integer;
    HEIGHT : integer;
  end;

type
  TForm1 = class(TForm)
    Panel1: TPanel;
    Button1: TButton;
    Button2: TButton;
    Button3: TButton;
    StatusBar1: TStatusBar;
    StringGrid1: TStringGrid;
    procedure Button1Click(Sender: TObject);
    procedure Button3Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
  private
    { Private declarations }
    MapHeader: TMapHeader;
    Terrain : array of array of TTerrainCell;
    f : file;  // of byte; (*glenn*)
    procedure ReadData;
    procedure FillData;
    procedure WriteData;
    procedure WriteMemo;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

const
  filename = 'TerrainCell.dat';

procedure TForm1.ReadData;
var w, h: integer;
begin
  assignfile(f, filename);
  reset(f, 1);  // ADDED ,1 (*glen*)
  // read header
  blockread(f, MapHeader, sizeof(MapHeader));
  // create dynamic array dimensions
  setlength(Terrain, MapHeader.WIDTH);
  for w:= Low(Terrain) to High(Terrain) do
    setlength(Terrain[w], MapHeader.HEIGHT);
  // read data
  for w:= Low(Terrain) to High(Terrain) do
    for h:= Low(terrain[w]) to High(Terrain[w]) do    (*glen*)
      blockread(f, Terrain[w, h], Sizeof(Terrain[w, h])); (*glen*)
  closefile(f);
end;

procedure TForm1.FillData;
var w, h: integer;
begin
  Terrain:= nil;
  MapHeader.WIDTH:= 4;
  MapHeader.HEIGHT:= 4;
  setlength(Terrain, MapHeader.WIDTH);
  for w:= Low(Terrain) to High(Terrain) do
    setlength(Terrain[w], MapHeader.HEIGHT);
  for w:= Low(Terrain) to High(Terrain) do
    for h:= Low(Terrain[w]) to High(Terrain[w]) do
      Terrain[w, h].Index:= random(32)
end;

procedure TForm1.WriteData;
var w, h: integer;
begin
  assignfile(f, filename);
  rewrite(f, 1);
  // write header
  blockwrite(f, MapHeader, sizeof(TMapHeader));
  // write to disk
  for w := Low(Terrain) to High(Terrain) do            (*glenn*)
    for h := Low(Terrain[w]) to High(Terrain[w]) do    (*glenn*)
      blockwrite(f, Terrain[w, h], sizeof(Terrain[w, h])); (*glenn*)
  closefile(f);
end;

procedure TForm1.WriteMemo;
var w, h: integer;
begin
  StringGrid1.FixedCols:= 1;
  StringGrid1.FixedRows:= 1;
  StringGrid1.ColCount:= MapHeader.WIDTH + 1;
  StringGrid1.RowCount:= MapHeader.HEIGHT + 1;
  for w:= 1 to StringGrid1.ColCount - 1 do
    StringGrid1.Cells[w, 0]:= format('w[%d]', [w-1]);
  for h:= 1 to StringGrid1.RowCount - 1 do
    StringGrid1.Cells[0, h]:= format('h[%d]', [h-1]);
  for w:= Low(Terrain) to High(Terrain) do
    for h:= Low(Terrain[w]) to High(Terrain[w]) do
      StringGrid1.Cells[w+1, h+1]:= format('%8d', [Terrain[w, h].Index]);
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  ReadData;
  WriteMemo
end;

procedure TForm1.Button3Click(Sender: TObject);
begin
  FillData;
  WriteMemo
end;

procedure TForm1.Button2Click(Sender: TObject);
begin
  WriteData;
  WriteMemo
 end;

end.

Happy coding!

Roo
Delphi Rules!

 

[Glenn9999]

I didn't catch onto this until roo0047 pointed it out. This thread has some material that might be relevant/interesting to the topic at hand as well.

thread102-1482542

----------
Measurement is not management.

 

[Glenn9999]

I did some more playing around - it seems that I'm not quite right. Delphi does use contiguous memory when it allocates the final static portion of a dynamic array. But it still seems to use an address translation to point at whatever it is you are interested in, and everything else is correct. Let's look:

To start, I defined a 2-dim dynamic array like so:

const
  dimx: integer = 20; // x dim of dynamic array
  dimy: integer = 10; // y dim of dynamic array
type
  mydata = packed record // 5 bytes instead of 8 unpacked.
    data1: integer;
    data2: char;
  end;
var
  myarray: array of array of mydata;

I used five bytes for the record to give an odd number so it would jump out at us. In the program I wrote to play around with this, I started by writing out the sizes (sizeof()) of different parts of the dynamic array.

    Size of Myarray: 4
   Size of Myarray[1]: 4
 Size of Myarray[1,1]: 5

Myarray and Myarray[1] are something different than mydata because it's not the same size (and this is why I made my record size an odd number not necessarily divisible by 4). My guess, given knowledge of general data types is that those are accessing the pointers associated with the memory allocation.

Myarray is a pointer?: C8 D9 97 00

That definitely appears to be a pointer address - guess, pointing at some parts of the table? Now, if we follow that pointer out and dump out a number of bytes (80 = 20 * 4 bytes per pointer), we can see:

98 66 97 00 
D8 66 97 00 
18 67 97 00 
58 67 97 00 
... (not all posted)
58 6B 97 00

Data that appears like the first pointer, which would definitely make it possible that we have:

pointer ---> table of pointers (stored as contiguous memory)

This explains the OP's access violation. With how blockread works, if you pass it Myarray or Myarray[x], the data on disk is placed into those spots in memory, making the pointer addresses invalid on a subsequent access to the structure.

Let's go on, though, and follow those pointers out to where they lead. We know sizeof(Myarray[1,1]) is 5, so this is likely where the real data begins. We can expect to find some of the data, so we'll try 50 bytes worth per address (5 bytes * 10). We'll find:

First element only, not all posted;

---- 0 ----
00 00 00 00 43 
01 00 00 00 43 
02 00 00 00 43 
.. (not all posted)
09 00 00 00 43 
 0C  1C  2C  3C  4C  5C  6C  7C  8C  9C 
-----------------------------------------------

I set myarray[x,y].data1 to x + y, and data2 to "C", which is consistent with what is shown here. To illustrate further, I wrote each record out in its proper data representations afterwards.

So we have as the dynamic array structure:

Pointer ----> Contiguous set of addressed data

Which is fine. But now, can that be used to avoid blockreading into each array spot? The answer turned out to be yes. The code below will work as the other code did.

// read data
  for x := Low(MyTerrain) to High(MyTerrain) do
    blockread(outfile, MyTerrain[COLOR=red][x, Low(MyTerrain)][/color], Sizeof([COLOR=red][x, Low(MyTerrain)][/color])*Header.Height);

I highlighted the array values to make clear what is going on: I used the first array value of each section of the table as the pointer for blockread. Of course, the problem with such code is the lack of clarity, as has been evident all through this thread in dealing with dynamic arrays.

The general assumption would be that you can work with dynamic arrays as you would static arrays, but that's not the case. Too much confusion, and definitely counterintuitive.

Hope this helps some more.

----------
Measurement is not management.

 

[lespaul]

all I can say is "wow"....

Leslie

Have you met Hardy Heron?

 

[majorkup]

Thanks a lot for such a great research.
But still, the only thing I'm interested is speed, even when it costs me additional 4 bytes for the pointer. But now when the reading goes in a loop, I'm not sure it's so fast anymore.
I need to test it.

 

[roo0047]

Glenn - I was about to post this...

With that, then it should be possible to truncate off the front of the array with Move:

To strip of the first 100 elements [0..99] :

  Header.Height:= Header.Height - 100;
  Move(MyTerrain[100, Low(MyTerrain)], 
       MyTerrain[0, Low(MyTerrain)],
       Sizeof([0, Low(MyTerrain)]) * Header.Height));

Correct? (untested)

...then decided to test it first myself.
It failed!!!
So next, I implemented your suggested change:

  // read data
  for x := Low(MyTerrain) to High(MyTerrain) do
    blockread(outfile, MyTerrain[x, Low(MyTerrain)], Sizeof([x, Low(MyTerrain)]) * Header.Height);

and a corresponding write...

  // write to disk
  for x := Low(MyTerrain) to High(MyTerrain) do
    blockwrite(outfile, MyTerrain[x, Low(MyTerrain)], Sizeof([x, Low(MyTerrain)]) * Header.Height);

That failed too. I've had a suspicion all along that if the data IS contiguous, then it must have HOLES in it. To prove it, rather that filling our data with random numbers or X+Y, I used this:

  for x:= Low(MyTerrain) to High(MyTerrain) do
    for y:= Low(MyTerrain[x]) to High(MyTerrain[x]) do
      MyTerrain[x, y].Index:= Integer(@MyTerrain[x, y].Index);

Each element now contains its own decimal address. Here's how it looks (8x8 matrix)
x[0] x[1] x[2] x[3] x[4] x[5] x[6] x[7]
y[0] 10102008 10102052 10102096 10112816 10112860 10112904 10112948 10112992
y[1] 10102012 10102056 10102100 10112820 10112864 10112908 10112952 10112996
y[2] 10102016 10102060 10102104 10112824 10112868 10112912 10112858 10113000
y[3] 10102020 10102064 10102108 10112828 10112872 10112916 10112960 10113004
y[4] 10102024 10102068 10102112 10112832 10112876 10112920 10112964 10113008
y(5] 10102028 10102072 10102116 10112836 10112880 10112924 10112968 10113012
y[6] 10102032 10102076 10102120 10112840 10112884 10112928 10112972 10113016
y[7] 10102036 10102080 10102124 10112844 10112888 10112932 10112976 10113020

Note the difference between each y value is 4, until you jump to the next col, then the difference is 16 (14 byte hole) consistently until you jump from [2,7] to [3,0].
Huge 10,692 (-4) byte hole. (Forgive the decimal notation, it was just easier to do the math. The app permits hex output. Also: longword(addr) produced same result as integer(addr). Adding "Packed" had no impact on output.)

I think that reducing the size of the array has given you a false positive with the block read.

Look at: "Packed (reserved word)" in Delphi Help:

Instances of a structured type hold more than one value. Structured types include sets, arrays, records, and files as well as class, class-reference, and interface types. Except for sets, which hold ordinal values only, structured types can contain other structured types; a type can have unlimited levels of structuring.

By default, the values in a structured type are aligned on word or double-word boundaries for faster access. When you declare a structured type, you can include the reserved word packed to implement compressed data storage. For example,

type TNumbers = packed array[1..100] of Real;

Using packed slows data access and, in the case of a character array, affects type compatibility (for more information, see [green]Memory management[/green]).

Then look at the referenced "Memory management":

On Windows, the memory manager is optimized for applications that allocate large numbers of small- to medium-sized blocks, as is typical for object-oriented applications and applications that process string data. Other memory managers, such as the implementations of GlobalAlloc, LocalAlloc, and private heap support in Windows, typically do not perform well in such situations, and would slow down an application if they were used directly.

To ensure the best performance, the memory manager interfaces directly with the Win32 virtual memory API (the VirtualAlloc and VirtualFree functions). The memory manager reserves memory from the operating system in [red]1Mb sections of address space[/red], and [red]commits memory as required in 16K increments[/red]. It decommits and releases unused memory in 16K and 1Mb sections. For smaller blocks, committed memory is further suballocated.

Memory manager blocks are always rounded upward to a 4-byte boundary, and always include a 4-byte header in which the size of the block and other status bits are stored. This means that memory manager blocks are always double-word-aligned, which guarantees optimal CPU performance when addressing the block.

The memory manager maintains two status variables, AllocMemCount and AllocMemSize, which contain the number of currently allocated memory blocks and the combined size of all currently allocated memory blocks. Applications can use these variables to display status information for debugging.

The System unit provides two procedures, GetMemoryManager and SetMemoryManager, that allow applications to intercept low-level memory manager calls. The System unit also provides a function called GetHeapStatus that returns a record containing detailed memory-manager status information

What that says to me is that the HOLES are not from a non-contiguous data stream, but from the memory manager placing the data into segmented boundries.

With hard coded 8x8 dynamic array, I tried writing with BlockWrite and reading with your BlockRead code: both failed.

I tried it by writing as before (element by element) then reading with your BlockRead code: That failed too.

Here's my code again. The matrix view makes this easier to follow. It will now optionally display the addresses as hex with added Checkbox. And the var names have been changed to be consistent with your x-y. I'd been staying with w-h for the benefit of the op up to now.

unit Unit5;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, ExtCtrls, ComCtrls, Grids;

type
  TTerrainCell = record
    Index : integer;
  end;

  THeader = record
    WIDTH : integer;
    HEIGHT : integer;
  end;

type
  TForm2 = class(TForm)
    Panel1: TPanel;
    Button1: TButton;
    Button2: TButton;
    Button3: TButton;
    Button4: TButton;
    StringGrid1: TStringGrid;
    CheckBox1: TCheckBox;
    procedure Button1Click(Sender: TObject);
    procedure Button3Click(Sender: TObject);
    procedure Button2Click(Sender: TObject);
    procedure Button4Click(Sender: TObject);
    procedure CheckBox1Click(Sender: TObject);
  private
    { Private declarations }
    Header: THeader;
    MyTerrain : array of array of TTerrainCell;
    outfile : file;  // of byte; (*glenn*)
    procedure ReadData;
    procedure FillData;
    procedure WriteData;
    procedure Truncate;
    procedure WriteMemo;
  public
    { Public declarations }
  end;

var
  Form2: TForm2;

implementation

{$R *.dfm}

const
  filename = 'MyTerrainCell.dat';

procedure TForm2.FillData;
var x, y: integer;
begin
  MyTerrain:= nil;
  Header.WIDTH:= 8;
  Header.HEIGHT:= 8;
  setlength(MyTerrain, Header.WIDTH);
  for x:= Low(MyTerrain) to High(MyTerrain) do
    setlength(MyTerrain[x], Header.HEIGHT);
  for x:= Low(MyTerrain) to High(MyTerrain) do
    for y:= Low(MyTerrain[x]) to High(MyTerrain[x]) do
      MyTerrain[x, y].Index:= Integer(@MyTerrain[x, y].Index); //x + y; //random(32)
end;

procedure TForm2.ReadData;
var x, y: integer;
begin
  assignfile(outfile, filename);
  reset(outfile, 1);  // ADDED ,1 (*glen*)
  // read header
  blockread(outfile, Header, sizeof(Header));
  // create dynamic array dimensions
  setlength(MyTerrain, Header.WIDTH);
  for x:= Low(MyTerrain) to High(MyTerrain) do
    setlength(MyTerrain[x], Header.HEIGHT);
  // read data
  (* new code - not working????
  for x := Low(MyTerrain) to High(MyTerrain) do
    blockread(outfile, MyTerrain[x, Low(MyTerrain)], Sizeof([x, Low(MyTerrain)])*Header.Height);
  *)
  for x:= Low(MyTerrain) to High(MyTerrain) do
    for y:= Low(MyTerrain[x]) to High(MyTerrain[x]) do    {*glen*}
      blockread(outfile, MyTerrain[x, y], Sizeof(MyTerrain[x, y])); {*glen*}
  closefile(outfile);
end;

procedure TForm2.WriteData;
var x, y: integer;
begin
  assignfile(outfile, filename);
  rewrite(outfile, 1);
  // write header
  blockwrite(outfile, Header, sizeof(THeader));
  // write to disk
  (* new code - not working????
  for x := Low(MyTerrain) to High(MyTerrain) do
    blockwrite(outfile, MyTerrain[x, Low(MyTerrain)], Sizeof([x, Low(MyTerrain)]) * Header.Height);
  *)
  for x := Low(MyTerrain) to High(MyTerrain) do            {*glenn*}
    for y := Low(MyTerrain[x]) to High(MyTerrain[x]) do    {*glenn*}
      blockwrite(outfile, MyTerrain[x, y], sizeof(MyTerrain[x, y])); {*glenn*}
  closefile(outfile);
end;

procedure TForm2.Truncate;
var n: integer;
begin
  n:= 5;
  Header.Height:= Header.Height - n;
  Move(MyTerrain[n, Low(MyTerrain)],
       MyTerrain[0, Low(MyTerrain)],
       Sizeof([0, Low(MyTerrain)]) * Header.Height);
end;

procedure TForm2.WriteMemo;
var x, y: integer;
begin
  StringGrid1.FixedCols:= 1;
  StringGrid1.FixedRows:= 1;
  StringGrid1.ColCount:= Header.WIDTH + 1;
  StringGrid1.RowCount:= Header.HEIGHT + 1;
  for x:= 1 to StringGrid1.ColCount - 1 do
    StringGrid1.Cells[x, 0]:= format('x[%d]', [x-1]);
  for y:= 1 to StringGrid1.RowCount - 1 do
    StringGrid1.Cells[0, y]:= format('y[%d]', [y-1]);
  for x:= Low(MyTerrain) to High(MyTerrain) do
    for y:= Low(MyTerrain[x]) to High(MyTerrain[x]) do
      if CheckBox1.Checked then
        StringGrid1.Cells[x+1, y+1]:= format('%s', [IntToHex(MyTerrain[x, y].Index, 8)])
      else
        StringGrid1.Cells[x+1, y+1]:= format('%8d', [MyTerrain[x, y].Index])
end;

procedure TForm2.Button1Click(Sender: TObject);
begin
  ReadData;
  WriteMemo
end;

procedure TForm2.Button3Click(Sender: TObject);
begin
  FillData;
  WriteMemo
end;

procedure TForm2.Button2Click(Sender: TObject);
begin
  WriteData;
  WriteMemo
 end;

procedure TForm2.Button4Click(Sender: TObject);
begin
  Truncate;
  WriteMemo
end;

procedure TForm2.CheckBox1Click(Sender: TObject);
begin
  WriteMemo
end;

end.

NOTE: Truncate does not work, it could be fixed doing it by element.


Roo
Delphi Rules!

 

[roo0047]

majorkup - Unfortunately, accuracy most always requires the sacrifice of speed.
See: Hick's or Fitts' Law.

Roo
Delphi Rules!

 

[Glenn9999]

How is the code failing with the blockread/blockwrite as I described it? How much data are you using?

As for what you have posted, your observations are thoroughly consistent with what I have written. Perhaps the confusion is that in a dynamic array of dimensions [x, y] (or [w, h]) you have x dynamic array structures placed into a dynamic array structure.

In your first example:

Move(MyTerrain[100, Low(MyTerrain)], 
       MyTerrain[0, Low(MyTerrain)],
       Sizeof([COLOR=red]MyTerrain[/color][0, Low(MyTerrain)]) * Header.Height));

(Assuming the change in red) You are moving Sizeof(MyTerrain[0, Low(MyTerrain) * Header.Height bytes from dynamic array structure MyTerrain[100] to dynamic array structure MyTerrain[0]. If the number of allocated bytes were less in the destination (i.e. SetLength was not used), I would expect an error.

The 2-dim dynamic array is NOT contiguous by any stretch of the imagination as you have clearly shown. A 2-dim dynamic array, is really a dynamic array of dynamic arrays, as the code to build one that we have both been using illustrates:

// sets dynamic array MyTerrain to Header.Width units
setlength(MyTerrain, Header.WIDTH);
for x:= Low(MyTerrain) to High(MyTerrain) do
// sets dynamic array MyTerrain[X] to Header.Height units
  setlength(MyTerrain[x], Header.HEIGHT);

However, a single 1-dim dynamic array IS contiguous - you (and I) have shown that as well. Notice that every x[index] unit is spaced 4 bytes apart in your example, which is the size of your TTerrainCell record.

Hope this clears things up a bit (or maybe not?). In the 2-dim array example, you have one pointer (MyTerrain) which holds a table of pointers in contiguous storage. Each of those pointers (MyTerrain[x]) points to a unique contiguous section of storage, not necessarily related to any other section of memory that comprises MyTerrain.

(Of course, I could not be understanding the specific error you are witnessing, either, so it's possible that's there's enough confusion to go around - that's how dynamic arrays seem to be)

----------
Measurement is not management.