Binary and Text File (all in one) :( 1

[litton1]

Hi all, I am attempting to stream a text file which also contains binary data. This in it’s self is not a problem. The problem is, I cannot keep track of where I am in the file. Hence I am not getting all the binary data.
So I suppose my problem is that I cannot find my position in the file where the binary data finishes . . .
The code I am using at the moment in brief.

long fileLen = streamReader.BaseStream.Length;
// Read and display lines from the file until the end of 
// the file is reached.
while ((line = streamReader.ReadLine()) != null)
{
   if (!gettingBinaryImage)
     filePositionPointer += line.Length + newline.Length;
   else
      filePositionPointer += line.Length;
}

When I get to the end of the file my filePositionPointer is only a third of the size of fileLen? Grrrr! deep breaths!! lol
Hope this makes sense

Any help appreciated.

Age is a consequence of experience

 

[jmeckley]

We'll need more information on the code to determine what's going on. for starters:[tt]
1. how are you using fileLen?
2. where does gettingBinaryImage come from? what does it represent?
3. where is newline defined? why do you only use it with 'gettingBinaryImage'?
4. do you have control over the file format?
5. could you post the complete code? this will help determine where the problem lies.
[/tt]

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

thx for the reply,
I suppose I should answer number 4 first. No i have no control over the file format. The file format is a type of postscript file that has images within which are converted to binary (from hex data) before writing.. an example of thisshort)

/CIP3PreviewImageCompression /None def
CIP3PreviewImage
... Binary Hex Data Here
...more text here
...more binary hex data
....

I apologies for some unreadable parts, there is a lot of test data in here.

Q1 I am Using filelen only to find the length of the file, that way when i am counting my way through the characters, when i get to the end of the file my counter should be the same as filelen?

Q2 I set gettingBinaryImage to true when i hit

CIP3PreviewImage

in the file cos this means that i am about to move to a binary part of the file.

Q3 newline is a string that = Environment.NewLine (\r\n),
I only use it when not getting binary image because they are hidden chars at the end of text, but i dont know if i should always use them. they are just to try to keep track of where i am in the file.

Q5 yep no probs.. need any bits explaining, let me know.

Going round in circles for the 4th day running.. so if its a bit messy sry in advance

using (StreamReader streamReader = new StreamReader(base.FileAndPath))
{
  long fileLen = streamReader.BaseStream.Length;
  while ((line = streamReader.ReadLine()) != null)
  {
    if (!gettingBinaryImage)
      filePositionPointer += line.Length + newline.Length;
    else
      filePositionPointer += line.Length;
       
      string[] words = line.Split(null);
      switch (words[0])
      {
        case CIP3_PREVIEW_IMAGE: // moving into binary image so get the ascii text
           binaryImageStartPosition = filePositionPointer + line.Length + newline.Length;
           asciiString.Append(line + newline);
           AddFileDataToArray( asciiString.ToString());
           asciiString = null;
           asciiString = new StringBuilder();
           gettingBinaryImage = true;
           break;
       case CIP3_END_SEPERATIONS:
           //need to get binary image here
           binaryfinishPosition = filePositionPointer + line.Length + newline.Length;
           testFileLength = streamReader. BaseStream.Position;
           hexString = GetHexImage(binaryImageStartPosition, binaryfinishPosition);
           asciiString.Append(line + newline);
           AddFileDataToArray(hexString);
                                 
          gettingBinaryImage = false;
          break;  
     case CIP3_END_OF_FILE_COMMENT:
          asciiString.Append(line + newline);
          AddFileDataToArray(asciiString.ToString());
          WriteTestFile();
          break;
          default:
     if (!gettingBinaryImage)
     {
       if (line.Length > 0)
           asciiString.Append(line + newline);
       //AddFileDataToArray(hexString);
     }
        break;
   }
 }
}

I guess I should re emphasize what I am trying to do. I am going through an ascii file (no problem) I count my characters (don’t know if I should). I find the beginning of a binary image (CIP3_PREVIEW_IMAGE that I need to convert back to hex data (no problem converting) . I try to find the end of the image so I carry on counting … I come to CIP3_END_SEPERATIONS: (“CIP3EndSeparation” in the file this is after the binary image) so I get the number of characters passed and I now call a function and pass the image start position and finish position so it will extract the binary data from the file which can then be converted to hex….. When I get all this data I will create a new file which will be duplicate of this one but with Hex data and not binary. Hope this is clearer?

Age is a consequence of experience

 

[jmeckley]

you don't need the stream length since your not using it within your code.

what is the format of the data? in other words does the data look like:
1.
hello world
CIP3_PREVIEW_IMAGE
sadjf8743095j2rys0f7qefj9s8df7sadf
CIP3_END_SEPERATIONS
hello world
CIP3_END_OF_FILE_COMMENT

or 2.
hello world CIP3_PREVIEW_IMAGEsadjf8743095j2rys0f7qefj9s8df7sadf CIP3_END_SEPERATIONS hello world CIP3_END_OF_FILE_COMMENT

or 3.
hello world
CIP3_PREVIEW_IMAGEsadjf8743095j2rys0f7
qefj9s8df7sadf CIP3_END_SEPERATIONS
hello world CIP3_END_OF_FILE_COMMENT

or 4.
hello
world CIP3_PREVIEW_IMAGEsadjf8743095j2rys0f7
qefj9s8df7sadf CIP3_END_SEPERATIONS hello
world CIP3_END_OF_FILE_COMMENT


or some other variation?
does the image always fall between CIP3_PREVIEW_IMAGE and CIP3_END_SEPERATIONS?
can the binary image string contain carriage returns?

if there is a standard format to the text document you could use this to parse the image data instead of the complex math positioning.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

it looks like 1
you asked:
does the image always fall between CIP3_PREVIEW_IMAGE and CIP3_END_SEPERATIONS?
Answer: Yes.

Age is a consequence of experience

 

[jmeckley]

here is a quick & dirty sample I put together

using System;
using System.Collections;
using System.Text;
using System.IO;

namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            string fileName = @"c:\sample.txt";
            Class1 test = new Class1();
            Console.WriteLine(test.TransformFile(fileName));
            Console.ReadKey();
        }
    }

    class Class1
    {
        public Class1() { } 

        public string TransformFile(string filePath)
        {
            //read data from file into an array list.  this minimizes the time the file is open.
            StringBuilder toReturn = new StringBuilder();
            ArrayList lines = new ArrayList();
            string line;
            using (StreamReader reader = new StreamReader(filePath))
            {
                while ((line = reader.ReadLine()) != null)
                {
                    lines.Add(line);
                }
            }

            //cycle through each item in the array list...
            string beginImage = "CIP3_PREVIEW_IMAGE";
            bool isImage = false;
            for (int i = 0; i < lines.Count; i++)
            {
                //convert to string
                line = (string)lines[i];

                //the string begins an image: write the line to the return value and flag an image is about to begin
                if (line.ToUpper().Trim() == beginImage)
                {
                    toReturn.AppendLine(line);
                    isImage = true;
                }
                //the string is an image: convert, append to return value, and flag the image is done
                else if (isImage)
                {
                    toReturn.AppendLine(this.ConvertBinaryToText(line));
                    isImage = false;
                }
                //just standard text: write to return value
                else
                {
                    toReturn.AppendLine(line);
                }
            }

            //cycle complete return string value
            return toReturn.ToString();
        }

        private string ConvertBinaryToText(string binaryData)
        {
            //do your acutal convesion here.
            return string.Format("[Converted {0} To Hex]", binaryData);
        }
    }
}

the text file data:

hello world
CIP3_PREVIEW_IMAGE
sadjf8743095j2rys0f7qefj9s8df7sadf
CIP3_END_SEPERATIONS
hello world
CIP3_PREVIEW_IMAGE
5d97asd5f9
CIP3_END_SEPERATIONS


CIP3_PREVIEW_IMAGE
ad9as7d5sdfg9sdfg
CIP3_END_SEPERATIONS
this is another line of text
and 1 more just to test
CIP3_PREVIEW_IMAGE
ad9as7d5sdfg9sdfg
CIP3_END_SEPERATIONS

CIP3_END_OF_FILE_COMMENT

this is the end result

hello world
CIP3_PREVIEW_IMAGE
[Converted sadjf8743095j2rys0f7qefj9s8df7sadf To Hex]
CIP3_END_SEPERATIONS
hello world
CIP3_PREVIEW_IMAGE
[Converted 5d97asd5f9 To Hex]
CIP3_END_SEPERATIONS
CIP3_PREVIEW_IMAGE
[Converted ad9as7d5sdfg9sdfg To Hex]
CIP3_END_SEPERATIONS
this is another line of text
and 1 more just to test
CIP3_PREVIEW_IMAGE
[Converted ad9as7d5sdfg9sdfg To Hex]
CIP3_END_SEPERATIONS
CIP3_END_OF_FILE_COMMENT

does this solution work, or at least point you in the right direction?

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

Gee thx! I am at home but will try in the morning

Age is a consequence of experience

 

[litton1]

Something I neglected to mention….. When I find the correct begin and end point of the Binary data, I call
hexString = GetHexImage(binaryImageStartPosition, binaryfinishPosition);
The reason I do this is so that I can pull this data from the file in a byte array which is as follows:

private string GetHexImage(int imageStartIndexInFile, int imageFinishPositionIndex)
{
  int imageSize = (imageFinishPositionIndex - imageStartIndexInFile);
  string hexString = string.Empty;
  string formattedHex = string.Empty;
  BinaryReader binaryReader = new BinaryReader(File.OpenRead(base.FileAndPath));
  long fileLen = binaryReader.BaseStream.Length;
  long fileLeft = fileLen - (imageSize + imageStartIndexInFile);
  byte[] dataBeforeImage = binaryReader.ReadBytes(imageStartIndexInFile);
  byte[] imageBytes = binaryReader.ReadBytes(imageSize);
  byte[] leftover = binaryReader.ReadBytes((int) fileLeft);
  binaryReader.Close();

  hexString = BitConverter.ToString(imageBytes);
  formattedHex = FormatHexString(hexString);
  return (formattedHex);
}

This may not be the best way? I am wondering if I was to use your way and pass in a string could I convert that to a byte array?


Age is a consequence of experience

 

[litton1]

yep i can see where you was going with this now...
if i did something like

private byte[] ConvertBinaryToText(string binaryData)
{
   byte[] byteToReturn = Encoding.ASCII.GetBytes(binaryData);
   return byteToReturn;
}

I will let you know how i am getting on later.
BTW Nicely laid out

Age is a consequence of experience

 

[litton1]

This is the bit that I cannot get straight, if I have my binary data in a string and I would like/need it in a byte array, I cannot convert from a string to a byte because its not actually an ascii type string, is it?
BTW the last method should have be
ConvertTextToBinary(string theBinaryData)

Age is a consequence of experience

 

[jmeckley]

a quick google search returned this:

string s = "this is a test";
byte[] b = new byte[20];
b = System.Text.Encoding.UTF8.GetBytes(s);
for (int i=0; i<b.Length;i++)
{
    Console.WriteLine((char)b[i]);
}

source:

http://mark.michaelis.net/Blog/ConvertStringToByte.aspx

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

It should be as simple as that... but after trying these

private byte[] ConvertBinaryToText(string binaryData)
{
   Byte[] byteToReturn1 = Encoding.UTF8.GetBytes(binaryData);
   Byte[] byteToReturn2 = Encoding.UTF32.GetBytes(binaryData);
   Byte[] byteToReturn3 = Encoding.ASCII.GetBytes(binaryData);
   Byte[] byteToReturn4 = Encoding.Unicode.GetBytes(binaryData);
}

I know they have the wrong data in them as the byte array at this point should contain mainly 255's. when i take from a file stream into a byte array this is the result (255's) and when converted to hex is FF FF FF FF etc?
any ides why this is?


Age is a consequence of experience

 

[jmeckley]

I'm sorry I don't follow. 255[binary] = FF[hex] is your binary data 255 or is it something else? My understanding is you want to turn.
byte[] { 1, 255, 160 } into 01FFA0.

if that's the case then what does the binaryData argument look like when you pass it to the conversion function?

if it looks like this: 001 174 222 033 175 255
it would convert to: 01AEDE21AFFF

string[] parts = binaryData.split(null);
byte[] bits =  new byte[parts.Length];
for(int i = 0; i < parts.Length; i++)
{
  bits[i] = int.Parse(parts[i]);
}

return new ASCIIEncoding().GetString(bits);

i also found a hex/binary converter on the code project website [

http://www.codeproject.com/csharp/hexencoding.asp

] this may help solve your conversion problem.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

yes thats correct, when i read the file from a stream straight into a byte array i get byteArray[mainly 255] = FFFFFFFFFFFFFFF (maiinly) but when i put into a first i get a lot of squares and other unreadable stuff, when i convert to hex it is all wrong

Age is a consequence of experience

 

[litton1]

sry this " but when i put into a first i get "
should say " but when i put into a string first "

Age is a consequence of experience

 

[jmeckley]

so the string contains the squares, or the hex output contains the squares? if the string contains the squares, they may be carriage returns, which should be stripped. If the hex contains the squares then the string contains characters that cannot be converted to binary/hex.

Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

Yes the string contains the squares. I will investigate a bit further by splitting
thx for all your help so far

Age is a consequence of experience

 

[litton1]

Ok I have tested the image by sending the start and finish position of the image ( bit of test stuff in the following method ) and the returned byte array contains 99% 255's. This is as expected. I may just be better using the FileStream Class, as I can then get the image start and finish position and then pass it to this method.
It will mean reading the file twice though

private string GetHexImage(int imageStartIndexInFile, int imageFinishPositionIndex)
{
   int imageSize = (imageFinishPositionIndex - imageStartIndexInFile);
   string hexString = string.Empty;
   string formattedHex = string.Empty;
   BinaryReader binaryReader = new BinaryReader(File.OpenRead(base.FileAndPath));
   long fileLen = binaryReader.BaseStream.Length;
   long fileLeft = fileLen - (imageSize + imageStartIndexInFile);
   byte[] dataBeforeImage = binaryReader.ReadBytes(imageStartIndexInFile);
   byte[] imageBytes = binaryReader.ReadBytes(imageSize);
   byte[] leftover = binaryReader.ReadBytes((int) fileLeft);
   binaryReader.Close();
   hexString = BitConverter.ToString(imageBytes);
   formattedHex = FormatHexString(hexString);
   return (formattedHex);
}

Age is a consequence of experience

 

[jmeckley]

to reduce the opening/closing of files for converting the image read the file into a byte array and close out the file. similar to reading each line into a string array.

then reference the byte array when you convert the binary to hex. this way you only need to read the file twice. once to get the string and another to get the binary.

something along the lines of
1. read file into byte array after reading file into string array.
2. track your position with in the file by using a long type variable
long position = 0;
//in your foreach string loop
posistion += line.Length;
3. when you get to an image pass the posistion of where you are and the length of the current string (the image).
4. use the byte array to convert the bytes to formatted hex.

this would change your ConvertBinaryToHex function because you would pass the starting point and length instead of the starting point and ending point. Or you could add the string length to the to the starting position and not change the signature.


Jason Meckley
Programmer
Specialty Bakers, Inc.

 

[litton1]

An excellent thought/solution ! I was going to keep reopening the file? Thx for all your help with this rather tedious problem.

Age is a consequence of experience