Running a console app from within the current console 3

[CestusGW]

I'm currently in the process of trying to make a VB app which runs from the command line. While I've found information on how to create new instances of the console, etc. (which all work quite well, thank you alt255 and others who made posts in the thread Console Applications), I need to use a currently running console. ie, the shell would look like this
C:\>myProgram.exe -h
These are the command line arguments for myProgram.exe
C:\>

How do I use the Windows API with VB to send my console output back to the console window that called it? Greg W
wolgemga@aecl.ca

 

[CestusGW]

Whoops, never mind - found some references to this elsewhere ... and have decided to just use a copy of VB.NET on another workstation to build this program. Why on earth doesn't MS provide what would appear to be the most base of functionalities to VB6? You'd think it would be easier to compile a console app as these applications came before GUI ones. Wait, that's logical and it doesn't apply to MS ... Greg W
wolgemga@aecl.ca

 

[strongm]

With a lot a certain amount of difficulty...

The problem is that VB applications compile as GUI applications, not console applications. During process startup, a VB application is foreceably disassociated from any console that might have launched it, and STDIN, STDOUT and STDERR are removed.

Now AllocConsole, which you've probably seen in the other thread(s) that you mention, kind of addresses this in that it creates a new console, and gives the missing standard streams - which point to this new console - back to VB

OK, well, how do you get something similar to work with an already existing console? Sadly, you can't (well, prior to XP). Even if you do something like a FindWindow to locate the console, there's no way to attach to it (Well, XP has an AttachConsole API, but no prior version of Windows does)

The lateral thinking method of dealing with this involves considering the fact that the problem is caused by that nasty initial startup code (which runs before any of your VB code). If there was some way of stopping the detachment of the launching console and the removal of the standard streams then all would be well.

Now in C++ there are compiler options that allow you to specify whether you want a console app or a GUI app. Unfortunately, MS designed VB as a closed environment so we can't make a similar choice. Does this stop us? Not necessarily. Making that choice in C++ may simply make some changes in the programs exe header.

A little bit of investigation reveals that it changes 1 byte, and if we could make the same change then we would have a VB console app!

So here's some self-modifying code that does just that (note that since it modifies a byte in the PE header it will only work on compiled applications):
[tt]
Option Explicit
Public Declare Sub CopyMem Lib "kernel32" Alias "RtlMoveMemory" (dst As Any, src As Any, ByVal Size As Long)

Public Sub Main()

' Is this app a console app? If it is not, then make a version that
' is and shell it instead
If Not AmIConsole Then
MsgBox "Converting into console application"
MakeConsoleVersion
Shell App.EXEName + "_iac.exe"
End
Else
' Whatever you want to do with your console app
End If
End Sub

Private Function SetConsoleApp(strFile As String) As Boolean
Dim hFile As Long
Dim strMZ_Header As String * 512
Dim lNewPE_Offset As Long
Const PE_FLAG_OFFSET As Long = 93&
Const DOS_FILE_OFFSET As Long = 25&


' Get the header
hFile = FreeFile
Open strFile For Binary Access Read Write As hFile
Seek #hFile, 1
Get hFile, , strMZ_Header
CopyMem lNewPE_Offset, ByVal Mid$(strMZ_Header, 61, 4), 4

' Rewrite the Application type byte to indicate console application
Seek #hFile, lNewPE_Offset + PE_FLAG_OFFSET
Put hFile, , 3

' Clean up
Close hFile

SetConsoleApp = True
End Function

Private Function MakeConsoleVersion()

FileCopy App.EXEName + ".exe", App.EXEName + "_iac.exe"
SetConsoleApp App.EXEName + "_iac.exe"

End Function

' Function that checks whether current VB application is a console app
Private Function AmIConsole() As Boolean
Dim hFile As Long
Dim strMZ_Header As String * 512
Dim lNewPE_Offset As Long
Dim lData As Long
Const PE_FLAG_OFFSET As Long = 93&
Const DOS_FILE_OFFSET As Long = 25&

hFile = FreeFile
Open App.EXEName & ".exe" For Binary Access Read As hFile
Seek #hFile, 1
Get hFile, , strMZ_Header
Close hFile

CopyMem lNewPE_Offset, ByVal Mid$(strMZ_Header, 61, 4), 4
lData = Asc(Mid$(strMZ_Header, lNewPE_Offset + PE_FLAG_OFFSET, 1))
If lData = 3 Then
AmIConsole = True
MsgBox "This is a console application"
End If

End Function

 

[CestusGW]

My question now would be how would this change the code I write in VB? Now that this executable is identified as a console app, how do I reference its parent console window? Will VB now have a builtin refernce to the parent console which I can call? Greg W
wolgemga@aecl.ca

 

[CestusGW]

OK, looking over some of your other threads I can see I'd want to write to the STDOUT pipe which should now be available to me because VB hasn't killed it? What sort of a command would I use to write to the STDOUT pipe from VB? Greg W
wolgemga@aecl.ca

 

[CestusGW]

And in the end I've pulled it off! This is the code which I used (a combination of some of StrongM's solutions) which, in essence, makes a console application (although my version just prints to output atm).

Private Const STD_ERROR_HANDLE = -12&
Private Const STD_INPUT_HANDLE = -10&
Private Const STD_OUTPUT_HANDLE = -11&
Private Declare Function GetStdHandle Lib "kernel32" (ByVal nStdHandle As Long) As Long
Private Declare Function WriteFile Lib "kernel32" (ByVal hFile As Long, ByVal lpBuffer As String, ByVal nNumberOfBytesToWrite As Long, lpNumberOfBytesWritten As Long, ByVal lpOverlapped As Long) As Long
Public Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
Option Explicit
Public Declare Sub CopyMem Lib "kernel32" Alias "RtlMoveMemory" (dst As Any, src As Any, ByVal Size As Long)

Public Sub Main()
Dim myOutputHandle, myString, theResult
' Is this app a console app? If it is not, then make a version that
' is and shell it instead
If Not AmIConsole Then
MsgBox "Converting into console application"
MakeConsoleVersion
Shell App.EXEName + "_iac.exe"
End
Else
myOutputHandle = GetStdHandle(STD_OUTPUT_HANDLE)
myString = "Hello world!"
theResult = WriteFile(myOutputHandle, ByVal myString, Len(myString), Len(myString), 0&)
Sleep 2000
End If
End Sub

Private Function SetConsoleApp(strFile As String) As Boolean
Dim hFile As Long
Dim strMZ_Header As String * 512
Dim lNewPE_Offset As Long
Const PE_FLAG_OFFSET As Long = 93&
Const DOS_FILE_OFFSET As Long = 25&


' Get the header
hFile = FreeFile
Open strFile For Binary Access Read Write As hFile
Seek #hFile, 1
Get hFile, , strMZ_Header
CopyMem lNewPE_Offset, ByVal Mid$(strMZ_Header, 61, 4), 4

' Rewrite the Application type byte to indicate console application
Seek #hFile, lNewPE_Offset + PE_FLAG_OFFSET
Put hFile, , 3

' Clean up
Close hFile

SetConsoleApp = True
End Function

Private Function MakeConsoleVersion()

FileCopy App.EXEName + ".exe", App.EXEName + "_iac.exe"
SetConsoleApp App.EXEName + "_iac.exe"

End Function

' Function that checks whether current VB application is a console app
Private Function AmIConsole() As Boolean
Dim hFile As Long
Dim strMZ_Header As String * 512
Dim lNewPE_Offset As Long
Dim lData As Long
Const PE_FLAG_OFFSET As Long = 93&
Const DOS_FILE_OFFSET As Long = 25&

hFile = FreeFile
Open App.EXEName & ".exe" For Binary Access Read As hFile
Seek #hFile, 1
Get hFile, , strMZ_Header
Close hFile

CopyMem lNewPE_Offset, ByVal Mid$(strMZ_Header, 61, 4), 4
lData = Asc(Mid$(strMZ_Header, lNewPE_Offset + PE_FLAG_OFFSET, 1))
If lData = 3 Then
AmIConsole = True
MsgBox "This is a console application"
End If

End Function



Greg W
wolgemga@aecl.ca