Category: Windows XP

It would be very nice to be able to make Windows applications that can act as either a GUI or console application depending on how they are used (i.e. act as a GUI application if double clicked in Windows Explorer or as a console application if called from a cmd.exe window).

Unfortunately the way Windows work, each exe application has field in the PE header that specifies which subsystem it should run under. This is set in Visual Studio by using one of the linker SUBSYSTEM option (i.e. Windows or Console). The subsystem is used by the Windows kernel to set up the execution environment for the application. If the application is built using SUBSYSTEM/Console then the kernel will connect the application to the parent console and the application’s stdout, stderr and stdin will be redirected to the parent console. If the program is built as a GUI application then the application detaches from the parent console and all output to stdout and stderr are are lost – basically the program runs, but doesn’t output anything to the parent console window.

People have attempted various hacks over the years to solve this problem. One solution proposed was to compile the program as a Windows application and then edit the PE header to mark the program as using the Console subsystem. The downside of this approach is it flashes a Console window on the screen when run as a GUI application that looks pretty unprofessional. The second hack commonly used is to create two separate binaries, for example, a myuselessprogram.com and myuselessprogram.exe. The .com version is built as a Console application while the .exe is built as Windows application. When you run myuselessprogram the Windows probing rule runs the .com version first and if there is nothing on the command line then the .com version calls the .exe Windows version. While both these approaches work, they are to say less than ideal.

A better approach is use the WINAPI AttachConsole function to attach the application to the parent console and then redirect stdout, stdin and stderr back to the parent console. This actually works very well except that when the application exits the parent console can’t detect this and hence release the command prompt. The end result is the parent console just sits there until the user presses the “enter” key.

There is no really elegant solution to this problem, but as applications can simulate the keyboard being used, a simple solution is to call the SendInput API function with the “enter” key just before the application exits. This simulates the user pressing the enter key and hence releases the command prompt.

To show how this approach works I have written a small test application (see below). The main limitations is that AttachConsole is only available on Windows XP and above. It does works under Cygwin which is nice.

Update. I have added a check to make sure that the console window is in focus before sending the enter key. It is a good check to make if you are running your console program in a background script or else you will end up with lots of “enter” key presses in whatever application you actually have in focus – lots of fun if you are working on the documentation while a script runs in the background :)

Update 2. Contrary to what has been posted on Stack Overflow this approach works with STDIN too. I don’t have any need to capture STDIN with my application (just the command line parameters), but all you need to do to capture stdin is treat STDIN as is done for STDOUT (i.e just redirect STDIN to the console) in the attachOutputToConsole function.

Update 3. This is MIT licensed if this is important to you :)

Update 4. Microsoft has broken the old approach I was using in VS2015. I have updated the code to use a different way of attaching STDOUT and STRERR to the parent console. This code works with VS2015 and all earlier compliers I was able to test (VS2013, VS2008, VS2005).

/*

 Copyright (c) 2013, 2016 Daniel Tillett
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 
 1. Redistributions of source code must retain the above copyright notice, this
 list of conditions and the following disclaimer.
 2. Redistributions in binary form must reproduce the above copyright notice,
 this list of conditions and the following disclaimer in the documentation
 and/or other materials provided with the distribution.
 
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
 ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

*/

#define WINVER 0x0501 // Allow use of features specific to Windows XP or later.
#define _WIN32_WINNT 0x0501
#define WIN32_LEAN_AND_MEAN
#include "windows.h"
#include "io.h"
#include "fcntl.h"
#include "stdio.h"
#include "stdlib.h"
#pragma comment(lib, "User32.lib")

// Attach output of application to parent console
static BOOL attachOutputToConsole(void) {
HANDLE consoleHandleOut, consoleHandleError;

if (AttachConsole(ATTACH_PARENT_PROCESS)) {
  // Redirect unbuffered STDOUT to the console
  consoleHandleOut = GetStdHandle(STD_OUTPUT_HANDLE);
  if (consoleHandleOut != INVALID_HANDLE_VALUE) {
    freopen("CONOUT$", "w", stdout);
    setvbuf(stdout, NULL, _IONBF, 0);
    }
  else {
    return FALSE;
    }
  // Redirect unbuffered STDERR to the console
  consoleHandleError = GetStdHandle(STD_ERROR_HANDLE);
  if (consoleHandleError != INVALID_HANDLE_VALUE) {
    freopen("CONOUT$", "w", stderr);
    setvbuf(stderr, NULL, _IONBF, 0);
    }
  else {
    return FALSE;
   }
  return TRUE;
  }
//Not a console application
return FALSE;
}

// Send the "enter" to the console to release the command prompt 
// on the parent console
static void sendEnterKey(void) {
 INPUT ip;
 // Set up a generic keyboard event.
 ip.type = INPUT_KEYBOARD;
 ip.ki.wScan = 0; // hardware scan code for key
 ip.ki.time = 0;
 ip.ki.dwExtraInfo = 0;

 // Send the "Enter" key
 ip.ki.wVk = 0x0D; // virtual-key code for the "Enter" key
 ip.ki.dwFlags = 0; // 0 for key press
 SendInput(1, &ip, sizeof(INPUT));

 // Release the "Enter" key
 ip.ki.dwFlags = KEYEVENTF_KEYUP; // KEYEVENTF_KEYUP for key release
 SendInput(1, &ip, sizeof(INPUT));
}

int WINAPI WinMain(HINSTANCE hInstance, 
HINSTANCE hPrevInstance, 
PSTR lpCmdLine, 
INT nCmdShow) {
int argc = __argc;
char **argv = __argv;
UNREFERENCED_PARAMETER(hInstance);
UNREFERENCED_PARAMETER(hPrevInstance);
UNREFERENCED_PARAMETER(lpCmdLine);
UNREFERENCED_PARAMETER(nCmdShow);
BOOL console;
int i;

//Is the program running as console or GUI application
console = attachOutputToConsole();

if (console) {
    // Print to stdout
    printf("Program running as console application\n");
    for (i = 0; i < argc; i++) {
         printf("argv[%d] %s\n", i, argv[i]);
         }

    // Print to stderr
    fprintf(stderr, "Output to stderr\n");
    }
else {
    MessageBox(NULL, "Program running as Windows GUI application",
    "Windows GUI Application", MB_OK | MB_SETFOREGROUND);
}

// Send "enter" to release application from the console
// This is a hack, but if not used the console doesn't know the application has
// returned. The "enter" key only sent if the console window is in focus.
if (console && (GetConsoleWindow() == GetForegroundWindow())){
    sendEnterKey();
    }
return 0;
}