Использование функции os.execute без показа окна windows cmd

В качестве временного решения

os.execute ("start /I /MIN /B program.exe")

Подскажите, а разве при этом само ДОС окно у вас запускается минимизированным?  

Нет.
Команда start помогает ДОС-окну закрыться до завершения приложения program.exe. Но ключ /MIN не помогает: ДОС-окно всё-равно мелькает на экране.

Проще всего создать  внешнюю библтотеку с этой функцией.
Или взять готовую. Можно вытряхнуть из архива http://www.bot4sale.ru/download-categories/2012-06-13-15-10-36/item/market-scaner.html библиотеку bot4sale.dll

require "bot4sale"

bot4sale.Shell(string file,string parameters,string directory,number show)

show:
define(`SW_HIDE', 0) # Hides the window and activates another window.
define(`SW_SHOWNORMAL', 1) # Activates and displays a window. If the window is minimized or maximized, Windows restores it to its original size and position. An application should specify this flag when displaying the window for the first time.
define(`SW_SHOWMINIMIZED', 2) # Activates the window and displays it as a minimized window.
define(`SW_SHOWMAXIMIZED', 3) # Activates the window and displays it as a maximized window.
define(`SW_MAXIMIZE', 3) # Maximizes the specified window.
define(`SW_SHOWNOACTIVATE', 4) # Displays a window in its most recent size and position. The active window remains active.
define(`SW_SHOW', 5) # Activates the window and displays it in its current size and position.
define(`SW_MINIMIZE', 6) # Minimizes the specified window and activates the next top-level window in the z-order.
define(`SW_SHOWMINNOACTIVE', 7) # Displays the window as a minimized window. The active window remains active.
define(`SW_SHOWNA', 8) # Displays the window in its current state. The active window remains active.
define(`SW_RESTORE', 9) # Activates and displays the window. If the window is minimized or maximized, Windows restores it to its original size and position. An application should specify this flag when restoring a minimized window.
define(`SW_SHOWDEFAULT', 10) # Sets the show state based on the SW_ flag specified in the STARTUPINFO structure passed to the CreateProcess function by the program that started the application. An application should call ShowWindow with this flag to set the initial show state of its main window.

возвращает код ошибки или 0