Using PythonService.exe to host python service while using virtualenv

Question

I've got a Windows 7 environment where I need to develop a Python Windows Service using Python 3.4. I'm using pywin32's win32service module to setup the service and most of the hooks seem to be working ok.

The problem is when I attempt to run the service from source code (using python service.py install followed by python service.py start). This uses PythonService.exe to host service.py - but I'm using a venv virtual environment and the script can't find it's modules (error message discovered with python service.py debug).

Pywin32 is installed in the virtualenv and in looking at the source code of PythonService.exe, it dynamically links in Python34.dll, imports my service.py and invokes it.

How can I get PythonService.exe to use my virtualenv when running my service.py?

Busyness

Related question: #27463082 – Busyness

Answer 1

It appears this used to work correctly with the virtualenv module before virtual environments were added to Python 3.3. There's anecdotal evidence (see this answer: https://mcmap.net/q/168946/-how-does-virtualenv-work) that Python's site.py used to look upward from the executable file until it found a directory that would satisfy imports. It would then use that for sys.prefix and this was sufficient for PythonService.exe to find the virtualenv it was inside of and use it.

If that was the behavior, it appears that site.py no longer does that with the introduction of the venv module. Instead, it looks one level up for a pyvenv.cfg file and configures for a virtual environment in that case only. This of course doesn't work for PythonService.exe which is buried down in the pywin32 module under site-packages.

To work around it, I adapted the activate_this.py code that comes with the original virtualenv module (see this answer: https://mcmap.net/q/358099/-how-can-i-activate-a-pyvenv-virtualenv-from-within-python-activate_this-py-was-removed). It is used to bootstrap an interpreter embedded in an executable (which is the case with PythonService.exe) into using a virtualenv. Unfortunately, venv does not include this.

Here's what worked for me. Note, this assumes the virtual environment is named my-venv and is located one level above the source code location.

import os
import sys

if sys.executable.endswith("PythonService.exe"):

    # Change current working directory from PythonService.exe location to something better.
    service_directory = os.path.dirname(__file__)
    source_directory = os.path.abspath(os.path.join(service_directory, ".."))
    os.chdir(source_directory)
    sys.path.append(".")

    # Adapted from virtualenv's activate_this.py
    # Manually activate a virtual environment inside an already initialized interpreter.
    old_os_path = os.environ['PATH']
    venv_base = os.path.abspath(os.path.join(source_directory, "..", "my-venv"))
    os.environ['PATH'] = os.path.join(venv_base, "Scripts") + os.pathsep + old_os_path
    site_packages = os.path.join(venv_base, 'Lib', 'site-packages')
    prev_sys_path = list(sys.path)
    import site
    site.addsitedir(site_packages)
    sys.real_prefix = sys.prefix
    sys.prefix = venv_base

    new_sys_path = []
    for item in list(sys.path):
        if item not in prev_sys_path:
            new_sys_path.append(item)
            sys.path.remove(item)
    sys.path[:0] = new_sys_path

One other factor in my troubles - there is a new pypi wheel for pywin32 that is provided by the Twisted folks that makes it easier to install with pip. The PythonService.exe in that package was acting oddly (couldn't find a pywin32 dll when invoked) compared to the one you get when installing the official win32 exe package into the virtual env using easy_install.

Busyness

Note, while I accepted this answer, the other provided one is very good also. I would contrast them by saying that this solution focuses on an approach that can be put in the application source code and doesn't require modifying the system (DLLs, registry settings, exe locations, etc.). – Busyness

python my_service.py debug is woking perfect, but python my_service.py install raises error, Error starting service: The service did not respond to the start or control request in a timely fashion – Buryat

Answer 2

Thanks very much for posting this question and a solution. I took a slightly different approach which might also be useful. It is pretty difficult to find working tips for Python services, let alone doing it with a virtualenv. Anyway...

Steps

This is using Windows 7 x64, Python 3.5.1 x64, pywin32-220 (or pypiwin32-219).

• Open an Administrator command prompt.
• Create a virtualenv. C:\Python35\python -m venv myvenv
• Activate the virtualenv. call myvenv\scripts\activate.bat
• Install pywin32, either:
  • From Pypi: pip install pypiwin32,
  • From http://www.lfd.uci.edu/~gohlke/pythonlibs/: pip install path\to\pywin32.whl
• Run the post-install script python myvenv\Scripts\pywin32_postinstall.py -install.
  • This script registers the DLL's in the system, and copies them to C:\Windows\System32. The DLL's are named pythoncom35.dll and pywintypes35.dll. So virtual environments on the same machine on the same major Python point release will share these... it's a minor tradeoff :)
• Copy myvenv\Lib\site-packages\win32\pythonservice.exe to myvenv\Scripts\pythonservice.exe
  • On the service class (whatever subclasses win32serviceutil.ServiceFramework), set the class property _exe_path_ to point to this relocated exe. This will become the service binPath. For example: _exe_path_ = os.path.join(*[os.environ['VIRTUAL_ENV'], 'Scripts', 'pythonservice.exe']).

Discussion

I think why this works is that Python looks upwards to figure out where the Libs folders are and based on that sets package import paths, similar to the accepted answer. When pythonservice.exe is in the original location, that doesn't seem to work smoothly.

It also resolves DLL linking problems (discoverable with depends.exe from http://www.dependencywalker.com/). Without the DLL business sorted out, it won't be possible to import from the *.pyd files from venv\Lib\site-packages\win32 as modules in your scripts. For example it's needed allow import servicemanager; as servicemanager.pyd is not in the package as a .py file, and has some cool Windows Event Log capabilities.

One of the problems I had with the accepted answer is that I couldn't figure out how to get it to accurately pick up on package.egg-link paths that are created when using setup.py develop. These .egg-link files include the path to the package when it's not located in the virtualenv under myvenv\Lib\site-packages.

If it all went smoothly, it should be possible to install, start and test the example win32 service (from an Admin prompt in the activated virtualenv):

python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py install
python venv\Lib\site-packages\win32\Demos\service\pipeTestService.py start
python venv\Lib\site-packages\win32\Demos\service\pipeTestServiceClient.py

The Service Environment

Another important note in all this is that the service will execute the python code in a completely separate environment to the one you might run python myservice.py debug. So for example os.environ['VIRTUAL_ENV'] will be empty when running the service. This can be handled by either:

• Setting environment variables from inside the script, e.g.
  • Find current path starting from the sys.executable, as described in the accepted answer.
  • Use that path to locate a config file.
  • Read the config file and put them in the environment with os.environ.
• Add registry keys to the service with the environment variables.
  • See Accessing Environment Variables from Windows Services for doing this manually with regedit.exe
  • See REG ADD a REG_MULTI_SZ Multi-Line Registry Value for doing this from the command line.

Antiserum

Installing pypiwin32 and running pywin32_postinstall.py does not work as it looks for pywin32_system32 directory, but in case of pypiwin32 it is pypiwin32_system32 (there's pi in the middle). – Succuss

Also one should set not _exe_path_, but _exe_name_ parameter. And most likely it should be wrapped in if not hasattr(sys, 'frozen'):. Thank you for your detailed instructions! – Succuss

You may not be able to start your service, if you happen to to have placed your venv in your local users directory like I did. You need to provide the username/password like so win32serviceutil.HandleCommandLine(cls, argv=argv, customInstallOptions='--username=domain\\user --password=secret') – Branson

I don't understand I followed all the steps but I can't see how do I install my py file and how do I start it – Mitsue

what is the pipeTestService.py file, how do I install my file ("my_service.py" - which is in totally different directory)? – Mitsue

FWIW, I found it also necessary to copy the pythonXX.dll file to the appropriate c:\windows subfolder in order for the pythonservice.exe file to run, in my case python39.dll to c:\Windows\Syswow64, since, for the moment, I am running 32-bit py3. – Rankle

Answer 3

I read all the answers, but no solution can fix my problem.

After carefully researched David K. Hess's code, I made some change, and it finally works.

But my reputation doesn't enough, so I just post the code here.

# 1. Custom your Project's name and Virtual Environment folder's name
# 2. Import this before all third part models
# 3. If you still failed, check the link below:
# https://mcmap.net/q/356054/-using-pythonservice-exe-to-host-python-service-while-using-virtualenv
# 2019-05-29 by oraant, modified from David K. Hess's answer.

import os, sys, site

project_name = "PythonService"  # Change this for your own project !!!!!!!!!!!!!!
venv_folder_name = "venv"  # Change this for your own venv path !!!!!!!!!!!!!!

if sys.executable.lower().endswith("pythonservice.exe"):

    # Get root path for the project
    service_directory = os.path.abspath(os.path.dirname(__file__))
    project_directory = service_directory[:service_directory.find(project_name)+len(project_name)]

    # Get venv path for the project
    def file_path(x): return os.path.join(project_directory, x)
    venv_base = file_path(venv_folder_name)
    venv_scripts = os.path.join(venv_base, "Scripts")
    venv_packages = os.path.join(venv_base, 'Lib', 'site-packages')

    # Change current working directory from PythonService.exe location to something better.
    os.chdir(project_directory)
    sys.path.append(".")
    prev_sys_path = list(sys.path)

    # Manually activate a virtual environment inside an already initialized interpreter.
    os.environ['PATH'] = venv_scripts + os.pathsep + os.environ['PATH']

    site.addsitedir(venv_packages)
    sys.real_prefix = sys.prefix
    sys.prefix = venv_base

    # Move some sys path in front of others
    new_sys_path = []
    for item in list(sys.path):
        if item not in prev_sys_path:
            new_sys_path.append(item)
            sys.path.remove(item)
    sys.path[:0] = new_sys_path

How to use it? It's simple, just paste it into a new python file, and import it before any third part model like this:

import service_in_venv  # import at top
import win32serviceutil
import win32service
import win32event
import servicemanager
import time
import sys, os
........

And now you should fix your problem.

Orthochromatic

Answer 4

Not use "pythonservice.exe", register python.exe to services directly:

import win32serviceutil
import win32service
import servicemanager
import sys
import os
import os.path
import multiprocessing

# 

def main():
    import time
    time.sleep(600)  

class ProcessService(win32serviceutil.ServiceFramework):
    _svc_name_ = "SleepService"
    _svc_display_name_ = "Sleep Service"
    _svc_description_ = "Sleeps for 600"
    _exe_name_ = sys.executable # python.exe from venv
    _exe_args_ = '-u -E "' + os.path.abspath(__file__) + '"'

    proc = None

    def SvcStop(self):
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        if self.proc:
            self.proc.terminate()

    def SvcRun(self):
        self.proc = multiprocessing.Process(target=main)
        self.proc.start()        
        self.ReportServiceStatus(win32service.SERVICE_RUNNING)
        self.SvcDoRun()
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)

    def SvcDoRun(self):
        self.proc.join()

def start():
    if len(sys.argv)==1:
        import win32traceutil
        servicemanager.Initialize()
        servicemanager.PrepareToHostSingle(ProcessService)
        servicemanager.StartServiceCtrlDispatcher()
    elif '--fg' in sys.argv:
        main()
    else:
        win32serviceutil.HandleCommandLine(ProcessService)

if __name__ == '__main__':
    try:
        start()
    except (SystemExit, KeyboardInterrupt):
        raise
    except:
        import traceback
        traceback.print_exc()

Its make python 3.5+ virtualenv support to work by pointing right iterpreter with service install.

Abelmosk

Answer 5

For anyone reading in 2018, I didn't have any luck with either solution above (Win10, Python 3.6) - so this is what I did to get it working. The working directory is in site-packages/win32 on launch, so you need to change the working directory and fix the sys.path before you try and import any project code. This assumed venv sits in your project dir, otherwise you may just need to hard code some paths:

import sys
import os
if sys.executable.lower().endswith("pythonservice.exe"):
    for i in range(4): # goes up 4 directories to project folder
        os.chdir("..")        
    # insert site-packages 2nd in path (behind project folder)
    sys.path.insert(1, os.path.join("venv",'Lib','site-packages'))

[REST OF IMPORTS]
class TestService(win32serviceutil.ServiceFramework):
    [...]

Susannesusceptibility

Answer 6

I hit this problem recently with the embeddable interpreter and came up with these instructions that work for Python 3.11 on both the embeddable interpreter and a normal virtual environment.

This solution differs from the current most popular answer in that it doesn't rely on any shared DLL's in C:\windows\system32.

Lastly, a Python script demonstrating the required environment setup is provided.

Virtual Environment

PS D:\dev\python_winsvc> C:\Python\Python311\python.exe -m venv venv_311
PS D:\dev\python_winsvc> . .\venv_311\Scripts\activate

(venv_311) PS D:\dev\python_winsvc> pip install pywin32
Collecting pywin32
  Using cached pywin32-306-cp311-cp311-win_amd64.whl (9.2 MB)
Installing collected packages: pywin32
Successfully installed pywin32-306

(venv_311) PS D:\dev\python_winsvc> deactivate

Make .\venv_311\Scripts look like this:

python.exe              # already there
python3.dll             # copy/link from venv source interpreter
python311.dll           # copy/link from venv source interpreter

pythoncom311.dll        # copy/link from .\venv_311\Lib\site-packages\pywin32_system32
pywintypes311.dll       # copy/link from .\venv_311\Lib\site-packages\pywin32_system32

pythonservice.exe       # copy/link from .\venv_311\Lib\site-packages\win32
servicemanager.pyd      # copy/link from .\venv_311\Lib\site-packages\win32

...

Perform the service install without activating the venv. Otherwise the wrong pythonservice.exe is used as service executable and it has to be the one in .\venv_311\Scripts as it needs to be in the same folder as the required DLL's.

PS D:\dev\python_winsvc> .\venv_311\Scripts\python .\winsvc.py install
Installing service python-winsvc
Service installed

PS D:\dev\python_winsvc> .\venv_311\Scripts\python .\winsvc.py start
Starting service python-winsvc

Embedded Interpreter

Note that we stage the pip packages for the embeddable interpreter by using a regularly installed one as the embeddable interpreter lacks the pip module.

PS D:\dev\python_winsvc> C:\Python\Python311\python.exe -m pip install --target embed_311\lib\site-packages pywin32
Collecting pywin32
  Using cached pywin32-306-cp311-cp311-win_amd64.whl (9.2 MB)
Installing collected packages: pywin32
Successfully installed pywin32-306

Make .\embed_311 look like this:

python.exe              # already there
python3.dll             # already there
python311.dll           # already there

pythoncom311.dll        # copy/link from .\embed_311\Lib\site-packages\pywin32_system32
pywintypes311.dll       # copy/link from .\embed_311\Lib\site-packages\pywin32_system32

pythonservice.exe       # copy/link from .\embed_311\Lib\site-packages\win32
servicemanager.pyd      # copy/link from .\embed_311\Lib\site-packages\win32

...

PS D:\dev\python_winsvc> .\embed_311\python.exe .\winsvc.py install
Installing service python-winsvc
Service installed

PS D:\dev\python_winsvc> .\embed_311\python.exe .\winsvc.py start
Starting service python-winsvc

Example Service

# winsvc.py

import sys
import pathlib
PYTHON_PATH = pathlib.Path(sys.executable).parent

import site
site.addsitedir(PYTHON_PATH.joinpath("lib/site-packages"))  # Only required when using the embedded interpreter

from typing import *

import logging, logging.handlers
import threading
import time

import win32event
import win32evtlogutil
import win32service
import win32serviceutil
import servicemanager

def configure_logger(filename: str) -> logging.Logger:
    logger = logging.getLogger(__name__)
    logger.setLevel(logging.DEBUG)

    formatter = logging.Formatter("%(asctime)s %(levelname)5.5s: %(message)s")
    handlers = [
        logging.handlers.RotatingFileHandler(pathlib.Path(__file__).parent.joinpath(filename), maxBytes=1024*1024, backupCount=0), 
        logging.StreamHandler()
    ]
    for handler in handlers:
        handler.setFormatter(formatter)
        logger.addHandler(handler)
    return logger

logger = configure_logger("winsvc.log")


class ApplicationThread(threading.Thread):
    
    def __init__(self) -> None:
        super().__init__()
        self._exit = False

    def stop(self) -> None:
        self._exit = True

    def run(self) -> None:
        logger.debug("service is running")
        while not self._exit:
            time.sleep(1)


class Win32ServiceWrapper(win32serviceutil.ServiceFramework):
    _exe_name_ = str(PYTHON_PATH.joinpath("pythonservice.exe"))
    _svc_name_ = "python-winsvc"
    _svc_display_name_ = "Python WinSvc"

    def __init__(self, args: Iterable[str]) -> None:
        super().__init__(args)
        self._stop_event = win32event.CreateEvent(None, 0, 0, None)
        self._thread = ApplicationThread()

    def SvcStop(self) -> None:
        self.ReportServiceStatus(win32service.SERVICE_STOP_PENDING)
        win32event.SetEvent(self._stop_event)

    def SvcDoRun(self):
        win32evtlogutil.ReportEvent(self._svc_display_name_, servicemanager.PYS_SERVICE_STARTED, 0, servicemanager.EVENTLOG_INFORMATION_TYPE, (self._svc_name_, ''))
        self._thread.start()
        win32event.WaitForSingleObject(self._stop_event, win32event.INFINITE)
        self._thread.stop()
        self._thread.join()
        win32evtlogutil.ReportEvent(self._svc_display_name_, servicemanager.PYS_SERVICE_STOPPED, 0, servicemanager.EVENTLOG_INFORMATION_TYPE, (self._svc_name_, ''))


if __name__ == "__main__":
    win32serviceutil.HandleCommandLine(Win32ServiceWrapper)

Shira

At least for my install (of python 3.12) the python executable is not in the venv root but in it's Script folder. The python3*.dll are in neither. Should the rest also go there? – Mcardle

Answer 7

The key thing to remember here (for those new to services) is that running a script via Windows Services uses PythonService.exe, and running it directly uses python.exe.

Command #1:

call .\env\Scripts\python3.exe server_windows10_app\\main.py

Result:

sys.executable.lower(): c:\u...\env\scripts\python3.exe

Command #2:

call .\env\Scripts\python3.exe .\server.py --startup=delayed install
call .\env\Scripts\python3.exe .\server.py start

Result:

sys.executable.lower(): c:\...\env\lib\site-packages\win32\pythonservice.exe

Sibeal