Problems running python script by windows task scheduler that does pscp

Question

Not sure if anyone has run into this, but I'll take suggestions for troubleshooting and/or alternative methods.

I have a Windows 2008 server on which I am running several scheduled tasks. One of those tasks is a python script that uses pscp to log into a linux box, checks for new files and if there is anything new, copies them down to a local directory on the C: drive. I've put some logging into the script at key points as well and I'm using logging.basicConfig(level=DEBUG).

I built the command using a variable, command = 'pscp -pw xxxx name@ip:/ c:\local_dir' and then I use subprocess.call(command) to execute the command.

Now here's the weird part. If I run the script manually from the command line, it works fine. New files are downloaded and processed. However, if the Task Scheduler runs the script, no new files are downloaded. The script is running under the same user, but yet yields different results.

According to the log files created by the script and on the linux box, the script successfully logs into the linux box. However, no files are downloaded despite there being new files. Again, when I run it via the command line, files are downloaded.

Any ideas? suggestions, alternative methods?

Thanks.

Ml

Thanks for these answers. It turns out both of you are right. Task Scheduler is weird in two ways: First; it doesn't like mapped drives. It works fine with UNC references. Second; it likes a full path reference for everything. Files, commands, syntax within scripts, everything. In fact, using these two techniques is how I fixed my problem. Now my only problem is deciding which answer to click correct since both work. – Ml

Answer 1

I had the same issue when trying to open an MS Access database on a Linux VM. Running the script at the Windows 7 command prompt worked but running it in Task Scheduler didn't. With Task Scheduler it would find the database and verify it existed but wouldn't return the tables within it.

The solution was to have Task Scheduler run cmd as the Program/Script with the arguments /c python C:\path\to\script.py (under Add arguments (optional)).

I can't tell you why this works but it solved my problem.

Donalt

this doesn't work for me either. every thing I try, it does nothing when I press Run. >:( – Pyrogallol

How do I prevent annoying "taskeng.exe" window from popping up every time script is executed (my script runs every 15 mins)? – Illicit

Answer 2

You can use the windows Task Scheduler, but make sure the "optional" field "Start In" is filled in.

In the Task Scheduler app, add an action that specifies your python file to run "doSomeWork" and fill in the Start in (optional) input with the directory that contains the file.. So for example if you have a python file in:

C:\pythonProject\doSomeWork.py

You would enter:

Program/Script: doSomeWork.py

Start in (optional): C:\pythonProject 

Fr

I think this is the best answer. Thank you. – Dredi

@Dredi Thanks. This was what worked best for me so I decided to post – Fr

@ErkinDjindjiev, I tried the above method but the script is not running. But it works correctly when i run the same in command prompt – Impatient

Answer 3

I'm having a similar issue. In testing I found that any type of call with subprocess stops the python script when run in task scheduler but works fine when run on the command line.

import subprocess

print('Start')
test = subprocess.check_output(["dir"], shell=True)
print('First call finished')

When run on command line this outputs:

Start
First call finished

When run from task scheduler the output is:

Start

In order to get the output from task scheduler I run the python script from a batch file as follows:

python test.py >> log.txt

I run the script through the batch file both on command line and through task scheduler.

Stripper

Answer 4

Brad's answer is right. Subprocess needs the shell context to work and the task manager can launch python without that. Another way to do it is to make a batch file that is launched by the task scheduler that calls python c:\path\to\script.py etc. The only difference to this is that if you run into a script that has a call to os.getcwd() you will always get the root where the script is but you get something else when you make the call to cmd from task scheduler.

Nealy

Answer 5

Last edit - start

After experiments... If you put there full path to python program it works without highest privileges (as admin). Meaning task settings like this:

program: "C:\Program Files\Python37\python.exe"
arguments: "D:\folder\folder\python script.py"

I have no idea why, but it works even if script uses subprocess and multiple threads.

Last edit - end

What I did is I changed task settings: checked Run with highest privileges. And task started to work perfectly while running python [script path]. But keep in mind, that title contains "Administrator: " at the begining... always...

P.S. Thanks guys for pointing out that subprocess is a problem. It made me think of task settings. I had similar problem when one script is running from Windows Task Scheduler, and another one doesn't. Running cmd with python [script path] didn't work for me on Windows 8.1 Embedded x64. Not sure why. Probably because of necessity to have spaces in path and issue with quotes. Hope my answer helps someone. ;)

Hittel

You mean like this? – Hittel

I was using the batch file (.bat) approach and having issues with sub-processes being killed. This fixed the issue, thank you. – Burner

Answer 6

Create a batch file add your python script in your batch file and then schedule that batch file .it will work . Example : suppose your python script is in folder c:\abhishek\script\merun.py first you have to go to directory by cd command .so your batch file would be like :

cd c:\abhishek\script python merun.py

it work for me .

Mayence

Answer 7

Just leaving this for posterity: A similar issue I faced was resolved by using the UNC (\10.x.xx.xx\Folder\xxx)path everywhere in my .bat and .py scripts instead of the letter assigned to the drive (\K:\Folder\xxx).

Pitchman

Correct. The default user that executes the task scheduler action is not aware there are mapped drives as they are specific to the logged in user. You can add net use S: "\\server\path" in a script beforehand to keep the letter mappings consistent in your code. – Hermaphroditism

Answer 8

I had this issue before. I was able to run the task manually in Windows Task Scheduler, but not automatically. I remembered that there was a change in the time made by another user, maybe this change made the task scheduler to error out. I am not sure. Therefore, I created another task with a different name, for the same script, and the script worked automatically. Try to create a test task running the same script. Hopefully that works!

Winding

Answer 9

For Anaconda installation of python in windows below solution worked for me

1. create a batch file with below "C:\Users\username\Anaconda3\condabin\activate" && python "script.py" && deactivate
2. Setup task to run this batch file

Putt

Answer 10

I recently had a similar problem when running a Python script under the Windows Task Scheduler.

The problem is with the data being requested. If you request the cmdline and then try and iterate:

pc = psutil.process_iter(['pid', 'name', 'cmdline'])
for process_details in pc:

You will get OSError: [WinError 998] Invalid access to memory location

The solution is to get the cmdline once you know it is a process you have permissions for, like this:

pc = psutil.process_iter(['pid', 'name'])
    for process_details in pc:
        if process_details.info['name'] == target_executable:
            pid = process_details.info['pid']
            # We have what may be the correct process, so now get the process and its cmdline
            pid_process = psutil.Process(pid)
            cmdline = pid_process.cmdline()

Hope that helps :)

Zenobiazeolite