Login/Register
Git operations occasionally hang in Jenkins on Windows
Asked Answered
windowsgitjenkinsassemblagitblit
I

4

15

We are running continuous Jenkins builds of a Git project hosted at Assembla. Jenkins is running on Tomcat 6 under its own user and generally works fine.

However every once in a while (say once in every 10 builds), the checkout operation at the beginning of the build job simply hangs. At other times the Git tag operation at the end of the build also hangs. I believe this did not ever happen in command-line operation (on the same host with the same user).

When hung, the Windows process tree shows taskhost.exe ? tomcat6.exe ? git.exe ? ssh.exe

When externally killing the Git and ssh processes of a hung tag command, the following stacktrace is seen in the job console output - the error is strange since the directory mentioned already exists and has the private key installed.

hudson.plugins.git.GitException: Command "C:/Program Files (x86)/Git/bin/git.exe push <repository> <tag-name>" returned status code 1:
stdout:
stderr: Could not create directory 'c/Users/<user-name>/.ssh'.
    at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:779)
    at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:741)

Here is a full listing of the processes and handles (except the long list of Tomcat threads). The build task is currently hung on pull. Another strange phenomenon here is that there is a scheduled SCM poll every 5 minutes, but it hasn't run for a few days - it is probably also stuck somehow.

Process                PID      CPU      Private   Working     Description                                      Company Name
                                         Bytes     Set
-----------------------------------------------------------------------------------------------------------------------------------------
System Idle          Process 0  84.09        0 K       24 K
 Interrupts            n/a       0.28        0 K        0 K    Hardware Interrupts
 DPCs                  n/a       0.85        0 K        0 K    Deferred Procedure Calls
 System                   4                112 K      300 K
  smss.exe              240                620 K    1,196 K    Windows Session Manager                                 Microsoft Corporation
sppsvc.exe             2664              3,312 K    9,100 K    Microsoft Software Protection Platform Service          Microsoft Corporation
csrss.exe               344              3,516 K    5,120 K    Client Server Runtime Process                           Microsoft Corporation
 conhost.exe           1316              1,184 K    2,804 K    Console Window Host                                     Microsoft Corporation
 conhost.exe           3148              1,140 K    2,696 K    Console Window Host                                     Microsoft Corporation
wininit.exe             396              1,944 K    4,624 K    Windows Start-Up Application                            Microsoft Corporation
 services.exe           496              5,096 K   10,444 K    Services and Controller app                             Microsoft Corporation
  svchost.exe           616              4,776 K    9,940 K    Host Process for Windows Services                       Microsoft Corporation
   WmiPrvSE.exe        2468              2,692 K    6,052 K    WMI Provider Host                                       Microsoft Corporation
   dllhost.exe         2180      2.27    2,160 K    5,392 K    COM Surrogate                                           Microsoft Corporation
  svchost.exe           692      0.28    4,512 K    8,568 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           776              9,804 K   12,528 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           832             22,052 K   34,980 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           888              7,988 K   14,528 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           944              8,844 K   15,740 K    Host Process for Windows Services                       Microsoft Corporation
   dwm.exe              800              1,692 K    4,636 K    Desktop Window Manager                                  Microsoft Corporation
   dwm.exe             3908              1,800 K    4,748 K    Desktop Window Manager                                  Microsoft Corporation
  svchost.exe           984             13,036 K   17,004 K    Host Process for Windows Services                       Microsoft Corporation
  svchost.exe           284              8,536 K   11,152 K    Host Process for Windows Services                       Microsoft Corporation
  spoolsv.exe          1064              9,808 K   16,696 K    Spooler SubSystem App                                   Microsoft Corporation
  svchost.exe          1168              1,116 K    2,740 K    Host Process for Windows Services                       Microsoft Corporation
  Tomcat6.exe          1308      0.28  331,512 K  302,568 K    Commons Daemon Service Runner                           Apache Software Foundation
   git.exe             1812              3,308 K    4,880 K
    ssh.exe            2996              2,940 K    5,792 K
  vmtoolsd.exe         1352              8,260 K   12,892 K    VMware Tools Core Service                               VMware, Inc.
  VMUpgradeHelper.exe  1416              2,452 K    6,588 K    VMware virtual hardware upgrade helper application      VMware, Inc.
  svchost.exe          1880              3,796 K    9,224 K    Host Process for Windows Services                       Microsoft Corporation
   rdpclip.exe         3020              2,584 K    7,076 K    RDP Clip Monitor                                        Microsoft Corporation
   rdpclip.exe         4072              1,948 K    6,236 K    RDP Clip Monitor                                        Microsoft Corporation
  svchost.exe          1932              1,964 K    5,532 K    Host Process for Windows Services                       Microsoft Corporation
  dllhost.exe          1996              4,500 K   11,340 K    COM Surrogate                                           Microsoft Corporation
  msdtc.exe            1284              3,604 K    7,880 K    Microsoft Distributed Transaction Coordinator Service   Microsoft Corporation
  taskhost.exe         2492              3,076 K    6,252 K    Host Process for Windows Tasks                          Microsoft Corporation
  taskhost.exe         3548              2,896 K    6,088 K    Host Process for Windows Tasks                          Microsoft Corporation
 lsass.exe              504              8,516 K   16,548 K    Local Security Authority Process                        Microsoft Corporation
 lsm.exe                512              3,468 K    6,480 K    Local Session Manager Service                           Microsoft Corporation
csrss.exe               408              1,836 K    3,796 K    Client Server Runtime Process                           Microsoft Corporation
winlogon.exe            436              1,780 K    4,392 K    Windows Logon Application                               Microsoft Corporation
 LogonUI.exe            784              7,344 K   14,460 K    Windows Logon User Interface Host                       Microsoft Corporation
csrss.exe              2184              2,756 K    7,532 K    Client Server Runtime Process                           Microsoft Corporation
winlogon.exe           2952              1,960 K    5,192 K    Windows Logon Application                               Microsoft Corporation
explorer.exe           1836             23,536 K   45,060 K    Windows Explorer                                        Microsoft Corporation
 VMwareTray.exe        2168              2,824 K    6,400 K    VMware Tools tray application                           VMware, Inc.
 regedit.exe           2772              6,212 K    9,584 K    Registry Editor                                         Microsoft Corporation
 procexp64.exe         3648      11.93  21,904 K   37,056 K    Sysinternals Process Explorer                           Sysinternals - www.sysinternals.com
csrss.exe              3140              2,732 K    5,612 K    Client Server Runtime Process                           Microsoft Corporation
 conhost.exe           2500              1,312 K    3,452 K    Console Window Host                                     Microsoft Corporation
winlogon.exe           3172              1,900 K    4,980 K    Windows Logon Application                               Microsoft Corporation
explorer.exe            868             28,840 K   45,200 K    Windows Explorer                                        Microsoft Corporation
 VMwareTray.exe        3300              2,672 K    6,252 K    VMware Tools tray application                           VMware, Inc.
 rundll32.exe          3328              1,828 K    5,584 K    Windows host process (Rundll32)                         Microsoft Corporation
 cmd.exe               2832              2,240 K    2,588 K    Windows Command Processor                               Microsoft Corporation

Process: Tomcat6.exe Pid: 1308

Type        Name
Desktop      \Default
Directory    \KnownDlls
Directory    \BaseNamedObjects
Event        \BaseNamedObjects\TOMCAT6SIGNAL
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\commons-daemon.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0
File    C:\Users\<username>\AppData\Local\Temp\hsperfdata_<username>\1308
File    C:\Program Files\Java\jre7\lib\rt.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\bootstrap.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\tomcat-juli.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\catalina.2012-05-30.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\localhost.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\manager.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\host-manager.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stderr.2012-05-24.log
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\logs\tomcat6-stdout.2012-05-24.log
File    C:\Program Files\Java\jre7\lib\ext\dnsns.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\annotations-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-ant.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-ha.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina-tribes.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\catalina.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\ecj-3.7.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\el-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jasper-el.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jasper.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\jsp-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\servlet-api.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-coyote.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-dbcp.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-es.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-fr.jar
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\lib\tomcat-i18n-ja.jar
File    C:\Program Files\Java\jre7\lib\resources.jar
File    C:\Program Files\Java\jre7\lib\ext\localedata.jar
File    C:\Program Files\Java\jre7\lib\jsse.jar
File    C:\Program Files\Java\jre7\lib\jce.jar
File    C:\Program Files\Java\jre7\lib\ext\sunec.jar
File    C:\Program Files\Java\jre7\lib\ext\sunmscapi.jar
File    C:\Program Files\Java\jre7\lib\ext\sunjce_provider.jar
File    \Device\Afd
File    \Device\KsecDD
File    \Device\Nsi
File    C:\Windows\System32\en-US\KernelBase.dll.mui
File    E:\jenkins\workspace\<job1>\.git\objects\pack\pack-f36e1122944b1a18c4f6a8dd9d38915125dffa9e.pack
File    C:\Windows\Fonts\symbol.ttf
File    \Device\Afd
File    \Device\Afd
File    C:\Program Files\Java\jre7\lib\charsets.jar
File    C:\Program Files\Java\jre7\lib\charsets.jar
File    \Device\Afd
File    \Device\Afd
File    E:\jenkins\jobs\<job2>\builds\2012-05-30_17-45-06\log
File    \Device\Afd
File    \Device\Afd
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aether-api-1.13.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aether-connector-wagon-1.13.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aether-impl-1.13.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aether-spi-1.13.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aether-util-1.13.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\ant-1.8.3.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\ant-launcher-1.8.3.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\aopalliance-1.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\commons-cli-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\commons-httpclient-3.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\commons-io-1.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\commons-net-3.0.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\doxia-sink-api-1.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\guava-11.0.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\jackrabbit-jcr-commons-2.2.5.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\jackrabbit-webdav-2.2.5.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\javax.inject-1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\jsch-0.1.44-1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\jsr305-1.3.9.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\lib-jenkins-maven-artifact-manager-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\lib-jenkins-maven-embedder-3.9.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-aether-provider-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-agent-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-artifact-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-compat-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-core-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-embedder-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-interceptor-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-model-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-model-<username>-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-plugin-api-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-reporting-api-3.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-repository-metadata-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-settings-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven-settings-<username>-3.0.4.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven2.1-interceptor-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven3-agent-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\maven3-interceptor-1.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\nekohtml-1.9.13.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-cipher-1.7.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-classworlds-2.3.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-component-annotations-1.5.5.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-interactivity-api-1.0-alpha-6.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-interpolation-1.14.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-sec-dispatcher-1.3.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\plexus-utils-2.0.6.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\sisu-guava-0.9.9.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\sisu-guice-3.1.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\sisu-inject-bean-2.3.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\sisu-inject-plexus-2.3.0.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\slf4j-api-1.6.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-file-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-ftp-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-http-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-http-shared-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-provider-api-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-ssh-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-ssh-common-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-ssh-external-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\wagon-webdav-jackrabbit-2.2.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\xercesImpl-2.9.1.jar
File    E:\jenkins\plugins\maven-plugin\WEB-INF\lib\xml-apis-1.3.04.jar
File    E:\jenkins\plugins\active-directory\WEB-INF\lib\active-directory-1.0.jar
File    E:\jenkins\plugins\active-directory\WEB-INF\lib\ado20-1.0.jar
File    E:\jenkins\plugins\active-directory\WEB-INF\lib\com4j-20080107.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\ant-filesystem-tasks-0.0.2.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\avalon-framework-4.1.3.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\commons-codec-1.4.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\commons-logging-1.1.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\log4j-1.2.9.jar
File    E:\jenkins\plugins\artifactdeployer\WEB-INF\lib\logkit-1.0.1.jar
File    E:\jenkins\plugins\git\WEB-INF\lib\annotation-indexer-1.2.jar
File    E:\jenkins\plugins\git\WEB-INF\lib\bridge-method-annotation-1.4.jar
File    E:\jenkins\plugins\git\WEB-INF\lib\joda-time-1.5.1.jar
File    E:\jenkins\plugins\git\WEB-INF\lib\jsch-0.1.44-1.jar
File    E:\jenkins\plugins\git\WEB-INF\lib\org.eclipse.jgit-0.12.1.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\activation-1.1.1.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\axis-1.4.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\commons-codec-1.4.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\commons-discovery-0.4.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\commons-logging-1.0.4.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\jaxrpc-api-1.1.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\log4j-1.2.9.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\saaj-api-1.3.jar
File    E:\jenkins\plugins\jira\WEB-INF\lib\wsdl4j-1.6.2.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\guava-r05.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\junit-4.7.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-archiver-1.0-alpha-11.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-classworlds-1.2-alpha-6.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-component-api-1.0-alpha-15.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-container-default-1.0-alpha-15.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-io-1.0-alpha-3.jar
File    E:\jenkins\plugins\periodicbackup\WEB-INF\lib\plexus-utils-1.5.1.jar
File    E:\jenkins\plugins\subversion\WEB-INF\lib\svnkit-1.3.4-jenkins-4.jar
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    \Device\Afd
File    E:\jenkins\workspace\<job2>\.git\objects\pack\pack-a25789d9a15085fdc370bf63603670b6ef0aa516.pack
File    E:\jenkins\workspace\<job3>\.git\objects\pack\pack-3debf683446a7b50138fa83d20b8a176adc40d74.pack
File    E:\jenkins\workspace\<job4>\.git\objects\pack\pack-08e86702e225df334aa0281cdc34f6fe04a1a896.pack
File    E:\jenkins\workspace\<job5>\.git\objects\pack\pack-691d5f70f196faae3152545cc4e8a0668ee43182.pack
File    E:\jenkins\workspace\<job6>\.git\objects\pack\pack-a1f605f6d4ea56878d88c9d85c40884f7c9dc2e9.pack
File    E:\jenkins\workspace\<job7>\.git\objects\pack\pack-77c1adab9d7dcd56337a59bf7aa6ab1fc5423f0c.pack
File    \Device\NamedPipe\
File    C:\Windows\Fonts\arial.ttf
File    C:\Windows\Fonts\wingding.ttf
File    \Device\Afd
File    E:\jenkins\workspace\<job4>\.git\objects\pack\pack-29e3fac2d55ff8f5851726098c1310848ba61982.pack
File    E:\jenkins\jobs\<job2>\scm-polling.log
File    C:\Windows\winsxs\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.7600.16385_none_a44af8ec57f961cf
File    \Device\Afd
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\work\Catalina\localhost\gitblit-0.9.3\wicketFilter-filestore\9946\429\2051F8E78B66AEB8107264447FDEE93E\pm-null
File    \Device\Afd
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\jenkins\WEB-INF\lib\jenkins-core-1.461.jar
File    E:\jenkins\workspace\<job3>\.git\objects\pack\pack-df631556f14db215a3790cdb2dbdc2d8ec6a4dce.pack
File    E:\jenkins\workspace\<job3>\.git\objects\pack\pack-41dc6597dd98bff4284be2be0437bc96b14a660f.pack
File    C:\Program Files\Apache Software Foundation\Tomcat 6.0\webapps\jenkins\WEB-INF\lib\jenkins-core-1.461.jar
Key    HKLM\SYSTEM\ControlSet001\Control\Nls\Sorting\Versions
Key    HKLM
Key    HKLM\SYSTEM\ControlSet001\Control\SESSION MANAGER
Key    HKU\S-1-5-21-1089811676-525746212-2675575413-1203\Control Panel\International
Key    HKU\S-1-5-21-1089811676-525746212-2675575413-1203
Key    HKLM\SYSTEM\ControlSet001\services\WinSock2\Parameters\Protocol_Catalog9
Key    HKLM\SYSTEM\ControlSet001\services\WinSock2\Parameters\NameSpace_Catalog5
Key    HKU
Key    HKU\S-1-5-21-1089811676-525746212-2675575413-1203\Software\Microsoft\Windows\CurrentVersion\Explorer
Key    HKU\S-1-5-21-1089811676-525746212-2675575413-1203\Software\Microsoft\Windows NT\CurrentVersion
Key    HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AppCompatFlags
Key    HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options
Process    git.exe(1812)
Section    \BaseNamedObjects\hsperfdata_<username>_1308
Section    \BaseNamedObjects\windows_shell_global_counters
WindowStation    \Windows\WindowStations\Service-0x0-12330$
WindowStation    \Windows\WindowStations\Service-0x0-12330$

Here is the SSH search (from a different point in time than the list of processes above). I don't know what to make of it.

Enter image description here

Involuntary answered 24/5, 2012 at 7:22 Comment(15)
Is user-name the same account that the one Jenkins is running with? – Inauspicious
Yes it is the same user. – Involuntary
and the tomcat6.exe process is owned by the same user as well? – Inauspicious
Yes, all the same user. I suspect that if the problem had to do with just the user, this would not have worked at all. However it does work - and only occasionally hangs. – Involuntary
Some kind of resource conflict then... It would be interesting, before killing the all process tree, to look for the handles on ssh.exe (through a process explorer: technet.microsoft.com/fr-fr/sysinternals/bb896653.aspx) – Inauspicious
will try next time it happens – Involuntary
Here is a full process tree and handles dump where jenkins is currently hung on a GIT checkout. – Involuntary
See above in the question body – Involuntary
BTW I think it was hung due to the SCM poll being hung - I killed the GIT and SSH processes, and all of a sudden new processes were created and the job started to actually pull from SCM and run. – Involuntary
Can you try a GIT_TRACE=1 in your job environment variables in order to see if there is any useful trace when that git ssh call hangs? And did you check the handles referencing those executable? (to check if there was any conflict with other processes) – Inauspicious
Will add the trace. Do you expect to see anything in the console output as a result or where else should the trace outputs be seen? As for the handles - not sure what you mean to check. The listing above has everything on the machine (procexp ran as admin). – Involuntary
The console should be enough. Regarding the handle, the idea is to check is any other process has an opened handle on ssh.exe: use in the Process Explorer the CTRL+F and type ssh (see geekyninja.com/archives/… for illustration) – Inauspicious
@Vonc added result of ssh find to question body. The console doesn't show anything different than before (single line - Started by user <username> and that's it). Again a build is stuck, and the poll for changes log is stale and ends with a "Fetching upstream changes" line. BTW a different job on the same jenkins server runs ok while this job is stuck. – Involuntary
Quite strange. I am sorry to not offer any conclusive answer, but I wanted to be sure to cover the usual debugging options. – Inauspicious
Is this issue fixed?? – Studding
S
13

I solved it by:

  • setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!) and
  • by setting the environment variable %HOME% to $USERPROFILE
    (otherwise HOME defaults to $HOMEDRIVE$HOMEPATH, which was H:\ in my case, but typically $HOME is set to $USERPROFILE).
Signalman answered 2/10, 2012 at 8:22 Comment(2)
Excellent. Thank you for taking the time and updating this old question. +1 – Inauspicious
I thought this had fixed my issue, but after a week of health the problem returned today =( – Cleptomania
S
1

In my case downloading plink from the PuTTY site and setting the GIT_SSH environment variable to use it solved the problem.

Substantive answered 8/6, 2012 at 16:4 Comment(0)
P
1

This is discussed at:

https://issues.jenkins-ci.org/browse/JENKINS-5413

You might run the script below in the Script Console:

Jenkins.instance.getTrigger("SCMTrigger").getRunners().each()
{
  item ->
    println(item.getTarget().name)
    println(item.getDuration())
    println(item.getStartTime())
    long millis = Calendar.instance.time.time - item.getStartTime()

    if(millis > (1000 * 60 * 3)) // 1000 millis in a second * 60 seconds in a minute * 3 minutes
    {
      Thread.getAllStackTraces().keySet().each()
      { 
        tItem ->
          if (tItem.getName().contains("SCM polling") && tItem.getName().contains(item.getTarget().name))
          { 
            println "Interrupting thread " + tItem.getName(); 
            tItem.interrupt()
          }
       }
    }
}

to at least clear up the issue ...

Procure answered 6/11, 2012 at 15:28 Comment(1)
What would this do? I just get: Result: [hudson.triggers.SCMTrigger$Runner@ff8174] – Shielashield
S
0

I may have solved this. I originally set up my SSH keys by generating them with ssh-keygen on a Linux machine. This generally worked, but Git would hang occasionally from Jenkins.

Now I've tried using the PuTTY tools to set up key-based authentication to my Git servers, using these instructions. I then set the GIT_SSH environment variable to the path to plink.exe. It seems to be working now, but it was always an intermittent issue. Hopefully it won't return...

Shielashield answered 9/1, 2014 at 20:11 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.