Re: My Jenkins builds fail checking out. Why?

["Zimoch Dirk \(PSI\) via Core-talk" <core-talk at aps.anl.gov>]

Hello Andrew and Torsten,

Thanks for your replies.

This is the Jenking log from the beginning:

19:02:09 Started by an SCM change
19:02:09 Building in workspace /jenkins/workspace/EPICS-3.15
19:02:09 +AFs-WS-CLEANUP+AF0- Deleting project workspace...
19:02:09 +AFs-WS-CLEANUP+AF0- Deferred wipeout is used...
19:02:09 +AFs-WS-CLEANUP+AF0- Done
19:02:09 Cloning the remote Git repository
19:02:09 Cloning repository https://git.launchpad.net/epics-base
19:02:09  +AD4- git init /jenkins/workspace/EPICS-3.15 +ACM- timeout+AD0-10
19:02:09 Fetching upstream changes from https://git.launchpad.net/epics-base
19:02:09  +AD4- git --version +ACM- timeout+AD0-10
19:02:09  +AD4- git fetch --tags --progress https://git.launchpad.net/epics-base +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
19:02:10  +AD4- git config remote.origin.url https://git.launchpad.net/epics-base +ACM- timeout+AD0-10
19:02:10  +AD4- git config --add remote.origin.fetch +-refs/heads/+ACo-:refs/remotes/origin/+ACo- +ACM- timeout+AD0-10
19:02:10  +AD4- git config remote.origin.url https://git.launchpad.net/epics-base +ACM- timeout+AD0-10
19:02:10 Fetching upstream changes from https://git.launchpad.net/epics-base
19:02:10  +AD4- git fetch --tags --progress https://git.launchpad.net/epics-base +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
19:02:10  +AD4- git rev-parse origin/3.15+AF4Aew-commit+AH0- +ACM- timeout+AD0-10
19:02:10  +AD4- git rev-parse 3.15+AF4Aew-commit+AH0- +ACM- timeout+AD0-10
19:02:10 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
19:02:10 +AFs-WARNINGS+AF0- Skipping publisher since build result is FAILURE
19:02:10 Sending e-mails to: dirk.zimoch+AEA-psi.ch core-talk+AEA-aps.anl.gov
19:02:10 Finished: FAILURE

So it does clean up the workspace.
I did the same manually to see what happens:

+ACQ- git init .
Initialized empty Git repository in /home/zimoch/testbuild/.git/
+ACQ- git --version
git version 1.8.3.1
+ACQ- git fetch --tags --progress https://git.launchpad.net/epics-base +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
remote: Counting objects: 122729, done.
remote: Compressing objects: 100+ACU- (32243/32243), done.
remote: Total 122729 (delta 92299), reused 118838 (delta 88605)
Receiving objects: 100+ACU- (122729/122729), 96.32 MiB +AHw- 33.05 MiB/s, done.
Resolving deltas: 100+ACU- (92299/92299), done.
From https://git.launchpad.net/epics-base
 +ACo- +AFs-new branch+AF0-      3.13       -+AD4- origin/3.13
 +ACo- +AFs-new branch+AF0-      3.14       -+AD4- origin/3.14
 +ACo- +AFs-new branch+AF0-      3.15       -+AD4- origin/3.15
 +ACo- +AFs-new branch+AF0-      3.16       -+AD4- origin/3.16
 +ACo- +AFs-new branch+AF0-      7.0        -+AD4- origin/7.0
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta0 -+AD4- R3.12.0-beta0
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta1 -+AD4- R3.12.0-beta1
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta10 -+AD4- R3.12.0-beta10
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta11 -+AD4- R3.12.0-beta11
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta12 -+AD4- R3.12.0-beta12
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta13 -+AD4- R3.12.0-beta13
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta2 -+AD4- R3.12.0-beta2
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta3 -+AD4- R3.12.0-beta3
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta4 -+AD4- R3.12.0-beta4
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta5 -+AD4- R3.12.0-beta5
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta6 -+AD4- R3.12.0-beta6
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta7 -+AD4- R3.12.0-beta7
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta8 -+AD4- R3.12.0-beta8
 +ACo- +AFs-new tag+AF0-         R3.12.0-beta9 -+AD4- R3.12.0-beta9
 +ACo- +AFs-new tag+AF0-         R3.12.1    -+AD4- R3.12.1
 +ACo- +AFs-new tag+AF0-         R3.12.1.1  -+AD4- R3.12.1.1
 +ACo- +AFs-new tag+AF0-         R3.12.1.2  -+AD4- R3.12.1.2
 +ACo- +AFs-new tag+AF0-         R3.12.1.3  -+AD4- R3.12.1.3
 +ACo- +AFs-new tag+AF0-         R3.12.1.4  -+AD4- R3.12.1.4
 +ACo- +AFs-new tag+AF0-         R3.12.2    -+AD4- R3.12.2
 +ACo- +AFs-new tag+AF0-         R3.12.2.1  -+AD4- R3.12.2.1
 +ACo- +AFs-new tag+AF0-         R3.12.2.2  -+AD4- R3.12.2.2
 +ACo- +AFs-new tag+AF0-         R3.13.0-alpha2 -+AD4- R3.13.0-alpha2
 +ACo- +AFs-new tag+AF0-         R3.13.0-alpha3 -+AD4- R3.13.0-alpha3
 +ACo- +AFs-new tag+AF0-         R3.13.0-alpha4 -+AD4- R3.13.0-alpha4
 +ACo- +AFs-new tag+AF0-         R3.13.0-alpha5 -+AD4- R3.13.0-alpha5
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta1 -+AD4- R3.13.0-beta1
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta10 -+AD4- R3.13.0-beta10
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta2 -+AD4- R3.13.0-beta2
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta3 -+AD4- R3.13.0-beta3
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta5 -+AD4- R3.13.0-beta5
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta8 -+AD4- R3.13.0-beta8
 +ACo- +AFs-new tag+AF0-         R3.13.0-beta9 -+AD4- R3.13.0-beta9
 +ACo- +AFs-new tag+AF0-         R3.13.1.1  -+AD4- R3.13.1.1
 +ACo- +AFs-new tag+AF0-         R3.14.0-alpha1 -+AD4- R3.14.0-alpha1
 +ACo- +AFs-new tag+AF0-         R3.14.0-alpha2 -+AD4- R3.14.0-alpha2
 +ACo- +AFs-new tag+AF0-         R3.14.0-beta1 -+AD4- R3.14.0-beta1
 +ACo- +AFs-new tag+AF0-         R3.14.0-beta2 -+AD4- R3.14.0-beta2
 +ACo- +AFs-new tag+AF0-         R3.14.1    -+AD4- R3.14.1
 +ACo- +AFs-new tag+AF0-         R3.14.10   -+AD4- R3.14.10
 +ACo- +AFs-new tag+AF0-         R3.14.10-pre1 -+AD4- R3.14.10-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.10-rc1 -+AD4- R3.14.10-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.10-rc2 -+AD4- R3.14.10-rc2
 +ACo- +AFs-new tag+AF0-         R3.14.11   -+AD4- R3.14.11
 +ACo- +AFs-new tag+AF0-         R3.14.11-pre1 -+AD4- R3.14.11-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.11-rc1 -+AD4- R3.14.11-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12   -+AD4- R3.14.12
 +ACo- +AFs-new tag+AF0-         R3.14.12-pre1 -+AD4- R3.14.12-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12-pre2 -+AD4- R3.14.12-pre2
 +ACo- +AFs-new tag+AF0-         R3.14.12-rc1 -+AD4- R3.14.12-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.1 -+AD4- R3.14.12.1
 +ACo- +AFs-new tag+AF0-         R3.14.12.1-rc1 -+AD4- R3.14.12.1-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.2 -+AD4- R3.14.12.2
 +ACo- +AFs-new tag+AF0-         R3.14.12.2-pre1 -+AD4- R3.14.12.2-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12.2-rc1 -+AD4- R3.14.12.2-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.3 -+AD4- R3.14.12.3
 +ACo- +AFs-new tag+AF0-         R3.14.12.3-pre1 -+AD4- R3.14.12.3-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12.3-rc1 -+AD4- R3.14.12.3-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.4 -+AD4- R3.14.12.4
 +ACo- +AFs-new tag+AF0-         R3.14.12.4-pre1 -+AD4- R3.14.12.4-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12.4-rc1 -+AD4- R3.14.12.4-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.5 -+AD4- R3.14.12.5
 +ACo- +AFs-new tag+AF0-         R3.14.12.5-pre1 -+AD4- R3.14.12.5-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12.5-rc1 -+AD4- R3.14.12.5-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.6 -+AD4- R3.14.12.6
 +ACo- +AFs-new tag+AF0-         R3.14.12.6-pre1 -+AD4- R3.14.12.6-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.12.6-rc1 -+AD4- R3.14.12.6-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.7 -+AD4- R3.14.12.7
 +ACo- +AFs-new tag+AF0-         R3.14.12.7-rc1 -+AD4- R3.14.12.7-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.12.8 -+AD4- R3.14.12.8
 +ACo- +AFs-new tag+AF0-         R3.14.12.8-rc1 -+AD4- R3.14.12.8-rc1
 +ACo- +AFs-new tag+AF0-         R3.14.2    -+AD4- R3.14.2
 +ACo- +AFs-new tag+AF0-         R3.14.3    -+AD4- R3.14.3
 +ACo- +AFs-new tag+AF0-         R3.14.4    -+AD4- R3.14.4
 +ACo- +AFs-new tag+AF0-         R3.14.5    -+AD4- R3.14.5
 +ACo- +AFs-new tag+AF0-         R3.14.6    -+AD4- R3.14.6
 +ACo- +AFs-new tag+AF0-         R3.14.7    -+AD4- R3.14.7
 +ACo- +AFs-new tag+AF0-         R3.14.8    -+AD4- R3.14.8
 +ACo- +AFs-new tag+AF0-         R3.14.8.1  -+AD4- R3.14.8.1
 +ACo- +AFs-new tag+AF0-         R3.14.8.2  -+AD4- R3.14.8.2
 +ACo- +AFs-new tag+AF0-         R3.14.9    -+AD4- R3.14.9
 +ACo- +AFs-new tag+AF0-         R3.14.9-pre1 -+AD4- R3.14.9-pre1
 +ACo- +AFs-new tag+AF0-         R3.14.9-pre2 -+AD4- R3.14.9-pre2
 +ACo- +AFs-new tag+AF0-         R3.14.9-rc1 -+AD4- R3.14.9-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.0.1  -+AD4- R3.15.0.1
 +ACo- +AFs-new tag+AF0-         R3.15.0.2  -+AD4- R3.15.0.2
 +ACo- +AFs-new tag+AF0-         R3.15.1    -+AD4- R3.15.1
 +ACo- +AFs-new tag+AF0-         R3.15.1-rc1 -+AD4- R3.15.1-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.2    -+AD4- R3.15.2
 +ACo- +AFs-new tag+AF0-         R3.15.2-pre1 -+AD4- R3.15.2-pre1
 +ACo- +AFs-new tag+AF0-         R3.15.2-rc1 -+AD4- R3.15.2-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.3    -+AD4- R3.15.3
 +ACo- +AFs-new tag+AF0-         R3.15.3-pre1 -+AD4- R3.15.3-pre1
 +ACo- +AFs-new tag+AF0-         R3.15.3-rc1 -+AD4- R3.15.3-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.4    -+AD4- R3.15.4
 +ACo- +AFs-new tag+AF0-         R3.15.4-pre1 -+AD4- R3.15.4-pre1
 +ACo- +AFs-new tag+AF0-         R3.15.4-rc1 -+AD4- R3.15.4-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.5    -+AD4- R3.15.5
 +ACo- +AFs-new tag+AF0-         R3.15.5-pre1 -+AD4- R3.15.5-pre1
 +ACo- +AFs-new tag+AF0-         R3.15.5-rc1 -+AD4- R3.15.5-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.6    -+AD4- R3.15.6
 +ACo- +AFs-new tag+AF0-         R3.15.6-rc1 -+AD4- R3.15.6-rc1
 +ACo- +AFs-new tag+AF0-         R3.15.7    -+AD4- R3.15.7
 +ACo- +AFs-new tag+AF0-         R3.16.0.1  -+AD4- R3.16.0.1
 +ACo- +AFs-new tag+AF0-         R3.16.1    -+AD4- R3.16.1
 +ACo- +AFs-new tag+AF0-         R3.16.1-rc1 -+AD4- R3.16.1-rc1
 +ACo- +AFs-new tag+AF0-         R3.16.2    -+AD4- R3.16.2
 +ACo- +AFs-new tag+AF0-         R3.16.2-rc1 -+AD4- R3.16.2-rc1
 +ACo- +AFs-new tag+AF0-         R7.0.1     -+AD4- R7.0.1
 +ACo- +AFs-new tag+AF0-         R7.0.1-pre1 -+AD4- R7.0.1-pre1
 +ACo- +AFs-new tag+AF0-         R7.0.1-rc1 -+AD4- R7.0.1-rc1
 +ACo- +AFs-new tag+AF0-         R7.0.1.1   -+AD4- R7.0.1.1
 +ACo- +AFs-new tag+AF0-         R7.0.2     -+AD4- R7.0.2
 +ACo- +AFs-new tag+AF0-         R7.0.2-rc1 -+AD4- R7.0.2-rc1
 +ACo- +AFs-new tag+AF0-         R7.0.2.1   -+AD4- R7.0.2.1
 +ACo- +AFs-new tag+AF0-         R7.0.2.2   -+AD4- R7.0.2.2
 +ACo- +AFs-new tag+AF0-         R7.0.3     -+AD4- R7.0.3
 +ACo- +AFs-new tag+AF0-         R7.0.3.1   -+AD4- R7.0.3.1
 +ACo- +AFs-new tag+AF0-         ca-4.13.1  -+AD4- ca-4.13.1
 +ACo- +AFs-new tag+AF0-         database-3.17.0 -+AD4- database-3.17.0
 +ACo- +AFs-new tag+AF0-         end-of-ca-master -+AD4- end-of-ca-master
 +ACo- +AFs-new tag+AF0-         end-of-core-master -+AD4- end-of-core-master
 +ACo- +AFs-new tag+AF0-         end-of-database-master -+AD4- end-of-database-master
 +ACo- +AFs-new tag+AF0-         end-of-libcom-master -+AD4- end-of-libcom-master
 +ACo- +AFs-new tag+AF0-         libcom-3.17.0 -+AD4- libcom-3.17.0
+ACQ- git config remote.origin.url https://git.launchpad.net/epics-base
+ACQ- git config --add remote.origin.fetch +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
+ACQ- git config remote.origin.url https://git.launchpad.net/epics-base
+ACQ- git fetch --tags --progress https://git.launchpad.net/epics-base +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
+ACQ- git rev-parse origin/3.15+AF4Aew-commit+AH0-
dbd6f7e8079bb158b9b30eaec6a60d0327752d6a
+ACQ- git rev-parse 3.15+AF4Aew-commit+AH0-
3.15+AF4Aew-commit+AH0-
fatal: ambiguous argument '3.15+AF4Aew-commit+AH0-': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git +ADw-command+AD4- +AFsAPA-revision+AD4-...+AF0- -- +AFsAPA-file+AD4-...+AF0-'

I have no idea what Jenkins is doing here (and why). I would simply do:
+ACQ- git checkout 3.15
Branch 3.15 set up to track remote branch 3.15 from origin.
Switched to a new branch '3.15'

However, when I do the previously failing line now, it works+ACE-
+ACQ- git rev-parse 3.15+AF4Aew-commit+AH0-
dbd6f7e8079bb158b9b30eaec6a60d0327752d6a

Also Jenkins is doing something different for 3.14.12 and 7.0:
19:18:21  +AD4- git rev-parse origin/3.14+AF4Aew-commit+AH0- +ACM- timeout+AD0-10
19:18:21 Checking out Revision 7dd1ea4cabade8c95cda5e677951e3b08f793487 (origin/3.14)
+AFs-...+AF0-

So instead of checking out the found revision, it tries another command to get the commit for 3.15, as if getting the
commit had failed. But it does not fail when I do it manually. Very strange. Unfortunately I don't see the (error)
output of the git commands Jenkins executes.

I have started the build manually now (normally it is started by the +ACI-Poll SCM+ACI- option on change only) and, lo and
behold, it works.

Dirk 


On Thu, 2020-02-06 at 10:52 -0600, Andrew Johnson wrote:
+AD4- Hi Dirk,
+AD4- 
+AD4- On 2/6/20 1:56 AM, Zimoch Dirk (PSI) via Core-talk wrote:
+AD4- +AD4- For 12 days now my Jenkins build fails for 3.15 checking out the sources:
+AD4- +AD4- 
+AD4- +AD4- 19:02:10 Fetching upstream changes from https://git.launchpad.net/epics-base
+AD4- +AD4- 19:02:10  +AD4- git fetch --tags --progress https://git.launchpad.net/epics-base +-refs/heads/+ACo-:refs/remotes/origin/+ACo-
+AD4- +AD4- 19:02:10  +AD4- git rev-parse origin/3.15+AF4Aew-commit+AH0- +ACM- timeout+AD0-10
+AD4- +AD4- 19:02:10  +AD4- git rev-parse 3.15+AF4Aew-commit+AH0- +ACM- timeout+AD0-10
+AD4- +AD4- 19:02:10 ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
+AD4- +AD4- 
+AD4- +AD4- Builds for 3.14 and 7.0 succeed. 3.16 fails as well.
+AD4- +AD4- 
+AD4- +AD4- Can somebody give me a hint what has changed and what I may have to change in my Jenkins configuration?
+AD4-  Nothing has changed as far as I know, but Launchpad does seem to be getting more unreliable. Have you tried wiping out the job's workspace to have Jenkins check out the git repo again from scratch? You can do that from the job's Workspace page.
+AD4- 
+AD4- +AD4- I have:
+AD4- +AD4- Repository URL https://git.launchpad.net/epics-base
+AD4- +AD4- Branch Specifier 3.15
+AD4- +AD4- 
+AD4- +AD4- For the other versions I have Branch Specifier 3.14, 3.16, and 7.0.
+AD4-  We plan to never commit anything to the 3.14 or 3.16 branches again. I have disabled my jobs for those but haven't deleted them.
+AD4- 
+AD4- +AEA-Torsten +IBQ- the git commands that Jenkins uses are usually generated internally by the Jenkins software, we only get to modify them by enabling and disabling features in the specific job configuration.
+AD4- 
+AD4- - Andrew
+AD4-