Jenkins missing many jobs

[avivkeller]

Many jobs are currently returning 404s on https://ci.nodejs.org, including node-test-pull-request. It seems as though these configurations have vanished.

FYI @nodejs/build / @nodejs/build-infra / @nodejs/jenkins-admins

[targos]

I've solved it with a reboot. It's the second time it happens (first time was on Jan 31).
Jenkins logs contain this error:

ng.NullPointerException: Cannot invoke "hudson.model.Run.isBuilding()" because "r" is null
-31 23:41:47.638+0000 [id=76025]        SEVERE        jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading item benchmark-node-micro-benchmarks
 at java.base/java.lang.Thread.run(Thread.java:1583)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
 at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
 at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
 at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
 at jenkins.model.Jenkins$5.runTask(Jenkins.java:1152)
 at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:304)
 at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:175)
 at jenkins.model.Jenkins$14.run(Jenkins.java:3499)
 at hudson.model.Items.load(Items.java:409)
 at hudson.model.Project.onLoad(Project.java:99)
 at hudson.model.AbstractProject.onLoad(AbstractProject.java:309)
 at jenkins.model.lazy.LazyBuildMixIn.onLoad(LazyBuildMixIn.java:132)
ng.NullPointerException: Cannot invoke "hudson.model.Run.isBuilding()" because "r" is null

[sxa]

Just saw this - thanks @targos - I took a look at it seems ok now after the restart. That is a little bit concerning though if it's now happened a second time. I wonder if we could get away with a "reload configuration from disk" instead of a full restart if we see it again. Curious as to what "r" is though 🤔

[sxa]

Related: #4239 from earlier in the month

[aduh95]

There seems to be some machine that are unable to clone the repo:

ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from git@github.com:nodejs/node.git
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:998)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
	at hudson.scm.SCM.checkout(SCM.java:540)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1250)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522)
	at hudson.model.Run.execute(Run.java:1842)
	at PluginClassLoader for matrix-project//hudson.matrix.MatrixBuild.run(MatrixBuild.java:331)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:456)
Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout: 
stderr: fatal: Unable to create '/home/iojs/build/workspace/node/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.

	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2848)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2773)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2768)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:2055)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:711)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:1098)
	at jdk.internal.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:986)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:961)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:907)
	at hudson.remoting.UserRequest.perform(UserRequest.java:225)
	at hudson.remoting.UserRequest.perform(UserRequest.java:50)
	at hudson.remoting.Request$2.run(Request.java:391)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:81)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:310)
	at java.base/java.lang.Thread.run(Thread.java:840)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 67.158.54.159/67.158.54.159:56068
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1916)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:384)
		at hudson.remoting.Channel.call(Channel.java:1108)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:306)
		at PluginClassLoader for git-client/jdk.proxy107/jdk.proxy107.$Proxy192.clean(Unknown Source)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:535)
		at PluginClassLoader for git//hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:46)
		at PluginClassLoader for git//hudson.plugins.git.extensions.GitSCMExtension.decorateFetchCommand(GitSCMExtension.java:296)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:994)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
		at hudson.scm.SCM.checkout(SCM.java:540)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1250)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522)
		at hudson.model.Run.execute(Run.java:1842)
		at PluginClassLoader for matrix-project//hudson.matrix.MatrixBuild.run(MatrixBuild.java:331)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:456)

E.g. https://ci.nodejs.org/job/node-test-commit-linux/68529/console

[sxa]

Hmmm so that lock file is not currently present on the mnx workspace machine, although the red status looks to be an out of space condition on the softlayer debian machine ... I've taken it offline for now Re-enabled after [log rotation](#4024 (comment) although it's still quite full).

Related issues:

• debian12-x64 cannot run CITGM (No space left on device) #4221
• No space left on test-softlayer-debian12-x64-1 #4024

E.g. https://ci.nodejs.org/job/node-test-commit-linux/68529/console

So far subsequent invocations of the job appear to be ok 🤞🏻

[sxa]

I think this has now been resolved so I'm going to close this for now.

[aduh95]

Reopening as it's showing up again

[sxa]

Resolved. Instead of triggering a full restart I used the jenkins "reload configuration from disk" which got it back very quickly. Nothing obvious wrong on the system but I've taken a copy of some of the logs from the time it occurred to take a closer look

[sxa]

There are multiple instances of Failed loading item <jobname> in the logs from shortly after the backup/cleanup process triggers a jenkins reload. In the one from the weekend this definitely includes node-test-pull-request as one of the ones it couldn't reload.