Skip to content

"concurrent map read and map write" crash in kubernetes ContainerManager.active #9081

Open
#10049
Open
"concurrent map read and map write" crash in kubernetes ContainerManager.active#9081
#10049
Labels
kind/bugSomething isn't workingpriority/p2May take a couple of releases
@mrwonko

Description

@mrwonko
mrwonko
opened on Sep 8, 2023

Expected behavior

No crash

Actual behavior

fatal error: concurrent map read and map write

Information

  • Skaffold version: v2.6.3
  • Operating system: macOS Ventura 13.5.1
  • Installed via: Homebrew
  • Contents of skaffold.yaml: I don't know, I think our Makefile generates them on-the-fly and I don't have the time to find out more

Here's the goroutine dump at the time of crash:

goroutine 609 [running]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).checkPod(0xc000cf6940, {0xc002137100, 0x8}, 0xc001036d80)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:125 +0x631
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:90 +0x2c7
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

goroutine 1 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/runner.(*SkaffoldListener).WatchForChanges(0xc000cbf280, {0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90}, 0xc0011e8810)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/runner/listen.go:75 +0x5a5
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/runner.(*SkaffoldRunner).Dev(0xc000572a80, {0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90}, {0xc000e70040, 0x1, 0x1})
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/runner/dev.go:433 +0x35fe
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.runDev.func5({0x1048efbb0, 0xc000572a80}, {0xc0005364d0, 0x1, 0x1})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/dev.go:71 +0x30f
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.withRunner({0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90}, 0xc00052d350)
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/runner.go:54 +0x278
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.runDev({0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/dev.go:66 +0x31a
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.runDebug({0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/debug.go:56 +0x72
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.NewCmdDebug.func1({0x1048e0f70, 0xc000d91b00}, {0x1048bc3c0, 0xc000cc1a90})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/debug.go:47 +0x4e
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.(*builder).NoArgs.func1(0x0, {0x0, 0x0, 0x0})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/commands.go:138 +0x9a
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app/cmd.applyPostRunHooks.func1(0xc000005800, {0xc000e82000, 0x0, 0xb})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/cmd/commands.go:185 +0x6e
github.com/spf13/cobra.(*Command).execute(0xc000005800, {0xc000d03ef0, 0xb, 0xb})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/spf13/cobra/command.go:940 +0xd02
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004600)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/spf13/cobra/command.go:1068 +0x991
github.com/spf13/cobra.(*Command).ExecuteContextC(0xc000004600, {0x1048e0f70, 0xc000d91b00})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/spf13/cobra/command.go:1001 +0x8e
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app.Run({0x1048bb1a0, 0xc000130008}, {0x1048bb1a0, 0xc000130010})
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/skaffold.go:51 +0x656
main.main()
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/skaffold.go:49 +0x2fa

goroutine 19 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc000155d00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/go.opencensus.io/stats/view/worker.go:292 +0x125
created by go.opencensus.io/stats/view.init.0
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/go.opencensus.io/stats/view/worker.go:34 +0xb7

goroutine 23 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x105e1c3c0)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/golang/glog/glog.go:882 +0x5b
created by github.com/golang/glog.init.0
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/golang/glog/glog.go:410 +0x1a9

goroutine 11 [chan receive]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2.newHandler.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/event/v2/event.go:61 +0x4b
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2.newHandler
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/event/v2/event.go:59 +0x1d8

goroutine 15 [select, 118 minutes]:
os/signal.NotifyContext.func1()
	/usr/local/Cellar/go/1.20.7/libexec/src/os/signal/signal.go:288 +0xa8
created by os/signal.NotifyContext
	/usr/local/Cellar/go/1.20.7/libexec/src/os/signal/signal.go:287 +0x2d9

goroutine 41 [runnable]:
sync.(*Mutex).Unlock(0xc0000dba18)
	/usr/local/Cellar/go/1.20.7/libexec/src/sync/mutex.go:212 +0x58
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event.(*eventHandler).handleExec(0xc0000db9e0, {0xc0028d7900, 0xc0028d7940})
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/event/event.go:722 +0x2f33
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event.newHandler.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/event/event.go:65 +0x9d
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event.newHandler
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/event/event.go:59 +0x191

goroutine 42 [syscall, 118 minutes]:
os/signal.signal_recv()
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sigqueue.go:149 +0x28
os/signal.loop()
	/usr/local/Cellar/go/1.20.7/libexec/src/os/signal/signal_unix.go:23 +0x1d
created by os/signal.Notify.func1.1
	/usr/local/Cellar/go/1.20.7/libexec/src/os/signal/signal.go:151 +0x2e

goroutine 29 [chan receive, 118 minutes]:
github.com/rjeczalik/notify.(*recursiveTree).dispatch(0xc0003e4910)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/rjeczalik/notify/tree_recursive.go:125 +0x51
created by github.com/rjeczalik/notify.newRecursiveTree
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/rjeczalik/notify/tree_recursive.go:119 +0x249

goroutine 31 [chan receive, 118 minutes]:
github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app.catchStackdumpRequests.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/stackdump.go:38 +0x3f
created by github.com/GoogleContainerTools/skaffold/v2/cmd/skaffold/app.catchStackdumpRequests
	/private/tmp/skaffold-20230805-5278-w0t7w2/cmd/skaffold/app/stackdump.go:36 +0xd6

goroutine 16 [select, 118 minutes]:
io.(*pipe).read(0xc0000dbce0, {0xc000e86000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/pipe.go:57 +0x18d
io.(*PipeReader).Read(0xc000b901b0, {0xc000e86000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/pipe.go:136 +0x7c
bufio.(*Scanner).Scan(0xc0008bf600)
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/scan.go:214 +0xa9c
github.com/sirupsen/logrus.(*Entry).writerScanner(0xc000cdca10, 0xc000b901b0, 0xc0003ba5a0)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/sirupsen/logrus/writer.go:59 +0x5b
created by github.com/sirupsen/logrus.(*Entry).WriterLevel
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/github.com/sirupsen/logrus/writer.go:51 +0x4e5

goroutine 255 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1358 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0017f67c8, 0x1dd)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sema.go:527 +0x14a
sync.(*Cond).Wait(0xc0017f67b8)
	/usr/local/Cellar/go/1.20.7/libexec/src/sync/cond.go:70 +0x78
golang.org/x/net/http2.(*pipe).Read(0xc0017f67b0, {0xc001d80001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/pipe.go:76 +0x427
golang.org/x/net/http2.transportResponseBody.Read({0xc0017f6780}, {0xc001d80001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2508 +0x126
encoding/json.(*Decoder).refill(0xc0005a83c0)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:165 +0x432
encoding/json.(*Decoder).readValue(0xc0005a83c0)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:140 +0x465
encoding/json.(*Decoder).Decode(0xc0005a83c0, {0x103817240, 0xc001977a40})
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:63 +0xff
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0012edb90, {0xc001eb8000, 0x8000, 0xa000})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x505
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc001107b80, 0x0, {0x1048ca608, 0xc0028c3f40})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x102
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0008447e0)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0xd6
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000af4a00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0x1a3
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x1be

goroutine 1309 [syscall]:
syscall.syscall6(0xc002003870?, 0xc0020037d8?, 0xc0017489c0?, 0xc002003a78?, 0xc0017489c0?, 0x1008c31d4?, 0xc002003b38?)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sys_darwin.go:45 +0x99
syscall.wait4(0x8b6, 0xc0020038a4, 0x0, 0xc002a60bd0)
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/zsyscall_darwin_amd64.go:43 +0xaf
syscall.Wait4(0x8b6, 0xc002003924, 0x0, 0xc002a60bd0)
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/syscall_bsd.go:144 +0x77
os.(*Process).wait(0xc000fc4ab0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec_unix.go:43 +0x1f5
os.(*Process).Wait(0xc000fc4ab0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec.go:132 +0x45
os/exec.(*Cmd).Wait(0xc0031dd4a0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:890 +0xd1
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).forward(0xc0017a4ec0, {0x1048dfb18, 0xc000e3b050}, 0xc0028165b0, 0xc0004a0f00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:155 +0xbf6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).Forward
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:83 +0x1b6

goroutine 1312 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).monitorLogs(0x0, {0x1048dfa70, 0xc000628aa0}, {0x1048adb00, 0xc000e3b5f0}, 0xc000cea6b8, 0xc0028165b0, 0xc0004a0f00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:223 +0x1a6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).forward
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:154 +0xbdc

goroutine 389 [chan receive, 116 minutes]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/trigger.(*manualTrigger).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/trigger/triggers.go:143 +0x4e
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/trigger.(*manualTrigger).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/trigger/triggers.go:142 +0xe7

goroutine 1310 [IO wait]:
internal/poll.runtime_pollWait(0x105cb3cb8?, 0x72)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/netpoll.go:306 +0x45
internal/poll.(*pollDesc).wait(0xc0004a1098, 0x72, 0x1)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x88
internal/poll.(*pollDesc).waitRead(0xc0004a1098, 0x1)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:89 +0x37
internal/poll.(*FD).Read(0xc0004a1080, {0xc002154c4e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:167 +0x42f
os.(*File).read(0xc000cea7a0, {0xc002154c4e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file_posix.go:31 +0x7e
os.(*File).Read(0xc000cea7a0, {0xc002154c4e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file.go:118 +0xdd
bytes.(*Buffer).ReadFrom(0xc000e3b5f0, {0x1048bb140, 0xc000cea7a0})
	/usr/local/Cellar/go/1.20.7/libexec/src/bytes/buffer.go:202 +0x142
io.copyBuffer({0x1048adb40, 0xc000e3b5f0}, {0x1048bb140, 0xc000cea7a0}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:413 +0x26b
io.Copy({0x1048adb40, 0xc000e3b5f0}, {0x1048bb140, 0xc000cea7a0})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:386 +0x88
os/exec.(*Cmd).writerDescriptor.func1()
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:560 +0x68
os/exec.(*Cmd).Start.func2(0xc002658f60)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:717 +0x39
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:716 +0x1206

goroutine 1206 [select, 30 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00182d800, 0xc000e3d300)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1438 +0xc5b
golang.org/x/net/http2.(*clientStream).doRequest(0xc00182d800, 0xc000e3d300)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1300 +0x25
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1229 +0x4d9

goroutine 390 [syscall, 2 minutes]:
syscall.syscall(0xc001700ea0?, 0xc0000c3a58?, 0x1048b0480?, 0xc00011a100?)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sys_darwin.go:23 +0x7f
syscall.read(0x0, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/zsyscall_darwin_amd64.go:1209 +0xd7
syscall.Read(0x0, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/syscall_unix.go:178 +0x6c
internal/poll.ignoringEINTRIO(0x10472cbb0, 0x0, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:794 +0x98
internal/poll.(*FD).Read(0xc000132000, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:163 +0x310
os.(*File).read(0xc000130000, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file_posix.go:31 +0x7e
os.(*File).Read(0xc000130000, {0xc001658000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file.go:118 +0xdd
bufio.(*Reader).fill(0xc000ba2300)
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:106 +0x270
bufio.(*Reader).ReadRune(0xc000ba2300)
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:300 +0x191
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/trigger.(*manualTrigger).Start.func2()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/trigger/triggers.go:150 +0x7f
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/trigger.(*manualTrigger).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/trigger/triggers.go:148 +0x209

goroutine 1359 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start.func5()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:104 +0x1e6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:99 +0x945

goroutine 1357 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0017f6780, 0xc000e12b00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1438 +0xc5b
golang.org/x/net/http2.(*clientStream).doRequest(0xc0017f6780, 0xc000e12b00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1300 +0x25
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1229 +0x4d9

goroutine 1308 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).monitorLogs(0x333238322d70756e, {0x1048dfa70, 0xc0003e4460}, {0x1048adb00, 0xc002548930}, 0xc001108e98, 0xc0027a40d0, 0xc0034b01e0)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:223 +0x1a6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).forward
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:154 +0xbdc

goroutine 283 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:81 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

goroutine 287 [select, 96 minutes]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/logger.(*LogAggregator).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/logger/log.go:135 +0x215
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/logger.(*LogAggregator).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/logger/log.go:129 +0x2bf

goroutine 1098 [select, 1 minutes]:
net/http.(*persistConn).writeLoop(0xc0008b7b00)
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/transport.go:2410 +0x16e
created by net/http.(*Transport).dialConn
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/transport.go:1766 +0x2ba7

goroutine 1361 [select]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0017f6c00, 0xc000e13400)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1438 +0xc5b
golang.org/x/net/http2.(*clientStream).doRequest(0xc0017f6c00, 0xc000e13400)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1300 +0x25
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1229 +0x4d9

goroutine 1311 [select]:
os/exec.(*Cmd).watchCtx(0xc0031dd4a0, 0xc001d20e40)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:757 +0xf4
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:743 +0x1074

goroutine 1165 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start.func5()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:104 +0x1e6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:99 +0x945

goroutine 667 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1362 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0017f6c48, 0x1ea)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sema.go:527 +0x14a
sync.(*Cond).Wait(0xc0017f6c38)
	/usr/local/Cellar/go/1.20.7/libexec/src/sync/cond.go:70 +0x78
golang.org/x/net/http2.(*pipe).Read(0xc0017f6c30, {0xc002118001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/pipe.go:76 +0x427
golang.org/x/net/http2.transportResponseBody.Read({0xc0017f6c00}, {0xc002118001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2508 +0x126
encoding/json.(*Decoder).refill(0xc001410b40)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:165 +0x432
encoding/json.(*Decoder).readValue(0xc001410b40)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:140 +0x465
encoding/json.(*Decoder).Decode(0xc001410b40, {0x103817240, 0xc000534438})
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:63 +0xff
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc000ce7710, {0xc00212a000, 0x8000, 0xa000})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x505
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000628f50, 0x0, {0x1048ca608, 0xc0000d9340})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x102
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc001cb0a20)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0xd6
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0018f9300)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0x1a3
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x1be

goroutine 1097 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x105cb3cb8?, 0x72)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/netpoll.go:306 +0x45
internal/poll.(*pollDesc).wait(0xc000bcea18, 0x72, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x88
internal/poll.(*pollDesc).waitRead(0xc000bcea18, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:89 +0x37
internal/poll.(*FD).Read(0xc000bcea00, {0xc002160000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:167 +0x42f
net.(*netFD).Read(0xc000bcea00, {0xc002160000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/fd_posix.go:55 +0x79
net.(*conn).Read(0xc0005976d0, {0xc002160000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/net.go:183 +0xaa
net/http.(*persistConn).Read(0xc0008b7b00, {0xc002160000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/transport.go:1943 +0x1df
bufio.(*Reader).fill(0xc002893320)
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:106 +0x270
bufio.(*Reader).Peek(0xc002893320, 0x1)
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:144 +0x178
net/http.(*persistConn).readLoop(0xc0008b7b00)
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/transport.go:2107 +0x27e
created by net/http.(*Transport).dialConn
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/transport.go:1765 +0x2b3b

goroutine 1164 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00182d848, 0x67a)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sema.go:527 +0x14a
sync.(*Cond).Wait(0xc00182d838)
	/usr/local/Cellar/go/1.20.7/libexec/src/sync/cond.go:70 +0x78
golang.org/x/net/http2.(*pipe).Read(0xc00182d830, {0xc0023c8001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/pipe.go:76 +0x427
golang.org/x/net/http2.transportResponseBody.Read({0xc00182d800}, {0xc0023c8001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2508 +0x126
encoding/json.(*Decoder).refill(0xc0005a9b80)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:165 +0x432
encoding/json.(*Decoder).readValue(0xc0005a9b80)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:140 +0x465
encoding/json.(*Decoder).Decode(0xc0005a9b80, {0x103817240, 0xc000614f90})
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:63 +0xff
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0011990e0, {0xc002456000, 0x8000, 0xa000})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x505
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000228960, 0x0, {0x1048ca608, 0xc0028d7880})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x102
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc001f2dd20)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0xd6
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc0024c8b00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0x1a3
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x1be

goroutine 605 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1205 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1204 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start.func5()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:104 +0x1e6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:99 +0x945

goroutine 1203 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00182d3c8, 0x68f)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sema.go:527 +0x14a
sync.(*Cond).Wait(0xc00182d3b8)
	/usr/local/Cellar/go/1.20.7/libexec/src/sync/cond.go:70 +0x78
golang.org/x/net/http2.(*pipe).Read(0xc00182d3b0, {0xc001d3e001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/pipe.go:76 +0x427
golang.org/x/net/http2.transportResponseBody.Read({0xc00182d380}, {0xc001d3e001, 0x7dff, 0x7dff})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2508 +0x126
encoding/json.(*Decoder).refill(0xc00015fcc0)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:165 +0x432
encoding/json.(*Decoder).readValue(0xc00015fcc0)
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:140 +0x465
encoding/json.(*Decoder).Decode(0xc00015fcc0, {0x103817240, 0xc0006145a0})
	/usr/local/Cellar/go/1.20.7/libexec/src/encoding/json/stream.go:63 +0xff
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc001e97380, {0xc001d46000, 0x8000, 0xa000})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x505
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc000236640, 0x0, {0x1048ca608, 0xc0028d7400})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0x102
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc001365640)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0xd6
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc000cbf440)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0x1a3
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x1be

goroutine 1166 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:81 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

goroutine 1030 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:81 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

goroutine 1307 [select]:
os/exec.(*Cmd).watchCtx(0xc000bd3e40, 0xc001780ae0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:757 +0xf4
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:743 +0x1074

goroutine 1202 [select, 30 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc00182d380, 0xc000e3cb00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1438 +0xc5b
golang.org/x/net/http2.(*clientStream).doRequest(0xc00182d380, 0xc000e3cb00)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1300 +0x25
created by golang.org/x/net/http2.(*ClientConn).RoundTrip
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:1229 +0x4d9

goroutine 801 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:81 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

goroutine 1235 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x105cb3cb8?, 0x72)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/netpoll.go:306 +0x45
internal/poll.(*pollDesc).wait(0xc002822f98, 0x72, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x88
internal/poll.(*pollDesc).waitRead(0xc002822f98, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:89 +0x37
internal/poll.(*FD).Read(0xc002822f80, {0xc0013fa600, 0x1300, 0x1300})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:167 +0x42f
net.(*netFD).Read(0xc002822f80, {0xc0013fa600, 0x1300, 0x1300})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/fd_posix.go:55 +0x79
net.(*conn).Read(0xc0012a2348, {0xc0013fa600, 0x1300, 0x1300})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/net.go:183 +0xaa
crypto/tls.(*atLeastReader).Read(0xc000e7f848, {0xc0013fa600, 0x1300, 0x1300})
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:788 +0xd5
bytes.(*Buffer).ReadFrom(0xc000634290, {0x1048b0160, 0xc000e7f848})
	/usr/local/Cellar/go/1.20.7/libexec/src/bytes/buffer.go:202 +0x142
crypto/tls.(*Conn).readFromUntil(0xc000634000, {0x106f842c0, 0xc0012a2348}, 0x5)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:810 +0x170
crypto/tls.(*Conn).readRecordOrCCS(0xc000634000, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:617 +0x1e5
crypto/tls.(*Conn).readRecord(0xc000634000)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:583 +0x2b
crypto/tls.(*Conn).Read(0xc000634000, {0xc001e26000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:1316 +0x207
bufio.(*Reader).Read(0xc001c757a0, {0xc0027ba740, 0x9, 0x9})
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:237 +0x37c
io.ReadAtLeast({0x1048adac0, 0xc001c757a0}, {0xc0027ba740, 0x9, 0x9}, 0x9)
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:332 +0x177
io.ReadFull({0x1048adac0, 0xc001c757a0}, {0xc0027ba740, 0x9, 0x9})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:351 +0xa5
net/http.http2readFrameHeader({0xc0027ba740, 0x9, 0x9}, {0x1048adac0, 0xc001c757a0})
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/h2_bundle.go:1567 +0xd6
net/http.(*http2Framer).ReadFrame(0xc0027ba700)
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/h2_bundle.go:1831 +0x125
net/http.(*http2clientConnReadLoop).run(0xc001dd1f80)
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/h2_bundle.go:9187 +0x1a8
net/http.(*http2ClientConn).readLoop(0xc00066ec00)
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/h2_bundle.go:9082 +0xdc
created by net/http.(*http2Transport).newClientConn
	/usr/local/Cellar/go/1.20.7/libexec/src/net/http/h2_bundle.go:7779 +0xf35

goroutine 1026 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1061 [runnable]:
crypto/tls.(*atLeastReader).Read(0xc000614618, {0xc0005d7c02, 0x83fe, 0x83fe})
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:784 +0x27a
bytes.(*Buffer).ReadFrom(0xc00199d410, {0x1048b0160, 0xc000614618})
	/usr/local/Cellar/go/1.20.7/libexec/src/bytes/buffer.go:202 +0x142
crypto/tls.(*Conn).readFromUntil(0xc00199d180, {0x12fb86010, 0xc0011e7728}, 0x4016)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:810 +0x170
crypto/tls.(*Conn).readRecordOrCCS(0xc00199d180, 0x0)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:662 +0xb9e
crypto/tls.(*Conn).readRecord(0xc00199d180)
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:583 +0x2b
crypto/tls.(*Conn).Read(0xc00199d180, {0xc00241a000, 0x1000, 0x1000})
	/usr/local/Cellar/go/1.20.7/libexec/src/crypto/tls/conn.go:1316 +0x207
bufio.(*Reader).Read(0xc002893bc0, {0xc001d98660, 0x9, 0x9})
	/usr/local/Cellar/go/1.20.7/libexec/src/bufio/bufio.go:237 +0x37c
io.ReadAtLeast({0x1048adac0, 0xc002893bc0}, {0xc001d98660, 0x9, 0x9}, 0x9)
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:332 +0x177
io.ReadFull({0x1048adac0, 0xc002893bc0}, {0xc001d98660, 0x9, 0x9})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:351 +0xa5
golang.org/x/net/http2.readFrameHeader({0xc001d98660, 0x9, 0x9}, {0x1048adac0, 0xc002893bc0})
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/frame.go:237 +0xd6
golang.org/x/net/http2.(*Framer).ReadFrame(0xc001d98620)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/frame.go:498 +0x125
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc001d37f80)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2225 +0x1a8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000c9e300)
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:2120 +0xdc
created by golang.org/x/net/http2.(*Transport).newClientConn
	/private/tmp/skaffold-20230805-5278-w0t7w2/vendor/golang.org/x/net/http2/transport.go:818 +0xf45

goroutine 1303 [syscall]:
syscall.syscall6(0xc001e37870?, 0xc001e377d8?, 0xc0024e5860?, 0xc001e37a78?, 0xc0024e5860?, 0x1008c31d4?, 0xc001e37b38?)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/sys_darwin.go:45 +0x99
syscall.wait4(0x8a7, 0xc001e378a4, 0x0, 0xc000193170)
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/zsyscall_darwin_amd64.go:43 +0xaf
syscall.Wait4(0x8a7, 0xc001e37924, 0x0, 0xc000193170)
	/usr/local/Cellar/go/1.20.7/libexec/src/syscall/syscall_bsd.go:144 +0x77
os.(*Process).wait(0xc0018635c0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec_unix.go:43 +0x1f5
os.(*Process).Wait(0xc0018635c0)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec.go:132 +0x45
os/exec.(*Cmd).Wait(0xc000bd3e40)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:890 +0xd1
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).forward(0xc0017a4ec0, {0x1048dfb18, 0xc000f7f7d0}, 0xc0027a40d0, 0xc0034b01e0)
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:155 +0xbf6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*KubectlForwarder).Forward
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go:83 +0x1b6

goroutine 1306 [IO wait]:
internal/poll.runtime_pollWait(0x105cb3cb8?, 0x72)
	/usr/local/Cellar/go/1.20.7/libexec/src/runtime/netpoll.go:306 +0x45
internal/poll.(*pollDesc).wait(0xc0034b0c78, 0x72, 0x1)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x88
internal/poll.(*pollDesc).waitRead(0xc0034b0c78, 0x1)
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_poll_runtime.go:89 +0x37
internal/poll.(*FD).Read(0xc0034b0c60, {0xc00015384e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/internal/poll/fd_unix.go:167 +0x42f
os.(*File).read(0xc001109060, {0xc00015384e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file_posix.go:31 +0x7e
os.(*File).Read(0xc001109060, {0xc00015384e, 0x3b2, 0x3b2})
	/usr/local/Cellar/go/1.20.7/libexec/src/os/file.go:118 +0xdd
bytes.(*Buffer).ReadFrom(0xc002548930, {0x1048bb140, 0xc001109060})
	/usr/local/Cellar/go/1.20.7/libexec/src/bytes/buffer.go:202 +0x142
io.copyBuffer({0x1048adb40, 0xc002548930}, {0x1048bb140, 0xc001109060}, {0x0, 0x0, 0x0})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:413 +0x26b
io.Copy({0x1048adb40, 0xc002548930}, {0x1048bb140, 0xc001109060})
	/usr/local/Cellar/go/1.20.7/libexec/src/io/io.go:386 +0x88
os/exec.(*Cmd).writerDescriptor.func1()
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:560 +0x68
os/exec.(*Cmd).Start.func2(0xc000da9160)
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:717 +0x39
created by os/exec.(*Cmd).Start
	/usr/local/Cellar/go/1.20.7/libexec/src/os/exec/exec.go:716 +0x1206

goroutine 1360 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:83 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/portforward.(*WatchingPodForwarder).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/portforward/pod_forwarder.go:77 +0x291

goroutine 1363 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start.func5()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:104 +0x1e6
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes.(*podWatcher).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/watcher.go:99 +0x945

goroutine 1364 [select]:
github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start.func1()
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:81 +0x20c
created by github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/kubernetes/debugging.(*ContainerManager).Start
	/private/tmp/skaffold-20230805-5278-w0t7w2/pkg/skaffold/kubernetes/debugging/container_manager.go:75 +0x265

pkg/skaffold/kubernetes/debugging/container_manager.go:125 is _, seen := d.active[key], that map is apparently being used from different goroutines simultaneously without synchronization.

Steps to reproduce the behavior

Keep using Skaffold successfully for months and then randomly get unlucky one day? I'm not aware of any reliable way to reproduce this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugSomething isn't workingpriority/p2May take a couple of releases

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions