Chenyo's org-static-blog

24 Jun 2024

A stupid debugging experience

1. What happened

  • Servers SA and SB have the same docker installation, and the same running container CA and CB.
  • A Go file G can be built on CA, but on CB it reports this error:

    runtime: failed to create new OS thread (have 2 already; errno=11)
    runtime: may need to increase max user processes (ulimit -u)
    fatal error: newosproc

2. What did I do

  1. I compared any related configurations between SA and SB. and between CA and CB, e.g., ulimit -a, /etc/security/limits.conf. They all look the same.
  2. I created a new container CN on SA with the same docker image, CN can compile G.
  3. I looked into the (complex) docker run script for CA/CB and figured out it was due to a resource constraint --pids-limit 100.
    • Increasing this limit to 200 seems resolve the issue, but I had no idea why the Go compiler needed so many resources (perhaps due to package I imported).
  4. Until this point, I realized, since the container did not support the compilation, why not just only transfer the compiled binary!
    • How silly that I didn’t even try this in the beginning!
  5. Since the program imports the net package, and there is a known issue of Alpine image running a Go binary file, I followed the post and disabled CGO on SA, then docker cp the binary to CA, and it worked.

3. Another issue of running RPC in docker

  • The other day, I also spent hours debugging a route unreachable error when I want to send a request from CA to SA.
  • The CA is using the bridge network, so it should talk to SA via SA’s interface docker0 within the subnet 172.17.0.0/16.
  • However, in my case, the docker by default rejects packages from any container as shown in SA’s tcpdump result:

    172.17.0.1->172.17.0.3 ICMP host unreachable- admin prohibited, length 68

  • By checking SA’s iptables, I found this rule:

      -A INPUT -j REJECT --reject-with icmp-host-prohibited
    
    • Strangely, the ping still works with this rule.
  • In the end, I need to append a new rule to make the RPC work.

      iptables -I INPUT 1 -i docker0 -p tcp --dport <port> -s 172.17.0.0/16 -j ACCEPT
    
Tags: docker go linux alpine