Technical explanation: it looks like ip_conntrack was watching some local (127.0.0.1 <-> 127.0.0.1) connections and it was dropping packets. We changed /etc/apf/fwd.rules and raised /proc/sys/net/ipv4/netfilter/ip_conntrack_max to 150000.
Yes, this is voodoo to me too. What we know is that:
- These problems are due to increasing traffic. Popularity has its price.
- It's not about the usual problems that hit web applications: database, or bandwidth. Our servers and networks are running nicely, with lots of spare capacity.
- Things are better but not fully fixed - I saw an error this morning.
Please let us know whether you also get problems. We're treating this as top priority.