[{"data":1,"prerenderedAt":770},["Reactive",2],{"content-/2026-08-05-go-oomkilled-in-kubernetes-gomemlimit":3},{"article":4,"related":759},{"_path":5,"_dir":6,"_draft":7,"_partial":7,"_locale":6,"_empty":7,"title":8,"description":9,"date":10,"tags":11,"cover":16,"level":17,"published":18,"body":19,"_type":754,"_id":755,"_source":756,"_file":757,"_extension":758},"/2026-08-05-go-oomkilled-in-kubernetes-gomemlimit","",false,"Your Go Service Is Getting OOMKilled and It Isn't a Leak","A highly concurrent Go service kept dying with exit code 137 in Kubernetes, but pprof showed no leak. The problem was that the Go runtime had no idea the pod had a memory limit. This is what GOMEMLIMIT fixes.","2026-08-05T00:00:00.000Z",[12,13,14,15],"Go","Kubernetes","Distributed-Systems","Performance","gomemlimit.png","warn",true,{"type":20,"children":21,"toc":744},"root",[22,43,63,89,94,101,106,126,160,171,176,182,200,205,217,222,228,245,263,284,289,295,307,318,323,329,334,415,420,546,579,604,610,622,631,655,663,674,680,685,695,711,717,728,739],{"type":23,"tag":24,"props":25,"children":26},"element","blockquote",{},[27],{"type":23,"tag":28,"props":29,"children":30},"p",{},[31,34],{"type":32,"value":33},"text","Art consists of limitation. The most beautiful part of every picture is the frame. ",{"type":23,"tag":28,"props":35,"children":36},{},[37],{"type":23,"tag":38,"props":39,"children":40},"b",{},[41],{"type":32,"value":42},"G. K. Chesterton",{"type":23,"tag":28,"props":44,"children":45},{},[46,48,54,56,61],{"type":32,"value":47},"A while back I was investigating a highly concurrent Go service that kept restarting in its Kubernetes pod during sudden bursts of traffic. Exit code 137 — ",{"type":23,"tag":49,"props":50,"children":51},"code-inline",{},[52],{"type":32,"value":53},"128 + 9",{"type":32,"value":55},", a ",{"type":23,"tag":49,"props":57,"children":58},{},[59],{"type":32,"value":60},"SIGKILL",{"type":32,"value":62}," from the kernel's OOM killer.",{"type":23,"tag":28,"props":64,"children":65},{},[66,68,73,75,80,82,87],{"type":32,"value":67},"My first guess was the obvious one: a memory leak. But ",{"type":23,"tag":49,"props":69,"children":70},{},[71],{"type":32,"value":72},"pprof",{"type":32,"value":74}," didn't agree. Under heavy load the in-use heap was fairly small, and the ",{"type":23,"tag":49,"props":76,"children":77},{},[78],{"type":32,"value":79},"inuse_space",{"type":32,"value":81}," profile looked healthy. It was the ",{"type":23,"tag":49,"props":83,"children":84},{},[85],{"type":32,"value":86},"alloc_space",{"type":32,"value":88}," profile that showed what was actually going on — the service was churning through enormous volumes of short-lived allocations, and nothing was telling the runtime to slow down.",{"type":23,"tag":28,"props":90,"children":91},{},[92],{"type":32,"value":93},"There was no leak. The runtime was doing exactly what it had been told to do. The problem was what it hadn't been told.",{"type":23,"tag":95,"props":96,"children":98},"h2",{"id":97},"two-accountants-no-shared-ledger",[99],{"type":32,"value":100},"Two accountants, no shared ledger",{"type":23,"tag":28,"props":102,"children":103},{},[104],{"type":32,"value":105},"For a Go service running in Kubernetes, two separate systems have an opinion about memory, and before Go 1.19 they barely spoke to each other.",{"type":23,"tag":28,"props":107,"children":108},{},[109,111,117,119,124],{"type":32,"value":110},"The first is the ",{"type":23,"tag":112,"props":113,"children":114},"strong",{},[115],{"type":32,"value":116},"cgroup",{"type":32,"value":118},". It enforces the pod's ",{"type":23,"tag":49,"props":120,"children":121},{},[122],{"type":32,"value":123},"resources.limits.memory",{"type":32,"value":125}," at the kernel level. It is not negotiable and it does not warn you — when the process crosses the line, the OOM killer takes it.",{"type":23,"tag":28,"props":127,"children":128},{},[129,131,136,138,143,145,151,153,158],{"type":32,"value":130},"The second is the ",{"type":23,"tag":112,"props":132,"children":133},{},[134],{"type":32,"value":135},"Go runtime",{"type":32,"value":137},". Its only real lever was ",{"type":23,"tag":49,"props":139,"children":140},{},[141],{"type":32,"value":142},"GOGC",{"type":32,"value":144},", which controls how large the heap may grow ",{"type":23,"tag":146,"props":147,"children":148},"em",{},[149],{"type":32,"value":150},"relative to the live set",{"type":32,"value":152},". The default, ",{"type":23,"tag":49,"props":154,"children":155},{},[156],{"type":32,"value":157},"GOGC=100",{"type":32,"value":159},", means \"let the heap roughly double before collecting again.\"",{"type":23,"tag":28,"props":161,"children":162},{},[163,165,169],{"type":32,"value":164},"Notice what's missing from that sentence: any absolute number. ",{"type":23,"tag":49,"props":166,"children":167},{},[168],{"type":32,"value":142},{"type":32,"value":170}," is a ratio, and a ratio has no ceiling. If the live set grows, the target grows with it. The runtime has no idea a 1 GiB limit exists, so it does the reasonable thing and keeps growing the heap — right through the limit and into the OOM killer.",{"type":23,"tag":28,"props":172,"children":173},{},[174],{"type":32,"value":175},"That's the whole bug. Not a leak: a units mismatch. The cgroup thinks in bytes, and the runtime was thinking in percentages.",{"type":23,"tag":95,"props":177,"children":179},{"id":178},"why-tuning-gogc-isnt-the-answer",[180],{"type":32,"value":181},"Why tuning GOGC isn't the answer",{"type":23,"tag":28,"props":183,"children":184},{},[185,187,191,193,198],{"type":32,"value":186},"The obvious workaround is to turn ",{"type":23,"tag":49,"props":188,"children":189},{},[190],{"type":32,"value":142},{"type":32,"value":192}," down. Set ",{"type":23,"tag":49,"props":194,"children":195},{},[196],{"type":32,"value":197},"GOGC=50",{"type":32,"value":199}," and the heap only grows 1.5x between collections instead of 2x.",{"type":23,"tag":28,"props":201,"children":202},{},[203],{"type":32,"value":204},"It works, in the narrow sense that the process stops dying. But it's the wrong instrument, for two reasons.",{"type":23,"tag":28,"props":206,"children":207},{},[208,210,215],{"type":32,"value":209},"It's still a ratio, so it still has no ceiling — you have made the OOM less likely without making it impossible. And it applies the same aggression at ",{"type":23,"tag":146,"props":211,"children":212},{},[213],{"type":32,"value":214},"all",{"type":32,"value":216}," times. A service sitting at 5% of its memory budget pays exactly the same GC tax as one sitting at 95%. You end up burning CPU on collections you didn't need, permanently, to protect against a burst that happens occasionally.",{"type":23,"tag":28,"props":218,"children":219},{},[220],{"type":32,"value":221},"What you actually want is a runtime that relaxes when there's headroom and gets serious as the budget runs out.",{"type":23,"tag":95,"props":223,"children":225},{"id":224},"gomemlimit-telling-the-runtime-about-the-budget",[226],{"type":32,"value":227},"GOMEMLIMIT: telling the runtime about the budget",{"type":23,"tag":28,"props":229,"children":230},{},[231,233,238,240],{"type":32,"value":232},"That is precisely what Go 1.19's ",{"type":23,"tag":49,"props":234,"children":235},{},[236],{"type":32,"value":237},"GOMEMLIMIT",{"type":32,"value":239}," provides. It tells the runtime: ",{"type":23,"tag":146,"props":241,"children":242},{},[243],{"type":32,"value":244},"this is roughly how much memory you're allowed to burn before Kubernetes gets angry.",{"type":23,"tag":28,"props":246,"children":247},{},[248,250,255,257,261],{"type":32,"value":249},"It's a ",{"type":23,"tag":112,"props":251,"children":252},{},[253],{"type":32,"value":254},"soft limit",{"type":32,"value":256}," on Go-managed memory — the heap, goroutine stacks, and runtime overhead. The behaviour is adaptive in the way ",{"type":23,"tag":49,"props":258,"children":259},{},[260],{"type":32,"value":142},{"type":32,"value":262}," isn't:",{"type":23,"tag":264,"props":265,"children":266},"ul",{},[267,279],{"type":23,"tag":268,"props":269,"children":270},"li",{},[271,273,277],{"type":32,"value":272},"While usage sits comfortably below the limit, GC behaves normally and ",{"type":23,"tag":49,"props":274,"children":275},{},[276],{"type":32,"value":142},{"type":32,"value":278}," governs as usual.",{"type":23,"tag":268,"props":280,"children":281},{},[282],{"type":32,"value":283},"As usage approaches the limit, the GC runs more frequently, trading CPU to keep the process inside its budget.",{"type":23,"tag":28,"props":285,"children":286},{},[287],{"type":32,"value":288},"So the runtime finally has the one piece of information the cgroup always had, and it can make its own trade-offs with that number in hand.",{"type":23,"tag":95,"props":290,"children":292},{"id":291},"picking-a-value",[293],{"type":32,"value":294},"Picking a value",{"type":23,"tag":28,"props":296,"children":297},{},[298,300,305],{"type":32,"value":299},"There's no universally correct value, and it depends on your workload. A reasonable starting point is ",{"type":23,"tag":112,"props":301,"children":302},{},[303],{"type":32,"value":304},"70–80% of the pod's actual memory limit",{"type":32,"value":306},".",{"type":23,"tag":28,"props":308,"children":309},{},[310,312,316],{"type":32,"value":311},"For a 1 GiB pod that lands around 700–800 MiB. The remaining 200–300 MiB isn't waste — it's the memory ",{"type":23,"tag":49,"props":313,"children":314},{},[315],{"type":32,"value":237},{"type":32,"value":317}," doesn't govern, and you need it: the binary itself, thread stacks, OS page cache, any cgo allocations.",{"type":23,"tag":28,"props":319,"children":320},{},[321],{"type":32,"value":322},"Set it too close to the pod limit and you've defeated the point, because the ungoverned memory pushes you over anyway. Set it too low and you're paying the aggressive-GC tax you were trying to avoid.",{"type":23,"tag":95,"props":324,"children":326},{"id":325},"setting-it",[327],{"type":32,"value":328},"Setting it",{"type":23,"tag":28,"props":330,"children":331},{},[332],{"type":32,"value":333},"As an environment variable, which is usually where it belongs — it lets you change the value without a rebuild, and keeps it next to the pod limit it's derived from:",{"type":23,"tag":335,"props":336,"children":341},"code",{"className":337,"code":339,"language":340,"meta":6},[338],"language-yaml","env:\n  - name: GOMEMLIMIT\n    value: \"800MiB\"\n","yaml",[342],{"type":23,"tag":343,"props":344,"children":345},"pre",{},[346],{"type":23,"tag":335,"props":347,"children":348},{"__ignoreMap":6},[349,367,392],{"type":23,"tag":350,"props":351,"children":354},"span",{"class":352,"line":353},"line",1,[355,361],{"type":23,"tag":350,"props":356,"children":358},{"class":357},"ct-268635",[359],{"type":32,"value":360},"env",{"type":23,"tag":350,"props":362,"children":364},{"class":363},"ct-183901",[365],{"type":32,"value":366},":\n",{"type":23,"tag":350,"props":368,"children":370},{"class":352,"line":369},2,[371,376,381,386],{"type":23,"tag":350,"props":372,"children":373},{"class":363},[374],{"type":32,"value":375},"  - ",{"type":23,"tag":350,"props":377,"children":378},{"class":357},[379],{"type":32,"value":380},"name",{"type":23,"tag":350,"props":382,"children":383},{"class":363},[384],{"type":32,"value":385},": ",{"type":23,"tag":350,"props":387,"children":389},{"class":388},"ct-141567",[390],{"type":32,"value":391},"GOMEMLIMIT\n",{"type":23,"tag":350,"props":393,"children":395},{"class":352,"line":394},3,[396,401,406,410],{"type":23,"tag":350,"props":397,"children":398},{"class":363},[399],{"type":32,"value":400},"    ",{"type":23,"tag":350,"props":402,"children":403},{"class":357},[404],{"type":32,"value":405},"value",{"type":23,"tag":350,"props":407,"children":408},{"class":363},[409],{"type":32,"value":385},{"type":23,"tag":350,"props":411,"children":412},{"class":388},[413],{"type":32,"value":414},"\"800MiB\"",{"type":23,"tag":28,"props":416,"children":417},{},[418],{"type":32,"value":419},"Or in code:",{"type":23,"tag":335,"props":421,"children":426},{"className":422,"code":424,"language":425,"meta":6},[423],"language-go","import \"runtime/debug\"\n\nfunc init() {\n    debug.SetMemoryLimit(700 \u003C\u003C 20) // 700 MiB\n}\n","go",[427],{"type":23,"tag":343,"props":428,"children":429},{},[430],{"type":23,"tag":335,"props":431,"children":432},{"__ignoreMap":6},[433,451,459,482,537],{"type":23,"tag":350,"props":434,"children":435},{"class":352,"line":353},[436,441,446],{"type":23,"tag":350,"props":437,"children":438},{"class":357},[439],{"type":32,"value":440},"import",{"type":23,"tag":350,"props":442,"children":443},{"class":363},[444],{"type":32,"value":445}," ",{"type":23,"tag":350,"props":447,"children":448},{"class":388},[449],{"type":32,"value":450},"\"runtime/debug\"\n",{"type":23,"tag":350,"props":452,"children":453},{"class":352,"line":369},[454],{"type":23,"tag":350,"props":455,"children":456},{},[457],{"type":32,"value":458},"\n",{"type":23,"tag":350,"props":460,"children":461},{"class":352,"line":394},[462,467,471,477],{"type":23,"tag":350,"props":463,"children":464},{"class":357},[465],{"type":32,"value":466},"func",{"type":23,"tag":350,"props":468,"children":469},{"class":363},[470],{"type":32,"value":445},{"type":23,"tag":350,"props":472,"children":474},{"class":473},"ct-253592",[475],{"type":32,"value":476},"init",{"type":23,"tag":350,"props":478,"children":479},{"class":363},[480],{"type":32,"value":481},"() {\n",{"type":23,"tag":350,"props":483,"children":485},{"class":352,"line":484},4,[486,491,497,502,508,512,517,521,526,531],{"type":23,"tag":350,"props":487,"children":488},{"class":363},[489],{"type":32,"value":490},"    debug.",{"type":23,"tag":350,"props":492,"children":494},{"class":493},"ct-049001",[495],{"type":32,"value":496},"SetMemoryLimit",{"type":23,"tag":350,"props":498,"children":499},{"class":363},[500],{"type":32,"value":501},"(",{"type":23,"tag":350,"props":503,"children":505},{"class":504},"ct-674544",[506],{"type":32,"value":507},"700",{"type":23,"tag":350,"props":509,"children":510},{"class":363},[511],{"type":32,"value":445},{"type":23,"tag":350,"props":513,"children":514},{"class":357},[515],{"type":32,"value":516},"\u003C\u003C",{"type":23,"tag":350,"props":518,"children":519},{"class":363},[520],{"type":32,"value":445},{"type":23,"tag":350,"props":522,"children":523},{"class":504},[524],{"type":32,"value":525},"20",{"type":23,"tag":350,"props":527,"children":528},{"class":363},[529],{"type":32,"value":530},") ",{"type":23,"tag":350,"props":532,"children":534},{"class":533},"ct-971041",[535],{"type":32,"value":536},"// 700 MiB\n",{"type":23,"tag":350,"props":538,"children":540},{"class":352,"line":539},5,[541],{"type":23,"tag":350,"props":542,"children":543},{"class":363},[544],{"type":32,"value":545},"}",{"type":23,"tag":28,"props":547,"children":548},{},[549,551,556,558,563,565,570,572,577],{"type":32,"value":550},"One warning about the environment variable: it takes size suffixes like ",{"type":23,"tag":49,"props":552,"children":553},{},[554],{"type":32,"value":555},"MiB",{"type":32,"value":557}," and ",{"type":23,"tag":49,"props":559,"children":560},{},[561],{"type":32,"value":562},"GiB",{"type":32,"value":564},", but ",{"type":23,"tag":112,"props":566,"children":567},{},[568],{"type":32,"value":569},"a bare number is interpreted as bytes",{"type":32,"value":571},". ",{"type":23,"tag":49,"props":573,"children":574},{},[575],{"type":32,"value":576},"GOMEMLIMIT=800",{"type":32,"value":578}," sets a limit of 800 bytes, not 800 MiB, and the runtime will accept it and immediately GC itself to death. This is an easy typo to make in a manifest and a confusing one to debug.",{"type":23,"tag":28,"props":580,"children":581},{},[582,584,596,598,602],{"type":32,"value":583},"Better still, derive it from the cgroup at startup rather than hardcoding it in two places that can drift apart. The ",{"type":23,"tag":585,"props":586,"children":590},"a",{"href":587,"rel":588},"https://github.com/KimMachineGun/automemlimit",[589],"nofollow",[591],{"type":23,"tag":49,"props":592,"children":593},{},[594],{"type":32,"value":595},"automemlimit",{"type":32,"value":597}," package reads the container's actual limit and sets ",{"type":23,"tag":49,"props":599,"children":600},{},[601],{"type":32,"value":237},{"type":32,"value":603}," to a percentage of it.",{"type":23,"tag":95,"props":605,"children":607},{"id":606},"what-it-looked-like",[608],{"type":32,"value":609},"What it looked like",{"type":23,"tag":28,"props":611,"children":612},{},[613,615,620],{"type":32,"value":614},"Before, with no limit defined — the heap climbs until the cgroup ends the process. ",{"type":23,"tag":49,"props":616,"children":617},{},[618],{"type":32,"value":619},"HeapSys",{"type":32,"value":621}," peaks past 1200 MiB, and the flat line at the right edge is the pod dying:",{"type":23,"tag":28,"props":623,"children":624},{},[625],{"type":23,"tag":626,"props":627,"children":630},"img",{"alt":628,"src":629},"Go heap growing unbounded until the pod is OOMKilled","https://res.cloudinary.com/function/image/upload/v1786020016/blog/gomemlimit-before.jpg",[],{"type":23,"tag":28,"props":632,"children":633},{},[634,636,640,642,647,649,653],{"type":32,"value":635},"After, with ",{"type":23,"tag":49,"props":637,"children":638},{},[639],{"type":32,"value":237},{"type":32,"value":641}," set — same workload, same traffic. ",{"type":23,"tag":49,"props":643,"children":644},{},[645],{"type":32,"value":646},"HeapAlloc",{"type":32,"value":648}," still swings hard with the bursts, which is expected and fine. What changed is that ",{"type":23,"tag":49,"props":650,"children":651},{},[652],{"type":32,"value":619},{"type":32,"value":654}," flattens out around 763 MiB instead of climbing forever:",{"type":23,"tag":28,"props":656,"children":657},{},[658],{"type":23,"tag":626,"props":659,"children":662},{"alt":660,"src":661},"Go heap staying within its budget once GOMEMLIMIT is set","https://res.cloudinary.com/function/image/upload/v1786020016/blog/gomemlimit-after.jpg",[],{"type":23,"tag":28,"props":664,"children":665},{},[666,668,672],{"type":32,"value":667},"The service stopped restarting. Note that the sawtooth in ",{"type":23,"tag":49,"props":669,"children":670},{},[671],{"type":32,"value":646},{"type":32,"value":673}," didn't go away, and it shouldn't — the allocation pattern was never the problem. The ceiling was.",{"type":23,"tag":95,"props":675,"children":677},{"id":676},"what-gomemlimit-does-not-cover",[678],{"type":32,"value":679},"What GOMEMLIMIT does not cover",{"type":23,"tag":28,"props":681,"children":682},{},[683],{"type":32,"value":684},"Two caveats worth knowing before you reach for it.",{"type":23,"tag":28,"props":686,"children":687},{},[688,693],{"type":23,"tag":112,"props":689,"children":690},{},[691],{"type":32,"value":692},"It only governs Go-managed memory.",{"type":32,"value":694}," Allocations made through cgo, memory-mapped files, and the OS page cache all sit outside its accounting. If a meaningful share of your footprint is non-Go, budget for it explicitly in the headroom you leave.",{"type":23,"tag":28,"props":696,"children":697},{},[698,703,705,709],{"type":23,"tag":112,"props":699,"children":700},{},[701],{"type":32,"value":702},"It's soft, and that matters when the live set genuinely doesn't fit.",{"type":32,"value":704}," If your live heap legitimately exceeds the limit, the runtime will not kill your process — it will keep collecting, harder and harder, trying to get under a line it cannot reach. That's a GC death spiral: the service stays alive but burns most of its CPU on garbage collection. Go mitigates this by capping GC at roughly 50% of CPU, so you get a service that's slow rather than one that's wedged. Still, a sudden CPU spike with no throughput to show for it is the signature to watch for, and it means your limit is too low for the work — not that ",{"type":23,"tag":49,"props":706,"children":707},{},[708],{"type":32,"value":237},{"type":32,"value":710}," is misbehaving.",{"type":23,"tag":95,"props":712,"children":714},{"id":713},"the-takeaway",[715],{"type":32,"value":716},"The takeaway",{"type":23,"tag":28,"props":718,"children":719},{},[720,722,726],{"type":32,"value":721},"Without ",{"type":23,"tag":49,"props":723,"children":724},{},[725],{"type":32,"value":237},{"type":32,"value":727},", the Go runtime behaves as though RAM is infinite. It isn't being reckless; nobody told it otherwise. Set the limit and it works within a defined budget and actively avoids growing past it.",{"type":23,"tag":28,"props":729,"children":730},{},[731,733,737],{"type":32,"value":732},"If you're running Go in Kubernetes and you've never set it, you are relying on ",{"type":23,"tag":49,"props":734,"children":735},{},[736],{"type":32,"value":142},{"type":32,"value":738},"'s ratio happening to stay under a ceiling it has never been shown. That works right up until a traffic burst, which is exactly when you'd rather it didn't fail.",{"type":23,"tag":740,"children":741},"style",[742],{"type":32,"value":743},".ct-268635{color:#F92672;}\n.ct-183901{color:#F8F8F2;}\n.ct-141567{color:#E6DB74;}\n.ct-253592{color:#A6E22E;}\n.ct-049001{color:#66D9EF;}\n.ct-674544{color:#AE81FF;}\n.ct-971041{color:#88846F;}",{"title":6,"searchDepth":369,"depth":369,"links":745},[746,747,748,749,750,751,752,753],{"id":97,"depth":369,"text":100},{"id":178,"depth":369,"text":181},{"id":224,"depth":369,"text":227},{"id":291,"depth":369,"text":294},{"id":325,"depth":369,"text":328},{"id":606,"depth":369,"text":609},{"id":676,"depth":369,"text":679},{"id":713,"depth":369,"text":716},"markdown","content:2026-08-05-go-oomkilled-in-kubernetes-gomemlimit.md","content","2026-08-05-go-oomkilled-in-kubernetes-gomemlimit.md","md",[760,765],{"_path":761,"title":762,"description":763,"date":764},"/2023-08-04-create-api-rate-limiter-in-go","Create an API Rate Limiter in Golang","Learn how to create an API Rate Limiter in GoLang.","2023-08-04T00:00:00.000Z",{"_path":766,"title":767,"description":768,"date":769},"/2023-07-11-production-ready-beanstalkd-with-laravel","Production-Ready Beanstalkd with Laravel Queues","Queues are one of the integral parts of a scalable system, this article introduces the users to Queues and how they can integrate them into their Laravel application to improve the system performance.","2023-07-11T00:00:00.000Z",1786025254299]