Debug Golang Memory Leaks with Pprof

Managing memory effectively is important for the performance of any application. While Golang's garbage collector typically does an excellent job of managing memory, memory leaks can still occur. A memory leak arises when an application doesn't release memory back to the operating system, even if it's no longer in use. In large applications, these leaks can lead to Out of Memory (OOM) errors and can impact application availability.

In Golang, memory leaks often happen due to infinite loops, improper use of goroutines, or not releasing references to memory once they're no longer needed. In this post, we'll discuss how pprof can be used for memory profiling and fixing leaks in Go.

Analyze the Signals

High memory usage might indicate a memory leak, but you need to confirm this first, since Linux uses an aggressive page caching mechanism which can lead to high reported memory usage (and it can be especially alarming in containerized environments where you have a fairly low resource ceiling). Here are some key signals that can help you understand if you have a problem:

• RSS (Resident Set Size): The amount of memory that the operating system has allocated to the process.
• HeapAlloc: The amount of memory allocated by the Go runtime.
• HeapInuse: The amount of memory in use by the application's live objects.
• HeapReleased: The amount of memory released back to the operating system.

By keeping an eye on these metrics and using the relevant commands, you can confirm whether your application is suffering from a memory leak or not.

Enable Profiling

If you have confirmed that you actually have a leak, the first debugging tool to reach for should be pprof, a built-in Go library for profiling Go programs.

The net/http/pprof package allows you to serve runtime profiling data in HTTP format. To use pprof, you need to import _ "net/http/pprof" in your main package, and start an HTTP server with http.ListenAndServe.

You can then use go tool pprof to interactively explore the data.

Here's a command to start a pprof session:

go tool pprof http://localhost:8080/debug/pprof/heap

The pprof tool provides various commands to help analyze the profile:

• top: Displays the top memory consumers.
• list: Shows the source code around a specific function.
• web: Opens a web-based visualization of the profile.

Let's Try It!

Consider the following Go HTTP server:

package main

import (
	"fmt"
	"net/http"
)

type UserCache struct {
	data map[string]string
}

func (uc *UserCache) handleRequest(w http.ResponseWriter, r *http.Request) {
	// Simulate memory leak by not removing old user data from the cache
	userID := r.URL.Query().Get("user_id")
	uc.data[userID] = "some data"
	fmt.Fprint(w, "Data stored in cache")
}

func main() {
	uc := &UserCache{
		data: make(map[string]string),
	}

	http.HandleFunc("/leaky-endpoint", uc.handleRequest)
	http.ListenAndServe(":8080", nil)
}

In this example, the server stores data for each user in the UserCache. On every request to /leaky-endpoint, new user data is created and added to the cache. However, there's no code to remove old user data from the cache.

You can simulate the leak by bombarding the server with a large number of requests using a tool like curl or ab.

Once the requests are completed, you can generate a heap profile by executing the following command in another terminal:

go tool pprof http://localhost:8080/debug/pprof/heap

As we can see, handleRequest is where the most allocations happen. It can also be confirmed by visual representation by doing:

go tool pprof -http=:8081 http://localhost:8080/debug/pprof/heap

Let's look at handleRequest more closely to identify where the leak comes from.

We were able to identify the exact line where the allocations happen, so now we can fix it by, for example, introducing a cache eviction policy.

Bonus: More Pprof Goodies

In addition to the techniques discussed earlier, pprof provides additional features and functionalities that can further enhance your profiling experience. Let's explore a few of these:

Profiling CPU Usage

You can profile your application's CPU usage using the goroutine and threadcreate profiles. To generate a CPU profile, execute the following command:

go tool pprof http://localhost:8080/debug/pprof/profile

Profiling Goroutines

You can profile the goroutines in your application using the goroutine profile. To generate a goroutine profile, execute the following command:

go tool pprof http://localhost:8080/debug/pprof/goroutine

Profiling Mutex Contention

You can profile the mutex contention in your application using the mutex profile. To generate a mutex profile, execute the following command:

go tool pprof http://localhost:8080/debug/pprof/mutex

These additional profiling features can help you gain deeper insights into the performance of your Go applications and identify any bottlenecks or issues that may be affecting their performance.

In conclusion, pprof is a powerful tool for debugging and profiling memory leaks in Golang applications. By utilizing its features and functionalities, developers can effectively manage memory and optimize the performance of their applications.