Profiling & Debugging
In any real world scenario, an app might start exhibiting undesirable behavior in terms of resource spikes. CPU/Memory spikes are not uncommon in most cases.
Dapr allows users to start an on-demand profiling session using
pprof through its profiling server endpoint and start an instrumentation session to discover problems and issues such as concurrency, performance, cpu and memory usage.
Dapr allows you to enable profiling in both Kubernetes and stand-alone modes.
To enable profiling in Standalone mode, pass the
--enable-profiling and the
--profile-port flags to the Dapr CLI:
profile-port is not required, and if not provided Dapr will pick an available port.
dapr run --enable-profiling true --profile-port 7777 python myapp.py
To enable profiling in Kubernetes, simply add the
dapr.io/enable-profiling annotation to your Dapr annotated pod:
annotations: dapr.io/enabled: "true" dapr.io/app-id: "rust-app" dapr.io/enable-profiling: "true"
Debug a profiling session
After profiling is enabled, we can start a profiling session to investigate what’s going on with the Dapr runtime.
For Standalone mode, locate the Dapr instance that you want to profile:
dapr list APP ID DAPR PORT APP PORT COMMAND AGE CREATED PID node-subscriber 3500 3000 node app.js 12s 2019-09-09 15:11.24 896
Grab the DAPR PORT, and if profiling has been enabled as described above, you can now start using
pprof to profile Dapr.
Look at the Kubernetes examples above for some useful commands to profile Dapr.
More info on pprof can be found here.
First, find the pod containing the Dapr runtime. If you don’t already know the the pod name, type
kubectl get pods:
NAME READY STATUS RESTARTS AGE divideapp-6dddf7dc74-6sq4l 2/2 Running 0 2d23h
If profiling has been enabled successfully, the runtime logs should show the following:
time="2019-09-09T20:56:21Z" level=info msg="starting profiling server on port 7777"
In this case, we want to start a session with the Dapr runtime inside of pod
We can do so by connecting to the pod via port forwarding:
kubectl port-forward divideapp-6dddf7dc74-6sq4 7777:7777 Forwarding from 127.0.0.1:7777 -> 7777 Forwarding from [::1]:7777 -> 7777 Handling connection for 7777
Now that the connection has been established, we can use
pprof to profile the Dapr runtime.
The following example will create a
cpu.pprof file containing samples from a profile session that lasts 120 seconds:
curl "http://localhost:7777/debug/pprof/profile?seconds=120" > cpu.pprof
Analyze the file with pprof:
You can also save the results in a visualized way inside a PDF:
go tool pprof --pdf your-binary-file http://localhost:7777/debug/pprof/profile?seconds=120 > profile.pdf
For memory related issues, you can profile the heap:
go tool pprof --pdf your-binary-file http://localhost:7777/debug/pprof/heap > heap.pdf
Profiling allocated objects:
go tool pprof http://localhost:7777/debug/pprof/heap > exit Saved profile in /Users/myusername/pprof/pprof.daprd.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
To analyze, grab the file path above (its a dynamic file path, so pay attention to note paste this one), and execute:
go tool pprof -alloc_objects --pdf /Users/myusername/pprof/pprof.daprd.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz > alloc-objects.pdf