Production Profiling Techniques
Real user traffic uncovers the most interesting performance problems. Profile carefully: minimise risk, capture actionable data, and feed insights back into tests and automation.
Production-safe profilers
- rbspy: Attaches to running processes, streams flamegraphs, minimal pause time.
- stackprof: Sampling profiler that can run inline or via environment variable.
- ruby-prof (wall/alloc modes): Great for staging or short production snapshots when overhead is acceptable.
- APM agents: Datadog, New Relic, Scout – combine transaction traces, DB timings, and error rates.
Staging vs production checklist
- Reproduce in staging with production-like data. If reproducible, profile there first.
- Snapshot traffic before profiling (requests/min, queue depth) to compare after fixes.
- Communicate in #dev-infra when attaching profilers; note PID and expected duration.
- Throttle sampling (rbspy `--frequency 50`, stackprof `--interval 1000`) to reduce overhead.
Profiling a running Sidekiq worker
# ssh into host
pid=$(pgrep -f "sidekiq 6")
rbspy record --pid "$pid" \
--duration 60 \
--format flamegraph \
--file /tmp/sidekiq.flamegraph
scp host:/tmp/sidekiq.flamegraph ./
rbspy flamegraph sidekiq.flamegraph > flamegraph.html
Attach only once per server, document start/end time, and remove generated graphs containing sensitive data after analysis.
Safety checklist
- Confirm profiling window and scope with the incident commander if during an outage.
- Throttle sample frequency and duration (1–2 minutes) to minimise user impact.
- Redact secrets before sharing dumps or flamegraphs in Slack or issues.
- Feed findings back into tests—add benchmarks, alerts, or documentation for the optimised path.