Hammerhead: moving performance testing upstream

September 30, 2008 10:07 pm | 54 Comments

Today at The Ajax Experience, I released Hammerhead, a Firebug extension for measuring page load times.

Improving performance starts with metrics. How long does it take for the page to load? Seems like a simple question to answer, but gathering accurate measurements can be a challenge. In my experience, performance metrics exist at four stages along the development process.

  • real user data – I love real user metrics. JavaScript frameworks like Jiffy measure page load times from real traffic. When your site is used by a large, diverse audience, data from real page views is ground-truth.
  • bucket testing – When you’re getting ready to push a new release, if you’re lucky you can do bucket testing to gather performance metrics. You release the new code to a subset of users while maintaining another set of users on the old code (the “control”). If you sample your user population correctly and gather enough data, comparing the before and after timing information gives you a preview of the latency impact of your next release.
  • synthetic or simulated testing – In some situations, it’s not possible to gather real user data. You might not have the infrastructure to do bucket testing and real user instrumentation. Your new build isn’t ready for release, but you still want to gauge where you are with regard to performance. Or perhaps you’re measuring your competitors’ performance. In these situations, the typical solution is to do scripted testing on some machine in your computer lab, or perhaps through a service like Keynote or Gomez.
  • dev box – The first place performance testing happens (or should happen) is on the developer’s box. As she finishes her code changes, the developer can see if she made things better or worse. What was the impact of that JavaScript rewrite? What happens if I add another stylesheet, or split my images across two domains?

Performance metrics get less precise as you move from real user data to dev box testing, as shown in Figure 1. That’s because, as you move away from real user data, biases are introduced. For bucket testing, the challenge is selecting users in an unbiased way. For synthetic testing, you need to choose scenarios and test accounts that are representative of real users. Other variables of your real user audience are difficult or impossible to simulate: bandwidth speed, CPU power, OS, browser, geographic location, etc. Attempting to simulate real users in your synthetic testing is a slippery, and costly, slope. Finally, testing on the dev box usually involves one scenario on a CPU that is more powerful than the typical user, and an Internet connection that is 10-100 times faster.

Figure 1 - Precision and ability to iterate along the development process

Given this loss of precision, why would we bother with anything other than real user data? The reason is speed of development. Dev box data can be gathered within hours of a code change, whereas it can take days to gather synthetic data, weeks to do bucket testing, and a month or more to release the code and have real user data. If you wait for real user data to see the impact of your changes, it can take a year to iterate on a short list of performance improvements. To quickly identify the most important performance improvements and their optimal implementation, it’s important to improve our ability to gather performance metrics earlier in the development process: on the dev box.

As a developer, it can be painful to measure the impact of a code change on your dev box. Getting an accurate time measurement is the easy part; you can use YSlow, Fasterfox, or an alert dialog. But then you have to load the page multiple times. The most painful part is transcribing the load times into Excel. Were all the measurements done with an empty cache or a primed cache, or was that even considered?

Hammerhead makes it easier to gather performance metrics on your dev box. Figure 2 shows the results of hammering a few news web sites with Hammerhead. By virtue of being a Firebug extension, Hammerhead is available in a platform that web developers are familiar with. To setup a Hammerhead test, one or more URLs are added to the list, and the “# of loads” is specified. Once started, Hammerhead loads each URL the specified number of times.

Figure 2 - Hammerhead results for several news sites

Figure 2 - Hammerhead results for a few news web sites

The next two things aren’t rocket science, but they make a big difference. First, there are two columns of results, one for empty cache and one for primed cache. Hammerhead automatically clears the disk and memory cache, or just the memory cache, in between each page load to gather metrics for both of these scenarios. Second, Hammerhead displays the median and average time measurement. Additionally, you can export the data in CSV format.

Even if you’re not hammering a site, other features make Hammerhead a useful add-on. The Cache & Time panel, shown in Figure 3, shows the current URL’s load time. It also contains buttons to clear the disk and memory cache, or just the memory cache. It has another feature that I haven’t seen anywhere else. You can choose to have Hammerhead clear these caches after every page view. This is a nice feature for me when I’m loading the same page again and again to see it’s performance in an empty or a primed cache state. If you forget to switch this back, it gets reset automatically next time you restart Firefox.

Figure 3 - Cache & Time panel in Hammerhead

Figure 3 - Cache & Time panel in Hammerhead

If you don’t have Hammerhead open, you can still see the load time in the status bar. Right clicking the Hammerhead icon gives you access for clearing the cache. The ability to clear just the memory cache is another valuable feature I haven’t seen elsewhere. I feel this is the best way to simulate the primed cache scenario, where the user has been to your site recently, but not during the current browser session.

Hammerhead makes it easier to gather performance metrics early in the development process, resulting in a faster development cycle. The biggest bias is that most developers have a much faster bandwidth connection than the typical user. Easy-to-install bandwidth throttlers are a solution. Steve Lamm blogged on Google Code about how Hammerhead can be used with bandwidth throttlers on the dev box, bringing together both ease and greater precision of performance measurements. GIve it a spin and let me know what you think.

54 Responses to Hammerhead: moving performance testing upstream

  1. Arnout | 30-Sep-08 at 10:39 pm | Permalink |

    It looks really great, Downloading it as we speak :)

  2. Niall Kennedy | 30-Sep-08 at 11:03 pm | Permalink |

    I would like a icon in a color other than red. Red makes me think there are errors on the page instead of communicating additional information available.

  3. Dan | 01-Oct-08 at 12:26 am | Permalink |

    Would there be any value in showing how much time was spend waiting for a connection prior to the first byte being recieived (TTFB), as well as a breakdown showing how much of the request time was spent doing the DNS lookup vs. the web server request time?

  4. Brendan Gibson | 01-Oct-08 at 8:14 am | Permalink |

    Anyone know how to get around:

    “Hammerhead” will not be installed because it does not provide secure updates

    when downloading?

  5. Brendan Gibson | 01-Oct-08 at 8:18 am | Permalink |

    I ‘spose reading the rest of the page wouldn’t hurt…

  6. Steve | 01-Oct-08 at 9:43 am | Permalink |

    Steve,

    Once again phenomenal post and very exciting measurement tool. I’ve already received value from it.

    Regards,

    Steve F.

  7. Another Steve F | 01-Oct-08 at 9:48 am | Permalink |

    This is definitely a useful tool but my perf testing needs have extended into processes run well into the lifetime of a page. I’d love to see this built out to support, for example, statistics about the processing of code between line A to line N marked similar to how breakpoints are marked on any loaded code accessible under firebug’s script tab. Thoughts?

  8. Alexander | 01-Oct-08 at 10:22 am | Permalink |

    Interesting idea, unfortunately I work on a lot of pages that include framebusting js code. Is there a way to gather these metrics without using an iframe around the page being monitored (yes, turning js off prevents the framebusting. It also prevents any dynamic content inclusions)?

  9. Steve Souders | 01-Oct-08 at 11:14 am | Permalink |

    @Niall – I’d like a b&w logo of a hammerhead shark. Know a good designer?

    @Dan – This is more for comparing code changes. Plus, looking at that detailed info at an aggregate level gets nonsensical. I recommend HTTPWatch for that information.

    @Steve F – In the FAQ I talk about a callback mechanism for doing something like this. Nothing definite yet.

    @Alexander – The frame is key for getting the onload callback.

  10. Dan Fabulich | 01-Oct-08 at 3:36 pm | Permalink |

    When I try to use HH to test multiple URLs, the timer halts after only a few loads, with a JS error on the (non-FB) JS console:

    Error: Permission denied to get property Object.selection
    Source File: XPCSafeJSObjectWrapper.cpp
    Line: 445

    I’m using FF3 on OSX 10.5.5: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.3) Gecko/2008092414 Firefox/3.0.3

  11. Dr. Known | 01-Oct-08 at 11:53 pm | Permalink |

    HH + Extended Status Bar Addon is an ideal combination.

    https://addons.mozilla.org/en-US/firefox/addons/versions/1433

  12. Derek | 03-Oct-08 at 1:55 pm | Permalink |

    Thanks, Steve! Heard about this at the Ajax Experience and it looks to be a wonderful tool with a lot of promise. I’m already using it to help identify problem areas. And your talk at the conference was outstanding. Keep up the great work!

  13. Kundan | 08-Dec-08 at 2:20 am | Permalink |

    “Hammerhead” will not be installed because it does not provide secure updates.

    It would be great if you can provide secure updates and list this fine tool on the mozilla addons website.

  14. Steve Souders | 08-Dec-08 at 11:25 am | Permalink |

    I would love host Hammerhead (and my many other FF extensions) on AMO, but it’s too difficult. You can’t just add an add-on to AMO. You have to add it to the Sandbox, then convince enough people to vote for it that it gets reviewed and hopefully makes it out of the Sandbox. Getting people to vote on something in the Sandbox is hard – you have to register and authenticate in a convulated way. Then whenever you want to do updates you have to get those approved (which is hard if you’re trying to coordinate that with a press release or web site update). Finally, in my previous experience with AMO the download stats were broken for months, and they don’t have the most important stat: # of people receiving updates. Finally, my best add-on, YSlow (with over 600K downloads) never made it into the list of recommended add-ons. I love Firefox and continue to extend it because the add-on interface is the best across all browsers, but AMO needs some serious work to make it easy for developers to host their extensions. This info is 1 year old. If these things are fixed now, please let me know and I’ll move Hammerhead there.

  15. Web Developer | 30-Dec-08 at 1:59 am | Permalink |

    Thank you for interesting post. I wonder where I can find additional information about bucket testing.

  16. Mary | 13-Jan-09 at 7:32 pm | Permalink |

    Thank You so much for the information! I appreciate the time you have taken here!

  17. Ragnar | 28-Jan-09 at 7:26 am | Permalink |

    Great post and great tool (and YSlow is also very nice).

    Performance testing is something that is done far too infrequently, or not at all, by many developers. They think they just have to produce lines of code to give to the testers and then fix whatever problems the testers find, but performance testing while developing is much more time-efficient. If you frequently test the code code you are writing, you will detect bad performance early and spend less time developing code that doesn’t perform well.

    Accessible and easy-to-use tools like Hammerhead are invaluable to get developers to start performance testing/optimizing their code as they write it.

    /Ragnar, Load Impact – http://loadimpact.com

  18. Alex | 11-Feb-09 at 11:07 am | Permalink |

    Great tool, thanks Steve.

    I had a couple of issue, does anyone know what might be the problem?
    1.Sometimes hammerhead will make 1 load and stop, it’s kind of random, so if I close and open the browser a few times eventually it will start hammering.

    2.Also randomly, primed cache values are showed slower than empty cache, maybe it’s just a bug that shifts the numbers.

  19. Tom K | 10-Jul-09 at 3:54 pm | Permalink |

    Hey Steve, seems that hammerhead isn’t working with Firefox 3.5 or the new Firebug update?

    Are you releasing a new version soon?

  20. Steve Souders | 11-Jul-09 at 5:26 am | Permalink |

    @Tom K: Thanks for the heads-up. I’ll work on it this week.

  21. Markku Laine | 22-Jul-09 at 3:53 am | Permalink |

    Hi Steve! I was just wondering is Hammerhead working with any combination of Firefox (version?) and Firebug (version?)?

    Measuring page load times works when loading a Web page (the small icon at the bottom right corner of the browser) but the Hammer tab in Firebug does not seem to work properly. It starts but it just does not stop… even thought I would set the value for # of loads to one.

    Is there another way to calculate the mean or average page load times using Hammerhead?

    What about the problems reported in comment #18. Does they still exist?

    I am using Firefox 2.0.0.20, Firebug 1.3.1, and Hammerhead 0.4 running on Mac OS X 10.5.7.

  22. Steve Souders | 22-Jul-09 at 6:57 am | Permalink |

    There is a bug with Hammerhead and Firefox 3.5. I’ll try to fix that today.

  23. Markku Laine | 24-Jul-09 at 4:56 am | Permalink |

    Okay, thanks for the information. Could you add a comment or something on this blog post when the new version of Hammerhead is available for download.

    Regards

    -Markku

  24. Steve Souders | 24-Jul-09 at 4:00 pm | Permalink |

    I fixed the incompatibilities with Firefox 3.5. You should receive an update notification next time you restart Firefox (after 24 hours) or can download Hammerhead 0.5 now from here:

    https://stevesouders.com/hammerhead/

    Thanks for your patience!

  25. Tom K | 24-Jul-09 at 10:31 pm | Permalink |

    Works great Steve thanks for the update!

    Tom

  26. Markku Laine | 25-Jul-09 at 2:40 am | Permalink |

    Thanks Steve!

  27. Justin | 05-Aug-09 at 5:25 pm | Permalink |

    Is there a way to store a list of urls? So I don’t have to add one by one every time I run the test.

    Thanks

  28. Steve Souders | 08-Aug-09 at 11:44 pm | Permalink |

    @Justin: I’ll be creating an open source project for Hammerhead soon. Please post this as an enhancement when the project is up and running. I’d be happy to work with you to implement it, too, if you want.

  29. Rolf Kaiser | 02-Sep-09 at 3:47 pm | Permalink |

    Great tool here – just what I was looking for!

  30. Florent Peyre | 30-Dec-09 at 12:57 pm | Permalink |

    Steve,
    Thanks a lot – this is an amazing tool to do comparison across the board.
    I saw that you were planning to fix the Export function. Any luck there? I know that it’s not critical but when you try to process very large amount of testing data, it would be incredibly helpful.
    Thx for the tool!
    Florent

  31. Julian | 25-Mar-10 at 4:17 am | Permalink |

    Nice plugin Steve! Sadly Firefox 3.6 has rejected it because ‘it doesn’t provide secure updates’. Do you have thoughts on an update to this?

  32. Julian (again) | 25-Mar-10 at 4:39 am | Permalink |

    Sorry ignore my previous stupid message I see you already answered this 2 years ago Steve :) I’m assuming it’s still an onerous process…

  33. Steve Souders | 25-Mar-10 at 5:19 am | Permalink |

    @Julian: No problemo!

  34. Matthew Sacks | 23-Apr-10 at 6:43 pm | Permalink |

    Actually, I am having the same problem as Julian with 3.6. I had 3.5, upgraded and the extensions.checkUpdateSecurity parameter was present and set to false. I tried modifying it, restarting and re-installing hammerhead but no dice. I also tried creating a new value checkUpdateSecurity, but Firefox still complains. Perhaps the name of this value has changed in 3.6? Perhaps I am doing something wrong as well.

  35. Dave Lozier | 27-May-10 at 12:17 pm | Permalink |

    Same issue as Mathew and Julian – Firefox 3.6.3

    about:config –
    extensions.checkCompatibility;false
    extensions.checkUpdateSecurity;false

    Still no dice.

    Google’s Page Speed and YSlow are both nice but neither seem to provide a basic display of the actual time it took to load the pages. Hammerhead looks promising and it also allows to compare multiple pages at the same time. (which is what I needed)

    I’m looking forward to seeing this open sourced and working well in the future. Thanks for all of your work so far Steve!

  36. Steve Souders | 27-May-10 at 4:46 pm | Permalink |

    @Matthew, @Dave: It works for me. I uninstalled Firefox including all my personal data and add-ons. Then I re-installed Firefox 3.6.3, Firebug 1.5.4, and Hammerhead 0.5. It worked (once I added checkUpdateSecurity to about:config). Is it possible some other add-on is in the way? Something else?

  37. Leslie | 30-May-10 at 10:39 am | Permalink |

    When I try to ‘Export Data’, a new blank tab comes up. Is this a known issue?

    I have disabled all my add-ons only enabling Firebug and Hammerhead. I am using FF 3.6.3

  38. adjie | 31-May-10 at 8:32 pm | Permalink |

    Hi Steve, I get what Leslie got.
    Export Data look like doesn’t work on FF 3.6.3

  39. Steve Souders | 05-Jun-10 at 11:40 pm | Permalink |

    I’ll investigate this as soon as I can after Velocity.

  40. Kamran | 07-Jun-10 at 5:36 pm | Permalink |

    Hi Steve,

    Thanks for the awesome tool. Do you know of anything similar for IE8?

    thx,
    Kamran

  41. Harpreet Singh | 04-Oct-10 at 8:30 am | Permalink |

    Hi,

    Is there a way I can save my list of urls and just run them each time without manually entering them?

    Thanks,
    Harpreet

  42. Tom K | 15-Oct-10 at 11:02 am | Permalink |

    Hi Steve,

    Still spreading the use of Hammerhead around. We are trying to measure the load of pages built from customized versions of Jive (example – http://www.jivesoftware.com/jivespace/community/support/jive_forums)

    When I went to select Hammer to measure it won’t load that tab to use Hammerhead.

    Any thoughts?

    Tom

  43. Steve Souders | 15-Oct-10 at 11:58 am | Permalink |

    @Harpreet: That’s a great enhancement, but right now you can’t store a list of URLs.

    @Tom K: Please send me more details through the contact form on my website.

  44. vijay rastogi | 26-Oct-10 at 6:22 am | Permalink |

    Hi Steve,

    Is there a way to use Hammerhead with another browser or is it a possibility to use it even with custom made browsers?

  45. Steve Souders | 26-Oct-10 at 10:07 am | Permalink |

    @vijay: Hammerhead is a Firefox add-on so only runs in Firefox.

  46. mario | 08-Nov-10 at 5:48 am | Permalink |

    Hi Steve,
    you mentioned years ago creating a project for this… I suppose the idea was given up right? because I also have issues with not saving the URL list… Also I couldn’t see a way how to trace Ajax requests – so I don’t even get why you presented it to the Ajax Experience :)

  47. Jason | 12-Nov-10 at 10:16 am | Permalink |

    When I have Firefox open and try to load this I get that this is an ‘imcompatible extension’ because of a lack of security updates and firefox will not allow installation. what gives?

  48. Jason | 12-Nov-10 at 10:17 am | Permalink |

    PS on you spam blocker, ‘twenty-for minus 3 = ” has no correct answer. Try “twentyfour minus 3 = “

  49. Steve Souders | 12-Nov-10 at 11:10 am | Permalink |

    @Jason: Please follow the directions on the Hammerhead page. The spam blocker is to avoid bots.

  50. Nabarun Mondal | 02-Dec-10 at 9:46 pm | Permalink |

    Steve, this is brilliant.

  51. Mondane | 06-Apr-11 at 1:18 am | Permalink |

    Will you make Hammerhead compatible with Firefox 4?

  52. Steve Souders | 06-Apr-11 at 8:12 am | Permalink |

    @Mondane: I’m not actively working on Hammerhead.

  53. John | 26-Apr-12 at 8:04 pm | Permalink |

    Firefox 12 please. :)

  54. Steve Souders | 26-Apr-12 at 8:23 pm | Permalink |

    @John: I watch my web server logs regularly and am surprised and pleased with the high number of hits on Hammerhead’s update.rdf (~1K). Unfortunately, I don’t have time to continue work on Hammerhead. Fortunately, the code is open source so it’s possible for you or someone else to take the project over. Please have at it! This tool has only scratched the surface of its potential.