App cache & localStorage survey
In preparation for my talk at HTML5 Dev Conf I surveyed the Alexa US Top 10 websites to see which ones use app cache and localStorage. I mostly focus on mobile these days so it’s natural to think I ran these tests from a mobile browser, which I did. But I also tested with a desktop browser.
Some people might ask, Why a desktop browser?
To which I would reply, Why not a desktop browser?
I blogged previously about how Google and Bing use localStorage on mobile devices. It’s a powerful performance technique. They break up their JavaScript and CSS into smaller blocks and save them in localStorage. Simultaneously they set a cookie so that the server knows not to send that payload on subsequent searches, a savings of 150-170 kB before gzipping.
In the comments on that post Simon asked:
I’m curious why the techniques with LocalStorage are used for the mobile versions of the search sites but not for the standard desktop versions – I would think that this technique would work well [on] all clients, no?
I agree – this and other HTML5 web storage techniques make sense for the desktop, too. There are some reasons why we’re seeing these used first on mobile:
- Mobile latencies are higher and connection speeds are lower, so clientside caching is more important on mobile.
- Mobile disk cache sizes are smaller than desktop sizes, so a better alternative is needed for mobile.
- There are still desktop browsers with significant market share that are missing many HTML5 capabilities, whereas mobile browsers have more support for HTML5.
Even though the motivation for app cache and localStorage are stronger for mobile, they should also be used when the website is accessed from a desktop browser. I did a quick survey to see which of the top 10 websites were using app cache or localStorage on either mobile or desktop. Here are the results:
| Website | Mobile | Desktop | ||
|---|---|---|---|---|
| app cache | localStorage | app cache | localStorage | |
| Google Search | no | yes | no | yes [1] |
| Bing | no | yes | no | no |
| no | yes | no | no | |
| Yahoo! Front Page | no | yes [2] | no | no |
| YouTube | no | yes | no | no |
| Amazon | no | no | no | yes [3] |
| yes | yes | no | yes | |
| no | no | no | no | |
| eBay | no | no | no | no |
| MSN.com | no | no | no | no |
[2] Yahoo! Front Page only stores two numbers.
[3] Amazon on the desktop only stores a ~30 character string.
Ignoring the small uses of localStorage, 5 of these top 10 websites use localStorage on mobile, but only 2 out of 10 use localStorage (or sessionStorage) on desktop. None of them use app cache, either on mobile or desktop except for Twitter on mobile. I’m surprised no one is using app cache. It’s not appropriate for all applications, such as search, but I would enjoy catching up on Twitter, Facebook, and LinkedIn on the plane – potentially from my laptop in addition to my mobile device. App cache not only brings this offline capability, but provides better caching than the browser’s disk cache.
I’ll repeat this survey in a few months to track the progress. I expect we’ll see the use of localStorage and app cache increase, and for desktop to catch up to mobile.
Manuel Deschamps | 26-Sep-11 at 10:21 pm | Permalink |
mobile twitter uses app cache and local storage even for
SSL
Rakesh Pai | 26-Sep-11 at 10:22 pm | Permalink |
I wouldn’t use appcache on the desktop because of the UX problem with Firefox, where the user is warned that the site is trying to store something on disk. That’s a scary looking warning, requires authorisation, and gives no immediate benefit to the user. To avoid the warning, I’ll have to UA sniff on the server and not serve the manifest attribute to Firefox. That’s just ugly and fragile.
I’m pretty sure that if Firefox stops showing that warning, more sites will use appcache. That’s my guess, at least. Sandboxing is the browser’s responsibility, and asking the user a scarily worded question doesn’t help anyone.
If you think about it, storing files in the appcache is not too different from storing cookies, but if the browser always asked if you want to allow the site to store data on disk everytime a cookie was set, the adoption for cookies would definitely drop.
Steve Souders | 26-Sep-11 at 10:37 pm | Permalink |
@Manuel: Are you using an app or browser? I’m on iPhone4 in the native browser going to http://twitter.com/ (then many redirects). I see a small amount of data in localStorage (2 items, 7kB) and didn’t see anything in app cache. How did you examine app cache?
@Rakesh: Great point about the warning in Firefox. There are other usage and design flaws in app cache. Certainly adoption will improve if these issues are addressed.
Davide Mendolia | 26-Sep-11 at 11:46 pm | Permalink |
@Steve,
Your are mentioning, that mobile latencies are higher, do you have seen some studies or benchmark ?
I was trying to reproduce it, comparing 3G connection and WIFI, I didn’t get any data validating or invalidating this.
Morgan Cheng | 26-Sep-11 at 11:50 pm | Permalink |
Before leveraging localStorage and app cache, there are basic performance techniques should be taken. Twitter doesn’t minify its hundreds-of-K bytes javascript. Surprise?!
Dave Orchard | 27-Sep-11 at 12:06 am | Permalink |
App cache breaks very hard if there’s a network failure. Most common reason I’ve heard, and why we don’t use it. It’s just not a good enough spec yet.
Patrick Mueller | 27-Sep-11 at 2:59 am | Permalink |
App cache is hard to develop for; you twittered yourself some issue about “clearing” the appcache or something a while back. Need better tools to get more control over it.
Sadly, if no one is using it, because it’s hard to use, then no one is going to be complaining about it either.
How are we going to fix this?
Rajaram | 27-Sep-11 at 4:14 am | Permalink |
One problem with appCache is that it caches master page too. So difficult to use in normal caching.
As the intent of appCache is offline availability, its only good for that.
Any good workaround?.
Pete | 27-Sep-11 at 7:42 am | Permalink |
Hey Steve,
I’m curious if you’ve looked at how or if AppCache affects performance of an application – not just the offline experience? If apps are making requests and pulling content from cache instead of from the network, I’d assume that there’d be a pretty big performance increase.
Steve Souders | 27-Sep-11 at 8:53 am | Permalink |
@Davide: Rajiv Vijayakumar (Qualcomm) gives some mobile latency stats in his talk from Velocity: Understanding Mobile Web Browser Performance. He doesn’t compare to wifi though.
@Patrick: The W3C is holding a Workshop on The Future of Off-line Web Applications on Nov 5 in Redwood City to discuss app cache. You and I will be in Berlin for Velocity Europe, but I encourage others to attend. Everyone should send your thoughts about app cache’s issues to them at team-uwa-review@w3.org.
@Rajaram: You could try the Fallback: section to see if you can NOT use the cached version of the master page when online.
@Pete: Yes, I encourage the use of app cache from a performance perspective because resources are read from cache. There are several issues, as raised in the comments here.
Aaron Barker | 27-Sep-11 at 9:40 am | Permalink |
@Davide I found these slides the other day that had some wifi vs mobile latency testing. See slide 7 for final results http://davidbcalhoun.com/present/mobile-performance/
Rajaram | 27-Sep-11 at 11:27 am | Permalink |
@Steve: My problem is if I include cache.manifest in a dynamic page the page gets cached along with files mentioned in manifest file and same page is served again and again if requested, even when page is changed. Only way around it looks like changing cache.manifest.
What I want is, only urls mentioned in cache.manifest file to be cached and other (including master) to be fetched from network when i’m online and served from cache if offline. (Is this possible at all? ).
Manuel Deschamps | 27-Sep-11 at 4:53 pm | Permalink |
@Steve: I examine the appcache using Desktop Safari (for debugging). if you look at the source of https://mobile.twitter.com, the html tag contains the manifest attr ()
You can see the manifest here: https://mobile.twitter.com/cache/manifest
Because of the redirects, I dont use the app cache for an “offline” client, I use it more for better caching of our assets and faster downloads.
Andy Walpole | 27-Sep-11 at 6:31 pm | Permalink |
“They break up their JavaScript and CSS into smaller blocks and save them in localStorage.”
How do they do that? Is there a tutorial about for this technique?
Niklas | 27-Sep-11 at 9:28 pm | Permalink |
@Manuel There might be a problem with the redirects. The 302 redirect from http://mobile.twitter.com to https://mobile.twitter.com causes the app cache to never be used. It will download all from server each time. This is a bug in Webkit (Fixed in latest versions, but that is not included in iOS yet)
We recently discovered this same thing for Yahoo! mail, and you can work around it with using other forms of redirects.
Andrew Hedges | 28-Sep-11 at 9:27 am | Permalink |
Wow, a lot of FUD here about HTML5 manifest caching (app cache). Yes, it’s a bit tricky to work with (e.g., a single syntax error in your manifest == nothing gets cached), but once you understand how it works (i.e. how it is intended to work and the actual implementations in the wild), it is a very useful tool. It’s not the perfect solution for every use case (no technology is), but don’t knock it until you try it.
Steve Souders | 28-Sep-11 at 11:49 am | Permalink |
@Aaron: Great pointer to those slides. Thanks.
@Rajaram: Again, let me suggest you try the Fallback section. Have you done that? Does it not work?
@Manuel: I’ll reply offline. We probably have testing issues.
@Andy: No OTS solution. This doesn’t seem complex. The answer varies depending on how often you change your JS & CSS.
@Andrew: FUD isn’t an accurate assessment of these issues. The behavior and issues are well documented. The world’s top mobile developers (Dion Almaer, Yehuda Katz, Scott Jehl) agree there are issues. You might be over the hump, but it’d be better to remove the hump for easier adoption by experts and non-experts alike.
Steve Souders | 28-Sep-11 at 4:56 pm | Permalink |
@Manuel: You’re right! I was able to confirm Twitter uses app cache on mobile. I confirmed the use of app cache in two ways, both of which are flawed:
1. I look at the document source using the Docsource bookmarklet which accesses documentElement.innerHTML. Unfortunately, documentElement doesn’t exist in mobile Safari, so I fallback to getElementsByTagName(‘head’)[0].innerHTML and document.body.innerHTML and thus am not able to see the HTML tag.
2. I spoof the UA on Chrome and look at the document source. Manuel pointed out that, for some reason, doing the UA switch only works in Safari. I tested it in Safari and do see the MANIFEST attribute.
Also, I realized I can confirm this by looking at mobile Safari’s settings under Safari -> Databases and see who uses app cache. I doubled check and indeed Twitter uses app cache.
Rajaram | 28-Sep-11 at 11:44 pm | Permalink |
@steve: Fallback is not working for me as expected.
Trunal | 29-Sep-11 at 10:15 am | Permalink |
@steve, LinkedIn mobile browser app uses localStorage for caching js and css. You can verify that by spoofing the UA in Desktop Safari and examining the localStorage in inspector.
Steve Souders | 29-Sep-11 at 10:30 am | Permalink |
@Rajaram: Unfortunate. A different approach would be needed – iframes or something. This is a major problem with app cache.
@Trunal: In the native browser on iPhone I logged in to LinkedIn – they don’t use localStorage.
Trunal | 29-Sep-11 at 10:46 am | Permalink |
@steve,
Sorry I should have been clearer in my previous comment, you need to login to http://touch.linkedin.com on your native iphone browsers to get the iPhone compatible version. The automatic redirect is not in place yet.
louisremi | 04-Oct-11 at 2:26 am | Permalink |
@Rakesh @Steve Mozilla is working toward removing the user prompt for appcache: https://bugzilla.mozilla.org/show_bug.cgi?id=648064
Rajaram | 04-Oct-11 at 5:07 am | Permalink |
@steve : I tried iframe, but its not working (checked on chrome). To me it looks like appCache best meant for single-page-apps.
Leonardo | 19-Mar-12 at 12:55 am | Permalink |
Hi Steve, I saw on gmail mobile code, that when he doesn’t take the CSS from the localStorage it take through a javascript that holds an object with the css string. Of course there is a reason for that. Do you know why?
Thanks Steve
Steve Souders | 19-Mar-12 at 7:25 am | Permalink |
@Leonardo: The CSS needs to be in a string eventually so it can be written to localStorage. Perhaps they found it easiest to start with it in that state for both inlining and later storage.