Google Serious Security Breach - Final Update - 25 July 07

Dear all,

Since last Saturday after Google Reader was patched (need confirmation from Google whether the Google Reader or local ISP proxy/cache played a bigger part in this security problem), I have not noticed any more security glitches. Two other Singapore users who contacted me also reported no more security issues since then. So, we have enough reasons to think that the security issues related to what I have reported have been eliminated.

So in this final update article, I will just document some of the information which I have missed out from my previous posts, particularly:

• Background articles (so that I don't have to repeat again)
  
• The common steps that I clicked through triggering the session-crossover
• List of services breached, which were affected, which were not, and to what extend
• Additional findings & references
• Personal view
  

Background articles:

• Google Reader Problem
• Google Serious Security Breach - update 1 - 20 July 07
• Google Serious Security Breach - update 2 - 20 July 07
• Google Serious Security Breach - update 3 - 20 July 07
• Google Serious Security Breach - update 4 - 22 July 070

The common steps:

I document this because I think it may be helpful for Google or anyone else who would like to have a better picture how it all occurred.

• Login (either of the steps below):
  
• using gmail.com
• using Google Toolbar
• via Bookmarks
• via Gmail
  
  
• Access Gmail
  
  
• From Gmail top left menu, select more -> Reader
  
  
• Access Reader
  
• If session-crossover hadn't occurred, click Settings (top right)
  
• from Reader Settings, click top left corner Google Reader Labs Logo instead of << Back to Google Reader, most of the session-crossover occurred after this, else, from Reader, click My Account (top right), if it occurred at this point, you would be asked to fill in your account password (which shouldn't happen when single-sign-on was working properly), repeat these steps if none had happened so far, if after 3 iterations and no crossover occurred, it was less likely to happen in this particular session.
  
  
• When it happened, I would notice that my account id jvyloh@gmail.com changed to other users. As mentioned, out of the popular 5, most likely it was victor.xxx.xx@gmail.com. I will touch on this later because another user from Singapore mark...@gmail.com contacted me telling me he did crossover to the same victor.xxx.xx@gmail.com a couple of times.
  
  
• When crossover occurred, you were literally in another user session rather than yours. In Reader, I would notice that none of the subscriptions were mine.
  
  
• At this point, new subscriptions could be added. Existing subscriptions could be deleted. One could add a new subscription to your own blog, rendering artificial visitor statistics. One could manage the subscriptions, performed rename, delete, change folder, new folder, add/remove tag, unsubscribe, delete all subscriptions, etc...
  
  
• One could view the trend in the other user's Reader Trend, got a feel of the user reading habit, like which topics were read, when most reading occurred in a day or week.
  
  
• One could mark certain items as shared, star/unstar them, add tags.
  
  
• The most serious violation was possible as one could open an item and click Email(Updated), a small ajax layer would pan out, one could gmail to arbitrary emails using the other user's account, click the email to: field, start typing from alphabet a to z to have a drop down of a list of Gmail contacts, send out the email with a customized note without having to save a copy in the Gmail sent box. The recipient would get that email.
  
  
• Sometimes the session would suddenly crossover to a different user, and one would be seeing things from the 'new users'. I suspect that most of this session-crossover happened in between users who were online and login to Google account at the same time. Besides the security bugs that Google Security Team had fixed with their Google Reader, whether there were other factors which contributed to this (e.g. ISP Proxy/Cache Server), was beyond me. Google did ask me about my ISP. More on that later.
  
  
• Access iGoogle:
  
• From Google Reader, clicked My Account(top right), single-sign-on wouldn't work, and I would be shown with my email id and prompted to enter password. At this point, even with my correct email id and password, I wouldn't be able to login most of the time.
  
  
• From here, click iGoogle, the iGoogle page would appear. For a couple of days, what happened was that I would be presented with whatever the user had added to their iGoogle.
  
  
• At this point, one could change the iGoogle theme, add stuff, re-layout gadgets, one could possibly add some of the more critical gadgets like Gmail, Google Docs, Google Calendar, etc. The list of Google services that could be added include: Google Finance Portfolios, Gmail, Google Calendar Viewer, Google Notebook, Googe Docs & Spreadsheet, Google Talk, Google Reader, My Google Groups, Google Map Search, Picasa Web Photos, etc...
  
  
• One of the most serious security breach I have encountered, I was crossovered to 1 such user who had already added his/her Gmail and Google Docs in iGoogle. The Gmail and Google Docs gadgets allowed one to change the number of messages to display up to a total of 9. So a summary of 9 Gmail messages would be presented. One couldn't have further access to Gmail services without additional login though. However, for the Google Docs gadgets, the security breach was the worst. Instead of just the docs title, one could click on the docs and had them open up in full editable mode. The whole document was exposed. Fortunately further access to Google Docs Home wasn't possible. Some other ToDo gadgets could be edited as well.
  
  
• Fortunately these didn't happen all the time to all the session-crossover. Most of the time when I was redirected to iGoogle, I would be redirected away 50% of the time, especially at the later stage, nearer to the date of Google solving the bugs.
  
  
• By clicking around Google services, one could easily figure out what services were granting access and what not. The fact was, one didn't have to do anything much, just clicking around the usual way you switching around Google services (when you were a frequent user of Google), was enough to send you roaming into other user's account, although we wouldn't want to, including services available through Google Toolbar and Firefox extension, e.g. Google Notebook.
  

List of services breached, which were affected, which were not, and to what extend:

• Google Reader - Action/Access Possible (to 3rd party):
  
• Access subscriptions (view, read, add)
• Manage subscriptions (delete, delete all, rename, change folder, new folder, add/remote tag, unsubscribe)
• Access Google Trends (view user reading habits)
• Individual Reader Items (share/unshare, star/unstar, add tags)
• Access Gmail contacts from Reader (view the whole list of contacts from a-z)
• Send item to arbitrary emails and attach a note, without saving in user sent box
  
  
• Google Reader - Action/Access Not Possible (to 3rd party):
• None
  
  
• iGoogle - Action/Access Possible (to 3rd party):
• View existing gadgets
• Change gadgets settings (e.g. number of items/summary to show)
• Add/remove gadgets
  
• Change theme, layout
• If not redirected away, Gmail could be added and up to 9 messages title would be available
• If not redirected away, Google Docs could be added and up to 9 docs title would be available. Individual docs could be open for editing, deletion and save.
  
  
• iGoogle - Action/Access Not Possible (to 3rd party):
• Full access to Gmail not possible.
  
• Further access to Google Docs Home not possible.
  
  
• Google Web History - Action/Access Possible (to 3rd party):
• Trends
• Interesting Items
• Bookmarks
  
  
• Google Web History - Action/Access Not Possible (to 3rd party):
• All History (Web, Images, News, Products, Sponsored Links, Video, Maps)
  
  
• Google Bookmarks (Toolbar/Web) - Action/Access Possible (to 3rd party):
• View, add, edit, remove, labeling, export
  
  
• Google Bookmarks (Toolbar/Web) - Action/Access Not Possible (to 3rd party):
• None
  
  
• Google Notebook(FireFox Ext/Web) - Action/Access Possible (to 3rd party):
• New note, delete note
  
  
• Google Notebook(FireFox Ext/Web) - Action/Access Not Possible (to 3rd party):
• None to my knowledge
  
  
• Google Groups - Action/Access Possible (to 3rd party):
• Access to My Groups and Recently Visited
  
  
• Google Maps - Action/Access Possible (to 3rd party):
• Access to Saved Location
  
  
• Google News / Finance - Action/Access Possible (to 3rd party):
• Viewable
  
  
• Gmail (Toolbar/Web) - Action/Access Possible (to 3rd party):
• Sending to arbitrary emails from Google Reader
• Can be added to iGoogle, summary viewable if not redirected away from iGoogle
  
  
• Gmail (Toolbar/Web) - Action/Access Not Possible (to 3rd party):
• Full access requires additional login
  
  
• Google Calendar (Toolbar/Web) - Action/Access Possible (to 3rd party):
• Can be added to iGoogle, not sure about summary access from iGoogle
  
  
• Google Calendar (Toolbar/Web) - Action/Access Not Possible (to 3rd party):
• Full access requires additional login
  
  
• Google Talk - Action/Access Possible (to 3rd party):
• Can be added to iGoogle, not sure about summary access from iGoogle
  
  
• Picasa Web Photos - Action/Access Possible (to 3rd party):
• Can be added to iGoogle, not sure about summary access from iGoogle
  
  
• Google Analytics, Blogger, Orkut - Action/Access Not Possible (to 3rd party):
• Full access requires additional login

Additional findings & references:

• As mentioned, the behaviors of the problem this round was different from the latest reported earlier this year:
  
• Details of Google’s Latest Security Hole
• Google Security Hole Allows Account Hijacking
  
  
• However, it might be slightly better than the previous case because one couldn't choose which user session to crossover to.
  
  
• But, it could be even worse solely because no known exploits had occurred previously whereas for this round, anyone that encountered a session-crossover could immediately perform serious damages to the other party (e.g. delete all the valuable bookmarks)
  
  
• The occurrence was particularly obvious in Singapore. It happened in a span of days to weeks. Especially when another user who read my blog emailed me that he also crossovered to the same user as mine.
  
  
• A few sources, including Tony Ruscoe, Ionut Alex Chitu from Googlesystem, Matt Cutts from Google, Google Security Team had first suspected a problem with ISP proxy/cache server, straydog from Singapore was having the same issue, both of us were using SingNet, an ISP belonged to SingTel who was launching a nationwide IPTV service in Singapore the same week this problem intensified, it fact, it was officially launched the same day (Friday) I reported these problems, refer here.
  
  
• Mark from Singapore and myself crossovered to the same user couple of times, the user was from Singapore too. I had in fact contacted the user victor.xxx.xx@gmail.com informing him that I had crossovered to his session and some modification I was making to myself were being reflected in his Google account.
  
  
• On 20 July, Mark emailed me: Yes, I'm in Singapore, and SingNet is my ISP. And I'm still crossing over other users.
  
  
• On 20 July, Tony Ruscoe wrote to me: I'm guessing you are connecting through an ISP that uses a proxy server.
  
  
• On 20 July, straydog left me a comment: I am a singtel user and since this morning I am experiencing the same thing as you with the reader. Gmail seems ok though.
  
  
• On 21 July, Ionut Alex Chitu from Googlesystem wrote a blog about this. Some of the comments in the blog were compiled below:
  
  
• Other Google user complains over at Google Groups: "Whenever I use Google Reader, I would 'cross-over' to another user's account."
  
  
• And another one: "I've been login to other users today, seeing their feeds instead of mine. I login to gmail and google reader. While reading the feeds halfway I would see my feeds change into other user's [feeds], my account will also change to other google user account."
  
  
• Other report from a regular reader of this blog: "While I was reading posts in Google Reader today, my account was switched to someone else's account. The account name on the upper right corner changed and I could see all his or her subscriptions in my Google Reader. I closed the Reader and open it again. Nice! I could read another person's subscriptions. I tried iGoogle and it was also changed."
  
  
• Update. Matt Cutts, from Google, posted this: "Given that most of these reports are coming from a single area (Singapore), it sounds like an ISP isn't handling their connections correctly. We've certainly seen ISPs mess up their proxies before. I'll still ask about this though."
  
  
• Adam Boeglin said on July 20, 2007 6:29:00 AM PDT:
  I had the same problem once in Google reader. I hit refresh, and it showed some different feeds. I refreshed the whole page, and it was someone else's account. After restarting Firefox, everything was back to normal. It was very odd.
  
  
• Tegla said on July 20, 2007 6:49:00 AM PDT:
  If I were you, I would definitely have a look at whether the university has some badly set-up proxy...
  
  
• Gustavo Caixeta said on July 20, 2007 7:09:00 AM PDT:
  i had the same problem at google groups at 29/06/2007
  it havens when i was managing the users of my group
  
  
• Mousuke said on July 20, 2007 7:21:00 AM PDT:
  I'm one of those being affected by this problem. I think it might be my ISP, because once I reached work, I have none of those problems, but I continue to have the problem at home.
  While I am from one of the named Singapore universities, I was not using the varsity connection, so I wouldn't know if it was in fact, isolated there.
  
  
• straydog said on July 20, 2007 7:30:00 AM PDT:
  I am this problem a couple of times already today on the reader. And yes, I am from Singapore.
  Can this problem be from the ISP?
  
  
• Jim LoVerde said on July 20, 2007 7:49:00 AM PDT:
  I have had this problem happen starting yesterday at a client site. If it's not on Google's server side (e.g. giving out redundant cookie session ids).
  I'd highly suspect that it is proxy server related. Either Google changed some of their cache directives, or some proxy servers have changed their behavior for some reason.
  In this case, the proxy server is Novell Border Manager, if that helps.
  It's also worth noting that it happened twice, and both times I ended up looking at the wrong Google Reader feeds but for the same user.
  
  
• Choo said on July 20, 2007 9:10:00 AM PDT:
  this is really bizarre, my friend ended up in my account!!
  luckily, he did not modify my feeds, though he did screw up some other guy's iGoogle... :(
  
  
• Justin said on July 20, 2007 10:10:00 AM PDT:
  I'm from Singapore too. I was using NUS network when I got the problem.
  But, on my IE, I have a different account other than the one that encountered the problem. When I used that account, there was no problem at all, even at the same time when I was having the same problem in the other account in Firefox! I logged off the problematic account in Firefox and logged in with the account from IE, there was no problem. I logged in again with that problematic account (also in Firefox) then the problem came back again!!!!
  I still don't know what had caused this problem. My testing with two accounts further confused the problem.
  
  
• Matt Cutts said on July 20, 2007 10:22:00 AM PDT:
  Given that most of these reports are coming from a single area (Singapore), it sounds like an ISP isn't handling their connections correctly. We've certainly seen ISPs mess up their proxies before. I'll still ask about this though.
  
  
• Matt Cutts said on July 20, 2007 11:41:00 AM PDT:
  Both the security team and the Google Reader team were already on it. Even though it's not on Google's side as near as we can tell, the Reader team are looking for a way to prevent faulty proxies/caching from affecting users.
  
  
• Brian said on July 20, 2007 1:53:00 PM PDT:
  It's totally a proxy problem...this used to happen when I was at a university and I would login to Yahoo! only to get someone else's Yahoo! account.
  
  
• ymerej said on July 20, 2007 8:17:00 PM PDT:
  This used to happen on my computer. I would log in to my "igoogle" back wen it was called Google Home Page and I would get my igoogle but with my girlfriends mail. She had the same problem. I wrote google but they didn't help except told to be sure we logged out.
  
  
• yasart said on July 21, 2007 1:30:00 PM PDT:
  I am from Turkey. I had this problem with IGoogle. Someone else's homepage is presented instead of mine.
  
  
• yasart said on July 21, 2007 1:30:00 PM PDT:
  I am from Turkey. I had this problem with IGoogle. Someone else's homepage is presented instead of mine.
  
  
• yasart said on July 21, 2007 1:30:00 PM PDT:
  I am from Turkey. I had this problem with IGoogle. Someone else's homepage is presented instead of mine.


• On 21 July, Google Security contacted me to ask me to furbish more information about my ISP.
  
  
• On 21 July, 3 and a half hours later, Google Security emailed me:
  
  Hi Jvy,
  We believe we've fixed the problem on our end for Google Reader. Please
  let us know specifics if you notice it happening again (on reader, or any
  other Google properties). Thank you again for reporting this to us.
  
  Regards,
  Will, The Google Team
  
  
• I did some testing and found some javascript error, so I feedback to them. Since then, no more problem occurred.
  
  
• On 22 July, Mark from Singapore emailed me: It seems that the problem has been resolved around sat night, sometime before midnight SG time.. No problems thus far.

Personal View:

• So far, I believe this problem has been fixed by Google, either that there was some bugs with Google Reader (which is still in lab version), or that Google made some changes to Google Reader so that it can better workaround some of the ISP proxy/cache server problems. May be Google can further clarify on this.
  
  
• If it was indeed a ISP proxy/cache server problem, we have gathered that at least 3 users (including myself) in Singapore had confirmed to be using SingNet, an ISP from SingTel (Singapore Telecom), who has just launched IPTV service in the same week (Friday) where the problem peaks. See the CNET article about the launch. Is SingTel IPTV service part of the contributor to this problem? If so, we have seen Google done their part, but has the Telco done anything yet? It may be affecting other services, including 1 yahoo service reported by 1 such user in googlesystem.blogspot.com.
  
  
• All software system are subjected to bugs especially a lab version, all trial user have to be aware of that bugs are inevitable at this level of release.
  
  
• I am highly impressed with Google's response time. They got the problem fixed within 3 and a half hours. My confidence with Google stays strong.

If you have any comments, do drop it here or email me: jvyloh@gmai.com. If you had this problem or still having this problem, do let us know too.

Feel free to link to this article.

Regards,
Jvy