By David Ackerman
December 14, 2011
On May 21, 2011, NYU successfully delivered Google Apps for Education to all students, faculty, staff, and alumni. Negotiating a contract that ensured the NYU community's privacy, security, legal compliance, and business continuity took years to hammer out, but the massive migration project was accomplished in a mere four months!
When the project officially kicked off, we first had to address the daunting challenge of copying 235 million e-mail messages—14 terabytes (14,680,064 megabytes) of data, stored in over 135,000 mailboxes—from NYU's old e-mail system to the new Google Mail servers. The three consulting firms that Google identified as possible partners for the project initially said it would be impossible. The main challenge lay in the fact that the messages needed to be transferred in a matter of days. A tight window was necessary because during the process of copying the messages to Google's system, community members needed to continue receiving, sending, and deleting messages. If we took too long to complete the first pass of copying the mail to Google's servers, it would take too long to go back and copy the changes in a second pass before too many new changes had occurred, requiring a large third pass, and so on.
Two of the consulting firms we spoke with suggested that we create a self-service mechanism for individuals instead of moving any of their mail for them. When we estimated that it would cost NYU more than a million dollars in lost productivity if each employee spent a few hours at this task, it was clear that we needed to find a better way. The other consulting firm, Appirio, won the bid by agreeing to tackle our goal of moving the mail. Though Appirio suggested that we only migrate NYU employees' mail, we weren't satisfied with leaving students to fend for themselves, and set out to solve the problem of how to migrate all NYU's mail.
One of the chief obstacles was that Google would only allow us to copy one message per second, per user. With 350 million messages, it would take at least 350 million seconds if we migrated community members one at a time—11 years! (And that's assuming we could actually copy one message each second, which wasn't possible for messages with large attachments.) Clearly, we needed to copy more than one person's mailbox at a time. But how many people could we migrate at the same time without bringing the production mail server to its knees?
In February, we began copying multiple mailboxes simultaneously on a test server. We started small and gradually increased the scale of the test: 100 mailboxes, one at a time, then 1,000 mailboxes, ten at a time. We eventually tried 10,000 mailboxes, 100 at a time; when that test was a success, I knew we were ready to tackle all 135,000 mailboxes.
We started timing the test runs, with the goal of being able to copy all the mail in under ten days. We were able to achieve sustained copying of one message every 1.2 seconds. At that speed, we determined that we could move the messages quickly enough to avoid major issues with changes to the mailbox during the transfer process (known as "deltas").
Another issue we faced was individuals with large inboxes. We planned to copy the messages in "streams" of 100 mailboxes at a time, and the messages for each individual had to be handled within a single stream. The largest mailbox was over 20 gigabytes, with about 300,000 messages. At one message per second, that one mailbox would take more than three days to copy! So that each stream wouldn't get bogged down by these large mailboxes, we figured out a way to handle them separately.
In the end, we used six powerful servers, each running five virtual machines, for a total of 30 virtual machines (each similar to an individual computer). Each virtual machine had 50 streams of mail, for a total of 1,500 simultaneous streams. After nearly three months of testing, we launched this mammoth effort on May 1, 2011—within nine days, we successfully made it through the first pass of copying every single mailbox.
With this huge success under our belt, it was just a matter of copying over the "deltas." We continued to run additional passes to copy these changes until we were ready to go live with the entire NYU Google Apps for Education suite early on May 21. From then on, all new mail was delivered to and sent from Google's server instead of NYU's.
For the NYU community, it may have appeared that ITS simply "flipped a switch" and NYU E-mail was magically moved to Google. In fact, it took a giant effort on the part of many team members to create this illusion.
Ten years ago, we moved NYU E-mail—all 73 gigabytes of it—from nine individual servers (is.nyu.edu through is9.nyu.edu, for any longtime NYU community members who might recall them) to the new mail.nyu.edu server. This required no less than nine e-mail outages—most of which lasted an entire evening. This May, we moved 200 times as much mail without a single outage. That's progress.
David Ackerman is Associate Vice President of ITS .edu Services and Executive Director for Digital Library Technology Services in NYU Libraries at New York University.
This Article is in the following Topics:
Connect - Information Technology at NYU