Infrastructure upgrades and migrations
The last 6 months, we have been extraordinarily busy with a major infrastructure upgrade that will make Clicky much faster and more resilient. This is by far the most major backend upgrade in our (almost) 10 year history. A lot of planning and testing has gone into this and we're excited to finally be under way with the database aspect. Our load balancers have already been on the new infrastructure for a month now, and web and tracking servers will follow up after the databases are done.
The database maintenance in late April was in prep for this migration, to make it as fast as possible. This weekend we will be doing 8 database servers - 6, 22, 28, 29, and 59-62. And each weekend thereafter we plan to do at least 20 more. We have 85 total database servers at the moment so we should be done or at least very close to it by the end of June.
Time-wise, migrations are similar to the database maintenance we do a few times a year. Most servers will be in the 4-10 hour range, but some of the bigger/oldest ones will take as long as 24 hours. As well, traffic processing unfortunately has to be halted during the migration. But these new servers scream and will catch back up to real time in no time at all once the migration is done.
We have gone out of our way to make this process as transparent as possible. When a db server is migrating, your affected sites' dashboards will show a message like this:
![](//web.archive.org./web/20161121061105im_/http://con.tent.network/media/screenshots/201606-db-migrate.png)
The message shows the time the migration started, and an ETA for when it will be done (both in your local timezone), based upon how many rows/second we know these go on average and how many rows total there are to migrate.
Once a server has been migrated, reports should load much faster than before. We are planning to increase the filtering/segmentation limits as well once this is all done, as those should have much less of an impact on real time processing.
We're also excited to be done with this because it means we can get back to the software side of things, which is our real passion.
Hope everyone has an amazing summer!
2 comments | Jun 12 2016 12:43pm
The database maintenance in late April was in prep for this migration, to make it as fast as possible. This weekend we will be doing 8 database servers - 6, 22, 28, 29, and 59-62. And each weekend thereafter we plan to do at least 20 more. We have 85 total database servers at the moment so we should be done or at least very close to it by the end of June.
Time-wise, migrations are similar to the database maintenance we do a few times a year. Most servers will be in the 4-10 hour range, but some of the bigger/oldest ones will take as long as 24 hours. As well, traffic processing unfortunately has to be halted during the migration. But these new servers scream and will catch back up to real time in no time at all once the migration is done.
We have gone out of our way to make this process as transparent as possible. When a db server is migrating, your affected sites' dashboards will show a message like this:
![](http://web.archive.org./web/20161121061105im_/http://con.tent.network/media/screenshots/201606-db-migrate.png)
The message shows the time the migration started, and an ETA for when it will be done (both in your local timezone), based upon how many rows/second we know these go on average and how many rows total there are to migrate.
Once a server has been migrated, reports should load much faster than before. We are planning to increase the filtering/segmentation limits as well once this is all done, as those should have much less of an impact on real time processing.
We're also excited to be done with this because it means we can get back to the software side of things, which is our real passion.
Hope everyone has an amazing summer!
![](/web/20161121061105im_/http://clicky.com/blog/media/comments.gif)
Recent Comments
That's correct. Yes, the new servers are wicked fast. Your user page will load much ... Sean (Clicky), Jun 14 2016 So, to be clear, when a DB server is being migrated there will not be any data loss ... Shaun, Jun 14 2016 I forgot to mention in the post, the "Pages" and "Query Parameters" filters were ... Sean (Clicky), Oct 22 2015 Yes, I've played around with the idea but I haven't come up with a way to put it ... Sean (Clicky), Oct 22 2015 Have you considered creating a button/link in the app to report spam-referrers? Jacob, Oct 22 2015 great concept to go with your great product, thanks Anthony, Oct 21 2015 Excellent work guys! Looks great and it must be rewarding to see this A+ rating for ... Beuk, Oct 09 2015 Nginx does wonders for you. I am glad you guys managed to move and reduce costs ... Angelica, Oct 06 2015 Is there a way to do this with Uptime? Ringo64, Jun 16 2015 Hi Sean, Great, thanks for the heads up I wasn't aware of this option :) This ... Chris, May 10 2015