Yesterday we pushed up a major update to how tracking works. The purpose of these changes is to make the code work even faster and smoother behind the scenes. 24 hours later, no sneaky issues have cropped up, so we're officially announcing it now.

We've migrated most tracking-related beacons to use navigator.sendBeacon instead of appending a [script] element to the HTML document. We've had our eyes on this for quite a while as its purpose is literally for making tracking faster and more reliable, but there's been very little discussion of it anywhere even though most browsers have officially supported it for years now. But earlier this year, Google announced "parallel tracking" for Google ads, which uses sendBeacon instead of redirects to track clicks on ads. Google relying on it for their revenue backbone was a sign that it was ready.

Unfortunately we had to keep the old [script] method for the initial page view, because it echoes back javascript code that needs to be executed in the scope of the page being tracked, e.g. on-site analytics, heatmaps, setting first-party cookies for inactive sites that tells our code to stop logging traffic - this isn't possible with sendBeacon, which works outside of the page scope and is a downside in this instance. But everything after that - pings, events, goals, heatmap data, clicks on outbound or download links - uses sendBeacon, which is pretty noice.

The [script] element method has also itself been updated though so that the [script] elements are appended to document.body instead of document.head, which should have a big impact to help speed up that initial page view tracking. This change should have been made ages ago but it wasn't until I was knee-deep in this upgrade that I learned that dynamically injected [script] elements still block, even if you create them with the 'async' parameter. They shouldn't, but they can and do. Sigh.


sendBeacon = No more pauses on download and outbound links
sendBeacon works outside of page scope which, as mentioned above, has one major downside for us. But the major upside is its reliability. sendBeacon works in the background even after a page has unloaded, so tracking should be much more accurate and reliable. We've entirely removed the 500 millisecond 'pause' that we were creating for tracking clicks on download and outbound links, which was needed to give enough time (most of the time) for a round trip to our tracking servers to log the click before the page was unloaded. It was a necessary evil 10 years ago when we added this feature.

How much of humanity's time have we collectively cost because of this pause? I'd lowball estimate we've tracked an average of 50 million clicks per day on download or outbound links over the last 10 years since we added this feature.
50M * 365 * 10 = 182,500,000,000 clicks x 0.5s = 91,250,000,000 seconds = 1,056,134 days = 2,893 years. Traveling back in time that many years would land you in the 9th century BC where you could travel to central Europe and witness the beginning of the Iron Age. Dude.


The code you copy and paste onto your site has also been changed. Instead of 3 versions of the code, there is now just 1, and it's fully asynchronous by default. And it's much simpler and more reliable than the old version of the (optional) asynchronous code. This one just uses a normal [script] element with the 'async' parameter embedded directly in it. It's now 2 lines instead of ~10.

Reinstalling the code is optional, but we do recommend it to get the full benefits. Also now we recommend the code go into your [head] element instead of at the bottom of the [body] since the default code is now fully asynchronous and does not include the badge or noscript image elements (which can be added back with the checkboxes, if you add those in then put the code in [body] instead).

As promised, by May 25 we are releasing tools to help you find and then export or delete personal data requested by your end users.

We cleaned up the user homepage a bit as you can see below. If there's a new blog post within the last 30 days, the link/title is in the top right corner. This gives more room to the main links on the left, which are now in a single row instead of 2 rows. A bit easier on the eyes. And the link to manage personal data requests is in the "Your web sites" area. Couldn't come up with a good icon for it since we already had a "person" icon for the sub-users link, so we just used a turtle which we had lying around because hey it's cute.





The personal data management page has some brief info at the top about the purpose and how to use it, then there's a form below that lets you search based on UID (tracking cookie), IP address, or a custom data tag if you're using that feature. The screenshot below shows the search results for a UID that I entered into the form at the bottom.




You can click on any of the session IDs to view the session, or click the export or delete buttons at the bottom. If you delete the sessions, you will be asked for confirmation first. If you choose export, it will export all the data from every matched session (visitor info and all actions) in a plain HTML table. It actually just shows it to you instead of being a "forced" download, it's kinda handy to just see right there. Use your browser's "save web page" feature to save it as an HTML file and send it to the person who requested it.

The search form has no max date range, unlike the rest of Clicky, so it can take quite a while if your site is high traffic. Let us know if you have any questions or issues about this tool.

We just pushed the fruit of our labor up to production servers and holy cow is it delicious! Take a big ol' bite and dig in boys and girls.

When you login you will see a message at the top that our terms and privacy policy updated so please review the changes to them by clicking the link. That page also covers the changes to Clicky itself, which are pretty much almost exactly what we mentioned in our last post.

As requested and as made sense as we did research, you are in control of your visitor privacy settings. The new default setting is to apply max privacy to "all visitors" and but you can change that to just "EEA visitors" or "no visitors" - but you can't select "no visitors" until the new terms have been accepted because that is the setting that will log "Personal Data" for visitors in the EEA (European Economic Area).

There's tons of new info all around the site so I will leave you with the following links.

Page to review all changes and accept new terms and privacy policy

Change your privacy settings on your user prefs page

Lots of juicy info has been added to our knowledge base


Our privacy shield certification is still pending, not sure it will be ready by May 25 but know that it's coming as soon as possible!

Lastly we haven't yet finished the tools to help you respond to Personal Data requests from EEA visitors. Those are in process though and will be ready by May 25.

Have fun!

Hello there! It's been way too long!

As the May 25 deadline for GDPR (General Data Protection Regulation) approaches, we've been getting a lot of questions about it. So without further ado:

Yes, Clicky will be GDPR compliant. In fact we can't wait. I mean, what company isn't jumping for joy at the prospect of crippling their own service to be compliant with a law made on another continent by politicians who have no idea how the internet works?

...Is this thing on?

Some features will have to be drastically changed or removed entirely, because it is now illegal anywhere in the world to log personally identifiable information (PII) of a European person without either consent or an extremely valid reason (e.g. logging a customer's address when taking their billing info). For example, we've offered the option to anonymize logged IP addresses for many years, but anonymized IPs are the future for all of you now.

A big reason we charge money is so that we don't have to sell your data to make money. If you're not paying for the product, you are the product. The only person who has access to a site's data on Clicky is the account owner. There's no cross referencing of visitors between sites, we don't build profiles of people for all the sites they visit and then sell it to advertisers like Facebook does. We could, but that's the kind of shit that free services pull, and we're not free. I would feel dirty anyways if we sold your data. The data is just... there... for site owner's to analyze. And that's it. Seriously!

Does GDPR care? Have you not been paying attention. If I gave you a nickel for every ounce that GDPR cared about legit services like Clicky, you would actually be poorer.

Think about the huge Facebook data leak that just happened and how fucking irresponsible Facebook was to let companies intentionally access that kind of data on their users in the first place. In terms of privacy violations, that's about as bad as it gets and that is the kind of thing that needs to be stopped. We just want to provide you with traffic reports for your site, but we're being grouped together with the likes of Facebook, who I now consider one of the most evil internet companies around. Cool.

---

Here is a partial list of the changes probably coming soon to a Clicky near you:

Viewing data on our site may require full user authentication (no more private links or "public access").

IP address anonymization might be forced rather than optional for all accounts.

Third party tracking cookies may be disabled. These are to help track the same visitor across multiple domains in your own account but this may be an issue now.

"Do Not Track" headers will be honored.

Public opt-out page for people to opt out of being tracked by all sites running Clicky.

Widgets that display individual visitor info will be changed/removed.

Custom data tracking - since this allows you to attach any data you want to visitor sessions, some of which may be PII (such as emails or usernames), something will need to change. Perhaps you'll need to check a box to assume full responsibility for any PII you choose to log before custom logging will work for your account.

Tools for you to comply with GDPR requests from your visitors (data you have about them, ability to delete it, etc).

Clicky will be Privacy Shield certified.

... and so much more! Subscribe today!

This is a partial list. We'll have more details within a few weeks as work continues on this.

I'm sorry I couldn't keep the snark out of this post, but rest assured we are taking the law very seriously as we know the future of Clicky depends on it. This is top priority and the only thing we're working on until it's done.

23 comments |   Apr 17 2018 4:03pm

The last 6 months, we have been extraordinarily busy with a major infrastructure upgrade that will make Clicky much faster and more resilient. This is by far the most major backend upgrade in our (almost) 10 year history. A lot of planning and testing has gone into this and we're excited to finally be under way with the database aspect. Our load balancers have already been on the new infrastructure for a month now, and web and tracking servers will follow up after the databases are done.

The database maintenance in late April was in prep for this migration, to make it as fast as possible. This weekend we will be doing 8 database servers - 6, 22, 28, 29, and 59-62. And each weekend thereafter we plan to do at least 20 more. We have 85 total database servers at the moment so we should be done or at least very close to it by the end of June.

Time-wise, migrations are similar to the database maintenance we do a few times a year. Most servers will be in the 4-10 hour range, but some of the bigger/oldest ones will take as long as 24 hours. As well, traffic processing unfortunately has to be halted during the migration. But these new servers scream and will catch back up to real time in no time at all once the migration is done.

We have gone out of our way to make this process as transparent as possible. When a db server is migrating, your affected sites' dashboards will show a message like this:




The message shows the time the migration started, and an ETA for when it will be done (both in your local timezone), based upon how many rows/second we know these go on average and how many rows total there are to migrate.

Once a server has been migrated, reports should load much faster than before. We are planning to increase the filtering/segmentation limits as well once this is all done, as those should have much less of an impact on real time processing.

We're also excited to be done with this because it means we can get back to the software side of things, which is our real passion.

Hope everyone has an amazing summer!