We are still on mixpanel because it is hard to extricate ourselves, but obviously that isn't the long term plan.
I can point to mistakes I made such as the higher posting. In our case some extra numbers are sent with each event and the values of those numbers are interesting. (One example is the volume level of the device.)
With Google Analytics where they term this custom variables all you get to see is the average value which is spectacularly useless. Mixpanel would show us the distribution, but only for a particular event type. We have about 20 different events, so working it out across all 20 would be too tedious.
Today we only use mixpanel as a receiver of events. We export the analytics data from them and then work on that locally. Unfortunately they only provide an export every 24 hours. We do not use any of their other functionality although we did try.
If you are hand-rolling your own analytics, it sounds like it might be simpler/cheaper for you to use SnowPlow (https://github.com/snowplow/snowplow) for your use case - especially as you can grab the data hourly from S3 rather than every 24 hours. Unfortunately we don't have mobile clients yet :-( (although we are working on them) - which platforms would you need? Feel free to reach out on alex@snowplowanalytics.com
Snowplow is only about web analytics as far as I can tell. There are a bazillion solutions out there for that.
For mobile app analytics you need a client library in Java (Android) and Objective C (iOS). The client library needs to record analytics events into a SQLite database, and then periodically try to upload them to a server (you don't always have connectivity). Attention must also be paid to things like roaming (do you want to burn user's data, it can be expensive in many parts of the world). You also need cleanup (eg if you can't send data for many days then you'll likely want to discard it). You'll want to make sure the client plays nice (eg not creating lots of threads and causing constant wakeups). It should also supply platform information since you'll want to analyze versions, screen sizes etc. There are various other little details that matter on the client.
On the server side it needs to correctly cope with data arriving days late, with "incorrect" time stamps from clients. And you'll want to easily do pointy clicky through the data as you'll have some common questions such as what are the most prevalent platform versions and devices and how does that correlate by country.
Thanks Roger, that's helpful input. I'm adding a "client-timestamp" to the SnowPlow querystring instead of just relying on the CloudFront timestamp ;-) Out of interest, is there anything about the MixPanel mobile client libraries (Android/iOS) that you would do differently if you were starting from scratch?
I've basically rewritten almost all of it now. One important note about our use case is that our product is a library that application developers add to their application. This means we do not control the application and we have to play very nice. Additionally any dependencies we include (such as an analytics library) reflect on us. This is also why it is hard to get apps rereleased on changes in our code - users punish app updates that have no visible functional differences.
Here is a list of things that mattered:
* The library needs to have a posture on how it is used by multiple different components in the same app. For example it can intend there to be one canonical source/package, or each component could make a private fork.
* If the canonical package is chosen then it must work with concurrent but different reporting ids and settings. (For example Google screw this up by having the tracker be a singleton.) It needs to be possible to find out the version number from tools so they can complain about being out of date.
* For the private fork posture it is easiest if the code is all one file (use nested classes). It should use a sqlite database name that differs per fork so they don't clash with each other.
* The library will have "slow" work that needs to be done. This includes updating the SQLite database with new events, clearing out too old events on startup, and sending event batches to the server. I updated the mixpanel code so that it returns Runnables for that work, and then my library can use existing slow work threads. Most however will want the library to work out where to run the slow work.
* I deleted the code that reads the unique device id (aka UDID). Some companies are happy grabbing that - our privacy policy is far stronger. We generate a random unique id string on first run. Even the device code being present but unused is enough to set off binary analyzers.
* You'll want to grab some other stuff by default (eg carrier information, device model, os version)
* Make debugging easy. For example the logcat mechanism on Android works nicely. Mixpanel were just logging their API call, not any detail. For example it would say "track" instead of "track: clicked" (where "clicked" is the event type)
* Sessions are what matters most. Mixpanel has no concept of sessions. For example when they purge unsent old events, they just delete the ones older than the time frame (was hard coded as 48 hours). However this means it could end up deleting the first half of a session but transmitting the rest. A better approach is to have a session id that is updated on each start, then delete all events belonging to old session ids.
In terms of implementation details, a comparison of Google Analytics to Mixpanel is useful.
Google only have one tracker instance, although you wouldn't know that from the API so multiple usage silently doesn't work. They have an extremely complicated custom variable scheme for adding extra data for each event. Ultimately their database stores a query string for each event. If there are 10 to send then they make 10 separate GET requests.
Mixpanel supports multiple instances, but almost everything was hard coded (eg dispatch intervals, expiry of old data). You supply events with arbitrary JSON data, including a list of "super properties" which are added to every event. This is a very good approach. The database stores the events. When submitting, a POST request is generated with a batch of events (up to 50, again it was hard coded as two different numbers in two different places).
If you use query strings (in the sense of a GET) then there is a danger of the data being logged by proxy servers, hitting URI length issues, and being unable to batch.
