alexatkeplar
Joined 15 years ago
Karma
948
Submissions
417
Member since
Aug 14
Recent submissions
Comment · 8 years ago
The event producer in this case is one of our Snowplow trackers (https://github.com/snowplow/snowplow-javascript-tracker) - and these tracking SDKs do indeed attach a UUID as a unique event ID at event creation time.
However, this event ID is not enough to identify and then dedupe all types of duplicate events. This blog post provides more information:
https://snowplowanalytics.com/blog/2015/08/19/dealing-with-d...
Big thanks to pragmacoders for putting this tutorial together! It's awesome seeing what you are doing with the Snowplow platform :-)
Comment · 8 years ago
Yes, Snowplow uses JSON Schema extensively for event definition and validation (https://github.com/snowplow/snowplow/).
Comment · 9 years ago
With Kinesis, you would just use the Kinesis Client Library (https://github.com/awslabs/amazon-kinesis-client-python) which would automatically handle committing the offsets to DynamoDB.
Home-rolling a checkpoint-free event pipeline is a rookie mistake; it's a pity they didn't come across our Snowplow project (Apache 2.0 event pipeline running on Kinesis, Kafka and NSQ, https://github.com/snowplow/snowplow/).
Comment · 9 years ago
Great write-up! I tried something similar for modelling out the Snowplow release dependency chain (https://github.com/snowplow/) - I gave up and started looking at SaaS alternatives, but inspired by this I am going to try again.
For the record, I didn't find any good product management or project management tools (SaaS or desktop) which acknowledge that roadmapping any sufficiently complicated platform or product family is best expressed by a directed acyclic graph. Casual.pm had the right concept (https://casual.pm/) but the implementation was very primitive.
Comment · 9 years ago
As far as I know, Snowplow is one of the largest active users of JSON Schema, and we maintain one of the larger repositories of JSON Schemas on the internet, https://github.com/snowplow/iglu-central
After something of a hiatus, we are re-starting active work on our schema registry technology (https://github.com/snowplow/iglu). We'd love to be involved in new design work on the JSON Schema spec. How do we get the old (or new!) band back together?
@alexcrdean https://github.com/snowplow/snowplow
Author of Event Streams in Action (Manning Publications), http://manning.com/dean/
My email is alex at my company domain.