suhailpatel

https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on...

In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data properly which led to data being 'missing' for an hour.

In a typical Cassandra cluster when scaled up, data moves around the ring a single node at a time. When you want to add multiple nodes, this can be an extremely time and bandwidth consuming process. There's a flag called auto_bootstrap which controls this behaviour. Our understood behaviour was that it will not join the cluster until operators explicitly signal for it to so (and this is a valid scenario because as an operator, you can potentially backfill data from backups for example). Unfortunately it was completely misunderstood when we originally changed the defaults many months prior to the scale up.

Fortunately, we were able to detect data inconsistency within minutes of the original scale up and we were able to fully revert the status of the ring to it's original state within 2 hours (it took that long because we did not want to lose any new writes so we had to carefully remove nodes in the reverse order that they came in and joined the ring).

Through a mammoth effort across the engineering team across two days, we were able to reconcile the vast majority of inconsistent data through the use of audit events.

This was a mega stressful day for everyone involved. On the plus side though, I've had a few emails telling me that the blog post has saved others from making a similar mistake.

It’s quite skewed towards the UK due to the social circles and reach but still useful nonetheless. I’m sure they would appreciate more data points!

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. We're running our own services to fill gaps in the transit network. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

We recently launched our new Smart Ride service, aimed at encouraging better shared mobility in cities. Read about The Responsive Network: https://medium.com/citymapper/the-responsive-network-part-3-.... This is super interesting and rewarding work from a technical perspective, we're constantly iterating and improving our planning, routing and simulation algorithms for Smart Ride to better serve our network. If you are interested in this sort of problem space, now is a fantastic time to get involved from the ground up.

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

Read our other blog posts at https://medium.com/@Citymapper. We've also launched an engineering blog: http://engineering.citymapper.com, if stuff like that interests you, definitely apply!

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. We're running our own services to fill gaps in the transit network. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

We recently launched our new Smart Ride service, aimed at encouraging better shared mobility in cities. Read about The Responsive Network: https://medium.com/citymapper/the-responsive-network-part-3-.... This is super interesting and rewarding work from a technical perspective, we're constantly iterating and improving our planning, routing and simulation algorithms for Smart Ride to better serve our network. If you are interested in this sort of problem space, now is a fantastic time to get involved from the ground up.

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

Read our other blog posts at https://medium.com/@Citymapper

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. We're running our own buses to fill gaps in the transit network. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

We recently launched our new Smart Ride service, aimed at encouraging better shared mobility in cities. Read about The Responsive Network: https://medium.com/citymapper/the-responsive-network-part-3-...

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

Read our other blog posts at https://medium.com/@Citymapper

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. We're running our own buses to fill gaps in the transit network. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

Read our blog at https://medium.com/@Citymapper

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. We're running our own buses to fill gaps in the transit network. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

Read our blog at https://medium.com/@Citymapper

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

https://citymapper.com/jobs

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

Read our blog at https://medium.com/@Citymapper

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Cities are complicated. We use the power of mobile and open transport data to help humans survive and master them. We are building the best public transit app, one that caters for the needs of commuters. We are building a routing engine which is truly multimodal. To power all of this, we're leveraging open data as well as building the tools necessary for agencies to add and fix data.

Read our blog at https://medium.com/@Citymapper

See all our open positions at https://citymapper.com/jobs

We're hiring for Backend (Python, Go, C/C++, AWS), Frontend (Web, React, ES6) and iOS/Android engineers as well as Data Science.

If you have any questions, feel free to drop me an email at suhail -at- citymapper -dot- com

Join us in our mission to make cities usable by building the ultimate transport app.

Hiring for ALL roles (Engineering, Design, Product), including:

-- Web Developer (React, Redux)

We build a lot with modern JS technologies. We have our web app, but also many systems behind the scenes that allow us to be the best source of transit data in our cities. We use React + Redux, CSS modules and PostCSS, Webpack, Django.

-- Android Developers & iOS Developers

We're particularly interested in developers who are passionate about UI, and/or using sensors & location efficiently.

-- Site Reliability Engineers

Help Citymapper scale its platform by orders of magnitude. We are currently in +30 cities, but we are going to be expanding to reach everyone who needs us.

-- Data Science

We're looking for data scientists to work on a variety of projects including improving the experience of the apps to make them more personal.

Read about our $40M Series B: https://medium.com/@Citymapper/getting-from-a-to-series-b-88...

Apply at https://citymapper.com/jobs/

Contact me at suhail at citymapper dot com if you have any questions.