Just went to @Scale and all I can say there are a lot of talks on companies dealing with scale. Many interesting stuff happens when you have to deal with a lot of data, traffic, users, etc. One thing to note is do NOT mind what fancy technology people use, think about what the capability/feature you need to provide, be naive by starting small. Your grand scheme in your head will not always be what is best for your users.
Here’s what I learned:
Events @ Box: Using HBase as a Message Queue
Box built desktop sync off HBase. They required guaranteed delivery and it’s possible for events to persist in the queue for days.
All event activity went into MySQL. Notification of file changes went to HBase.
They naively chose to key off timestamps; however noticed that they weren’t able to discern what events that was not read when there were multiple parallel writes to the queue for a user.
They improved it by time-bounding the queue which means they would guarantee delivery of events in a certain time window so they can keep advancing the message queue. They also added back-scanning which put a limit to how many messages they could write into the users queue.
To improve availability, they replicated the events to another HBase cluster. They had to make sure their clients were aware of which HBase was master/slave. More or less they replicated master/slave concept from MySQL.
Box looked at other queue tech like Kafka. However they decided at their given scale they would not be able to create enough queues per user. Kafka had a limit of some tens of thousands queues that can be created. They required millions of queues. Data replication is also an issue too as they couldn’t make it transparent enough to the client.
Scaling Instagram Data Systems
They recommend to start small and iterate!
Case Study - taking Database snapshots and pushing to Hive is hard.
v0 (naive approach)
spawn new db instances
load data from backups
export to csv
import to hive
… this was slow and high maintanence
v1 (automated process)
dedicated db instances
continuous stream of update with incremental snapshots
export to csv
import to hive
… they wanted to provide search, but search to them was more than just reading a row in PostgreSQL. It’s also difficult to make sure the data is up-to-date given their size.
They created a system called Slipstream which delivers events via Thrift to a system called Firehouse. From this, real-time indices can be created by filtering only the relevant events they need to process. This allowed them to create users search, hashtag search, media search and many others. All the data is structured so it would be easy to create new indices (or new features). Also with Slipstream they still deliver their data to PostgreSQL.
A huge lesson they learned was to be free to change
The Motivation for a Monolithic Codebase: Why Google Stores Billions of Lines of Code in a Single Repository
Yes. Google manages one huge monolithic repo Kind of reminds me of svn.
Their rate of change is huge, exponentially huge. Sorry no photo. Just imagine a graph that has an exponential curve on it.
Google loves automation and they showed a graph between requests via automation and humans. Tooling to support testing, measuring quality and more was growing at higher rate than humans. (Humans are slow!)
Google’s workflow is users select code they want to work on and put it in their own workspace (in the cloud). They all commit to head and branching is only done for releasing. They also use Cherry-picks to update release branches. (Weird I swear there’s a company that has done this… IBM’s ClearCase!)
Advantages (Yes they advantages to share which are quite nice)
change can be distributed to all the code in one commit. No longer does a developer need to submit many PRs to fix people’s broken code.
single source of truth. Everyone commits to head. Can’t debate that. Also there’s only one version.
Allows for code modernization (aka eliminate tech debt). Every commit has to ensure it works for every project.
Disadvantages to the monolithic approach…
You need to invest in tooling that enables you to do this effectively. Do this early on in the team. There maybe more but this is really an interesting developer paradigm shift. Code at your fingertips without managing many repos in a certain order.
Facebook: Structuring and Adopting a GraphQL Server
Build a query language that aligns with product managers’ vision. Don’t build APIs that are constrained and force every client to update. This is a framework that allows you the client to ask for what you want (with parameters) and get what you need.
No more creating a RESTFul API that is limited to whatever is provided at the time, this is query language that allows you to get the data you really need in the structure you desire.
Read the README for an overview of its capabilities.
Other facets:
They have adapters for other languages other than JavaScript like graphQL-Ruby and graphQL-Java.
There is even a way to add more data types that allows you to really just literally have a heterogenous set of data back ends. You provide the keys and fields that are available and the GraphQL can iterate through to gather the data needed.
Summary
Start small, do NOT over-engineer, and iterate fast! Think about your users. They need small wins now.
This is a pretty good starter for those learning security. Remember, security is everyone’s problem – it is not just for your security department to worry about.
Wow, what an amazing experience that I just had at Chef Conf 2015 held at Santa Clara, CA. This was my first time and I’m totally blown away by the number of people that came, the diversity of the community, and the common passion to #automateallthethings.
Here is my rundown of what happened:
Chef Community Open Summit
This an Open SpaceSummit, which may seem very improv; however, the intention is to really discuss topics of that are of great interest at the moment in time. The one law they have is law of two feet which states
If at any time during our time together you find yourself in any situation where you are neither learning nor contributing, use your two feet, go someplace else.
Thinking about this really applies to life in general. If you aren’t able to contribute, don’t stay – move!
The one topic that hit home is the one on burnout. Many feel this sense that they have to complete work or get it done right right away. From what I gather, in operations, we tend to internalize this stress and think we have to keep fixing stuff or more or less be a hero. I am wondering if production engineers have to start standing up and thinking critically of what’s healthy as one team (be less two sides) and stop have the mentalitly of throwing the grenade to others. Automation can help reduce stress, but we all will need to think as one team.
Awesome keynotes! Awesome video about delivering eggs 356 days!
Which led into the intro of Christ Barry’s keynote bringing a new product to help with CD called Chef Delivery.
Really awesome workflow. Really great to see how Chef can play a huge role in contionus delivery. By the way, they even opensourced this as well which you can see here.
Other topics that I saw:
Kitchen Flows: Test Kitchen Usage Patterns by Fletcher Nichol. Really good to know it supports Windows. Also really good to know people are using this to support cooking multi node cookbook all locally on your workstation.
Chef Provisioning a Chef Server Cluster by Joshua Timberman. Its like inception where you are provisioning a Chef Server with Chef Server. BTW, Chef Provisioning Rocks! F typing in 100x commands to provision something. F manual sh!t!
Building And Releasing an Online Game From Nothing With NoOps by Jamie Winsor. He defines NoOps as No Dedicated Operations as this is everyone is responsibility. Very good to listen as it can help you shape your team to move at velocity (aka with speed).
Conclusion
Automation Rocks! Collaborate more! Be responsible for your code and product! Lets cook awesome products together!
“…the difference between a line cook, who can cook a few specific recipes…and a master chef, who truly understands how food and taste work. He can create recipes.”
