From where we started, Gojek has grown to be a community of more than one million drivers with 3 Million+ orders every day in almost no time. To keep supporting this growth, hundreds of microservices run and communicate across multiple data centers to serve the best experience to our customers.

In this post, we’ll talk about our approach of assembling Infrastructure As Code that simplifies the maintenance of an increasingly complex microservices architecture for our company.

Motivation

Building infrastructure is, without a doubt, a complex problem evolving over time. …


Stories from the land of data at GO-JEK

gojek.jobs

At GO-JEK, we build products that help millions of Indonesians commute, shop, eat and pay, daily. The Data Engineering team is responsible to create a reliable data infrastructure across all of GO-JEK’s 18+ products.

Mission

We aim to provide disproportionately large advantages to GO-JEK over its competitors by making data available, accessible, reliable and actionable at scale.

The GO-DATA platform allows internal teams to build on our products to develop innovative solutions; creating new opportunities for our customers and expanding on what’s possible.

Approach

Some informal philosophies and approaches we follow during our time working on data infrastructure:

Scale

Data at GO-JEK doesn’t…


How and why we designed one of the best internal product wiki

gojek.jobs

At GO-JEK, the Data Engineering team is responsible for crafting reliable data infrastructure across all of GO-JEK’s 18+ products. We aim to provide disproportionately large advantages to GO-JEK over competitors by making data available, accessible, reliable and actionable at scale.

Our GO-DATA platform allows teams to build on our products to develop innovative solutions, creating new opportunities for our customers and expanding what’s possible.

Why do you need an internal product website?

In the real world, when we buy a product, it comes with a quick guide or some other form of documents which help you get going with the product. …


Given that our internal services generate billions of events a day, our data infrastructure must be highly scalable, available, and flexible enough to keep up with rapid product iteration and exponential data growth. This results in spending a substantial portion of our time in infrastructure provisioning, load testing, and system recovery. This article describes how we envisioned Asgard, our toolbox for automating end-to-end data infrastructure challenges.

Product Goal

Every product needs a vision. This can come from many sources; most important being solving existing pain points and fulfilling a real, deeply-felt human need. …


Building confidence in system behaviour through disaster simulation experiments

It’s important for us to build a resilient and scalable data infrastructure when we have 18+ products. This means designing systems that can sustain operations in the face of failure.

Built on the principles of Chaos Engineering, and tailored to our specific use cases, Loki is our internal disaster simulation and load testing tool. It helps ensure our data infrastructure can tolerate random instance failures. Exposing engineers to failures frequently incentivises them to build these resilient services.

Features:

  • Disaster simulation: Loki can randomly terminate virtual machine instances, containers and cluster nodes that run inside our production/performance environment.
  • Load testing: Loki can…

At GO-JEK, we build products that help millions of Indonesians commute, shop, eat and pay, daily. The ‘Data Engineering’ (DE) team is responsible to create a reliable data infrastructure across all of GO-JEK’s 18+ products. In the process of realising this vision, here are some learnings:

  • Scale: Data at GO-JEK doesn’t grow linearly with the business, but exponentially, as people start building new products and logging new activities on top of the growth of the business. We currently see 6 Billion events daily and rising.
  • Automation: Working at such a large scale makes it really important to automate everything from…

Aggregation and visualization of billions of data points daily

Atlas showing demand/supply ratios in Jakarta and nearby cities

At GO-JEK, location is built into the fabric of all our products, powering dozens of products that are used by millions of customers and drivers, daily.

Here’s a tweet from one of our investors from when this post was written to help contextualise our scale.

As a result, we have billions of GPS points flowing through our data pipelines daily in real-time. This data drives decisions like driver allocation, surge pricing, driver incentives and many more.

At GO-JEK’s Data Engineering team we built Atlas in an attempt to make it easy for teams within GO-JEK to visually explore this…


In a hyper-growth organization like GO-JEK, technology plays a vital role. However, as an organization matures, the hard part is not scaling the technology, but paying heed to culture.

For us at GO-JEK, culture is a collective philosophy about how to build products that change lives. We all do our bit to make sure it‘s transparent and open to innovation.

Trust

A great way to understand an organization is to ask, “Why should someone work there?”

For me, it’s the people. And it’s not just me, most of us at GO-JEK will have a similar answer. …


It’s a mindset, not a skillset

Throughout my career, I’ve have changed job roles more frequently than I have changed buses.

I started out being a tutor, then sold health insurance, marketed embedded products, designed interfaces, developed embedded systems, wrote software, wrote stories as a data journalist, was an entrepreneur and much more.

As a software developer, specifically, I’ve been doing prototyping, design, frontend development, backend development and now exploring devops and data engineering. And that must be the reason that I’m frequently asked: How to do it all and how to be a full stack dev.

So, here are a few techniques that have been…


Just another day in the life of one.

Shot by Mr. Geek

August 2014, I was in the process of evaluating the tech stack for my next project. For the last few projects, I used MEAN stack, the most recent being Resumite, and I loved working on Express and Angular, but now it was time to move on, as the future of both seems very doubtful.

5:00 am, and I was exploring the newly discovered world of Isomorphic JS with React, flux, and whatnot. Being a JS lover, I am fascinated with the future of the web's isomorphic/universal era.

Web apps started with full…

Ravi Suhag

Data Engineering, Data Visualization, Infrastructure Automation, Distributed systems

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store