Categories: Distributed Systems

Lessons from Google on Distributed Storage System

The article lists down the lessons learnt by Google Engg. team while they implemented Google BigTable, a distributed storage system, which is used to manage structured data of more than 60 Google products or so. Read further about Google BigTable on this page.

KISS Principle for Simpler Design & Coding

With distributed systems bound to be complex and the related codebase expected to evolve over a period of time, it may be good idea to keep the design and coding simple for ease of code maintenance and debugging. One could apply the KISS principle by breaking down the problem into smaller pieces and do the design and coding appropriately. Read more about KISS principle on some of the following pages:

 

YAGNI Principle to Avoid Complexity

While working with distributed systems, one might want to delay adding new features until it is clear how the new features will be used. This is similar to what is mentioned by YAGNI principle – “Always implement things when you actually need them, never when you just foresee that you need them.”

 

Failures are bound to happen. Plan for it.

Large distributed systems are vulnerable to many types of failures, some of which are listed below. One should, therefore, plan to take care of each one of them in a diligent manner and not make any assumptions whatsoever.

  • Memory and network corruption
  • Large clock skew
  • Hung machines
  • Extended and asymmetric network partitions
  • Bugs in related systems
  • Overflow of disk quotas
  • Planned and un- planned hardware maintenance.

 

Monitoring is the Key

While working with distributed systems, one may want to setup proper system-level monitoring to do a regular check on some of the following aspects:

  • Lock contention
  • Slow writes
  • Hung accesses to one or more tables
  • Clusters
Nidhi Rai

Nidhi has been been actively blogging in different technologies such as AI / machine learning and internet technologies. Her field of interest includes AI / ML, Java, mobile technologies, UI programming such as HTML, CSS, Javascript (Angular/ReactJS etc), open-source and other related technologies.

Share
Published by
Nidhi Rai

Recent Posts

Agentic Reasoning Design Patterns in AI: Examples

In recent years, artificial intelligence (AI) has evolved to include more sophisticated and capable agents,…

3 weeks ago

LLMs for Adaptive Learning & Personalized Education

Adaptive learning helps in tailoring learning experiences to fit the unique needs of each student.…

4 weeks ago

Sparse Mixture of Experts (MoE) Models: Examples

With the increasing demand for more powerful machine learning (ML) systems that can handle diverse…

1 month ago

Anxiety Disorder Detection & Machine Learning Techniques

Anxiety is a common mental health condition that affects millions of people around the world.…

1 month ago

Confounder Features & Machine Learning Models: Examples

In machine learning, confounder features or variables can significantly affect the accuracy and validity of…

1 month ago

Credit Card Fraud Detection & Machine Learning

Last updated: 26 Sept, 2024 Credit card fraud detection is a major concern for credit…

1 month ago