EngineSmith's Blog

Engineering Craftsman

Posts Tagged ‘mysql’

SSD Rules

Posted by EngineSmith on June 22, 2011

Deal with your own servers with freaking bad-ass disk I/O can save your infrastructure complexity and sometimes a life saver for a startup. We have been using Intel X-25E SSD drives as our only MySQL storage for a year now. And it is simply amazing and cost effective. If you want to make your life a living hell and deal with turtle speed I/O, tons of EC2 instances and constant fails, try EC2.

Here is a video talking about the big win of SSD. My favorite quote: “50TB SSD in one machine for $80K, and you can fucking skip sharding”.

Posted in Engineering, Hardware, Operations, Startup | Tagged: , , , | 2 Comments »

Your car has more than 10M lines of code

Posted by EngineSmith on February 14, 2011

Luxury car has more than 100M lines of code, while F-22 Raptor has only 1.7M lines of code. Mercedes-Benz S-class only the navigation system contains more than 20M lines of code.

From somewhere else I found that average car from Ford contains more than 10M lines of code. I think we are doomed with this trend. Everybody seems like to reinvent the wheel again and again, no matter how lame they are with it.

Same thing happening in the web world, every several months, there are some new kids on the block trying to solve an old problem with a completely new approach. “You don’t throw away the baby with the bath water”, there is some quote like from from Drizzle project (the re-vamp of MySQL project), if I remember correctly. So many NoSQL products are trying to replace the existing rock solid MySQL, backed by remarkable marketing hype (and propaganda campaign machines). Only time will tell, just like how MySQL survived so many years.

[update] A colleague found this link, amazing: F22 got zapped by International Date Line.

Posted in Engineering, Software | Tagged: , , , , | Leave a Comment »

MySQL read write split myth, and why I wouldn’t use it

Posted by EngineSmith on September 11, 2010

In MySQL scale-out practices, many people chose to split their read and write to different nodes. This may work well for a read-most environment, but actually significantly painful and wrong in write-most settings, like ours.

  • Two MySQL nodes, master-master replication
  • Only the active node is taking writes (writer)
  • Two readers, round-robin between the two nodes
  • MMM is used to monitor replication delay and handle role switching

This seems to be the common sense approach, make sure only one node is writer to guarantee consistency, while utilize both nodes as readers to share the load. We did that for the last 6 months, and found out in practice this is wrong:

  • Plan your capacity for the worst case scenario. The point of using two nodes is to handle fail-over. When one node dies, all read and write traffic will go to the other node. You should plan to have one node handling all traffic, the read/write split gives you an illusion that you have the capacity until the failover, it is too dangerous.
  • Splitting read/write actually made your application logic super complicated due to replication latency. Regardless of your tolerance level, at some critical point, you have to write some ugly code like this:  a = reader.get(); if (a == null) a = writer.get();
  • MMM is not designed very well to handle role switching during failure cases (I will write another post about it later), using two roles make the situation fairly messy.

So, here is what we did recently, in one cluster always have only one active node taking both reads and writes. MMM only handles fail-over (not replication delay). Then over-sharding on the cluster (prepare to split the shard in case it can’t handle the load). Life is much much simpler now.

Posted in Engineering, Operations | Tagged: , , | Leave a Comment »