EngineSmith's Blog

Engineering Craftsman

Archive for August, 2009

Grails – The Savor of Java Web Applications

Posted by EngineSmith on August 28, 2009

Let’s face it, developing web application in Java/JSP is really really painful, regardless which framework you work with. You got to restart your application server very often, for some simple Java classes or layout/configuration changes. But we are kind of stuck with it, since most of our domain logic are already written in Java, using PHP or Rails will require us to either rewrite domain logic, or expose them as HTTP services. Both sounds a bit crazy unless you got a web guru in house.

Finally Grails comes to the rescue, there are couple things which are just so amazing about it, you get the best from both world: the strictness of compiling language as Java for domain logic, the dynamics and convenience of scripting language Groovy. It is truly a god send:

  • The only thing deployed to Production is a web application (WAR file) which contains Java classes only. Everything runs in compiled Java class form. You do NOT need Grails or Groovy environment in Production. You got all the benefits of mature JVM: multi-threading, heap management, garbage collection and great scalability
  • Most things are dynamic loaded at Development time. So you don’t need to restart server all the time
  • Seamless integration with existing Java domain logic, Spring is native. Intellij IDEA contributed the mighty compiler which compiles Groovy file and Java file at the same time, and, you can debug Groovy class inside IDEA

Furthermore, Grails has tons of convenience stuff to make developers life super easier (like Rails), for example, they have a wonderful example which takes 10 lines of code to add full text search ability for a database entity by using Lucene. Those don’t contribute to my decision, but icings on the cake are just so sweet!

Unfortunately, Grails has too many code-by-convention stuff which makes it hard to be wrapped into existing big Java projects. There are some tricks and hacks we did to make it work, I will cover them in future blogs.

Posted in Engineering, Software | Tagged: | 2 Comments »

SSD faults finally resolved

Posted by EngineSmith on August 28, 2009

We finally got our Production environment stabilized with MySQL, Open Solaris, ZFS, SSD/JBOD. The performance is just super awesome! With sysbench, we could reach 21K QPS stable! It took us quite some time to get it done, especially we faced a nasty SSD fault issues.

We were inspired by this SmugBlog: Success with OpenSolaris + ZFS + MySQL in production! and ordered JBOD with 2 Intel X-25E 32GB SSD and 10 SAS 15K drives for our MySQL se rver. The SSDs are configured as ZFS Cache (write cache). The SAS controller is LSI 3801E.

After some load testing, we found that there are many “scsci bus reset” errors in the log and “iostat -E” show “Hard Errors” for the SSD drives. Whenever “scsi bus reset” error happens, everything comes to a halt on MySQL side, thus we got very in-consistent load testing results. The “reset” happens every 30 minutes or 1 hour. It was just 3 days before our launch date!

We upgraded OpenSolaris to 2009.06, LSI and SSD firmware, didn’t help. Finally our mighty consultant found that turn off OpenSolaris Fault Management can ease off the errors. It took us through the first couple weeks’ Production launch period. Eventually our vendor Silicon Mechanics helped out and we updated the LSI driver. Wowla, not a single SSD error after that, performance is simply amazing!

Fault Management

  • fmadm unload disk-transport   # turn it off
  • fmadm -eV      # show status

Sysbench Load Testing

  • sysbench –test=oltp –oltp-table-size=100000 –mysql-db=test –mysql-user=xxxx –mysql-host=host –mysql-password=xxxx prepare
  • sysbench –test=oltp –oltp-table-size=100000 –mysql-db=test –mysql-user=xxxx –mysql-host=host –mysql-password=xxxx –max-time=60 –oltp-read-only=on –max-requests=0 –num-threads=8 run

Update LSI Driver

root@opensolaris:~# wget
root@opensolaris:~# uncompress itmpt-x86-XXX.tar.Z
root@opensolaris:~# tar -xvf itmpt-x86-XXX.tar
root@opensolaris:~# cd install
root@opensolaris:~# pkgadd -d .
root@opensolaris:~# vim /etc/driver_aliases (Or the editor of your choice)
Here you will need to comment out the mpt driver aliases and uncomment
the itmpt driver at the end of the file.
#mpt "pci1000,30"
#mpt "pci1000,50"
#mpt "pci1000,54"
#mpt "pci1000,56"
#mpt "pci1000,58"
#mpt "pci1000,62"
#mpt "pciex1000,56"
#mpt "pciex1000,58"
#mpt "pciex1000,62"

itmpt "pci1000,30"
itmpt "pci1000,50"
itmpt "pci1000,54"
itmpt "pci1000,56"
itmpt "pci1000,58"
itmpt "pci1000,62"
itmpt "pci1000,621"
itmpt "pci1000,622"
itmpt "pci1000,624"
itmpt "pci1000,626"
itmpt "pci1000,628"
itmpt "pci1000,640"
itmpt "pci1000,642"
itmpt "pci1000,646"

Reboot OS after this

Posted in Engineering, Hardware, Operations | Tagged: | 7 Comments »