Diary of a mysql database problem in 15 minutes

1:42pm - frantic instant message comes in from a smallish project that their site is down.

1:42pm - I ssh into both the webserver and the database server.  Database server takes awhile to actually get a login prompt.

1:43pm - Email containing frantic instant message with importance set to high ( side note- people still use importance in emails? people still email after they've contacted you another way?  people still email? )

1:44pm - I run "top" on both the webserver and the database server.   ( There isn't any other monitoring since it's a smallish project.   there probably should be.  Will have to get client to pay for that.)  Webserver looks fine.  Database server load is really high, around 5 when it's usually around 0.30.  This is a mysql linux server.

1:45pm - exit "top" and open a mysql command line prompt.

1:46pm - execute "show processlist" in mysql.  A ton of queries show up, all behind one sinister looking one on a new table that's been set up recently.

1:47pm - execute "explain sinister-looking query" in mysql to see how mysql is executing that query.

1:48pm - possible_keys = NULL

1:49pm - Ruh roh.  Execute "alter table add index..."

1:50pm - database working.

1:51pm - database working.

1:52pm - database working.

1:53pm - database done working.  execute "show processlist" to see 2 queries left on stack.

1:54pm - Execute "top" command, load is down to 2.7 and falling. 1.9 now.  0.70 now.

1:55pm - Declare crisis averted via instant message and email.  Wonder again if people still use email.

1:56pm - Assign developer responsible for non-indexed table the task of making sure the schema changes get into source code control.

1:57pm - Go back to regular work.

Write helpful code comments or none at all.

Stumbled onto the following comment in some code I'm working on:

//Code updated by Crappy Developer - 06/22/2011. Fix for Prod problem.

This is a terrible code comment.  I would rather you not even put this comment into the code.

First off, I've replaced the name of the developer with "Crappy Developer" to protect their public persona and since that's how they are now known to me.

The comment "Fix for Prod problem" tells me nothing.  Aren't all problems Production problems?  Otherwise, they're not problems.  What was the actual bug?  Under which circumstances was it reproducible?  How did you fix it?  Any tricky business logic involved here?

And then there's the "//Code updated by.."  You might think that giving the date of the fix and the developer's name is at least a little bit helpful, but all of that can be discovered with "git blame."  This clown probably doesn't know "git blame" exists.  I'll have to send him the man page on that.

So the next time you leave a comment in the code, make sure its worth the time of the next developer that might see it.  And make sure they aren't just going to blog about how terrible your comment was.  You're wasting time with lame code comments, so please stop it.

Using Gmail "Details" link to discover older IP addresses

For one project that I'm working on, they have a very strict firewall setup which restricts SSH access to only internal machines and those temporary machines outside the network that need access ( like if you're working from home ).  So as my IP address changes at home, I need to notify the service that my IP address has changed from A to B.  The service prefers if I can give them both the old address and the new address because they'll take out the old address from the firewall rules and add access for the new IP address ( yes, they are that anal ).  I was keeping this list on a pad by my desk but today I found myself downstairs, away from that pad, so I couldn't find my previous IP address, just the new one.

Fearing that I would have to walk ALL the way upstairs to get that pad, I thought for a few seconds and remembered the "Details" link in the footer of my gmail page.  Clicking "Details" in the footer of gmail will show you the last 10 instances of logins/refreshes to your gmail page.  Sure enough, I was able to find the number 7 entry was from last night and showed my previous IP address.  I copied it into my firewall request and got back to my work.

MySql tip for displaying query results vertically

I have a need at the day job to view mysql records via the mysql command line client.  Some of the tables I'm looking at have 100 to 200 columns so it's pretty annoying to view this data horizontally across the screen.  If you end your queries in the mysql command line client with \G instead of a semicolon, then it will display the results vertically down the page.  Try it out!

SELECT * FROM users WHERE id = 72 \G