Monthly Archives: July 2009

The very, very poor man’s Google Analytics: tail, cut, sort, uniq & wc

I still have what most would call an unfounded fear of privacy when it comes to Google. They may receive a copy of every email I send to my friends who use gmail, they may place every call to me via Google Voice, they may server every ad from Double Click (which I then block) and I sure as heck never stray from their bad-ass search on google.com, but I don’t host anything with them directly.

I’ve run my share of web analizer tools, but some times I wanna know, right now, “how many people subscribe to my blog feed?”. Now, I probably should be using FeedBurner (No shit – I did not know, ’til just this second, that they too are now owned by Google. Oh, the irony!), but my site, despite its claims, is still a bit of the cobbler’s child when it comes to analytics. Heck, I still don’t have mod_usertrack on!

Enter tail, cut, sort, uniq and wc!

tail -10000 access_log|grep /blog|cut -d" " -f 1|sort|uniq|wc

In layman’s term’s that’s “get the last 10000 lines of my access log, cut each line into fields separated by the space character, grab the first field (the IP address in this case), sort the resulting lines of now just an IP address per line, remove the duplicates and count the number resuling lines (or IP addresses)”. Presto! 388 of you out there, including all the bots, spiders, crawlers, trolls and goblins. Thanks for the interest!

Ruby-less way to add key frames to flv videos for the likes of JWPlayer

Leave a reply

At work we’ve been working on a good way to roll our own videos. We initially started with the generic off the shelf swf player + flv to make our videos go. This was OK, but was lacking some key features folks were used to, primarily full screen play back and the ability to seek to a specific spot in a video. Additionally, for videos that were over 20 minutes being viewed over slower connections, precious apache children would be chewed up potentially causing a slow down for the web head doing the serving.

Enter JWPlayer! This is a great, easy to configure, free for non-commercial use, flv player that offers the features we’re looking for. Further, we could offload the flv’s to our image server, lighttpd . (By the way lighttpd is extremely awesome; I can not recommend it enough. I first learned about from my friend over at WikiSpaces where they had dropped apache entirely in favor or lighttpd. Noteworthy is that lighttpd was first concieved as an attempted answer to the c10k problem. I’ve personally seen it handle over 2000 connections per minute and the server load didn’t go above 0.5 (though the full meaning of load average is a curious one (sorry for the double, now triple parenthetical statements )).) Simply add a little flv streaming foo to lighttpd, and you’re good to go.

At this point, we hit a stumbling block. In order for JWPlayer to seek, it would send a request for the FLV to the web server and give it a starting point, like so:

10.1.6.221 lighttpd.domain.dom - [29/Jul/2009:12:46:37 -0700] 
"GET /web_assets/video/your.flv?start=17038076 HTTP/1.1" 200
 17765145 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; 
en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1"

However, if the flv you were serving didn’t have key frames then the web server would simply ignore the query string, and indeed, the click all together.

We do most of our development and video authoring on a mac and then serve our files off a load balanced linux server. At first, we used flvtool2 on linux to embed the key frames on our flvs. This kinda works, but it’d be easier to have the video author be able to add them himself with out needing to download and install ruby and rails and all that crazy server scripting foo. As his flv creator/codec/authoring app/chumpy wasn’t playing nice, I was given the task to find a simple solution.

Now, finally, we get to the point of the post. If you’ve gotten this far and wanna know how to embed keyframes with out ruby, get thee to the multi-platform, command line tool called flvmeta and go home happy!

Update: This post would not be complete without mentioned our use of swfobject to render the flash HTML.

Secret Jumps of Tunnel

2 Replies

If you ride off road, you’re always looking for new ways to get from point A to point B and stay on the dirt. Today is Thursday which means that it’s leave work early and ride day! H and K and I were exploring off of Tunnel and found a full on hidden BMX track. I never road BMX as a kid, which makes my MTB skills a bit lacking. BMX skills or no, none of actually road this track. 4′ tall jumps with a 5′ gap? No thanks!

Macchiato!

Leave a reply

I will be the first to confess that I love the ritual around coffee as much as I love the the actual drinking of coffee. Today, joining my pal from twtitw, I went to the wonderful Cento Cafe. It was sunny, the two macchiatos looked lovely and I was as happy as, well, as a me drinking phenomenal coffee with a good friend in the sun!

Rogue MySQL queries

Leave a reply

Do you you ever have those moments where you’re trying out a query in MySQL and realize after you executed it that it’s going to kill your database server? If you’re like me, you’re often working in a LAMP stack in which case you’ve executed the query via a web page. The first thing you do is hit “esc” to stop the page from loading. MySQL doesn’t head the “full stop” call from Apache, if issued. If you’re silly and you did this on some sort production machine, you don’t exactly want to restart Apache or even worse, restart MySQL which can often take a while.

Enter show processlist; and kill PID! Yeah, mysql has it’s on version of “ps” and “kill -9”. In our case, this is extremely handy because it saves us the headache of rebooting MySQL and taking the DB offline for a minute. Connect to your DB as root and type:

mysql> show processlist;
+------+------+-----------+------+---------+------+----------------------+--------------+
| Id   | User | Host      | db   | Command | Time | State                | Info         |
+------+------+-----------+------+---------+------+----------------------+--------------+
| 1693 | root | localhost | rei  | Query   |   75 | Copying to tmp table | SELECT     di| 
| 1695 | root | localhost | NULL | Query   |    0 | NULL                 | show processl| 
+------+------+-----------+------+---------+------+----------------------+--------------+

Indeed this one query is killing the CPU:

top - 08:57:50 up 30 days, 20:59,  2 users,  load average: 0.53, 0.14, 0.04
Tasks:  72 total,   2 running,  70 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.1% us, 30.1% sy,  0.0% ni, 49.8% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2074628k total,  2010740k used,    63888k free,   142648k buffers
Swap:   524280k total,      192k used,   524088k free,  1549692k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                              
 3119 mysql     16   0  549m 115m 5076 S 99.8  5.7   9:53.40 mysqld

All we have to do is kill it and the box is back to idle. Sweet!

mysql> kill 1693;
Query OK, 0 rows affected (0.00 sec)

mysql> show processlist;
+------+------+-----------+------+---------+------+-------+------------------+
| Id   | User | Host      | db   | Command | Time | State | Info             |
+------+------+-----------+------+---------+------+-------+------------------+
| 1695 | root | localhost | NULL | Query   |    0 | NULL  | show processlist | 
+------+------+-----------+------+---------+------+-------+------------------+
1 row in set (0.00 sec)