Category Archives: Linux

New Love: Meego

0 minutes, 44 seconds

A bit ago I read a post about Meego 1.0 being available. I had a Acer great netbook that was suffering from a slow slow install of XP. I’d been thinking of going to Linpus, which originally shipped with the Aspire Ones. However, Meego had great live, bootable USB download which allowed me to give the whole OS a spin on my hardware. Everything just works: webcam, USB bluetooth dongle, wifi, NIC and internal SD Card.

Last night, I took the plunge and installed it over XP.

Today, I’m happy to report I’m never going back to XP on this lil’ guy. I got Dropbox, KeepassX, Synergy and sshd all working with out a lot of hassle. The boot time is insanely fast. Google Chrome is WAY faster than FF3.6 in XP. I am a very happy camper.

Go Meego today! Full disclosure – sleep looks to be broken :(

Update: It looks like AIM is broken too. That’s a real bummer because it’s my main IM network.

Another Very Poor Man’s Google Analytics Post

1 minute, 15 seconds

A bit ago I wrote a post about using command line tools to get stats of this blog. I recently wrote another version of this to get the most popular posts here, sorted by the most popular at the top. I love that this can be done in all in one command.

Here’s the command:

tail -1000000 access_log|grep 'GET /blog'|cut -d" " -f 7|egrep -v '.png|.jpg|wp-includes|.css|/page/|/category/|xmlrpc|wp-trackback|/feed/|wp-login|/wp-content/|/trackback/|wp-comments|wp-app.php|wp-admin|comment-page|index.php|?p=|page_id|comments|feed'|sort|cut -d"/" -f 3|uniq -c|grep -v ' 1 '|sort -nr>plip.blog.tops.txt

This breaks down into the following:

  • get the last 1000000 of the blog access log
  • look for requests to “/blog”
  • split by space, and get the 7th field, the URL being requested
  • exclude a ton of items
  • sort the results
  • split by the “/” slash and get the 3rd field, the blog name in the URL
  • get the unique list of blog names with a count for each URL
  • remove the singletons
  • reverse sort so the most popular is at the top
  • write it all to a file called plip.blog.tops.txt

The results are in! The winner is currently chocolate-crinkle-cookie-photos! W00T

   137 chocolate-crinkle-cookie-photos
   119 two-loves-css-recaptcha
   109 24-hours-in-photos
   104 our-pet-venus-fly-trap
   103 ruby-less-way-to-add-key-frames-to-flv-videos-for-the-likes-of-jwplayer
    94 toss-your-salad-code
    91 update-firefox-does-have-reset-more
    91 firefox-reset-is-really-launch-in-safe-mode
    84 keep-those-passwords-safe
    81 photos-food-bikes-sunsets-and-stars
    79 thoughts-on-very-large-monitors
    78 when-the-cat-is-away-the-worms-will-play
    76 photos-from-around-the-bay
    76 our-tree
    75 one-foggy-morning-in-my-commute
    74 wordpress-exploit-fog-fruit-plants-and-plates
    72 recaptcha-now-google-recaptcha-will-help-google-books
    72 from-burning-man-town-to-oaktown
    67 gmaps-pedometer-google-calc-8-94607843-minutes-per-mile
    66 the-massive-compost-tower
    65 on-theft-privacy-and-data-loss
    64 pizza-and-dough-from-scratch
    60 this-is-not-an-ipad
    60 go-faster-encoding
    57 fixed-theme-wp-updated-more-wp-hacks
    44 every-vehicle-is-a-prius
    42 photorec-to-the-rescue
    41 the-very-very-poor-mans-google-analytics-tail-cut-sort-uniq-wc
    41 on-comcast-internet
    38 taking-the-plunge-safari-4-full-time
    35 secret-jumps-of-tunnel
    35 i-got-four-cores-but-a-distributed-load-aint-on-one
    34 stir-fry-dinner
    33 tasty-comfort-food
    32 fancy-diff
    26 how-to-fix-zend-studio-5-5-zde-in-os-10-6-snow-leopard
    24 ping-traceroute-and-quotes
    22 wordpress-rich-mans-blog-poor-mans-cms
    21 new-news-old-open-source
    20 old-broken-usb-hub-ipod-charger
    19 gmail-contest
    19 alternate-way-to-have-google-analytics-track-pdfs
    17 this-is-what-makes-a-happy-saturday
    17 macchiato
    16 american-born-chinese
    15 rogue-mysql-queries
    15 fixed-gear-slipped-chain-thankful-for-brake
    13 simple-wp
    13 plip-is-no-longer-a-cobblers-child
    11 plix-plixing-better
    11 itunes-imovie-on-lenovos-new-media-center-pc
    10 wonderful-bike-lane-signs
    10 this-is-what-makes-a-happy-sunday
    10 plip-ts-on-your-back
     9 plipgo-01-released
     9 bart-speaks
     8 yet-another-redesign
     7 update-plip-content
     7 plixing-for-pleasure
     7 plip-for-peace
     7 long-be-gone
     7 kodiak-11-released
     7 dot-com-casualty
     7 dont-just-commit-commit-intelligently
     6 verge-works-solves-all-your-woes
     6 simpsons-for-ever
     6 simple-is-better
     6 plip-gets-its-own-dictionary
     5 aids-ride-completed

Fixed theme, WP updated, more WP hacks

0 minutes, 32 seconds

Back when I fixed this blog’s instance of wordpress, I broke the theme. Today I fixed it. W00T! Sadly, this is because I found even more instances of a hacked wordpressery. This was based on the remv.php hacks. I can not stress enough how important it is to keep your software up to date. Go subscribe to the wordpress release feed if you need to. Just stay up to date.

While finding the instances of remv, I took a peek at it. It’s actually pretty neat! If you have LAMP stack handy and are interested, here’s a copy. Enjoy!

Update: Here’s the home page of PHPRemoteView (aka remv.php) translated from russian. Checkout the rockin windows 95 style windows!

Keep those passwords safe

1 minute, 52 seconds

A problem with online security is that there’s no standards for passwords. You may come up with the silly simple single password you use for all site. This works well, until you encounter a site that wants one that is, say, 2 characters longer than the one you use. What then? Or, maybe you’re a bit better and use a scheme where you “encrypt” the domain name into your password. Again, this works fine until a site forces you to break this scheme, and then you’re shit out of luck. The net result is that you either A) have extremely simple passwords or B) forget the passwords often or C) write them down next to your computer.

In case you didn’t think so, options A, B and C suck. Don’t do it. Be smart and be safe.

The way to do this involves some pain in the ass security, which I’ve said before different levels are acceptable. I feel that protecting your passwords are critical, so I’m willing to accept a somewhat higher PITA level. My PITA of choice for passwords is KeePass. Being an open source project (W00T!), some of the execution of the user interface is left to the developer, so you may find some ports are better than others. However, the vanilla OS X and Windows flavors I use at work and home respectively, simply rock. The Android port I use is the icing on the cake. I can download a copy of my password file and have all my passwords on the go.

KeePass, much like TrueCrypt, has really thought about how to store passwords. Here’s a list of some of the great features:

  • generate a secure password based a given site’s rules (8+ letters, 1+ number etc.)
  • hot keys to quickly copy username and password
  • android version puts the username in alert menu so you can easily copy and paste it into a web form
  • encrypt notes for extra info like security question you also won’t remember and wrote down next to your password
  • ported to just about every platform, including iphone and android

The net result of this is that you never forget a password, you use secure passwords and no one can get at your passwords. This is secure and this is how you should do it! For the forward thinking, store your password file on a USB cary with you or, if you’re like me, you’ll put it on your dropbox account, and then you can seamlessly use it on all your computers. Doubly handy!

On theft, privacy and data loss

4 minutes, 58 seconds

I’ve recently taken a serious look at the reality of theft of computers as well as ensuring privacy and reducing data loss should such a theft occur. Take a moment and and take inventory of where you store you data and how accessible and backed up it is. What would happen if:

    • You dropped your cell phone in the toilet? (data loss)
    • Your cell phone got stolen? (data loss & theft)
    • What about that ‘it will never be stolen’ desktop computer at home? What if some one stole that? Do you have a password on login, do you have your email password saved and your browser remember all your passwords? Do you file your taxes online and store copies? Do have resume with references, previous address and social security number? (Data loss, theft, and loss of privacy)

With the sheer number of accounts we create at every new site we register with, we’ve become lazy and no longer want to remember passwords. Most folks either check ‘remember password’ in their browser of choice, use the same password for every site, write down the password or all three. Further, most folks don’t password protect their smart phones. Compliment with all personal data stored on a laptop or a desktop, this creates a recipe for catastrophic data loss, serious violation of privacy potentially leading to identity theft.

In this post, I’m going to outline a number of suggestions to help fight data loss and identity theft while protecting your privacy. I’ll give each suggestion a PITA rating of how hard and how long it will take to implement. A PITA of 1 is easy a PITA of 10 is, well a real PITA!

Master Passwords

PITA: 1
If there’s one thing you actually do, do this one. I use both Thunderbird and Firefox. As I said, most folks are lazy and want all their passwords stored and remembered for them as needed. This is all fine and dandy until your laptop walks an the thief can use all your accounts with out ever touching the keyboard. Both Firefox and Thunderbird offer the ability to set a master password. Every time you open your browser or mail client and a password is needed, you will be prompted for a your master password once. Then all other passwords will be filled in for as normal. Note: close your browser and mail client often ; )

OS Level Passwords

PITA: 1
A no brainer. Both Windows and OS X (video) have it.

Set a phone password

PITA: 9, then less
When I first set a password on my phone it was a real pain in the ass. I had to enter it every time I wanted to make a call or check my email. I have a Palm OS phone (not a Web OS phone), so it’s not that sophisticated. I know that Android based phones have a quick pattern you can trace which is quite easy. The iphone has a number pad you can use. Any which way, over time, this becomes second nature, so the PITA rating will fade from 9 to a lower number.

Remote Backup

PITA: 7
I’ll explain my backup technique in a sec, but take this one seriously. If your house were to burn down while you were out of town, how much data would you loose? I hate to be all doom and gloom here, but most folks don’t shoot analog any more, so all photos are digital. Maybe you upload to flickr or the like, but there’s nothing like have all your photos organized just so. I strongly suggest you look at commercial providers for this.

Our household has both local and off site backup. We start by backing up all devices (two laptops and a desktop) to our local qnap server via robocopy (think rsync for windows). The qnap is just linux, so it runs all sorts of great things like rsync and secure copy (scp). A friend has a qnap as well. We first connected an external drive to our own qnaps and made a backup of all our backups. We then swapped external drives and connected them to our respective qnaps. We now have a remote backup that we can rsync our data to over ssh and the initial gigs of data are already there.

Remote, Secure Backup

PITA: 10
I have a subset of my data that is hundreds of PDFs. I generate them via my trusty and some what spendy sheet feed scanner. This guy creates searchable PDFs that have the OCR text embedded in them. Genius. In comes a bill, tax return or sensitive document, out comes a PDF and some shredded paper to recycle. Cross cut, of course.

This data set is a treasure trove. Should my desktop with hundreds of megs of PDFs walk out my front door one sunny afternoon in a thief’s hands I’d be up you know what creek with out a paddle. Data loss aside, it would be little effort to apply for a credit card after a little address change. Bad times indeed.

Enter Drop Box. This is an excellent free service (for the first 2 gigs) that allows you to do what I would call very rich man’s rsync. Store all your PDFs in this folder, and now they’re not only backed up, but remotely accessible!

Wait – what about the stolen desktop? The default behavior of Drop Box is to remember your password. Should someone take your computer and gain access to it, the PDFs on Drop Box are good as local on the drive.

Now enter the second layer: TrueCrypt. TrueCrypt is the no-joke way to store data securely. They support both an encrypted boot drive as well as the spook spy stuff: plausible deniability. Ignoring the more advanced features, TrueCrypt’s quick start guide will walk you through creating an encrypted volume that’s encapsulated in a single file. This file can be any size, thus the drive can be any size. You could then store you hundreds of PDFs in a TrueCrypt volume in a file on Drop Box. True secure offsite backup.

I’ve even gone so far as to create a small TrueCrypt volume that has all my passwords. It’s the keys to the kingdom, but I’m going to be vigilant about protecting this file and only the one closest to my heart knows how to get in there.

Caveat emptor

I’m not a security expert, take my advice with a big grain of salt. There are ways of hacking the master password for firefox and thunderbird. OS Level passwords are trivial to bypass for a skilled IT professional or evil intentioned googlist. Even TrueCrypt can be accessed via social engineering or a sloppy operator who writes down their password.

Good luck and happy securing!

The very, very poor man’s Google Analytics: tail, cut, sort, uniq & wc

1 minute, 6 seconds

I still have what most would call an unfounded fear of privacy when it comes to Google. They may receive a copy of every email I send to my friends who use gmail, they may place every call to me via Google Voice, they may server every ad from Double Click (which I then block) and I sure as heck never stray from their bad-ass search on google.com, but I don’t host anything with them directly.

I’ve run my share of web analizer tools, but some times I wanna know, right now, “how many people subscribe to my blog feed?”. Now, I probably should be using FeedBurner (No shit – I did not know, ’til just this second, that they too are now owned by Google. Oh, the irony!), but my site, despite its claims, is still a bit of the cobbler’s child when it comes to analytics. Heck, I still don’t have mod_usertrack on!

Enter tail, cut, sort, uniq and wc!

tail -10000 access_log|grep /blog|cut -d" " -f 1|sort|uniq|wc

In layman’s term’s that’s “get the last 10000 lines of my access log, cut each line into fields separated by the space character, grab the first field (the IP address in this case), sort the resulting lines of now just an IP address per line, remove the duplicates and count the number resuling lines (or IP addresses)”. Presto! 388 of you out there, including all the bots, spiders, crawlers, trolls and goblins. Thanks for the interest!

Ruby-less way to add key frames to flv videos for the likes of JWPlayer

2 minutes, 2 seconds

At work we’ve been working on a good way to roll our own videos. We initially started with the generic off the shelf swf player + flv to make our videos go. This was OK, but was lacking some key features folks were used to, primarily full screen play back and the ability to seek to a specific spot in a video. Additionally, for videos that were over 20 minutes being viewed over slower connections, precious apache children would be chewed up potentially causing a slow down for the web head doing the serving.

Enter JWPlayer! This is a great, easy to configure, free for non-commercial use, flv player that offers the features we’re looking for. Further, we could offload the flv’s to our image server, lighttpd . (By the way lighttpd is extremely awesome; I can not recommend it enough. I first learned about from my friend over at WikiSpaces where they had dropped apache entirely in favor or lighttpd. Noteworthy is that lighttpd was first concieved as an attempted answer to the c10k problem. I’ve personally seen it handle over 2000 connections per minute and the server load didn’t go above 0.5 (though the full meaning of load average is a curious one (sorry for the double, now triple parenthetical statements )).) Simply add a little flv streaming foo to lighttpd, and you’re good to go.

At this point, we hit a stumbling block. In order for JWPlayer to seek, it would send a request for the FLV to the web server and give it a starting point, like so:

10.1.6.221 lighttpd.domain.dom - [29/Jul/2009:12:46:37 -0700] 
"GET /web_assets/video/your.flv?start=17038076 HTTP/1.1" 200
 17765145 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; 
en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1"

However, if the flv you were serving didn’t have key frames then the web server would simply ignore the query string, and indeed, the click all together.

We do most of our development and video authoring on a mac and then serve our files off a load balanced linux server. At first, we used flvtool2 on linux to embed the key frames on our flvs. This kinda works, but it’d be easier to have the video author be able to add them himself with out needing to download and install ruby and rails and all that crazy server scripting foo. As his flv creator/codec/authoring app/chumpy wasn’t playing nice, I was given the task to find a simple solution.

Now, finally, we get to the point of the post. If you’ve gotten this far and wanna know how to embed keyframes with out ruby, get thee to the multi-platform, command line tool called flvmeta and go home happy!

Update: This post would not be complete without mentioned our use of swfobject to render the flash HTML.

Rogue MySQL queries

1 minute, 32 seconds

Do you you ever have those moments where you’re trying out a query in MySQL and realize after you executed it that it’s going to kill your database server? If you’re like me, you’re often working in a LAMP stack in which case you’ve executed the query via a web page. The first thing you do is hit “esc” to stop the page from loading. MySQL doesn’t head the “full stop” call from Apache, if issued. If you’re silly and you did this on some sort production machine, you don’t exactly want to restart Apache or even worse, restart MySQL which can often take a while.

Enter show processlist; and kill PID! Yeah, mysql has it’s on version of “ps” and “kill -9”. In our case, this is extremely handy because it saves us the headache of rebooting MySQL and taking the DB offline for a minute. Connect to your DB as root and type:

mysql> show processlist;
+------+------+-----------+------+---------+------+----------------------+--------------+
| Id   | User | Host      | db   | Command | Time | State                | Info         |
+------+------+-----------+------+---------+------+----------------------+--------------+
| 1693 | root | localhost | rei  | Query   |   75 | Copying to tmp table | SELECT     di| 
| 1695 | root | localhost | NULL | Query   |    0 | NULL                 | show processl| 
+------+------+-----------+------+---------+------+----------------------+--------------+

Indeed this one query is killing the CPU:

top - 08:57:50 up 30 days, 20:59,  2 users,  load average: 0.53, 0.14, 0.04
Tasks:  72 total,   2 running,  70 sleeping,   0 stopped,   0 zombie
Cpu(s): 20.1% us, 30.1% sy,  0.0% ni, 49.8% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:   2074628k total,  2010740k used,    63888k free,   142648k buffers
Swap:   524280k total,      192k used,   524088k free,  1549692k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                              
 3119 mysql     16   0  549m 115m 5076 S 99.8  5.7   9:53.40 mysqld                                                                                                                                

All we have to do is kill it and the box is back to idle. Sweet!

mysql> kill 1693;
Query OK, 0 rows affected (0.00 sec)

mysql> show processlist;
+------+------+-----------+------+---------+------+-------+------------------+
| Id   | User | Host      | db   | Command | Time | State | Info             |
+------+------+-----------+------+---------+------+-------+------------------+
| 1695 | root | localhost | NULL | Query   |    0 | NULL  | show processlist | 
+------+------+-----------+------+---------+------+-------+------------------+
1 row in set (0.00 sec)

Fancy Diff

0 minutes, 53 seconds

simplediff
simple diff

Let’s say you’ve edited a long HTML file and want find what’s changed against the original. If you were on the command line, you’d turn to ‘diff’, right? This is all fine and dandy, if not mundane and some what obfuscated as to what has changed.

 
fancydiff
fancy diff

However, if you wanted to view the entire files side by side, have the changes appear in context of the rest of the file and easily see where lines have been removed or add, you would use diff with the flags: ‘-yiEbwB’. Now that is a I can get used to.

 
fancydiff
megawide fancy diff

Now if you had a massive file, say a WSDL (say “wizdull”) file from a certain CRM you use at work and you wanted to compare differences, diff to the rescue again. However because we’re seeing the entire file which is over 450K, the only way to parse through this is via ‘less’. If you couple this with increasing the width to 300 columns, it makes reviewing the changes a breeze. Further, you can search with in the diff by using the search features in less.

diff -yiEbwB --width=300 enterprise.wsdl enterprise2.wsdl |less