Category Archives: Linux

Timekpr-nExT Remote

Leave a reply

tl;dr – Timekpr-Next Remote is an easy to use web app to add or remove time to users of linux login time tracking app, Timekpr-nExT

Recently, one of my kids got a drawing tablet and wanted to use it with Krita. Given they were on a Chromebook, we decided to repurpose an old Intel NUC i3 server with a clean Ubuntu 22.04 Desktop. Not only would this allow Krita to run well, but it would also enable video editing via KDEnlive and other hard-to-run on ChromeOS softwares.

For some time now, we’ve been happily running Legba to track computer usage. However, we wanted something with a bit more teeth, so we settled on Timekpr-nExT (henceforth just “Timekpr”, but it’s the “nExT” one, not this out of date one or this waaay out of date one, k?). This is a great app that allows for a finite amount of time to be used per day, and it is relatively easy to add more time. Well, easy if you’re on a desktop. And you have SSH installed. And you know the login and password to each computer you want to control. So, not at all easy if you’re a busy parent who’s juggling managing kids and helping with school work and cooking dinner. So, not at all easy if you’re a parent, amiright?!

Enter Timekpr-Next Remote! This is a Dockerized Python app that allows you to easily update update your kids’ computer time right from the nearest parental phone or desktop device:

As you can see, for any given user (only one sample user, “Muhammad”, is shown here) you can easily add more time (or remove time if you fat fingered the add time). Given how ubiquitous phones are, having a self hosted, non-cloud way to easily control time has been a win for us. Video chat with grandma after you’ve done your homework and used all your time allotment? Add -> 30 min -> Save, it only takes 3 seconds \o/

SSH FTW

Let’s have a look under the covers at how all this works.

I should start out by saying that Timekpr is licensed GNU GPL v3 and they post the code online. I did consider adding an HTTP server to handle REST requests to core package up stream. Then I realized I’d have to do the securely and that I’d have to deal with certificates and such. A greenfield approach would be quicker (grok only my code, not someone else’s) and more secure, but not as clean (I used SSH vs REST). With that out of the way…

Timekpr Remote uses SSH to communicate! This is clunky, indeed. But, Python has a wonderful SSH library in the form of Fabric (which turn uses the awesome Paramiko) which took a lot of the really clunky parts out and made them pretty elegant.

Here’s the flow of data when we load the page and want to get the current usage of Mumammad:

This “handwritten” diagram compliments of JS Sequence Diagrams – thank you!

All data flows this way, and there’s three AJAX endpoints which the web client sends via this flow:

  • Get all Users and IPs
  • Get usage for a user and IP pair
  • Add/Remove time for a user and IP pair

This isn’t a perfect REST API, but it’s OK enough. It was my first time writing an app in Flask, so it was fun to figure how to do different URL handling and JSON returning and such, even if my REST uses some GETs instead of POSTs/PUTs

“s” in Timekpr-next Remote is for “Security”

While we an trust SSH between Timekpr Remote and Server, you may note a lack of authentication between the mobile handset and Timekpr Remote. Indeed, there is none. Here’s what I recommend:

Run something like Traefik or Caddy in another docker container. From there you can bind the timekpr-next remote server to the host docker IP with something like TIMEKPR_IP=172.17.0.1 docker compose up -d. It will no longer be available on the network, only only via the revers proxy you set up.

You can then either use basicauth (eg in Caddy) or what I did is make a host name that is un-guessable like https://user-time-8957446623432192758492038.domain.com. Everyone just bookmarks this. Even if your kids see the URL, they won’t be able to remember it.

For those that give SSH the stink eye (smells like an injection attack, eh?), you can harden this too. Ensure the SSH user on the server can not do anything more than run timekpra by restricting it in the authorized_keys file on each client. This will ensure if extra variables are passed (though we do explicitly protect against this), they won’t do any more harm.

Like and Subscribe that video

Here’s a 9 second video demonstrating it in situ!

Jokes on you though, there’s no like and subscribe because it’s not actually YouTube (though GitHub is pretty close to a social media network…)

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Keys-To-The-Tunnel 1.1.0 released

Leave a reply

Hey hey! I’d been meaning to consolidate some of my VMs I’m paying for and semi-accidentally deleted the place where I was hosting my work’s instance of Keys-To-The-Tunnel (KTTT). With this VM deletion, I decided to leverage my recent dabblings with Caddy and do a big refactor of KTTT by replacing Apache with Caddy.

As a reminder, KTTT is an easy way for an organization which both develops web apps and uses GitHub to share their local dev app on the internet via a publicly accessible URL all protected with solid TLS and SSH encryption.

Let’s dive into what the update means and why I did it!

Parenthetical aside about TLS for local Android development

Originally I asked if potential KTTT users :

ever need to test an Android application against a web server such that you need a valid TLS certificate?

Not so good idea on my original KTTT post

This is actually a bad idea if all you need is a valid TLS cert for Android testing. The reason is that you’ll literally be holding your phone in your hand which is less than a foot from the computer hosting your app you want to test against. By introducing KTTT into the mix, you send your traffic thousands of miles/kilometers (you pick which) and back just to get a TLS cert. Crazy times!

A much better approach is to use something like local-ip.co . If you want an easy way to keep your traffic entirely local, you can use nginx-local-ip with a one line docker compose call to set up everything you need to locally run a local-ip.co TLS. It’s a really sweet set up! (I’m biased because I help author some of it ;)

KTTT is still a great idea if you need to share your local dev environment though!

With that out of the way, back to KTTT updates…

What’s new with KTTT

For an end user of KTTT who just wants to share their app, it’s now WAY easier to figure out which SSH command to run and which URL to share. This is thanks to the handy web app which walks you through three easy questions:

  1. What is your GitHub Username?
  2. What port is your app running on locally?
  3. Is your local app using http or https?

This is MUCH better than wading through a list of random ports and other GitHub usernames which you only cared about your username and port. Here’s what the updated web app looks like in action (15 second video):

For the administrators of KTTT, you’ll note that KTTT now uses Caddy instead of Apache. While there’s nothing wrong with Apache, Caddy is a simpler take on the needs of an app like KTTT that requires a bunch of small reverse proxies. Caddy is 7 years old and came to being in the world of Docker, containers and micro-services. Where as Apache is 27 years old and came being near the birth of world wide web.

Ironically, a key feature of Caddy, the ability to automatically provision and renew TLS certs, is NOT being used. Instead, the opportunity to use a wildcard TLS cert came up via acme-dns.io and I took it.

That all said, it’s a joy to use Caddy because I can create a simple JSON file with four lines to define a reverse proxy:

mrjones-plip.awesome-tunnel.plip.com {
   tls /etc/certs/fullchain.pem /etc/certs/privkey.pem
   reverse_proxy 127.0.0.1:3089
}

Love it!

Putting it all together

The web app above gives a pretty good idea of the improvements, but since I added a demo video of the whole KTTT experience on GitHub, may as well post it here in case you’re curious (44 second video):

Other odds and ends in 1.1.0

There’s also a bunch of other fixes and improvments I made while in there. Here’s the notes from the 1.1.0 release:

  • Replace Apache with Caddy
  • Add mini web app to help devs figure which URL and SSH command to use
  • Unify mutli SNI certs to one wildcard with acme-dns.io (still using Let’s Encrypt though) per #1
  • Don’t overwrite existing user’s ports every time you run setup per #3
  • Don’t regenerate TLS certs every time you run setup per #3
  • Don’t rewrite vhosts in web server every time you run setup per #3
  • Update MOTD on login per #4
  • Try my hand at being an artist and create a KTTT logo
FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Legba the Net-tracker

1 Reply

Intro

I’d been meaning to learn how to write an app using something more than CSV files, but less than MariaDB, to store files – I’m thinking SQLite of course! Then along came the desire to have a simple way to track when a computer was on a network as a proxy for kids’ daily screen time. After all, the network is the computer, right?

While there’s so very many ways to solve detecting if a computer is online (more on this later), I thought it’d be fun to write a simple app that could correlate multiple IPs to a single person, and then give a histogram of minutes per day per person. Given this is just a proxy for screen time, it’s fine if it doesn’t have alerting, password protection or even a way to prevent going over the allotted time per day. The goal will be for any interested parties to see how long a device has been on for the current day. It’s then up to the family to have a discussion about what it means to go over your daily allotment.

Ok, let’s do this! We have a requirement to track computers being online and to write and read the results to a SQLite DB. I’ve been groovin’ on learning Python, so let’s double down and use that. I did some Wikipedia exploring and read about Papa Legba, and thought it made a mighty fine sounding name. Finally, after some nudging from a friend, we’ll package it up in Docker so it’s easy to try out and host in an isolated container.

Ping FTW

The first step to using Legba, is to define a list of users and which IPs they’ll be on. Very likely the best way to do this is to either use static IPs on your LAN clients, or have your DHCP server set the same IPs per MAC every time.

Then you’ll create a conf.py file copied from the conf.example.py file and fill it out. Here we see Jon and Habib have one IP each, where as Mohamed has 2:

trackme = {
    'Jon': ["192.168.1.82"],
    'Habib': ["192.168.1.12"],
    'Mohamed, ': ["192.168.1.240", "192.168.1.17"]
}

The code to track if a device is achieved via the subprocess module via a ping() function with just two lines that send a single ICMP packet:

# thanks https://stackoverflow.com/a/10402323
def ping(host):
    """ Ping a host on the network. Returns boolean """
    command = ["ping", "-c", "1", "-w1", host]
    return subprocess.run(args=command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL).returncode == 0

Back in the main() function, we then read in config, loop over each person and try and ping() each of their IPs. If we see them online, we write to the DB via record(). It ended up, just as I’d hoped, that Python’s SQLite libraries are robust and it’s just 6 lines to insert a row:

sql = ''' INSERT INTO status(name,state,date) VALUES(?,?,?) '''
cur = sqlite.cursor()
activity = (name, state, datetime.now())
cur.execute(sql, activity)
sqlite.commit()

return cur.lastrowid

Just before the end of the loop we call probably the most complex function of the lot output_stats_html(). This function is responsible for reading the day’s active users, getting each users activity by hour, the total for the day and finally output static HTML as well as a static JSON file that will get called via AJAX so the stats will auto-refresh.

At the end of the loop we sleep for 60 seconds. In theory if you had hundreds (thousands?!) of IPs to track and they were on connections with >500ms latency, it would take way longer than 60 seconds. Legba will not scale to this level. It’s currently been comfortably tested with 5-10 devices on a LAN where each device has ~20ms of latency.

A histogram is worth a 1000 words

After you’ve done a bit of a git clone with a lil pip3 install and fleshed out your own config.py and done a little systemd love, you’ll have some sweet sweet histograms! (Some keen eyed readers may note this histogram looks familiar ;)

It’s interesting to note that mobile devices, as seen withe “Adnon Cell”, are effectively on all the time. In this sense, Legba is not much use to track a cell phone. Meanwhile, Bobby Table’s desktop, Adnon’s Laptop and Chang’s Nintendo Switch all work as expected (NB – I didn’t actually test with a Switch).

Existing Solutions

I’ve been running this for solution for just about 4 months now. It’s been a great way for our family to have an open discussion about what it means to spend too much time on the computer and it’s been rock solid. Checking ls and select count(*) from status; I see my DB is 23MB and has 487,069 rows.

Given the simplicity of this app, could this DB and rows be easily stored and retrieved elsewhere? When I wrote the app, I didn’t care – I just wanted to write it for the fun of writing it! However, I was listening to episode 171 of Late Night Linux and they mentioned how utilitarian Telegraf is. It struck me that, indeed, if you had a Telegraf, InfluxDB and Grafana (aka “the TIG stack”) already set up, it would be pretty trivial to capture these same stats. I would do this by setting up a centralized instance of Telegraf and either use the built in Ping plugin, or possibly the more extensible [[inputs.exec]] input type. With the latter, you could even re-use parts of Legba to pretty trivially input the data to InfluxDB. Then, it would be equally trivial, to slice up the ping counts per hour, per user and have a slick dashboard. Just food for thought!

Otherwise, I hope some else than me gives Legba a try!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

DockStat: Docker stats in a simple to use and easy to read Bash script

Leave a reply

Intro

At work I’ve been doing a lot of Docker based projects. I ended up writing a neat little Bash utility which I then recently extended into what I’m calling DockStat. It shows running containers and their related resources. You could use it if you’re repeatedly upping, downing and destroying docker containers over and over like I was. Or maybe you just want a nice little dashboard to see what’s running on your server?

DockStat at work

However, if you have more than say a dozen active containers, this script might not scale nicely (oh, perhaps a monitor in portrait mode might fix this? ;)

Being the good little open source nerd that I am, this is of course available for download with a permissive license in the hopes that some one will find it useful or possibly even offer a PR with some improvements to my nascent Bash coding skills.

Background

With countless primers on how to use Docker out there, I won’t get into what the commands all mean, but the impetus for this script was repeatedly running docker ps to show a list of the active containers. A bit later I remembered you could run endless Bash loops with a one-liner which made the process a bit nicer as it auto-refreshed:

while true; do clear;date;docker ps;sleep 5; done

A bit after that I stumbled upon the glorious watch command! Wow – just when you think that that you know and OS, they come and show you that there’s this awesome command they’ve been hiding from you all these years. Thanks Linux!

watch greatly improved on my Bash one-liner as it was an even short one-liner, could trivially be configured to refresh at what ever frequency you wanted and show a header or not. The icing on the cake was that prevents the flash of a redraw upon refresh:

watch -t -n 1 docker ps

About now I got more cozy with the --format feature built into most Docker command line calls. This was handy because I could reduce the number of fields shown in docker ps that I wasn’t interested in. Here’s maybe the simplest of them which shows JUST the container name and if how long the it has been running:

docker ps --format='{{.Names}} {{.Status}}'

Research continued on how to architect the helper script. I had need to show different data than docker ps had to offer. I branched out into docker inspect as well as finding other Dockeristas one-liners that I shamelessly co-opted (I’d be honored if anyone did the same of my work!!). This allowed me to joined ps and inspect like as seen with this fave that shows all the running containers’ and their internal IPs:

docker ps --format='{{.Names}}'|xargs docker inspect --format='{{.Name}} - {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'

I was ready to assemble all the docker Data dukets I’d gathered into a nice CLI dashboard so our app developers could see the status of our containers booting. Finding a solution for this enabled both the the helper script to spring forth and simultaneously created the nascent DockStat. This was a Bash utility that was both easy to use, automates flash-less refreshes and introduced basic terminal layout functionality with a near zero learning curve (assuming you know Bash): Bash Simple Curses

Thanks

Thanks to James and Russ for reviewing my code and an early draft of this post. I’ve been trying to improve both my posts and code and this won’t happen without folks kind donation of their time and input!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Portable, headless Raspberry Pi development

Leave a reply

The Problem

Back when I was writing Cattrotar on a Raspberry Pi (or really, any other of the myriad times I was hacking on a small embedded device), I often faced some sort of problem or another:

  • I thought I set up the network for a device, but can no longer remotely access it over WiFi
  • I’ve just written a new OS to a microSD card, want to further configure the OS but it’s not yet reachable on the network
  • I want to take my set up to the hacker space to be social while I work on an issue
  • I’m waiting in the library for my kids to finish up a class and can’t easily get my Pi on the library WiFi to hack on it from my laptop
  • I don’t want to carry around, or even bother setting up, a keyboard and monitor to get access to my Pi

The Solution

As many of you lovely readers may know, I’m already a fan of small travel routers made by GL.iNet, like the fine GL-MT300N-V2. For a scant $20 dollars (shipped!) you get DHCP server you can not only bring with you, but will happily run off a battery. Better yet, because it has a USB port, you can actually power your Pi off the wee router. Daisy chain USB power for the win! The final icing on the cake is that you can make the two Ethernet ports on be both LAN (instead of LAN & WAN). While these are great devices, their WiFi stack is a bit flaky, so being able to hardwire your network devices is a great, if not slightly cumbersome, work around.

The net result is that your around town bag can have:

  • A USB Battery
  • Two very short Ethernet cables
  • A USB Ethernet dongle (w/ USBC adapter tah boot)
  • A travel router (GL-MT300N-V2 in this case)
  • Two micro USB cables
  • Pi (In the picture below I have an Orange Pi Zero with a temp sensor and a screen threaded through one side of a case)

When you put it all together, you get a nice tidy mess and it works just great, solving all of the above problems. Your laptop can be on two networks so it can have local access to the Pi and to the internet too. Further, if the WiFi your on isn’t too hostile with their captive portals and such, you can actually have the GL.iNet router act as a WiFi repeater and backhaul that bandwidth to your Pi. Run your apt update away!

This set up is not only small and lightweight, but lets me work un-tethered on my Pi setup and Python code. Now if this frickin pandemic would just be solved, I’d actually be able to take this out of the house instead of just co-working with my partner in our office in our house!

Happy Pi Hacking!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Replacing two iPods with a Bash script

Leave a reply

13 years ago we got an iPod to actually listen to music on the go. It was awesome! Some time later we had our first kid and some time after that smart phones became prevalent, so our iPod fell out of use. But it was about 9 years ago that we started to use the old iPod as an easy way to play the same a bedtime playlist for our kids (oh yeah, we had a 2nd kid at some point too ;). Soon, the kids split into their own bedrooms so we picked up a used iPod on eBay and loaded up the same bedtime playlist onto it. These two iPods dutifully played the same songs day in and day out every night for years. They’re even more utilitarian than designed, but worked well at their finite task (Photo by Nicnicoleleeolee):

However, over time things started to not work. First one of the fancy docks we used as a speaker stopped charging which ever iPod was in it. This meant every week or so we’d have to swap it over to the good dock to charge up. But then one started to lock up and had to be rebooted. We feared we’d have to replace them soon.

It was around this time that I starting using the Cast All The Things software (aka catt) in my quest to both (finally) learn Python and to easily control the volume on a chromecast. Checkout my cattmate and cattrotar projects! catt is a command line script and python library that allows you to easily control and play videos/music on Chromecasts. It was also around the time that Chomecast Audio’s were stopped being made so I’d stocked up them. We have about 4 or 5 plugged in here and there about the house, including one in each kids room. They work really well!

By now you can likely see how this post is going to resolve itself, but lets find out, shall we?

Yes, that’s right, I wrote a full featured web site that you could pull up and easily play music in which ever room needed to hear the play list. It was simple, yet fancy ;) On the front end there was a series of 2 buttons, one for each room. When you pressed a button, it sent an AJAX call to a to the single endpoint on the back end. The back end then made an exec call to a bash script. This in turn used catt to play a MP3 on the specified Chromecast.

This worked great for a day or so. But then a bug reared it’s ugley had. Let me tell you, the cost of a software bug in your web app which involves waking your child up at 10.30pm is EXTREMELY high. Like, unacceptably high. Sleeping children are gold. Parents get grown up time, the kids get much needed rest and we’re all better for it the next day. Don’t fuck with a kid’s sleep.

After some unfruitful debugging, I got lazy and realized I’d already installed the wonderful termux on my phone, along with a $2 add on to have a bash script launcher widget on my phone’s desktop. So, after a dozen or so minutes of coding and a little ssh-keygen -t ed25519 for good measure, I had this on my phone:

Here’s what happens when you press one of those four links:

  1. Call a local script on the phone with the same name shown above
  2. Each script has the same contents but just calls the remote server with a different argument: ssh napserver controlmusic.sh CHILD PLAY_STOP. So for the first one above, that’d be ssh napserver controlmusic.sh e play
  3. The remote server, (hardened of course to only allowing one command to be run for that SSH key,) inside controlmusic.sh runs a catt command: /usr/bin/catt -d DEVICE COMMAND OPTIONAL_COMMAND. Again for the first button that’d look like: /usr/bin/catt -d "E's Chromecast" cast songs.mp3

So, while not nearly has fancy as the web app I initially wrote, it works every time, has saved me the time of debugging the web app (aka let me be lazy ;) and most importantly, does not wake the kids up after they’ve gone to sleep! Icing on the cake is that continued my lazy streak and bought the app for my partner’s phone so they could activate the playlist while I’m out of town (instead of me VPNing in and activating it remotely on request ;).

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Kids DNS in the Time of Covid

Leave a reply

Like all of you parents lucky enough to still have a job during the COVID19 layoffs, I’ve been struggling to balance time at work, personal time, family time and being the family’s IT person. With school closed, and now all summer camps closed, our use of kids screen time (aka internet time) has gone up from 0.5hrs/day to 3hrs+/day. How do we ensure we have safe computing environment for them?

DoT by Design

Originally, we had a MacOS workstation for the kids with parental controls enabled. This allowed us to do things like set up a 30 min per day limit, create separate accounts for each kid, limit which apps they could use and, most importantly, limit which URLs they could use (deny all, allow some). When coupled with my love of LXD/Pi-Hole/Quad9, that looked like this:

In this scenario the kid’s single shared workstation would get an IP lease from the DHCP server running on the pfSense router. This lease would specify the house wide Pi-Hole which sent all it’s DNS clear text queries to Stubby which in turn sent them encrypted to Quad9 via DNS over TLS (DoT). This is really nice as not only do we do get LAN wide ad blocking, but we get LAN wide encrypted DNS too. Score!

The kids workstation gets no special treatment on the network and is a peer of every other DNS lease on the LAN. However, with them needing to do school work and have fun and learn over the summer, they’ve since each gotten their own workstations. Now we have three workstations! This is starting to be a hassle to maintain the lock down on which sites they can browse. As a result, we just told them “be good” and let them use their new workstations with out any filters. This is sub-optimal!

.* Blacklist

How can we improve this situation to make it more tenable and more secure? By adding more instances of Pi-Hole, of course! It’s trivial to add a new instance of Pi-Hole with LXD. Just add a new container lxc launch ubuntu: pi-hole2 and then install Pi-Hole on the new container with curl -sSL https://install.pi-hole.net | bash . It’s two one liners that take all of 5 minutes.

For those of you like me that want an easy way to export their existing whitelist from MacOS’s parental controls, check out the “Directory Service command line utility” aka dscl. With this command you can create a file with all the URLs you’ve whitelisted. You can then easily import them into your new Pi-Hole instance (be sure to swap out USERNAME for your user):

dscl . -mcxexport /Users/USERNAME com.apple.familycontrols.contentfilter|grep http|cut -d'>' -f2|cut -d'<' -f 1|sort|uniq

Back to the new Pi-Hole instance, if we set the upstream DNS server to be the initial Pi-Hole, this means the kids DNS gets all the benefits of the existing encrypted infrastructure, but can add their own layer of blocking. Here I configure their Pi-Hole to just use the existing Pi-Hole as the resolver:

Specifically, if you add .* as a blacklist, EVERY site on the internet will fail to resolve. Then you can incrementally add sites you want resolve to your whitelist:

Once we hard code each of the three workstations to use the new Kids DNS, we’re good to go! And, this indeed works, but the savvy technologist will see the time suck of a flaw in my plan: If you whitelist example.com, there’s 5 or more sites you need to whitelist as well in order for example.com to work. This is because 99% of all sites use 3rd party javascript via content delivery networks (CDNs), have integrations with social media and of course often use the ever present Google Analytics. It gets even more tricky because if you want to keep your kid from searching on Google, you can’t think, “Oh, I’ll just whitelist *.google.com and then all it’ll save a bunch of time!”. Along with that will come Gmail and who knows what ever else. I knew this issue would be there going in, so I wasn’t afraid to take the time to get it to work. But caveat emptor!

Teaching Kids to be Smart

Speaking of caveats of a plan – all parents should know that this plan is VERY easy to bypass once your kids starts to figure out how the internet and their specific devices work. I’ve literally told my kids what I’m doing (stopping just about every site from working) why I’m doing it (the internet can be a horrible place) and that they can likely figure a way around it (see Troy Hunt’s tweet – as well as his larger write up on parenting online).

Like Troy Hunt, I’ll be super proud when they figure a way around it – and that day will come! But I do want to prevent them from randomly clicking a link and ending up somewhere we don’t want them to be. They can then ask us parents about why they can’t access a site or when it might allowed.

Being honest with your kids about what you’re doing is the way for them to be aware that this is for their benefit. The end goal is not to lock the entire internet away forever, it’s actually the opposite. The end goal is to prepare them to be trusted with unfettered access to the internet. This will happen soon enough whether we parents want it or not!

Banning 8.8.8.8 et al.

While I was in there tuning up the DNS, I remembered that some clients on my network (I’m looking at you Roku!) weren’t listening to the DHCP rules about using my preferred, encrypted DNS and going direct to Google’s DNS (8.8.8.8) or others I didn’t like. After a little research I found I could redirect all outbound TCP and UDP DNS traffic so that all devices use my Pi-Hole/Stubby/Quad9 DNS* whether they thought they were or not. For others running pfSense and want to do this, see the steps to “Blocking DNS Queries to External Resolvers” and then “Redirecting all DNS Requests to pfSense” (both thanks to this Reddit thread).

* We shall not speak of how devices will soon speak DNS over HTTPs (DoH), thus ruining this idea.

What about product X?

Some of you may be thinking, “this seems like a lot of work, why don’t you just implement an existing off the shelf solution?” Good question! For one, I like to DIY so I control my data and what’s done with it instead of letting a 3rd party control it. As well, while there’s home based solutions, I prefer open source solutions. To put my money where my mouth is, I’ve just donated for the 2nd (3rd?) time to Pi-Hole. I encourage you to do the same!

To be clear though, this set up is a pretty crude tool to achieve the end result. It looks like there’s some quite polished solutions out there if you’re OK with closed source, cloud hosted solutions. As well, there’s of course other variations on the “Use Pi-Hole For Parental Controls“.

Wrapping Up

Now that we have all in this in place, we can trivially support N clients which we want to force to use the kids more lock down DNS set up. This looks like exactly like it did before, but we have an extra container in the LXD server (and, some what orthogonally, a fancier pfSense DNS blocking setup):

I suspect this set up won’t last for more than a year or two. As more and more sites get added to the white list, it will be harder and harder to maintain. Maybe after that I’ll give each kid their own Pi-Hole instance to run on an actual Raspberry Pi and let them do with it as they please ;)

(Of course just after I deployed this, Pi-Hole 5.0 came out which offer the concept of groups, so you can likely do this idea above in a single instance instead of multiple. A bummer for me now, but a win for all other Pi-Hole users, including my future use!)

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Asterisk, LXD, Wireguard VPN and Remote “Office”

2 Replies

You may remember that a while ago, I set up a fun little PBX for my kids. It was awesome! That setup allowed my partner and I to use our cell phones as SIP clients to the Asterisk instance running on the LXD server and my kids each had an analog phone going through the ATA:

Since then, I decided it would actually be pretty cool to have a phone in our kitchen so we could call upstairs to the kids. If I was gonna wire up 1 phone, I may as well wire up 3 phones and I may as well make them all awesome. Yes, you know it, I’m talking about deploying 3 of the venerable Cisco 7960s:

These phones, according to my research, will be 20 years old in August of next year. That’s 10 years older than my oldest kid. That’s….really old! Especially in internet time! Yet, these phones are indeed venerable. They simply work and won’t quit. Even when they do quit, all you need is a little cardboard and they’ll keep on goin’. I had a few laying around and they’re often posted for sale for $5-15 online. I won’t get into it in this post, but it is some what of an art to get them on the right SIP (not SCCP!) firmware. This guide has some good info as does Loligo’s. tl;dr – set up an TFTP server, set your DHCP with the TFTP option, tie your phones MAC to the right conf file, and away you go. Feel free to email me if you get stuck!

But, we’re getting ahead of ourselves. Before we could plug the phones in though, we had to string some Ethernet. This means that my kids learned the important life skill every 7 year old needs to know, how to crimp RJ45 cable ends:

After all 3 phones were physically connected to the network (and running SIP firmware per above), they could connect to the Asterisk instance on the LXD box. Now our set up looks like this (only two SIP phones are shown, we have 3 (actually I put one on my office desk recently, so now we have 4 :))

At this point, I nuked the vanilla Asterisk instance and installed the latest version of FreePBX. Now the kids no longer get to learn about busy signals, instead they get to learn about conference calls, hold music (but not THAT hold music sadly), voice mails and a house wide paging system. It is SO much fun! And, honestly, it’s super practical too.

I was talking to my sister recently and she’d heard the kids talk about their phones and how much they loved them. I asked if she wanted one at her house. Given our kids don’t have email or a cell phone, this would give my sister a direct way to contact her niece and nephew with no middle parent man. Let’s do it! But…how?

Let’s assume we just go for it. We’ll just program another phone we picked up off craig’s list to talk to the public IP of my house (no static IP, but that’s what Dynamic DNS is for), and we’ll punch a whole in the NAT Firewall Router thingy (a fanless doodad running pfSense). Asterisk uses SIP as we know, which is on port UDP 5060, so it’s pretty easy. We do a port forward like this – see red arrow:

This is a bad idea. On so many levels. First off, these hella old phones use only unencrypted tech. I mean, why use SSH when you have telnet? Why use TLS when you have good ol’ HTTP? SIP itself is unencrypted which means that any one of the many hops the traffic goes through will be able to trivially sniff the UDP packets used to authenticate against the Asterisk instance. Not only could they get on to my LAN, they could listen to all the calls. Nitpickers may note that Wikipedia speaks of SIP encryption – but that’s impossible on these old phones.

These types hacks are no theoretical either. Security researcher Ang Cui has made quite a name for him with all the vulns he’s found in these phones. In a Defcon 21 talk called “Stepping P3wns: Adventures in full spectrum embedded exploitation (and defense!)” he demonstrated how sending a resume (PDF) which would get printed on a (vulnerable) HP printer would allow a reverse tunnel to open up which could then be used hack the phone on the desk and silently enable the mic so he could listen to you discuss his “resume”. Awesome!! And scary ;) The same nitpicker as above will not this was the 7961, not the 7960 – still my OLDER phone is very likely less secure than the NEWER one.

Maybe I should encrypt the traffic? Like, what if we put a VPN server behind the firewall, do a port forward to it, and a VPN client at the remote “office”? That way the SIP traffic is never seen on the internet! Yeah!! Very similar to the diagram above, but with two more devices:

Now instead of unencrypted packets being forwarded to the Asterisk server, we only have encrypted packets being forwarded to the VPN server (again, see red arrow below). Further the remote phone uses the VPN (blue arrow) and thinks it’s on my home network – un-routable IP and all!

But where as we spent $15 before, we’ve reused existing phones with the new setup and VPNs sound hard and possibly expensive to deploy. Maybe it can’t be done the cheap-cheap? Dun dun dun!! Enter Wireguard! This insanely simple, radically secure and Sys Admin friendly VPN is great. I’ve deployed a bunch of instances now and can’t get enough of it. But what about the price of the hardware? Here’s where the final piece of this Asterisk, LXD, Wireguard VPN and Remote “Office” puzzle is put in place:

For just over $20 shipped you too can have an awesome VPN server aka the GL-MT300N-V2 made by GL Technologies (aka GL.iNet). They also work as clients too! While we’ve had to reboot the remote VPN and Phone once or twice, we’ve had months of up time using this set up. The router supports a slick GUI (what I ended up using) but if you’re retro, you can do it all manually too.

An added bonus to this whole set up is by adding a Wireguard client on my phone, I can now VPN in and use the SIP client where ever I am to call or be called.

Postscript: A few weeks ago we decided we’d experiment with letting the kids be at home alone for short periods. Per above, they have no cell phones and we have no land line. But with a perfectly good PBX in place already, I spent $4 getting a LocalPhone SIP trunk. We now pay $0.005 per outgoing call. Yes, you read that right, half a cent per call. Read more over at Ward Mundy’s site!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

Happy Hacker Halloween!

1 Reply

Last year I wanted to be a “Hacker” and code up a solution to show near by access points and nearby phones. I failed. However, I did a good job of brushing up on what I needed to do over the past year and so this year I was a hacker for reals. Here I am in the final get up:

hacker.jones.jpg

Let’s break it down! Here’s the hardware list (affiliate links to Amazon):

My final build out looked like this:

IMG_20191031_160240.jpg

A quick write up of the software is:

  1. install latest Rasbpian on your MicrSD card
  2. Install latest YANPIWS in /var/www/html/YANPIWS
  3. Install howmanypeoplearearound as the pi user. Ensure it’s path is /home/pi/.local/bin/howmanypeoplearearound
  4. Ensure you have all the libs for YANPIWS python scripts installed so you can talk to the BME280
  5. Hook up the BME280 to the right 4 pins on the Pi using the jumper cables
  6. Hook up the monitor to the Pi’s HDMI and the External WiFi adapter a USB port
  7. Have the Pi boot into kiosk mode with a browser that points to http://127.0.0.1 by following this awesome guide on pimylife.com. Note that you’ll only use the one URL and have no while loop in the kiosk bash script.
  8. Install Apache and PHP with sudo apt install apache2 php
  9. In /var/www/html/ put all of the files I just published on this gist. Basically it’s a small web app to show the data we’re collecting as well as some bash scripts that get run in cron.
  10. Install a bunch of cron jobs that gather the data as the pi user. This will use wlan0 (built in) to look for nearby access points using the venerable iw command. It will use wlan1 (USB adapter) to look for phones and such in monitor mode using howmanypeoplearearound. Finally, it will get the temp and humidity using the python script from YANPIWS. You may need to make /var/www/html writable by pi user to make this work.

It’s not my finest code, but if everything worked correctly, the Pi will boot up every time and show something like this:

IMG_20191031_190702.jpg

As you can see it got cold tonight on our walk – by the time we got home at 8pm it was 45. Happy Hacker Halloween!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail

NAT and Macvlan on production LXD (plus reverse proxy & SSH Config)

8 Replies

Intro and LXD install

At work recently I was charged with rebuilding a bare metal host. Beyond needing to follow our security best practices and be well documented, it was left up to me how to do it. I had my own needs for test VMs and there was a pending request for a VM* for semi-production instance. This meant some VMs* would be fine in a traditional NATed environment, where they had no publicly accessible interfaces, and others would need full fledged public IPs. (* – I’m using “VM” liberally in this post. These are technically LXD containers which use the host kernel.)

Given my penchant for LXD, I’m guessing you can see where this is going ;) If you don’t know my penchant, check out these posts, specifically, “From zero to LXD: Installing a private compute cloud on a Cisco C220 M4SFF“.

I won’t go as into nitty-gritty detail on the hardware setup (this time with an older c220 M3 LFF instead of the new M4 SFF), but I set up the system very similarly, but was forced to use a RAID10 set up on 4 drives – no fancy ZFS set up this time. I’ll see some performance and features lost as LXD was configured to just use filesystem (/var/lib/lxd), but given I have bare metal in a colo with as many VMs as I want, I’m happy ;)

After installing Ubuntu 18.04, giving it a static IP and running our Ansible hardening roles against it, I was ready to configure LXD. The nice thing about LXD is that you can have as many container profiles as you want. This means I can zip through the default lxd init process to have VMs which are behind NAT and then trivially add a new profile that allows hosts to have a public IP after that.

The initial config of LXD looks like this:

Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]:
Create a new BTRFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=15GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

After that, and HUGE thanks to this concise post by Simos Xenitellis, we can now configure a new profile with Macvlan for VMs that need a public IP. Simos’ post really covers this nicely (I even use their same code snippets ;), but by copying the default profile (lxc profile copy default lanprofile) and then setting the the nictype (lxc profile device set lanprofile eth0 nictype macvlan) and the parent (lxc profile device set lanprofile eth0 parent enp5s12) on the new profile, we’re ready to go. Note that this assume your bare metal’s nic is enp5s12 and your LXD VMs use eth0 (the default).

Network types: NAT, Bridge & Macvlan

But wait, what is Macvlan? And, just so we’re all clear, how does it differ from the default NAT set up or the fancy bridged set up in my earlier post? Let’s break it down:

  • Network Address Translation (NAT) – You’re very likely using this right now to read this post ;) NAT is what enables us to easily share a connection to the Internet with out everyone having a public IP. Have you seen IPs that start with 10.x.x.x, 172.x.x.x or 192.x.x.x? While not exclusive to NAT, they’re the most common IP ranges used in conjunction with it (See RFC 1918 for TMI). NAT allows a gateway to hand out these IPs which then can send traffic out to the Internet and, by modifying the ports used, send the responses back to the NATed host who originally made the requested.

    NAT is what LXD uses when you accept all the defaults in lxd init. This is super handy for testing and development! As well, we can use it to our advantage with a reverse HTTP proxy in production – more on this below.

  • Bridge – Bridges are a layer 2 connection that makes it appear as all devices are on the same network. This is convenient when you want all devices to work with the same IP range, either with public IPs or in your NATed network. This is how I set up LXD in the prior article. Any time a VM is created in LXD, it can see all hosts, but it does take a slightly more complex network set up on the bare metal.

  • Macvlan – I’ll quote this great write up on hicu.be to describe Macvlan, “[it] allows you to configure multiple Layer 2 (i.e. Ethernet MAC) addresses on a single physical interface. Macvlan allows you to configure sub-interfaces (also termed slave devices) of a parent, physical Ethernet interface (also termed upper device), each with its own unique (randomly generated) MAC address, and consequently its own IP address.”. This achieves the same result as bridges with one major caveat: host and VMs can not talk to each other. That is, your VMs won’t be able to talk to you bare metal LXD host and vice versa – caveat emptor!

Now that you know what the three setups are, and how easy it was to set up NAT (just accept LXD defaults) and how easy it is to set up Macvlan (3 command line calls) – let’s see what we can do with them!

Again per Simos’ post, we can easily create a new NATed VM and then a Macvlan VM like so:

lxc launch ubuntu: natVM
lxc launch -p lanprofile ubuntu: lanVM

To set a static IP on either host, assuming your running Ubuntu 18.04 like me, you’d just edit /etc/netplan/50-cloud-init.yaml. So let’s say I wanted to give the natVM IP .10 in the 10.x.x.x range that LXD gave me and use Quad9 for DNS. I’d edit50-cloud-init.yaml to look like this:

network:
   version: 2
   renderer: networkd
   ethernets:
     eth0:
      dhcp4: no
      addresses: [10.0.0.10/24]
      gateway4: 10.0.0.1
      nameservers:
        addresses: [9.9.9.9]

This ends the part of the post where we talk about NAT and Macvlan both easily co-existing on LXD. Now on to what you might do with that set up! Specifically, how you might use Apache to forward on HTTP requests on a public IP to a NATed VM.

Apache reverse proxy

If you wanted to run lots of VMs, none of which needed a public IP, but a few needed to run a public service, you might wonder how to best do this? In my case, I had a small amount of public IPs, so burning one for every VM was a big waste. A better way is to just selectively forward some HTTP traffic from the bare-metal host’s public IP to a NATed VM’s IP. I’m an Apache kinda person, but this could be done with your web server of choice. It goes with out saying, but this trick will only work with HTTP traffic. I’ll speak to being able to SSH “directly” to any NATed hosts below!

Let’s get started by installing apache2 on the Ubuntu bare-metal host and enable some key modules: 

apt install apache2
a2enmod ssl rewrite proxy proxy_http

Now edit /etc/apache2/ports.conf  so that it’s listening on any ports you need – in our example it’s 3000 (Grafana) and 8086 (InfluxDB) so we’ll add just two lines:

<IfModule ssl_module>
   Listen 443
   Listen 3000
   Listen 8086
</IfModule>

Assuming you want to run a service on 8086 (InfluxDB) and a service on 3000 (Grafana) on the VM we configured above on .10, you’d create a vhost file called /etc/apache2/sites-available/influxdb-int.conf and it would look like this:

<VirtualHost *:3000>
         ServerName grafana-int.example.com
         LogLevel warn
         SSLEngine on
         SSLCertificateFile /etc/httpd/ssl.crt/your.crt
         SSLCertificateKeyFile /etc/httpd/ssl.key/your.key
         ProxyRequests Off
         <Proxy *>
             Require all granted
         </Proxy>
         ProxyPass / https://10.0.0.10:3000/
         ProxyPassReverse / https://10.0.0.10:3000/
 </VirtualHost>
 <VirtualHost *:8086>
         ServerName influxdb-int.example.com
         LogLevel warn
         SSLEngine on
         SSLCertificateFile /etc/httpd/ssl.crt/your.crt
         SSLCertificateKeyFile /etc/httpd/ssl.key/your.key
         ProxyRequests Off
         <Proxy *>
             Require all granted
         </Proxy>
         ProxyPass / http://10.0.0.10:8086/
         ProxyPassReverse / http://10.0.0.10:8086/
 /VirtualHost>

Note that this assumes you’re running everything over TLS (you should!!). As well, it assumes that your cert (SSLCertificateFile) and key (SSLCertificateKeyFile) are in /etc/httpd/ssl.key . Change these according to your specifc set up.

From here, you would follow the set up your apps to ensure they’re working locally on .10 and they should work on the public ip of your bare metal. Of course these all need to be configured to use TLS over the default HTTP. Huh – sounds like a whole “How to harden your TIG deployment” might be in order! (Of course, store any passwords encrypted when automating your deployments.)

Secure SSH to NATed LXD hosts

A final note on this set up is how to securely SSH to LXD hosts. Of course you can just SSH to your bare metal host and then bash in (eg lxd exec natVM bash), but how do you run your Ansible roles against these NATed VMs or another automation tool? SSH config files to the rescue!

Let’s assume your public IP of your bare metal is 1.2.3.4 and you want to ssh to the 10.0.0.10 IP we just set up above. All you need to do is create a file in your .ssh folder called “config” with 3 lines like this:

Host natVM
   Hostname 10.0.0.10
   ProxyCommand ssh -W %h:%p 1.2.3.4

With this set up, you can run ssh natVM and your config will automatically see the configuration to securely proxy the command through the 1.2.3.4 host through to your internal only .10 host. This works especially well when you have SSH Keys set up with SSH Agents.

Drop me a note if you have any questions!

FacebooktwitterredditpinterestmailFacebooktwitterredditpinterestmail