Intro

For some time I’ve had all my home lab systems running on LXD. For me at least, LXD predated Docker and so I’ve stuck with it. The containers are a bit more pet like and less cattle like – but that’s OK. They’re there for me to learn with – plus I can totes do docker in LXD inception if I wanna! (I’ll figure my way over to Incus some other month)

So years and years ago I found out bout FreshRSS as a great way to self host an RSS reader. This meant I didn’t have to install any apps on my phone, my feeds could stay synced and since I already had a VPN setup, it was trivial to access while on the road. Lovin’ it! I’d set this up around late 2018.

Fast forward to yesterday, I flippantly upgraded FresshRSS to the latest, as I occasionally do, and the site started having a fatal error, as it often does after I upgrade ;) I pulled up the error logs on the FressRSS LXD container running Apache and MariaDB and immediately saw some sort of unexpected { character in function blah blah on line 1-oh-smang-thirty error. Huh, what’s that about?

Turns out in the latest dev release of FreshRSS, they’d formally removed support for PHP 7.x and required PHP >8.0. Specifically, this was because they’re using Union Types in their function declarations. This is cool stuff! you can see the int|string values in the sample function here:

// Declaring a function with Union Type
function getMixedValue(int|string $value): int|string {
   return $value;
}

This is no problem! I’ll just update PH….P…. oh yeah – I’m on hella old Ubuntu 18…so then I’ll just find some third party apt repo and add that and then…. Hrrmm…might be more trouble than it’s worth. That’s cool! I’ll just deploy a new Ubuntu container on 24.04. Oh, well, that’s gonna take a chunk of disk space – I’ve been using a bit of Alpine to build small Docker images at work, what about a pet Alpine in LXD? They have an image – why not!

Preparing the leave the old system

Before we sudo rm -rf / on the old box (er, container), let’s get our data outa there. We need to first make a dump of the database. We’re root so we can just zing right through any permissions with a one liner. Next up we can zip up our old files into one big ol’ honkin zip file. That looks like this:

mysqldump freshrss > ~/freshrss.sql
cd /var/www/localhost/htdocs/
zip -r ~/freshrss.zip .

Finally, we can generate an SSH key for this root user to easily copy to the new container – I knowingly didn’t add a password because I’m about to delete the container and we’re all friends here:

ssh-keygen -t ed25519 
cat /root/.ssh/id_ed25519.pub

Ok – I’ll hold on to that pub key for the “One last trip to the old digs” section below.

Bless the new hotness

Here’s how to bloop out an Alpine container in LXD:

lxc launch images:alpine/3.20 freshrss
lxc shell freshrss

That’s it! You’re now sitting as root on the new instance. Let’s install the base packages including Apache, MariaDB, OpenSSH and PHP with all it’s libraries:

apk add \
   mariadb mariadb-client openssh \
   apache2 apache2-http2 php83 \
   php83-cli php83-apache2 php83-session php83-curl \
   php83-gmp php83-intl php83-mbstring php83-sqlite3 \
   php83-xml php83-zip php83-ctype php83-fileinfo \
   php83-dom php83-pdo

Now let’s ensure the three services start at boot and then we can start Apache and OpenSSH (MariaDB will have to wait):

rc-update add apache2
rc-update add sshd
rc-update add mariadb
rc-service apache2 start
rc-service sshd start

As well, that pub key you got from the old server? Let’s add that in on the new server:

mkdir ~/.ssh
echo "ssh-ed25519 AAAAC3Nz-SNIP-NplQ3 root@freshrss-old" > ~/.ssh/authorized_keys
chmod 700 ~/.ssh/
chmod 600 ~/.ssh/authorized_keys 

One last trip to the old digs

As final hurrah at the old server, now that our SSH key is on the new server, let’s copy over the zip archive and the SQL dump. Be sure to replace 192.168.68.217 with your real IP!

scp ~/freshrss.* 192.168.68.217:

Go SQL!

Now that we have our server with all the software installed and all the data copied over, we just need to pull together all the correct configs. First, let’s run setup for MariaDB and then harden it. Note that it’s called mysql… , but that’s just for backwards compatibility:

rc-service mariadb setup
rc-service mariadb start
mysql_secure_installation  

That last command will ask questions – default answers are all good! And the initial password is empty, so you can just hit return when prompted for the current password. Maybe check out passphraseme if you need a password generation tool? Let’s add the database, user and perms now. Be sure to not use password as your password though!

echo "CREATE USER 'freshrss'@'localhost' IDENTIFIED BY 'password';" | mysql
echo "GRANT ALL PRIVILEGES ON *.* TO 'freshrss'@'localhost' WITH GRANT OPTIO" | mysql
echo "GRANT ALL PRIVILEGES ON *.* TO 'freshrss'@'localhost' WITH GRANT OPTION;" | mysql

Now we can load up the SQL and move all the PHP files to their correct home. Again, we’re root so no SQL password, and again, this is actually MariaDB, not MySQL:

mysql  freshrss < freshrss.sql 
unzip freshrss.zip 
mv * /var/www/localhost/htdocs/.
mv .* /var/www/localhost/htdocs/.

Go Apache!

Apache just needs four updates – easy!

Edit /etc/apache2/httpd.conf with your favorite editor. Find these three lines and uncomment them – they won’t be next to each other:

#LoadModule rewrite_module modules/mod_rewrite.so
#LoadModule session_module modules/mod_session.so
#LoadModule remoteip_module modules/mod_remoteip.so

Now find the one line where DocumentRoot is set and change it to this value and add two more lines. One to allow encoded slashes and one to set the server name. Be sure to use the IP address or FQDN of the server – don’t use rss.plip.com!

DocumentRoot "/var/www/localhost/htdocs/p"
AllowEncodedSlashes On
ServerName rss.plip.com:80

Now that apache has been configured, let’s restart it so all the settings are loaded:

rc-service apache2 restart

Conclusion

The old FreshRSS install should now be running on your new Alpine based container – congrats! This has been a fun adventure to appreciate how Alpine works as compared to Ubuntu. This really came down to two main differences:

• systemd vs OpenRC – Ubuntu has used systemd for sometime now and the primary interface to that is systemctl. Alpine on the other hand uses OpenRC which you interface with rc-update and rc-service. Alpine picked this up from when it split off from Gentoo.
• apt vs apk – Package management is slightly different! I found this to be an inconsequential change.

There’s plenty of guides out there that do the same as this one. Heck, you’re likely better of just using a pre-built docker image (though the top results were pinned to PHP7)! However, I wanted to document this for myself and hopefully I’ll save someone a bunch of little trips off to this wiki or that FAQ to understand how to migrate off of Ubuntu to Alpine.

Cheers!

tl;dr – An open source bash script which provisions a server to terminate TLS traffic with a valid certificate and reverse proxies the traffic back to a web based development instance via an SSH tunnel. Enables sharing of a dev instance and testing Android apps.

Intro

Are you an ngrok user (or one of its many competitors)? Do you use GitHub (GH) or work in a GH Organization? Is part of your application a web server? Do you ever need to test an Android application against a web server such that you need a valid TLS certificate? Have you ever wished it’d be easy to share you local dev instances with a colleague? If yes, then maybe Keys-To-The-Tunnel is for you!

Keys-To-The-Tunnel converts a newly provisioned, dedicated Ubuntu instance into a multi-user server that can both terminate inbound TLS traffic as well as gives developers easy to follow instructions on how to connect their web based instance of their app to it. By using easy to get public SSH keys for your users, Keys-To-The-Tunnel takes the legwork out of setting up a reverse proxy.

This post discusses the impetus and some of the background ideas that make Keys-To-The-Tunnel possible. If you’re all ready to get started, the GitHub repository and the FAQ have everything you need!

DNS & TLS via Let’s Encrypt

The first cornerstone to Keys-To-The-Tunnel is that Let’s Encrypt offers free TLS certificates. Once the script creates the certificates, they’re automatically renewed because that’s how certbot works. Free certs with free renewals – love it!

By using valid certificates with root CAs already deployed in Android, the resulting vhosts in a Apache allow you to test your Android apps against the URLs. Further, if you’re working with other members of your organization on the project, they can easily access your local instance via your URL. As well, while desktops allow responsive mode testing, sometimes you just need to see your app/site in a true mobile browser to properly test.

A critical part of making this all work is that you need both an A record and a wildcard in your DNS for the host you’re on. This allows Let’s Encrypt to verify your certificate no matter which hostname it is as hostnames are dynamically generated off GH handles.

Usernames & Keys

The second cornerstone of Keys-To-The-Tunnel is that GH allows anyone to know your public SSH key(s) (e.g. here’s mine). When you couple this with the fact that GH has an API to retrieve all the members of an organization (needs a token), it means that given just the organization name, the script can provision dozens of accounts with just a one liner:

./installTunnelServer.sh domain.com you@domain.com

Any GH users that do not have an SSH key are skipped, but you can simply re-run the script to add a user who has since added an SSH key to their account. In a soon to be released version, you can run the script in a cronjob so it will both add and remove users when your GH organization changes.

SSH

SSH is the final of three corner stones of the project. By using SSH, specifically an SSH tunnel, something which every developer has already installed, it’s easy to quickly set up an ad hoc tunnel. Here’s example command that your users will be given:

ssh -T -R REMOTEPORT:127.0.0.1:LOCALPORT GH-HANDLE@domain.com

So if you had a web app running on http://localhost:8080 and your GH Handle was mrjones-plip and Keys-To-The-Tunnel had assigned you a port of 1234, you would run this SSH command to set up a the tunnel:

ssh -T -R 1234:127.0.0.1:8080 mrjones-plip@domain.com

When you run the SSH command, you know you’re successfully connect when you’re greeted with this text:

Connected to SSH tunnel server
Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-51-generic x86_64)
Press 'ctrl + c' to exit

When Keys-To-The-Tunnel is done running, it will write an index.html to the root of your bare domain instructing users how to use the service:

Caveat Emptor

There are some gotchas to this script which you should be aware of:

• Unlike ngrok which generates a random hostname every time your connect, Keys-To-The-Tunnel always uses the same hostnames. While handy for on-going use, it means the URLS may be discovered and scraped/crawled. As developer instances often have weak or no passwords, care should be used to tear down the tunnel when not in use (or use strong passwords – or both!).
• Highly distributed teams may consider deploying multiple instances of Keys-To-The-Tunnel so they’re closer to the team members. If a developer is in Kenya and is connecting to a server in Canada, their traffic might be MUCH slower than if the server was deployed very close to Kenya from a packet’s perspective.
• While this should Just Work, this script was only tested on LXD containers and Digital Ocean (referral link) Ubuntu droplets. Please open a ticket if it doesn’t work!

Like all of you parents lucky enough to still have a job during the COVID19 layoffs, I’ve been struggling to balance time at work, personal time, family time and being the family’s IT person. With school closed, and now all summer camps closed, our use of kids screen time (aka internet time) has gone up from 0.5hrs/day to 3hrs+/day. How do we ensure we have safe computing environment for them?

DoT by Design

Originally, we had a MacOS workstation for the kids with parental controls enabled. This allowed us to do things like set up a 30 min per day limit, create separate accounts for each kid, limit which apps they could use and, most importantly, limit which URLs they could use (deny all, allow some). When coupled with my love of LXD/Pi-Hole/Quad9, that looked like this:

In this scenario the kid’s single shared workstation would get an IP lease from the DHCP server running on the pfSense router. This lease would specify the house wide Pi-Hole which sent all it’s DNS clear text queries to Stubby which in turn sent them encrypted to Quad9 via DNS over TLS (DoT). This is really nice as not only do we do get LAN wide ad blocking, but we get LAN wide encrypted DNS too. Score!

The kids workstation gets no special treatment on the network and is a peer of every other DNS lease on the LAN. However, with them needing to do school work and have fun and learn over the summer, they’ve since each gotten their own workstations. Now we have three workstations! This is starting to be a hassle to maintain the lock down on which sites they can browse. As a result, we just told them “be good” and let them use their new workstations with out any filters. This is sub-optimal!

.* Blacklist

How can we improve this situation to make it more tenable and more secure? By adding more instances of Pi-Hole, of course! It’s trivial to add a new instance of Pi-Hole with LXD. Just add a new container lxc launch ubuntu: pi-hole2 and then install Pi-Hole on the new container with curl -sSL https://install.pi-hole.net | bash . It’s two one liners that take all of 5 minutes.

For those of you like me that want an easy way to export their existing whitelist from MacOS’s parental controls, check out the “Directory Service command line utility” aka dscl. With this command you can create a file with all the URLs you’ve whitelisted. You can then easily import them into your new Pi-Hole instance (be sure to swap out USERNAME for your user):

dscl . -mcxexport /Users/USERNAME com.apple.familycontrols.contentfilter|grep http|cut -d'>' -f2|cut -d'<' -f 1|sort|uniq

Back to the new Pi-Hole instance, if we set the upstream DNS server to be the initial Pi-Hole, this means the kids DNS gets all the benefits of the existing encrypted infrastructure, but can add their own layer of blocking. Here I configure their Pi-Hole to just use the existing Pi-Hole as the resolver:

Specifically, if you add .* as a blacklist, EVERY site on the internet will fail to resolve. Then you can incrementally add sites you want resolve to your whitelist:

Once we hard code each of the three workstations to use the new Kids DNS, we’re good to go! And, this indeed works, but the savvy technologist will see the time suck of a flaw in my plan: If you whitelist example.com, there’s 5 or more sites you need to whitelist as well in order for example.com to work. This is because 99% of all sites use 3rd party javascript via content delivery networks (CDNs), have integrations with social media and of course often use the ever present Google Analytics. It gets even more tricky because if you want to keep your kid from searching on Google, you can’t think, “Oh, I’ll just whitelist *.google.com and then all it’ll save a bunch of time!”. Along with that will come Gmail and who knows what ever else. I knew this issue would be there going in, so I wasn’t afraid to take the time to get it to work. But caveat emptor!

Teaching Kids to be Smart

Speaking of caveats of a plan – all parents should know that this plan is VERY easy to bypass once your kids starts to figure out how the internet and their specific devices work. I’ve literally told my kids what I’m doing (stopping just about every site from working) why I’m doing it (the internet can be a horrible place) and that they can likely figure a way around it (see Troy Hunt’s tweet – as well as his larger write up on parenting online).

Like Troy Hunt, I’ll be super proud when they figure a way around it – and that day will come! But I do want to prevent them from randomly clicking a link and ending up somewhere we don’t want them to be. They can then ask us parents about why they can’t access a site or when it might allowed.

Being honest with your kids about what you’re doing is the way for them to be aware that this is for their benefit. The end goal is not to lock the entire internet away forever, it’s actually the opposite. The end goal is to prepare them to be trusted with unfettered access to the internet. This will happen soon enough whether we parents want it or not!

Banning 8.8.8.8 et al.

While I was in there tuning up the DNS, I remembered that some clients on my network (I’m looking at you Roku!) weren’t listening to the DHCP rules about using my preferred, encrypted DNS and going direct to Google’s DNS (8.8.8.8) or others I didn’t like. After a little research I found I could redirect all outbound TCP and UDP DNS traffic so that all devices use my Pi-Hole/Stubby/Quad9 DNS* whether they thought they were or not. For others running pfSense and want to do this, see the steps to “Blocking DNS Queries to External Resolvers” and then “Redirecting all DNS Requests to pfSense” (both thanks to this Reddit thread).

* We shall not speak of how devices will soon speak DNS over HTTPs (DoH), thus ruining this idea.

What about product X?

Some of you may be thinking, “this seems like a lot of work, why don’t you just implement an existing off the shelf solution?” Good question! For one, I like to DIY so I control my data and what’s done with it instead of letting a 3rd party control it. As well, while there’s home based solutions, I prefer open source solutions. To put my money where my mouth is, I’ve just donated for the 2nd (3rd?) time to Pi-Hole. I encourage you to do the same!

To be clear though, this set up is a pretty crude tool to achieve the end result. It looks like there’s some quite polished solutions out there if you’re OK with closed source, cloud hosted solutions. As well, there’s of course other variations on the “Use Pi-Hole For Parental Controls“.

Wrapping Up

Now that we have all in this in place, we can trivially support N clients which we want to force to use the kids more lock down DNS set up. This looks like exactly like it did before, but we have an extra container in the LXD server (and, some what orthogonally, a fancier pfSense DNS blocking setup):

I suspect this set up won’t last for more than a year or two. As more and more sites get added to the white list, it will be harder and harder to maintain. Maybe after that I’ll give each kid their own Pi-Hole instance to run on an actual Raspberry Pi and let them do with it as they please ;)

(Of course just after I deployed this, Pi-Hole 5.0 came out which offer the concept of groups, so you can likely do this idea above in a single instance instead of multiple. A bummer for me now, but a win for all other Pi-Hole users, including my future use!)

You may remember that a while ago, I set up a fun little PBX for my kids. It was awesome! That setup allowed my partner and I to use our cell phones as SIP clients to the Asterisk instance running on the LXD server and my kids each had an analog phone going through the ATA:

Since then, I decided it would actually be pretty cool to have a phone in our kitchen so we could call upstairs to the kids. If I was gonna wire up 1 phone, I may as well wire up 3 phones and I may as well make them all awesome. Yes, you know it, I’m talking about deploying 3 of the venerable Cisco 7960s:

These phones, according to my research, will be 20 years old in August of next year. That’s 10 years older than my oldest kid. That’s….really old! Especially in internet time! Yet, these phones are indeed venerable. They simply work and won’t quit. Even when they do quit, all you need is a little cardboard and they’ll keep on goin’. I had a few laying around and they’re often posted for sale for $5-15 online. I won’t get into it in this post, but it is some what of an art to get them on the right SIP (not SCCP!) firmware. This guide has some good info as does Loligo’s. tl;dr – set up an TFTP server, set your DHCP with the TFTP option, tie your phones MAC to the right conf file, and away you go. Feel free to email me if you get stuck!

But, we’re getting ahead of ourselves. Before we could plug the phones in though, we had to string some Ethernet. This means that my kids learned the important life skill every 7 year old needs to know, how to crimp RJ45 cable ends:

After all 3 phones were physically connected to the network (and running SIP firmware per above), they could connect to the Asterisk instance on the LXD box. Now our set up looks like this (only two SIP phones are shown, we have 3 (actually I put one on my office desk recently, so now we have 4 :))

At this point, I nuked the vanilla Asterisk instance and installed the latest version of FreePBX. Now the kids no longer get to learn about busy signals, instead they get to learn about conference calls, hold music (but not THAT hold music sadly), voice mails and a house wide paging system. It is SO much fun! And, honestly, it’s super practical too.

I was talking to my sister recently and she’d heard the kids talk about their phones and how much they loved them. I asked if she wanted one at her house. Given our kids don’t have email or a cell phone, this would give my sister a direct way to contact her niece and nephew with no middle parent man. Let’s do it! But…how?

Let’s assume we just go for it. We’ll just program another phone we picked up off craig’s list to talk to the public IP of my house (no static IP, but that’s what Dynamic DNS is for), and we’ll punch a whole in the NAT Firewall Router thingy (a fanless doodad running pfSense). Asterisk uses SIP as we know, which is on port UDP 5060, so it’s pretty easy. We do a port forward like this – see red arrow:

This is a bad idea. On so many levels. First off, these hella old phones use only unencrypted tech. I mean, why use SSH when you have telnet? Why use TLS when you have good ol’ HTTP? SIP itself is unencrypted which means that any one of the many hops the traffic goes through will be able to trivially sniff the UDP packets used to authenticate against the Asterisk instance. Not only could they get on to my LAN, they could listen to all the calls. Nitpickers may note that Wikipedia speaks of SIP encryption – but that’s impossible on these old phones.

These types hacks are no theoretical either. Security researcher Ang Cui has made quite a name for him with all the vulns he’s found in these phones. In a Defcon 21 talk called “Stepping P3wns: Adventures in full spectrum embedded exploitation (and defense!)” he demonstrated how sending a resume (PDF) which would get printed on a (vulnerable) HP printer would allow a reverse tunnel to open up which could then be used hack the phone on the desk and silently enable the mic so he could listen to you discuss his “resume”. Awesome!! And scary ;) The same nitpicker as above will not this was the 7961, not the 7960 – still my OLDER phone is very likely less secure than the NEWER one.

Maybe I should encrypt the traffic? Like, what if we put a VPN server behind the firewall, do a port forward to it, and a VPN client at the remote “office”? That way the SIP traffic is never seen on the internet! Yeah!! Very similar to the diagram above, but with two more devices:

Now instead of unencrypted packets being forwarded to the Asterisk server, we only have encrypted packets being forwarded to the VPN server (again, see red arrow below). Further the remote phone uses the VPN (blue arrow) and thinks it’s on my home network – un-routable IP and all!

But where as we spent $15 before, we’ve reused existing phones with the new setup and VPNs sound hard and possibly expensive to deploy. Maybe it can’t be done the cheap-cheap? Dun dun dun!! Enter Wireguard! This insanely simple, radically secure and Sys Admin friendly VPN is great. I’ve deployed a bunch of instances now and can’t get enough of it. But what about the price of the hardware? Here’s where the final piece of this Asterisk, LXD, Wireguard VPN and Remote “Office” puzzle is put in place:

For just over $20 shipped you too can have an awesome VPN server aka the GL-MT300N-V2 made by GL Technologies (aka GL.iNet). They also work as clients too! While we’ve had to reboot the remote VPN and Phone once or twice, we’ve had months of up time using this set up. The router supports a slick GUI (what I ended up using) but if you’re retro, you can do it all manually too.

An added bonus to this whole set up is by adding a Wireguard client on my phone, I can now VPN in and use the SIP client where ever I am to call or be called.

Postscript: A few weeks ago we decided we’d experiment with letting the kids be at home alone for short periods. Per above, they have no cell phones and we have no land line. But with a perfectly good PBX in place already, I spent $4 getting a LocalPhone SIP trunk. We now pay $0.005 per outgoing call. Yes, you read that right, half a cent per call. Read more over at Ward Mundy’s site!

Intro and LXD install

At work recently I was charged with rebuilding a bare metal host. Beyond needing to follow our security best practices and be well documented, it was left up to me how to do it. I had my own needs for test VMs and there was a pending request for a VM* for semi-production instance. This meant some VMs* would be fine in a traditional NATed environment, where they had no publicly accessible interfaces, and others would need full fledged public IPs. (* – I’m using “VM” liberally in this post. These are technically LXD containers which use the host kernel.)

Given my penchant for LXD, I’m guessing you can see where this is going ;) If you don’t know my penchant, check out these posts, specifically, “From zero to LXD: Installing a private compute cloud on a Cisco C220 M4SFF“.

I won’t go as into nitty-gritty detail on the hardware setup (this time with an older c220 M3 LFF instead of the new M4 SFF), but I set up the system very similarly, but was forced to use a RAID10 set up on 4 drives – no fancy ZFS set up this time. I’ll see some performance and features lost as LXD was configured to just use filesystem (/var/lib/lxd), but given I have bare metal in a colo with as many VMs as I want, I’m happy ;)

After installing Ubuntu 18.04, giving it a static IP and running our Ansible hardening roles against it, I was ready to configure LXD. The nice thing about LXD is that you can have as many container profiles as you want. This means I can zip through the default lxd init process to have VMs which are behind NAT and then trivially add a new profile that allows hosts to have a public IP after that.

The initial config of LXD looks like this:

Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]:
Create a new BTRFS pool? (yes/no) [default=yes]:
Would you like to use an existing block device? (yes/no) [default=no]:
Size in GB of the new loop device (1GB minimum) [default=15GB]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like LXD to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]:

After that, and HUGE thanks to this concise post by Simos Xenitellis, we can now configure a new profile with Macvlan for VMs that need a public IP. Simos’ post really covers this nicely (I even use their same code snippets ;), but by copying the default profile (lxc profile copy default lanprofile) and then setting the the nictype (lxc profile device set lanprofile eth0 nictype macvlan) and the parent (lxc profile device set lanprofile eth0 parent enp5s12) on the new profile, we’re ready to go. Note that this assume your bare metal’s nic is enp5s12 and your LXD VMs use eth0 (the default).

Network types: NAT, Bridge & Macvlan

But wait, what is Macvlan? And, just so we’re all clear, how does it differ from the default NAT set up or the fancy bridged set up in my earlier post? Let’s break it down:

• Network Address Translation (NAT) – You’re very likely using this right now to read this post ;) NAT is what enables us to easily share a connection to the Internet with out everyone having a public IP. Have you seen IPs that start with 10.x.x.x, 172.x.x.x or 192.x.x.x? While not exclusive to NAT, they’re the most common IP ranges used in conjunction with it (See RFC 1918 for TMI). NAT allows a gateway to hand out these IPs which then can send traffic out to the Internet and, by modifying the ports used, send the responses back to the NATed host who originally made the requested.
  
  NAT is what LXD uses when you accept all the defaults in lxd init. This is super handy for testing and development! As well, we can use it to our advantage with a reverse HTTP proxy in production – more on this below.
  
  
• Bridge – Bridges are a layer 2 connection that makes it appear as all devices are on the same network. This is convenient when you want all devices to work with the same IP range, either with public IPs or in your NATed network. This is how I set up LXD in the prior article. Any time a VM is created in LXD, it can see all hosts, but it does take a slightly more complex network set up on the bare metal.
  
  
• Macvlan – I’ll quote this great write up on hicu.be to describe Macvlan, “[it] allows you to configure multiple Layer 2 (i.e. Ethernet MAC) addresses on a single physical interface. Macvlan allows you to configure sub-interfaces (also termed slave devices) of a parent, physical Ethernet interface (also termed upper device), each with its own unique (randomly generated) MAC address, and consequently its own IP address.”. This achieves the same result as bridges with one major caveat: host and VMs can not talk to each other. That is, your VMs won’t be able to talk to you bare metal LXD host and vice versa – caveat emptor!

Now that you know what the three setups are, and how easy it was to set up NAT (just accept LXD defaults) and how easy it is to set up Macvlan (3 command line calls) – let’s see what we can do with them!

Again per Simos’ post, we can easily create a new NATed VM and then a Macvlan VM like so:

lxc launch ubuntu: natVM
lxc launch -p lanprofile ubuntu: lanVM

To set a static IP on either host, assuming your running Ubuntu 18.04 like me, you’d just edit /etc/netplan/50-cloud-init.yaml. So let’s say I wanted to give the natVM IP .10 in the 10.x.x.x range that LXD gave me and use Quad9 for DNS. I’d edit50-cloud-init.yaml to look like this:

network:
   version: 2
   renderer: networkd
   ethernets:
     eth0:
      dhcp4: no
      addresses: [10.0.0.10/24]
      gateway4: 10.0.0.1
      nameservers:
        addresses: [9.9.9.9]

This ends the part of the post where we talk about NAT and Macvlan both easily co-existing on LXD. Now on to what you might do with that set up! Specifically, how you might use Apache to forward on HTTP requests on a public IP to a NATed VM.

Apache reverse proxy

If you wanted to run lots of VMs, none of which needed a public IP, but a few needed to run a public service, you might wonder how to best do this? In my case, I had a small amount of public IPs, so burning one for every VM was a big waste. A better way is to just selectively forward some HTTP traffic from the bare-metal host’s public IP to a NATed VM’s IP. I’m an Apache kinda person, but this could be done with your web server of choice. It goes with out saying, but this trick will only work with HTTP traffic. I’ll speak to being able to SSH “directly” to any NATed hosts below!

Let’s get started by installing apache2 on the Ubuntu bare-metal host and enable some key modules:

apt install apache2
a2enmod ssl rewrite proxy proxy_http

Now edit /etc/apache2/ports.conf  so that it’s listening on any ports you need – in our example it’s 3000 (Grafana) and 8086 (InfluxDB) so we’ll add just two lines:

<IfModule ssl_module>
   Listen 443
   Listen 3000
   Listen 8086
</IfModule>

Assuming you want to run a service on 8086 (InfluxDB) and a service on 3000 (Grafana) on the VM we configured above on .10, you’d create a vhost file called /etc/apache2/sites-available/influxdb-int.conf and it would look like this:

<VirtualHost *:3000>
         ServerName grafana-int.example.com
         LogLevel warn
         SSLEngine on
         SSLCertificateFile /etc/httpd/ssl.crt/your.crt
         SSLCertificateKeyFile /etc/httpd/ssl.key/your.key
         ProxyRequests Off
         <Proxy *>
             Require all granted
         </Proxy>
         ProxyPass / https://10.0.0.10:3000/
         ProxyPassReverse / https://10.0.0.10:3000/
 </VirtualHost>
 <VirtualHost *:8086>
         ServerName influxdb-int.example.com
         LogLevel warn
         SSLEngine on
         SSLCertificateFile /etc/httpd/ssl.crt/your.crt
         SSLCertificateKeyFile /etc/httpd/ssl.key/your.key
         ProxyRequests Off
         <Proxy *>
             Require all granted
         </Proxy>
         ProxyPass / http://10.0.0.10:8086/
         ProxyPassReverse / http://10.0.0.10:8086/
 /VirtualHost>

Note that this assumes you’re running everything over TLS (you should!!). As well, it assumes that your cert (SSLCertificateFile) and key (SSLCertificateKeyFile) are in /etc/httpd/ssl.key . Change these according to your specifc set up.

From here, you would follow the set up your apps to ensure they’re working locally on .10 and they should work on the public ip of your bare metal. Of course these all need to be configured to use TLS over the default HTTP. Huh – sounds like a whole “How to harden your TIG deployment” might be in order! (Of course, store any passwords encrypted when automating your deployments.)

Secure SSH to NATed LXD hosts

A final note on this set up is how to securely SSH to LXD hosts. Of course you can just SSH to your bare metal host and then bash in (eg lxd exec natVM bash), but how do you run your Ansible roles against these NATed VMs or another automation tool? SSH config files to the rescue!

Let’s assume your public IP of your bare metal is 1.2.3.4 and you want to ssh to the 10.0.0.10 IP we just set up above. All you need to do is create a file in your .ssh folder called “config” with 3 lines like this:

Host natVM
   Hostname 10.0.0.10
   ProxyCommand ssh -W %h:%p 1.2.3.4

With this set up, you can run ssh natVM and your config will automatically see the configuration to securely proxy the command through the 1.2.3.4 host through to your internal only .10 host. This works especially well when you have SSH Keys set up with SSH Agents.

Drop me a note if you have any questions!

At work the other day I was testing our Ansible instructions on how to get a development environment set up. Given that this was supposed to be platform agnostic and that I exclusively develop on Ubuntu and LXD, I found an old Mac Mini on Craigslist to run VirtualBox on. As it came with only 2GB of RAM, I was happy to discover you can actually upgrade to 16GB per the Everymac site:

Officially, this model supports 8 GB of RAM, but … it actually is capable of supporting 16 GB of RAM using two 8 GB modules.

– EM

Add an old 500GB SSD I had kicking around, and now the machine is pretty responsive for being 7 years old and costing me $190 all in!

Given I didn’t want to dedicate a keyboard, monitor and mouse to this, the very first thing I did was to enable Remote Desktop, specifically VNC, and stuffed it with my other mini servers in the “server room”:

I then went about zipping through installing Ansible, VirtualBox and Vagrant .

When I went to boot my first VM, I got weird error on the command line (I didn’t save it, sorry). After some trouble shooting, I decided to just re-install, and more slowly this time, and the GUI showed me this:

Baffled, I tried again and again, failing the same way every time. Researching the problem, I found a post on Medium suggesting I hadn’t allowed the correct permissions in the Security & Privacy settings. None of these suggestions helped. Finally, I read the comments at the bottom of the page, including the one from Elias Politakis which said,

Please note that if you are using a VNC connection (or similar remote access software) you won’t be able to click the [Allow] button because OSX requires that Process ID pressing the Allow button is zero (0) which is the system PID. You would need to physically visit the Mac and click the Allow button with the physical mouse.

– EP

Oh, OK! But…now I had to extract the Mini from the server room :( Then I remembered I had a spare wireless mouse! What I did was plug the mouse in to the mini, then back to my desktop worsktation where I connected to the Mini over VNC and the mouse was able to still work all the way back to the closet. Then I could click the button with a real mouse, but without using a real monitor or real keyboard, or even moving the mini:

So – if you happen to be like Elias or me, just use a real mouse! Happy computing.

Skip to the install guide if you just want to know how to set up your Pi easily ;) Otherwise, read on for a little background.

Introduction

I’ve been deep in DNS land of late. At work I’m working on DNS Stats and helping QA/release/document a packet capture tool for DNS stats. I even, just today, automated a complete Pi-Hole install to have a reliable dev environment for DNS stats. At the shop, I’ve set up the same encrypted DNS + Pi-Hole + LXD + Quad9 as I have at home. It’s all DNS all the time here is what I’m trying to say ;)

I’ve yet to find the magic sauce to compile Stubby on the Orange Pi Zero board though. It’s so cheap ($20 shipped), has a built in Ethernet, and is just so dang cute! So, I was looking around at stubby posts and Linux posts and Ubuntu posts and found this great write up on Ubuntu 18.04 and stubby and it said,

Stubby is in Ubuntu 18.04 repository

-linuxbabe.com

This was awesome! This means my previous trickery of having to compile stubby from source on Ubuntu wasn’t needed! However, the revelations about easy DNS set up and encryption were only just getting started.

The next one I found was that the 4.0 release of Pi-Hole from early August, had a new feature: custom ports can be used for upstream servers. Wham! This double awesome! Now, in the GUI of Pi-Hole, you could safely add a the IP of stubby and specify a random port to use! But we’re done yet, no sir, two more revelations to go.  Hold on.

The penultimate revelation was BOTH the Orange Pi Zero AND the Raspberry Pi 3 b had a release of Ubuntu 18.04 for them. This means that you not only don’t have to compile stubby for your x86 LXD environment, but you don’t have to do it for ARM SoC setups either! Yay!

The final revelation dates back to a long, LONG time ago, and I’m just late to the party.  I’m talking proto-internet long time ago.  The legend Jon Postel decided that not only would IPv4 have a reserved IP address of 127.0.0.1 for localhost, but in RFC 790 in 1981, he said it would actually be a /8, so you get just over 16 million localhost IPs just for your bad ass self. This means you don’t actually need the new port specifying feature of Pi-Hole – you can just set up Stubby on port 53 on 127.1.1.1 and Pi-Hole on port 53 on 127.0.0.1. Ugh – this makes it so much easier – if I only was more a network guy!

Now that my rambling background on my recent revelations is done, let’s get to the technical write up.  Though, honestly, this part will be pretty short and sweet.

Installation Guide

This guide assumes you’ve downloaded and installed Ubuntu 18.04 for either your Orange Pi Zero or Raspberry Pi 3 B. Note that both the official download page of both Orange Pi and Raspberry Pi Foundation, do not list 18.04 options. It also assumes you’re running everything as root. The instructions are identical for both boards:

1. Ensure you’re up to date:
   apt-get update&&apt-get upgrade
   
2. Install Stubby: apt-get install stubby
3. Edit /etc/stubby/stubby.yml so that it’s listening on 127.1.1.1:
   listen_addresses:
 - 127.1.1.1
4. Restart Stubby: systemctl restart stubby
   
5. Install Pi-Hole. Use what ever upstream DNS server you want when prompted, we’re going to override it with Stubby:
   curl -sSL https://install.pi-hole.net | bash
   
   IMPORTANT – See Troubleshooting below if you get stuck on “Time until retry:” or “DNS resolution is not available” when installing Pi-Hole

6. Log into your new Pi-Hole at YOUR_PI_IP/admin and go to Settings -> DNS. Uncheck any DNS servers and enter a Custom 1 (IPv4) of 127.1.1.1:
   

Coming full circle, the Reddit thread I cited in my original write up, now has a comment that Ubuntu 18.04 has a Stubby package.

Quad9

If you want to use Quad9 (and I think you should ;), in step 3, while you’re in stubby.yml, comment out all the other servers in upstream_recursive_servers: and un-comment Quad9 so it looks like this::

upstream_recursive_servers:
  - address_data: 9.9.9.9
    tls_auth_name: "dns.quad9.net"
  - address_data: 2620:fe::fe
    tls_auth_name: "dns.quad9.net

Full disclosure, I work for PCH which sponsors Quad9.

Troubleshooting

A few things I found while researching this post that might help you:

• The login on the Raspberry Pi is Ubuntu with password is Ubuntu. The login on the Orange Pi Zero is root and password is 1234. Check out my SSH Bootstrap trick as well.
• The Orange Pi Zero didn’t get an IP via DHCP the first boot. A reboot solved that.
  

• The Pi-Hole script gave me a headache when installing. Near the end of the install it said, “Starting DNS service” and then was waiting to retry. I found a post on the Pi-Hole boards that solved it perfectly. To work around this, edit /etc/init.d/pihole-FTL so that this line:
  
  su -s /bin/sh -c "/usr/bin/pihole-FTL" "$FTLUSER"
  
  is replaced by this line:
  
  /usr/bin/pihole-FTL
  
  

  • 

  • 

  
  After that, be sure to reload your init script with:
  
  systemctl daemon-reload

  Finally you should be able to complete your install just by restarting Pi-Hole:
  
  systemctl restart pihole-FTL
 
  
  
• Even though I followed step 4, during one my tests stubby was still blocking port 53 on 127.0.0.1. If that happens, restart stubby:
  
  systemctl restart stubby
 
  
  
• At any point you can test that stubby or pi-hole is working. These are good to intersperse with each install and configuration change:
  
  dig @127.1.1.1 plip.com +short # stubby
dig @127.0.0.1 plip.com +short # pi-hole

I was playing around with Snaps and I wanted to try out Ubuntu Core as well. I still had some of those  Orange Pi Zero boards laying around and when you go to the Core Download page – there’s an option for Orange Pi right there – sweet! I downloaded the .img file, wrote it to my microsd card with dd, slapped it in my Orange Pi Zero, found the new IP in my DHCP server and off I went to SSH in.

Then I saw this step on install docs:

you will be asked to enter your Ubuntu SSO credentials

– Core Install Docs

Whhhaaat? Oh, I see, Core’s whole shtick is that it’s secure by default. They say, “Secure by default – Automatic updates ensure that critical security issues are addressed in the field, even if a device is unattended.”.  Cool, I can get behind that. IoT needs some thought leaders in IoT security. However…I still just want to SSH in and poke around a bit – I don’t want to have to set up an account at Ubuntu. 

Then I thought, “What if I just create a .ssh directory in /root/ and put my public key in the authorized keys file?” Assuming you’re on an Ubuntu system, logged in as mrjones and just stuck in your microSD card with the core image on it, that’d look like this:

cd /media/mrjones/writable/system-data/root/
mkdir .ssh
chmod 700 .ssh/
vim .ssh/authorized_keys 
chmod 700 .ssh/authorized_keys

After unmounting the card, inserting it into and rebooting my Pi, I SSHed as root and it just worked – that’s awesome! Now you know how to do it as well!  In fact, this likely will work with the Raspbian, Armbian and Ubuntu images for all kinds of Pi boards as well.

Stay tuned for my next post where I’ll massively simplify my stubby and pi-hole how to!

The other day the kids and I were at our local, awesome Goodwill store. I guess they’re all awesome, eh? I was looking over the pile of small electronics while the kids picked out a shirt (“any shirt you want!!”), and stumbled across a device with Ethernet jacks (RJ45) and phone jacks (RJ11). It’s a router with two ATA ports – cool! It was only $5 bucks!! A quick internet search suggests that it may be locked to a provider like Vonage, but that there’s lots of folks unlocking it. Given this was Goodwill, there were was a plethora of POTs devices handy – two more phones at just $5/ea – no problem!

Now that I had the whole kit home, it was time to see what the ATA was up to. It’s a Linksys RTP300 – and the voice section was definitely locked down :( But that means it’s time to start hacking! I dug up the site I initially found before buying it – luckily all the downloads still work! The post is 8 years old, forever in ‘net time, so I was impressed.

I’m pretty pleased with myself because I fully understood everything and was able to improve the process because of this. Mainly:

• Being a web developer I already had a web server running. This is much easier to use instead of setting up a new TFTP server as they suggested.
• Being a web developer (still) I was kinda in awe that I’d already forgotten how amazeballs Firebug was, and how I take for granted that modern browsers have such good developer tools built in.
• Knowing all the linux commands used (wget, chmod, cd & dd) I was able to explore the device a bit
• I figured out that the guide wanted you to download into /var/tmp which didn’t exist on my device, but /var worked
• They have you download some *.img, but running it through strings and checking the size, it appeared to match the 3.1.24 version. Completing the install confirmed this. However, md5sum outputs don’t match.

After all this I had a generic, if not quite dated, ATA that was ready to talk to a VoIP provider. Time to install Asterisk! This is where LXD comes in. I already had my home server running Stubby and Pi Hole – time to add another. After spinning up an Ubuntu 16.04 box, hitting a snag, then spinning up an 18.04 box, a quick apt-get install asterisk and I was ready to go!

I created 4 extensions, one for each of us, by following some simple guides I’d found (edit /etc/asterisk/sip.conf, /etc/asterisk/extensions.conf and a reload). I plunked in the LXD Asterisk IP, extension and password and BAM! both lines registered easy peasy on the ATA.

With two hard phones set up (just plug ’em in to the ATA), I set up me and the wife using the built in client on our phones. Now we all had a phone and could call each other. See the gallery below for the visual story!

On a personal note, this was pleasing for a two reasons:

1. I saved 3 small electronic devices from the landfill.  More and more I’m trying to be conscientious about buying less or buying used.
2. My kids got to look up at me and say, “it’s making a weird sound!?” I explained to them that it was a “busy signal”.  See, cell phones, office phones and 99% of home phones don’t do this any more. They have either call waiting or voice mail, or both ; )

As part of putting some good hardware to use, I taught a class on how to use SSH Keys the other week. As this was the first time I taught this class, it took a good long while to do the prep for it. I figured it’d take 6 hours (3 sessions @ 2hr), but I think I ended up putting in closer to 15 hours. There’s still room for improvement!

The slides are available (PDF), as is the video (below). I hope to redo this class so that it’s a generic “learn about SSH Keys” which then could be open to the general public. That said, I’m really glad to see we’re up to 16 containers and the load average is only at 0.37 right now ;)

https://vimeo.com/270495638

PS – I know, I know, those are not VMs, they’re actually containers!