Short Attention Span Theatre

Arch and ext4 on prgmr.com

2014-10-13T13:25:45Z

Getting Arch installed on a prgmr.com VPS is not insanely complex, but I wanted to make sure my method is documented somewhere I could find it again. The goal of this is to document the steps taken, then flesh it out into a full set of scripts to install Arch on a VM. Ideally this would be relatively simple, and provide an installation with two separate partitions for the file systems (/boot and /). Having /boot in a separate partition will allow me to upgrade / to btrfs when prgmr.com upgrades to grub2 in a few months.

This method is based on the old guide that details installing Arch, with some changes for my particular requirements. I needed a bit more space than specified, so I created a VM with 2.5 GiB on the hard drive. Use the Arch Linux ISO to boot in the VM from the "Boot Arch Linux (x86_64)" grub menu item (should be first).

Perform the Arch install on the VM. To create a set of scripts for the install paste the commands below into two separate scripts (they are marked 1 and 2 for easy use). Note that the provided scripts and commands are separated by commands you must execute manually, assume a connection to the Internet, require you to actually change the values assigned to some variables to make them work properly, set your timezone to PST, and may kill kittens.

In the first set of commands the hard drive is prepared with an MBR, an ext4 /boot and /, a base Arch install with wget and base-devel installed as well, and an automatically generated /etc/fstab.

#!/bin/sh
# Script 1 (pre-chroot)
echo -e 'o\nn\np\n1\n2048\n+256M\nn\np\n2\n\n+512M\nt\n2\n82\nn\np\n3\n\n\nw' \
  | fdisk -u /dev/sda
mkfs.ext4 -F -L boot /dev/sda1
mkfs.ext4 -F -L root /dev/sda3
mkswap -L swap /dev/sda2
swapon /dev/sda2
mount /dev/sda3 /mnt
mkdir -p /mnt/boot
mount /dev/sda1 /mnt/boot
pacstrap /mnt base wget base-devel openssh
genfstab -p /mnt >> /mnt/etc/fstab

Next, enter the Arch chroot environment in your new system.

arch-chroot /mnt

Now, to the minimal things needed to prepare the system for actual use. Replace all the "CHANGE_THIS_????????" values below with those that apply to your VPS. Once run the commands below will

set your short and fully qualified host name,
set your timezone to PST,
generate locales for US english,
set the default locale to UTF-8,
create a static network assignment for eth0,
enable the systemd network daemon,
enable the systemd name resolution daemon,
change the port on which systemd listens for ssh connections,
add the modules necessary for running as a Xen domU to the kernel,
generate a new kernel,
enable multilib compilation,
set the root password,
create a normal user and set its password,
install yaourt along with its dependency,
install grub-legacy using yaourt,
install grub to /dev/sda,
and create a link from /boot/boot to /boot (used when booting within the VPS).

#!/bin/sh
# Script 2 (post-chroot)
export THEUSER=CHANGE_THIS_USERNAME
export THEHOST=CHANGE_THIS_HOSTNAME
export THEDOMN=CHANGE_THIS_DOMNNAME
export THEIPVF=CHANGE_THIS_IPVFADDR
export THEGATE=CHANGE_THIS_GATEIPVF
export THEPORT=CHANGE_THIS_PORTNUMB
echo $THEHOST > /etc/hostname
perl -0777 -pi \
  -e 's/me\>\n/me\>\nTHEIPVF    THEHOST.THEDOMN THEHOST\n/' \
  /etc/hosts
perl -0777 -pi -e "s/THEIPVF/$THEIPVF/" /etc/hosts
perl -0777 -pi -e "s/THEHOST/$THEHOST/g" /etc/hosts
perl -0777 -pi -e "s/THEDOMN/$THEDOMN/" /etc/hosts
-e 's/me\>\n/me\>\nTHEIPVF
ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime
perl -0777 -pi -e 's/\n#en_US/\nen_US/g' /etc/locale.gen
locale-gen
echo LANG=en_US.UTF-8 > /etc/locale.conf
cat > /etc/systemd/network/eth0_static.network <<EOF
[Match]
Name=eth0

[Network]
Address=$THEIPVF/24
Gateway=$THEGATE
DNS=71.19.145.215
DNS=71.19.155.120
DNS=208.67.222.222
DNS=208.67.220.220
EOF
systemctl enable systemd-networkd
systemctl enable systemd-resolved
mkdir -p /etc/systemd/system/sshd.socket.d
cat > /etc/systemd/system/sshd.socket.d/sshd_new_port.conf <<EOF
[Socket]
ListenStream=
ListenStream=0.0.0.0:$THEPORT
FreeBind=true
EOF
perl -0777 -pi -e 's/\nMODULES="/\nMODULES=" xen-netfront xen-fbfront/' \
  /etc/mkinitcpio.conf
perl -0777 -pi -e 's/\nMODULES="/\nMODULES=" xenfs xen-kbdfront/' \
  /etc/mkinitcpio.conf
perl -0777 -pi -e 's/\nMODULES="/\nMODULES="xen-blkfront/' \
  /etc/mkinitcpio.conf
mkinitcpio -p linux
perl -0777 -pi -e 's/\n#\[multilib\]\n#Inc/\n\[multilib\]\nInc/' \
  /etc/pacman.conf
pacman -Syu
passwd
useradd -m -G wheel $THEUSER
perl -0777 -pi -e 's/\n# %wheel/\n%wheel/' /etc/sudoers
passwd $THEUSER
sudo -u $THEUSER mkdir -p /home/$THEUSER/aur
cd /home/$THEUSER/aur
sudo -u $THEUSER wget https://aur.archlinux.org/packages/pa/package-query/package-query.tar.gz
sudo -u $THEUSER wget https://aur.archlinux.org/packages/ya/yaourt/yaourt.tar.gz
sudo -u $THEUSER tar xf package-query.tar.gz
sudo -u $THEUSER tar xf yaourt.tar.gz
cd package-query
sudo -u $THEUSER makepkg -s
pacman -U package-query-1.4-*
cd ../yaourt
sudo -u $THEUSER makepkg -s
pacman -U yaourt-1.5-*
sudo -u $THEUSER yaourt -Sa grub-legacy
grub-install /dev/sda
cd /boot
ln -s . boot
rm -f /var/cache/pacman/pkg/*
rm -fr /home/dave/aur
perl -0777 -pi -e 's/sda/xvda/g' /etc/fstab
perl -0777 -pi -e 's/sda/xvda/g' /boot/grub/menu.lst

After that is complete exit the chroot environment, link /etc/resolv.conf to the file managed by systemd-resolved (so name resolution will work after boot on the VPS), and unmount the file systems used during installation.

exit
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
umount /mnt/boot /mnt

Now boot your VPS with the new "Debian Wheezy Live" rescue ISO (it should be the second option on the first grub boot menu). Once logged in as root (no password required), prepare the VPS hard drive with the same partitions as used in the VM.

echo -e 'o\nn\np\n1\n2048\n+256M\nn\np\n2\n\n+512M\nt\n2\n82\nn\np\n3\n\n\nw' \
  | fdisk -u /dev/xvda
mkswap -L swap /dev/xvda2

Next, create a privilege separation directory for sshd, insert a copy of your public ssh key into /root/.ssh/authorized_keys, the run sshd on a particular port (this also only allows one connection at a time).

mkdir /tmp/sshd
ln -s /tmp/sshd /var/run/sshd
/usr/sbin/sshd -d -p PORT -f /etc/ssh/sshd_config

Now, on the VM on your local machine, initiate a connection to copy the local ext3 and btrfs volumes to the VPS using the following commands.

dd if=/dev/sda1 | ssh -p 21773 root@CHANGE_THIS_IPVF "dd of=/dev/xvda1"
dd if=/dev/sda3 | ssh -p 21773 root@CHANGE_THIS_IPVF "dd of=/dev/xvda3"

Once the transfers are complete, the only tasks remaining are to verify the system boots, can connect to the network, and enable/start your sshd.socket service to accept incoming connections. Reboot the VPS, then execute the following commands to enable and start your sshd.socket.

systemctl enable sshd.socket
systemctl start sshd.socket

Now ping something off the local network to make sure you can communicate with the rest of the world, and you should be in business.

Stop gpg-agent from running on login

2014-09-19T06:48:03Z

Every time I install Arch Linux I am surprised when XFCE insists on running gpg-agent when I log on. When it further doesn't work properly with ssh-keys (admittedly, probably because I don't know how to use it properly) and invokes pinentry to input another passphrase so Gnome Keyring can protect the key the agent is supposed to protect... well... that's when I start to get irritated. Irritation leads to anger, and anger to research, and research to a solution.

Luckily the solution is easy to find every time, but this time I'm making it easier to find. This makes XFCE use ssh-agent even if gpg-agent is installed.

$ xfconf-query -c xfce4-session -p /startup/ssh-agent/type -n -t string -s ssh-agent

The solution was (shockingly) located in the Arch Linux wiki.

Solarized cheat sheet

2014-01-03T08:34:41Z

I grew tired of scrolling up and down the solarized web page to figure out which color was which hex code for HTML use, so I created a cheat sheet.

Here is the LaTeX source (compile with lualatex) and the PDF file produced.

Switching to a website generator that fits my schedule

2013-12-23T23:47:06Z

In working with Hakyll the biggest problem was my lack of familiarity with Haskell, and my lack of time to learn it sufficiently to feel comfortable with the software. After a while I started to get irritated by my lack of understanding, and decided that (unless I dedicated more time to learning Haskell) I needed to move over to something in a language I was more familiar with (Python)... thus the move to Acrylamid.

Installing on FreeBSD turned out to be relatively simple, given that the software is available for install using easy_install or pip. I chose pip because I have an issue with software that cannot be easily uninstalled. Acrylamid was installed using the following incantation.

# pip install acrylamid

On my system, that actually installed a few other packages in addition to acrylamid, specifically Jinja2, Markdown (in python), and translitcodec.

Configuration took a bit more finagling than I expected because the tokens for the hour and minute of publication were not available when generating each post. As a result I requested a fix via pull request which was accepted within a day. Unfortunately the fix only applied to the current development branch, not the release available via pip. The lack of a release which could be installed via pip was not a show-stopper, because the procedure to have pip install from a git repository was well documented and only required pointing pip at my forked repository.

# pip install git+git://github.com/sinecure/acrylamid.git

Another speed-bump was how to set the timezone within each post. Because of previous conversions I had laboriously converted all time/date stamps to UTC time, and it was not obvious how to indicate the timezone within the post ("UTC" was not accepted with one error, "+0000" was not accepted with another). Luckily, I found others had encountered a similar difficulty, and the solution to my problem (setting TZINFO = "UTC" in conf.py) was in the comments.

The next piece of the puzzle was getting all the ancillary files which may be with a post (image/video/text files) into the same directory as the post. This was easily solved using the STATIC definition in the conf.py file, and translating the (hand-named) post directories to the standard "slug" format. It turned out the slugs are generated using a public domain python function.

Overall, this was a relatively painless and simple process requiring minimal time before the site was transferred. Most of the delay was in the actual conversion of date/time information to a format without timezone information, and a few errors by the operator (me) along the way.

Mathopd and reprepro

2013-04-29T04:21:35Z

Although the method of serving an APT repository I discovered when setting up Debian on MacBook Pro 5.2 is sufficient, it is not very robust. Thus I began looking for something more useful for multiple repositories, and which is packaged for debian itself.

The search led me to a post that demonstrated how to configure a private repository using reprepro. Although that gave me almost everything I needed, one final bit was required: permission adjustment. As I use Mathopd, the easiest way to prevent visitors from accessing the conf and db directories was to restrict their permissions to that of my user alone.

$ chmod go-rx conf db

Setting hostname in FreeBSD when using dhclient

2013-04-27T08:55:03Z

I experienced a problem for years on FreeBSD, but have never been motivated to solve it until now. The problem presents when attempting to set FQDN of the system (nastie.weller-fahy.com). I set hostname to the proper value in /etc/rc.conf.

hostname="nastie.weller-fahy.com"

When I type hostname at the prompt, I am given the unqualified hostname, nastie. When it bit me tonight, I finally tried to find a solution, and (shock of all shocks) found one!

So, to sum up, create an executable file /etc/dhclient-enter-hooks with the following content.

#!/bin/sh
check_hostname(){
    hostname nastie.weller-fahy.com
}

Next time you reboot, the hostname command will result in the FQDN of your system, as it should.

Leaving bread crumbs and playing with Hakyll

2013-03-28T07:41:34Z

One of the things I wanted most in my conversion to Hakyll was a way to show the location breadcrumbs of the current page. For those unfamiliar with the concept, these do not show the actual path the user took to get to the page, but they do show the location of the current page in the website hierarchy. For example, if I were on a post called "Having my bread crumbs and eating them too!", then the bread crumb URLs on the page would be as follows.

sast :: posts :: Having my bread crumbs and eating them too!

Having bread crumbs like these allows users to have a better feel for "where" they are in a website, and they are neat. Determining how best to create those bread crumbs using hakyll was not easy, and I am certain my implementation is overly complicated, but I finally have something that works! The key insight which led to a working implementation was the following:

An element of the page does not need to be visible.

That insight is not rocket science, and web designers are probably snickering under their breath, but I had been struggling with how to get various levels of the crumbs to appear and disappear depending on the visible level. The insight started me down a path that ended with the idea of combining templates, inline styles, and the CSS display property.

First, I assume that the website has three different levels: main (index and facets), category (indexes of posts and projects), and page (individual posts and projects). Changing the levels would require changing the code, but with this pattern in place adding levels should (for certain values of should) be trivial.

Next, I include the following HTML in the default template (default.html).

<div id="headercrumbs">
    <a class="headercrumb" id="headerpagetitle" href="$url$">$title$</a>
    <span class="headercrumb" style="display:$categoryIs$">::</span>
    <a class="headercrumb" style="display:$categoryIs$"
        href="/$category$">$category$</a>
    ::
    <a class="headercrumb" id="sitelink" href="/">sast</a>
</div>

Note that one of the crumb separators ("::") is bare, not surrounded by a span: this is because there will always be a "sitelink" (the link to the main page of this website), and there will always be a "headerpagetitle". Thus, there will always be at least one crumb separator and two links in the bread crumbs.

Now comes the clever bit (or, at least I like to think it is the clever bit). Contexts are generated which contain two metadata fields. If the page is part of a category and not a category index, then the "category" field is set to the category name and the "categoryIs" field is set to "inline". If the page is not part of a category or is a category index, then the "category" field is set to the empty string and the "categoryIs" field is set to "none". All this magical manipulation happens with the help of three new contexts and two new functions, listed below.

defaultCtx :: Context String
defaultCtx = mconcat
    [ crumbCtx
    , crumbIsCtx
    , defaultContext
    ]

-- | Return String with the HTML code for bread crumbs of the current item.
-- This assumes three levels to the site: main (index, facets), category
-- (indices of posts, projects), page (individual posts, projects).
--
crumbCtx :: Context a
crumbCtx = field "category" $ \item -> do
    metadata <- getMetadata $ itemIdentifier item
    let crumb = findCategory (itemIdentifier item) categories
    return crumb

crumbIsCtx :: Context a
crumbIsCtx = field "categoryIs" $ \item -> do
    metadata <- getMetadata $ itemIdentifier item
    let crumbIs = findCategoryIs (itemIdentifier item) categories
    return crumbIs

findCategory :: Identifier -> [String] -> String
findCategory id cats
    | notElem '/' $ idPath                     = ""
    | hcat == []                               = ""
    | hcat == head (splitDirectories $ idPath) = hcat
    | otherwise                                = findCategory id $ tail cats
    where idPath = toFilePath id
          hcat = head cats

findCategoryIs :: Identifier -> [String] -> String
findCategoryIs id cats
    | notElem '/' $ idPath                     = "none"
    | hcat == []                               = "none"
    | hcat ++ "/index.md" == idPath            = "none"
    | hcat == head (splitDirectories $ idPath) = "inline"
    | otherwise                                = findCategoryIs id $ tail cats
    where idPath = toFilePath id
          hcat = head cats

The defaultCtx is only used when applying the default.html template, and thus should have minimal impact on other pages.

With the above code (and a bit of CSS) I have a happy little breadcrumb navigation section on the upper-right corner of my web site which updates based on the current page. It certainly is not the most elegant solution: the two functions have a lot of duplication, as do the two contexts. For now though, until I figure out how to use the output of findCategory in findCategoryIs, it will do.

Changing the expiration date of my GPG key

2013-03-05T00:33:30Z

It is considered good practice to set an expiration date on your GPG/PGP keys. Setting the expiration date prevents the possibility of losing the private key leaving a valid, non-expired, key out on the key servers that you cannot revoke. Revocation certificates may be one solution, but I found it more useful to simply set the expiration date.

It is useful if one can also remember how to extend the expiration date of the GPG key in question when the expiration date sneaks up without warning. Luckily, I was able to find an article by George Notaras on How to change the expiration date of a GPG key, which allowed me to change the date with minimal fuss.

The short version follows, but requires some reading up... so do so before execution.

$ gpg --edit-key [key-id-of-key-to-edit]
gpg> key 0
gpg> expire
...
gpg> key 1
gpg> expire

Conversion from ikiwiki to Hakyll

2013-01-23T08:25:21Z

Ikiwiki is a great idea in theory. For those unfamiliar with the concept, it is basically a fully functional wiki integrated with a version control system (VCS), all done in such a way that the pages are served statically unless you are modifying the configuration or one of the pages.

An excellent solution for those who need that level of interactivity, but it is a bit much for my purposes. It is a fairly complex system designed to be a wiki, but without some of the downfalls of other wikis which serve all pages dynamically. It was this very complexity which led me to eventually realize that I do not actually want a wiki. I want website generation software with the following features:

Allows me to write my posts and pages in Markdown.
Allows fully static pages (off site generation of pages, not "code as website").
Allows files associated with the posts to be with the posts without having to jump through too many hoops. The perfect example for this is a post which has pictures: unless the pictures are being referenced from an external gallery, they should be physically located in the same directory as the post.
Does not require any on line configuration (think blob of pages with no preferences).
Allows a hierarchical layout of the source files based on date.
Allows any VCS to be used, without requiring extraordinary efforts to be made.
Allows a portion of the site to be formatted as a blog, where the most recent post is on top, with the rest in reverse chronological order all the way down (or, if one desires, with separate pages for each chunk of posts).
Lets me play with a new programming language
- This is my comfort food: some people like brownies, but I like new programming languages... and brownies.
Felt fun.

There are quite a few website generation software packages available which provide most of the above features, so the big two ended up being new programming language and fun. Haskell has had my attention for a while, even if that attention was hidden behind thesis work, research, classes, promotion testing, family, children, and life. This seemed the perfect time to allow that attention to come to the fore, and learn a bit about a language which seems interesting.

Fortunately, I am approaching my defense, so this provided something for me to do when my brain was screaming, "STOP WITH THE RESEARCH STUFFESSES!"

Unfortunately, I am approaching my defense, so I had to take the "get it working" route instead of the "learn the language thoroughly" route.

With that, here is an accounting of the conversion, lessons learned, modifications required, and result (which, with any luck, is what you are reading right now).

First, I discovered that cabal is not a package manager when Hakyll bumped from version 3.X to version 4.X. My experience involved fumbling about in the dark trying to figure out why my cabal install hakyll was not working properly, and nothing would compile, all the way to having to remove all ports beginning with hs- using pkg delete hs-. That resulted in starting over with portmaster devel/hs-haskell-platform, which led me to the second part of the conversion...

How the heck do I convert these 148 posts to the markdown with metadata format used by Hakyll, and determine the proper published and updated dates? This was complicated by the fact that the website has gone through so many transitions, including converting from HTML (straight, no chaser) to Blosxom to WordPress to ikiwiki. The last conversion including mass conversion from a MySQL database to text files in Markdown format, with the entire thing added to git and metadata adjusted to reflect the actual date as recorded in the database. Not exactly impossible, but not something I wanted to tackle without a bit of script-fu. Unfortunately there is no single, "one script to rule them all," to share with you, the audience. Instead it took a series of three to five line scripts which allowed me to extract my dates, convert all similarly formatted posts, and then manually handle the 20 or so posts which did not match the common format. The total conversion took about 2 hours, including building each script as needed and looking up metadata within the historical MySQL dump, which is not bad.

In the process, however, I discovered the dangers of depending on file system metadata to record modified dates when dealing with a DVCS: when you checkout a file, the file metadata changes!!! Yep... all those preciously guarded modified timestamps were lost whenever the file was checked out. That leads to my first lesson from the conversion:

Always store your metadata explicitly.

Following the extra time the lesson imposed in digging out metadata from the backups of the old website (which, thank goodness, I had), the next step was determining a structure for the website.

Each example of a possible layout will be representing a post named, "Editing my navel," that was (notionally) published on the 12th of January, 2013 at 23:48:09 UTC.

The default structure Hakyll uses is as follows.

posts/2013-01-12-Editing_my_navel.md

The default, being the default, seemed much less appealing on multiple grounds. The primary being that it violated the, "keep the files with the post," requirement, with another, more minor issue being the lack of specificity: What if I wanted to make two posts in the same day? Or hour? Or minute? Would the two posts properly sort within the day (given that the file path is usually used to sort in Hakyll)? Or would they be sorted arbitrarily? Three alternative layouts occurred to me, illustrated by the following three examples.

posts/2013/01/12/23/48/09/Editing_my_navel.md
posts/2013-01-12-23-48-09-Editing_my_navel.md
posts/2013/01/12/23/48/09/Editing_my_navel/index.md

While all three would be ideal with regards to specificity, they are awkward as regards to human access of a given post. This led to further navel gazing, and some searching, where I came across another Hakyll blog by Ian Ross which used a limited version of one of my schemes.

posts/2013/01/12/Editing-my-navel.md

While this solved most of the problems I was anticipating (mostly from having run into them on other website platforms), I wanted something that resulted in clean URLs and allowed files that go with posts to be located with them on disk (not all files of the same day go with all posts of the same day), which led to the final version.

posts/2013/01/12/Editing_my_navel/index.md

With this layout most of the sorting can be done at the file system level, files that go with posts can be located with the post on disk, and the URLs to refer to the posts do not require the un-site-ly (yuk) index.html at the end. As to clean URLs - I do not know why, but I have disliked having the *.html file at the end of the URL for years. This does not solve the specificity problem, but as each post will have an explicit published metadata field this will do until I figure out how to sort using the published field.

Alright, the layout for posts has been figured, now what about the layout for projects? The projects layout was much easier, as projects are supposed to be longer, less time oriented pages, thus the date is omitted from the path and the following title might be a new HOWTO on navel editing.

projects/Navel_editing_HOWTO/index.md

After that, and with the example of the site.hs of the author, the only significant problem was designing a sort routine to keep the posts in chronological order, which turned out to be a non-issue. The previously mentioned blog also uses a similar layout and had already rewritten the chronological function. With that as an example, generating my own chronological function was relatively simple, resulting in the following.

import System.FilePath (splitDirectories, joinPath)
import Data.List (sortBy)
import Data.Ord (comparing)

-- | Sort pages chronologically. This function assumes that the pages have a
-- @posts/year/month/day/title/index.md@ naming scheme.
chronological :: [Item a] -> [Item a]
chronological = sortBy $ comparing $
    joinPath . take 3 . drop 2 . splitDirectories . toFilePath . itemIdentifier

Note that I use this by importing Hakyll as follows.

import Hakyll hiding (chronological)

Of course, shortly after I was able to get the above function functional, Hakyll was upgraded to version 4.2.X, which automatically sorts based on "published" metadata if it exists. Harumph.

The last item on the agenda was allowing access to the source files from within the website itself (copy the source files along with the compiled files). Making the original markdown files available turned out to be simpler than it originally seemed, requiring only addition of another match and specification of the version when loading snapshots. Just make sure to define the postPattern.

postPattern = "posts/*/*/*/*/index.md"

Copy the raw posts and define the version.

-- Make sure to copy the raw posts
match postPattern $ version "raw" $ do
route $ idRoute
compile $ getResourceBody

When loading snapshots later on, just make sure to refer to the snapshots not associated with a version... otherwise, the result will probably not be what you intend.

loadAllSnapshots (postPattern .&&. hasNoVersion) "content"

With that, all the basics were in place, with nothing to do except tweak and work on items that are not absolutely required (breadcrumbs being a good example of something desired but not required). As it is, you should be reading this on the new and improved site. I will put a TODO list up on the facets page to indicate what is left to be done.

Excellent example of WebGL programming

2013-01-14T18:51:59Z

Just came across this excellent example of WebGL programming, and lost some productive time playing with it. This is a superb demonstration of what the web could become, or, at least, could aspire to become... plus, it is fun!

Installing LXDE on HEAD

2012-12-18T08:18:34Z

When installing LXDE on FreeBSD-HEAD (at least, until imake is eliminated from the ports tree, if that ever happens) the following addition to the make.conf will help immensely in preventing stress and hair tearing.

# clang fallout: lxde has ports using imake, imake does not like clang
.if !empty(.CURDIR:M/usr/ports/*) && exists(/usr/local/bin/gcc46)
.if ${.CURDIR:M*/textproc/docbook-to-man} || ${.CURDIR:M*/x11/lxpanel}
CC=gcc46
CXX=g++46
CPP=cpp46
.endif
.endif

Note this only works if you have installed gcc 4.6, otherwise modify as necessary.

Automate ZFS install on NAStie using mfsBSD

2011-11-26T17:04:17Z

When performing zfs perambulations after install (as noted in ZFS and NAStie and i386 and not reading the specification), the chance of finger fumbling errors is high. To prevent those errors, I modified the zfsinstall script included in the mfsBSD 8.2-RELEASE-p2 ISO to configure the paths to different dataset objects as described on the FreeBSD Wiki.

A patch is available for future use/reference, and the contents of the patch are displayed below.

ZFS and NAStie and i386 and not reading the specification

2011-11-22T09:30:27Z

Ahem.

Apparently one should not skim the docs when putting together a server. Apparently one should actually read the portion which tells you what type of processor you have (for example, if you have a 64-bit processor). Occasionally that type of information would be useful... like... say... when you are trying to get your FreeBSD installation using ZFS to be stable, which it is NOT on 32-bit (ok, some small portion of the population has aligned the stars, but I did not)! One should not depend on ones memory when one thinks, "Self, that system is old so the processor must be 32-bit," and does not check to be sure.

Yep... I noticed a few days ago the Pentium 4 2.8 GHz processor in my server is indeed 64-bit, silly me. So I immediately attempted to load 9.0-RC2, only to have it fail to boot. RC1 (off of PC-BSD) failed as well, and then I was off and running in an attempt to find out what caused this PR (filed recently).

After digging a bit I discovered the mfsBSD images, a ZFS boot howto, and re-discovered the FreeBSD Wiki page on ZFS booting. Those gave me everything I needed to get back up and running using an amd64 image.

So, first thing was to download the ISO from the mfsBSD page (8.2p2 special edition) and boot... it works! The zfsinstall command line I used follows.

# zfsinstall -d ad6 -t /cdrom/8.2-RELEASE-p2-amd64.tar.xz -s 8G -p nastie

After install I rebooted (instead of chrooting into /mnt) into the new environment, and it had that new car smell... mmmm. Anyway, tasks I did right after reboot.

I created lots of other mountpoints (the default only creates /, /var, /tmp) following the FreeBSD Wiki page.

Note about this: Remember to make sure you move any directories/files out of the way before creating the mount point... confusion reigned for a moment until my brain caught up with my fingers.

New mountpoints created (from the wiki - all commands copied from the wiki and modified for my setup) follow. BUYER BEWARE: I am only showing the commands necessary to create mountpoints and set options, not those necessary to destroy the already created mountpoints or move files out of the way while creating a mountpoint which was already created as a directory when installing mfsBSD.

Set the checksum algorithm.

# zfs set checksum=fletcher4 zroot

Create /tmp and set the proper permissions.

# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/tmp
# chmod 1777 /tmp

Create the /usr and /usr/home.

# zfs create nastie/root/usr
# zfs create nastie/root/usr/home
# cd / ; ln -s /usr/home home

Create the ports hierarchy.

# zfs create -o compression=lzjb -o setuid=off nastie/root/usr/ports
# zfs create -o compression=off -o exec=off -o setuid=off nastie/root/usr/ports/distfiles
# zfs create -o compression=off -o exec=off -o setuid=off nastie/root/usr/ports/packages

Use the src, Luke!

# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/usr/src

A variety of various subdirectories of var.

# zfs create nastie/root/var
# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/var/crash
# zfs create -o exec=off -o setuid=off nastie/root/var/db
# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/var/db/pkg
# zfs create -o exec=off -o setuid=off nastie/root/var/empty
# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/var/log
# zfs create -o compression=gzip -o exec=off -o setuid=off nastie/root/var/mail
# zfs create -o exec=off -o setuid=off nastie/root/var/run
# zfs create -o compression=lzjb -o exec=on -o setuid=off nastie/root/var/tmp
# chmod 1777 /var/tmp
# zfs set readonly=on nastie/var/empty

Finally the service NAStie will provide.

# zfs create -o compression=lzjb -o exec=off -o setuid=off nastie/root/srv

That sets up the file systems. Now all that needs to be done is add a normal user and start setting up the services on the machine. It is interesting that this was much faster than installing normally. Have to look into building my own custom release sometime.

I will add anything I notice must be done, but between this post, Getting NAStie with ZFS, and Rebuilding NAStie I should be able to reconstruct this server.

Cygwin terminate batch annoyance

2011-10-21T05:23:02Z

When installing CYGWIN in the future, remember to visit this page. It tells you how to use bash.exe --login -i instead of the batch file.

Thunderbird defaults reminder

2011-10-16T07:50:41Z

A tiny reminder to (when playing with Thunderbird) do the following in the config editor.

mail.check_all_imap_folders_for_new = true
mailnews.default_sort_order = 22
mailnews.default_sort_type = 1
mailnews.default_view_flags = 1
mailnews.headers.showSender = true
mailnews.headers.showUserAgent = true

Sigh... much better.

Oh, yeah, found at http://www.everything-mdaemon.com/general/threading-messages-in-thunderbird.

Xorg font follow-up

2011-10-07T20:20:48Z

In Debian on Macbook Pro 5.2 I mentioned using the xterm*font resource to get rid of the nasty fonts, and then could not find the TTF fonts in xfontsel/xlsfonts (despite using my previous post about xorg font hell), and began to get very frustrated.

Apparently, the frustration was not necessary. Apparently, while I have spent my time in OS X land, the Linux universe was moving further and further from having X provide the fonts, instead using fontconfig and other tools. Also xterm has a method to access ttf fonts separate from the normal X fonts.

The following (in combination with a recent article on the top 10 programming fonts) fixed me up.

xterm*faceName:     DejaVu Sans Mono
xterm*faceSize:     8

I just plopped it in my .Xresources file, removed the xterm*font line, merged the resources using xrdb -merge ~/.Xresources, and relaunched my xterms.

Now, why is FireFox giving me fits?

Debian on Macbook Pro 5.2

2011-10-07T08:16:23Z

So, Mac OS X got on my last nerve and led me back to the holy land (linux, what else?). While FreeBSD beckoned there are certain pieces of software I must have working for this Quarter. It is the first of my Master Degree classes, so I want to get in the proper habits (such as not staying up all night getting Mathematica or MATLAB working in FreeBSD, when Linux versions are available ;)).

Regardless, some notes about stumbling blocks would be helpful if I need to do this again, so here goes.

WIFI - The wireless is a Broadcom Corporation BCM4322.

$ lspci -s 04:00.0
04:00.0 Network controller: Broadcom Corporation BCM4322 802.11a/b/g/n Wireless LAN Controller (rev 01)

Checking the debian wiki page about Broadcom wireless cards indicated the fix is not difficult... oh, unless you do not have a connection to the Internet so you cannot get the software so you do not have conne... ahem.

The solution (some might have simply plugged in upstairs by the router and continued on their merry way - but certainly not I) was to download the needed debs on another computer (thanks wif!), and create a small temporary repository to hold them until I got connected. Why did I not just use dpkg to install the debs? Good question: The dpkg method does not handle dependencies, the local respository method does.

The method used was from a handy obsolete page about repositories and allowed me to install the two packages ([broadcom-sta-common and broadcom-sta-source]) with all their dependencies, and move on to the configuration (remember, every time the kernel changes, you must rebuild the broadcom-sta module).

Configuring wireless is fairly painless... at least it is once you realize the wlan0 interface does not actually exist! That is right folks, it is a red herring to trick you into typing silly things like sudo ifup wlan0 (when what you really want is sudo ifup eth1). Heh.

Next annoying thing to attract my wrath was the function keys... whoever decided it was a good idea to make the function keys act like the cute little alternate functions... well, let us just say there is a reason I left Mac OS X. I will probably pick up an MS Natural Keyboard just to have an MS product hooked to a Mac... well, that and (other than the plus shaped extra arrow keys useless for gaming) it is my favorite keyboard of all time.

While a fix to the function key problem was on a page about the Apple Keyboard, someone also created a daemon which fixed all the problems I had with function keys (without updating initramfs) and the brightness adjustment which was my next stop on the way to insanity! Check out pommed if you are installing on a Mac.

Next to fix was... erm... oh, yeah, nvidia drivers... wait! I almost forgot, the annoyingly small font in xterm. The line I used in my .Xresources file follows.

xterm*font:     -misc-fixed-medium-r-normal-*-17-*-*-*-*-*-iso10646-1

NOW for the nvidia drivers which was very anticlimactic. First, I am running testing, so YMMV if you are not. All I did was install the nvidia-kernel-dkms package, then put the following in an /etc/X11/xorg.conf.

Section "Device"
    Identifier  "My MASSIVE GPU"
    Driver      "nvidia"
EndSection

That was it, and it worked. Weird.

Last problem: Fonts do not want to show up with the new "modularitalized" xorg, and my xorg font hell post did not help, so I will have to go digging. Oh, yeah, firefox all of a sudden starts giving me fits (CPU hogging, machine lagging, but never crashing fits). So, I have installed chromium (little brother of chrome) until I can figure it out... wonder if adblock plus is available for chromium. ;)

That is all for this set of adventures. Hopefully I will remember to look here next time I install on a Mac.

Stability for i386 and ZFS

2011-10-01T07:32:17Z

According to various sources, stability of ZFS on i386 boxes is going to be... not a priority. Regardless, I decided to take up the challenge.

My first attempt to get everything stable was to use zero settings in the loader.conf file. That lasted until my first big ports build. That did not work.

A similar (but not the actual: I have received plenty and just chose the last one) error message follows.

panic: kmem_malloc(86016): kmem_map too small: 520044544 total allocated
cpuid = 0
KDB: enter: panic
[ thread pid 0 tid 100228 ]
Stopped at      kdb_enter+0x3a: movl    $0, kdb_why

This is consistent every time I load the disks heavily (such as trying to copy the entirety of the HEAD svn repository to another directory). Now, my goal is to attempt to get things stable under heavy load. I started using the following loader.conf settings.

# /boot/loader.conf excerpt
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="128M"
vfs.zfs.vdev.cache.size="5M"

That did not work. To avoid such quoting throughout, I will refer to such settings in the future as vm.kmem_size/vm.kmem_size_max/vfs.zfs.arc_max/vfs.zfs.vdev.cache.size. Using that the above values would be 512M/512M/128M/5M. Now, since that did not work, and after some IRC goodness on #freebsd, I decided to just try 384M/384M/64M/5M. Now, there is no real empirical reason to use those, but they are close to the 330M/330M/40M/5M which worked when NAStie had 784M of memory, so perhaps this will do the trick.

We will see, now to rsync and compile a lot!

Update: Hrm... after transfering the latest Mathematica download (about 800 MB), and then removing the Mathematica 7 ISO for Mac (I do not remember how much), system dumped again. I am going back to the 330M/330M/40M/5M which worked for me when I only had 768 MB of memory. We will see how that works.

Update: Odd, had a middle of the night shutdown, did not notice until this evening. I will reboot and see what is what. No change in values.

Update: Even odder, I can not get the server to stay stable for longer than (about) a day. Every time I stress the drives it dies (technical term). Going to try moving to AMD64 (more in the new post).

New chassis for NAStie

2011-09-02T06:37:39Z

It is always exciting (and scary!) when a new system arrives, and arrive it did: A brand spanking seller-refurbished Dell Optiplex GX620 with 512 MB of RAM and a 40 GB hard drive. Wooo!

Ahem.

Anyway, the first task was to confirm it worked, which I did and it did. The second task was to move NAStie (of building NAStie) over to the new system.

Surprisingly (with a few burps because of SATA connected to the wrong port and other fumbles) everything worked right away. I just made sure to zpool export and zpool import as recommended on a forum post about similar hijinks and the system is operational 20 minutes after the switch!

Now, to upgrade the 512 MB of memory (barely useful) to a slightly bigger value (4G should do ;)).

Getting NAStie with ZFS

2011-08-24T22:51:53Z

While my priorities have not changed a "shiny" moment had me read all about ZFS and how it will change the world, save all the children, and eliminate hunger for all time.

With advertisements like those, how could I resist trying to run it on a Compaq SR1103WM with a lofty 768MB of RAM? Luckily, I found a post which provided almost everything needed to do a ZFS-mostly install. As it had only been three days since the server rebuild, and everything I did to setup the server aspects (vs. the actual install and filessystem aspects) was still in my head, I figured it was now or when the next reinstall needed to be done. Needless to say I went with now.

Anywho, all the previous guidance regarding Rebuilding NAStie applies, but use the From sysinstall to ZFS-only configuration post to install. As I am limited on RAM I found the FreeBSD Wiki ZFS Tuning Guide particularly helpful (especially the settings from CyShubert). Also useful during the install and for the sake of research were Installing FreeBSD Root on ZFS (Mirror) using GPT and Chapter 4 of the Solaris ZFS Administration Guide.

To summarize, follow the instructions in the From sysinstall to ZFS-only configuration, use the loader.conf values from FreeBSD Wiki ZFS Tuning Guide, and use the rest of the links as reference and knowledge building.

Now to see how Samba likes serving files off of a ZFS platter.

htop on FreeBSD

2011-08-24T20:58:19Z

Installing htop on FreeBSD requires a particular directory under the linux compatibility tree be mounted, but I did not want linux compatibility installed. Luckily, I found a thread about htop on FreeBSD which covered that issue, and another which covered loading the linux kernel module.

Just create /compat/linux/proc, add the requisite entry to /etc/fstab, and # mount linproc. If you have problems, try putting linux_enable="YES" in your /etc/loader.conf.

Done!

Rebuilding NAStie

2011-08-21T08:28:16Z

After a recent spat with NAStie (my home NAS built in building NAStie) and its grub infestation, I decided to go back to FreeBSD for my home server needs. I like Linux, but mostly for desktop/remote server stuff. For home server on old hardware, I likes me some *BSD.

So, I started my research at my post about RAID1 on FreeBSD (FreeBSD 5.2.1 and cheap RAID1), learned about the RAID-1 options in FreeBSD, and needed to decide whether there was any use in trying for a volume management option.

Answer: NO.

As to why: In my home network server upgrades are driven not by space requirements, but by old hardware breaking down. My servers are usually hand-me-downs which have been abused by people who (speaking broadly) are not computer savvy. This leads to interesting life-cycles for the hardware, which tends to have had encounters with small humans for extended periods of time before they are "rescued" by me. Since I have never had to add storage to the home server, and I always overestimate the total required space for us, it makes sense to set my priorities as follows.

Secure - the data stored should be safe from casual efforts to gain access. I am not trying to stop a determined effort by a major player, but I am trying to stop accidental disclosure and any non-targeted attacks. In this case non-targeted means not targeting me (Dave), it does not mean the attacks would not target an IP address on which my computer happens to be accessible.
Safe - the data stored should be reasonably safe from hard drive failure. I am not looking for perfect safety (impossible!) but expect to be able to recover from a single failed drive. Any data stored on this system will be either a backup of a laptop drive (so, three copies extant), or non-critical data which can be recovered (copies of CD/DVDs, working papers put there temporarily, etc.), thus even failure of both drives at once should just be inconvenient, not catastrophic.
Accessible - the data stored should be accessible by me anywhere I have Internet access (sshfs or openvpn), and by my wife at home. Speed of access outside the home is not critical, but being able to access it is.
Growth - in the unlikely event my data outgrows my storage capacity (1.5 TB), adding storage should be relatively simple.

As growth is my last priority, and adding hard drives (and thus more space) is not that difficult even with separate partitions to deal with (instead of growing a file system using LVM), I shall not give it much thought.

Now, asides aside, here is a brief "here's how I did it" install description. This should help in the event I need to look back on how I did it in the future.

Do a standard FreeBSD install but set up gmirror while installing. This was remarkably easy to do once I found the Using Software RAID-1 with FreeBSD page.

Update and install some software.

# freebsd-update fetch install
# portsnap fetch extract
# cd /usr/ports/ports-mgmt/portmaster
# make install
# portmaster shells/bash security/sudo editors/vim-lite www/lynx
# portmaster sysutils/cdrecord sysutils/multitail net/rsync devel/git
# portmaster x11/xorg x11/xlockmore x11-wm/i3 x11/dmenu www/firefox

Note: Some would frown at the presence of X11 on a server, and in a perfect world they are correct. However, I also use this system for playing with FreeBSD and I like three/four uxterms going at once.

Edit /etc/ntp.conf and /etc/rc.conf to make NTP behave. I used the Arch Linux wiki NTP page and FreeBSD Handbook NTP page for reference. NAStie will not be used as the home NTP server, this server will just be used to keep proper time.

# portmaster net/samba35

Configure Samba in a way similar to building NAStie... same references, slightly different internal structure: I will have a single big partition to store things on, but they will be presented as two separate shares to anyone connecting via the network. As I control all the computers writing to these shares, any issues regarding uncontrolled usage of space should not happen (if they do I can fix the problem as they are my computers ;)).

Finally, we will need some sort of email system on here which does not store the email locally, as I prefer all mail to be sent through my mail server. My preference has been qmail, but I have had thoughts of allowing the default installed MTA to do its job... Hrm. I will try it for a while, and see how things go. The only thing I had to do was modify /etc/mail/aliases, so we will see how well it works.

That is it! The server is running well, the Wif can access her files as can I, and the mirror is fully synced and running. We will see how long that lasts... perhaps my next item will be on how to save the contents of a Time Machine volume to NAStie, or something like that.

Building mutt from mercurial repository on FreeBSD

2011-08-21T04:10:46Z

Just a note to remind myself of the ports needed to build mutt from the mercurial repository on FreeBSD (because I recently had to figure that out again).

# portmaster --force-config devel/mercurial devel/automake \
    devel/gettext textproc/docbook-xml textproc/docbook-xsl \
    textproc/libxslt databases/tokyocabinet www/lynx \
    security/cyrus-sasl2

There are other combinations which might do the job, but IWFM.

LFS coreutils test errors

2011-05-30T17:49:10Z

A short note to remind me of the patch for coreutils 8.10 which allows its tests to pass when building Linux From Scratch (LFS) version 6.8.

LVM booted using grub2 in Arch

2011-05-07T23:24:47Z

Unlike my previous experiment with building NAStie installing this variant of linux onto LVM-only partitions proved difficult. My attempts left my sleep schedule a bit tweaked, and my disgust for "automated" configuration-file-building-scripts found new purpose. That aside, what worked when I installed Arch linux on LVM partitions is below.

Install Arch using the standard install tool.
While doing this, refer to the Arch Wiki page on LVM and follow its instructions until it talks about changing the grub configuration file.
Assigning (VolumeGroupLabel-LogicalVolumeLabel) to GRUB_DEVICE in /etc/default/grub will cause the /boot/grub/grub.cfg-building script to insert the parameter 'root=(VolumeGroupLabel-LogicalVolumeLabel)' when building the kernel's command-line. This does not work. Once the /boot/grub/grub.cfg file is build by grub-mkconfig, edit the /boot/grub/grub.cfg file to change the kernel's root parameter to 'root=/dev/mapper/VolumeGroupLabel-LogicalVolumeLabel' (gathered from various sources including the LVM section on the Arch Wiki GRUB2 page).
Boot happily ever after.

This is quick and dirty, but I wanted it documented for the next time I try this.