Category Archives: Tools

Vim Overriding Settings Woes

Published / by Andrew

Probably every good program has a few irritating anti-features. One of Vim’s aggravating quirks is that it loads plugins after vimrc, which means that some of those plugins may or may not override some settings. (The most frustrating one for me is that I like to have formatoptions set to nothing, but many plugin devs have decided that they know my typing habits better than I do.)

Customizability is extremely important to me, and these plugins really begin to wreak havoc on my workflow.

Probably the best thing to do would be to track those plugins down and edit them manually. Whenever you find a setting that’s being overridden, in this example we’ll use formatoptions, run :verbose set formatoptions?. This will tell you what offending script set it.

Unfortunately, this will likely only be an option if you have root access, because the plugins are in /usr/share/vim/vimXX/* (where “XX” is the Vim version). I have this problem in Termux’s default Vim installation, but that’s resolvable because I have root access to edit these files. However, I’m currently in a situation with a client in which I don’t have root access.

There is a good workaround that I found online, but it’s not perfect. Unfortunately, the website has been down since before I even found the site, but the search results looked promising enough that I checked Google’s cache, and it revealed a pretty good solution. Kind of an obvious solution, but a good one nonetheless.

The direct link is here, but, as I write this, I haven’t been able to reach it (including while using a proxy server).

At any rate, since I don’t know how long Google keeps their cache up, I thought I’d go ahead and duplicate it here (with a few minor changes), in case the website is gone for good.

What we need to do is take advantage of Vim’s “after” functionality, so it will reload $HOME/.vimrc after every plugin is loaded. (You may have some luck with $MYVIMRC, but I’ve found that there are many cases in which $MYVIMRC isn’t even set.) There’s a simple BASH script to do this, and it’s so short that I usually just type it directly into the terminal instead of making an individual script file.

(TL;DR version: Start here)

Please note that there’s a good chance that you’ll need to change the directory path from /usr/share/vim/vim74 to something else. It was only recently that Vim updated to 8.0, and a lot of systems (mine included) don’t use it, and some are still on 7.1. You’ll just need to figure out what directory to use.

    mkdir -p ~/.vim/after/ftplugin
    rm -i ~/.vim/after/ftplugin/*
    for file in /usr/share/vim/vim74/ftplugin/*.vim; do
        ln -s ~/.vimrc-after ~/.vim/after/ftplugin/`basename $file`
    done

Now, Vim will automatically load whatever’s in $HOME/.vimrc-after after loading any plugin. I just put source $HOME/.vimrc in mine. This means it’ll resource my vimrc every time it loads a plugin.

This is not ideal because it means resourcing a settings file every time you open a new file, but, from a pragmatic standpoint, it helps a whole lot more than it hurts.

FastMail Programming Woes

Published / by Andrew

I went through a headache today when I discovered that emails were no longer being sent from Threadstr. I couldn’t figure out what had changed because I haven’t actually touched it in quite awhile. Both my local development and threadstr.com had stopped working and with the same error message: “Error: Invalid login: 535 5.7.0 Incorrect username or password.” I had changed no settings since the last time it was working on either, I had made no changes to the domain registration, and the username and password were correct.

Well, it turns out that something changed on FastMail’s side. I’m not exactly sure specifically what they did, but they no longer let you programmatically send mail using the same password that you use to log in to the frontpage. You now need to create a new password specifically for your application.

It’s actually a good idea, but I sure wish they had let me know about this before they implemented it, because it completely broke Threadstr functionality. I can’t even find an announcement for it– I had to dig through the settings in a wild madcap adventure to find something, anything that could have caused the service to suddenly stop working.

(Irritated as I am, I’m making no plans to change to another email provider, because they’ve still been great so far.)

At any rate, I thought I’d make a post in case this helps anybody else. If you can’t get FastMail to work with something like NodeMailer or PHPMailer, or if it suddenly stopped, it could be this. Below are screenshots of the FastMail settings that will solve the problem.

Use the password generated here instead of the regular password in your application, and you’re golden.

Reco: Vagrant

Published / by Andrew

You may remember the post that I had before on using VirtualBox as a testing environment. Well, I recently learned about a program that just blows that out of the water.

Vagrant makes creating, destroying, using, customizing, ssh-ing into, etc., of virtual machines so ridiculously easy that I’m now using a VM for every project (which is actually the intended purpose of the program). These aren’t just for testing code periodically, but for every step in the development process. I never run the code on the host computer– It’s always in a VM.

That means that I understand all dependencies that are being used. I don’t need to worry that code that works on my machine won’t work on a DO droplet because of an unknown dependency that I installed on my box two years ago. I also don’t need to worry about filling my box with weird and even conflicting software, because the environments are encapsulated in virtual machines. (Case in point– I’m currently teaching myself C#/ASP.Net, etc., and I haven’t bothered to install Mono on my laptop because I’ve installed it in a Vagrant VM instead.)

Vagrant is essentially a program that extends VirtualBox functionality (though it also works with VMware Workstation and Parallels). Everything that I do with Vagrant is possible using VirtualBox on its own, but Vagrant makes it much faster and easier. In Threadstr, I create and destroy VMs on a whim to make sure that every little change that I’ve made to ubuntusetup.sh will work correctly. I rarely even run the node command on the host machine anymore.

Like calculus, Vagrant is hard to learn but easy to use once you’ve learned it. I think it’s well worth the effort, and I’d highly suggest it to any dev that’s working in a compatible project. This means projects that don’t depend on Visual Studio, XCode, or anything else that’s incompatible with Linux, FreeBSD, and other FOSS operating systems. (Piracy laws won’t allow non-free operating systems like Windows, MacOS, etc. to be distributed the way that Vagrant VMs are.)

(If you’re interested in my Vagrant notes, you can find them here, or the original plaintext file here.)

Setting up a VirtualBox VM for Ubuntu Server testing

Published / by Andrew

Something really useful about virtualization, besides getting those one-off Windows programs running without having to dual-boot, is the ability to set up a virtual server.  I use VirtualBox to test Ubuntu Server projects that I plan to put on Digital Ocean.  This way I can test it on the virtual machine before actually spending money to spin up a droplet.

For the sake of brevity and getting to bed at a decent hour, I’ll only explain in detail the one part that was tricky to me the first time I did it.  The easy parts are installing VirtualBox, then installing Ubuntu Server Edition (I prefer an LTS version, currently 16.04)  onto a new virtual machine.  Though I will say these two things:

  • Installation can take a long time, so I would also suggest making a backup copy of the newly-created VM so that you can make a copy of that backup instead of reinstalling every time you want to make a new one.
  • You’ll want to install OpenSSH so that you can a.) SSH into the VM so you can control it from a terminal (from any machine on your network, no less, when we’re done with it) and b.) SFTP files onto the VM (again, from any machine on your network).  You can do this while installing Ubuntu or, if you accidentally hit Enter instead of Space, which I do every time, you can install it after-the-fact with sudo apt-get install openssh-server.

Now for the part that I had a hard time figuring out, but, with some internet research, I did, eventually, figure out– Setting the VM to appear as a separate machine on the network. It’s actually really easy if you know what to do and can be done before or after the operating system is installed– Go to the settings of the VM, go to “Network,” and, for “Attached to,” select “Bridged Adapter.”

Now when you start up the VM, it will have its own ip address on your LAN that’s independent of the host machine.  You can find it with the command ifconfig from within the VM’s window: It’s going to be the “inet addr,” which, in the example picture below, is 192.168.1.111.

Now that this is done, while the VM is powered on, you can use SSH to log into it, SFTP to transfer files to it, install Git and download a project onto it, or do anything else that you need to.  This is a virtual machine on your network.  Use it as a test webserver, use it for Zoneminder, use it for anything that you would use a server for.

The Freedom of Software Development

Published / by Andrew

One of the worst things of using a computer is having to work with tools that you don’t like. Sometimes it’s just because you don’t think the same way that the original devs did, or because the devs are stubbornly holding to a feature that’s widely-acknowledged to be broken.

The most freeing part of software development is the ability to make your own tools. That’s not to say that you’ll never be dependent on other people’s work, of course, but, if you don’t like the tool that you’re using, you have the freedom to make a new one if you’re willing to spend the time to do it. A good example of this would be NeoVim.

Case in point, the image above. The autocorrect in Google has been driving me absolutely insane for years. I’ve been using DuckDuckGo for a long time because the autocorrect is less aggressive, but it’s still pretty bad.

Then I noticed that the only thing that Google does in the get query (though I don’t know if it’s really a “get” query, since it starts with a ‘#’ instead of a ‘?’) to disable autocorrect is a simple flag: “nfpr=1”. That’s it. Adding that programatically is a very simple task.

So, I set out to create that very simple webpage. Apart from the Rage comic that I added for kicks, it’s just a single HTML file. I didn’t even separate the CSS and JS files. It was really that easy to do, but I wouldn’t have known that if I didn’t already know how to use JS. When I set Firefox to use this as my default search, it uses Google without autocorrect.

And that webpage is public. There are a couple of reasons for this. The first is that, I may as well share it, because I know I’m not the only one aggravated by this. The second reason is that I can now use it anywhere. From my phone to my office (if I can convince IT to let me through to it).

I relish the freedom that I have as someone with knowledge of software development, and I hope to learn more on a wide area of topics so that I can continue to create and tweak tools to my own purposes.

Shell languages are the gifts that keep on giving

Published / by Andrew

I am currently downloading every episode of The British History Podcast. I wrote a small BASH script to do this:

function downloadFullPodcast(){
    xhtml=$(wget -qO- $1)
    regex='.*?\.mp3'
    readarray links < <(grep -oP "$regex" <<<"$xhtml")
    arrLen=${#links[@]}

    i=0
    tempStr=${links[i]}
    #dmz1
    while [ "$tempStr" != "" ]; do
        regex='>.*<'
        tempStr=$(grep -oP "$regex" <<<"$tempStr")
        len=${#tempStr}
        mp3url=${tempStr:1:len-2}
        numDum=$(($arrLen-$i))
        number=$(printf "%0*d" 4 $numDum)
        wget -qO $number\.mp3 $mp3url
        ((i++))
        tempStr=${links[i]}
    done
}

downloadFullPodcast "https://feeds.feedburner.com/TheBritishHistoryPodcast"

This isn't exactly a novel idea, but I'm surprised at how few devs I come across that have any interest at all in using the command line languages or other scripting languages.

Even before I became a developer, I recognized how powerful and useful the command line could be. When I was an undergrad (with no coding experience whatsoever at the time), someone on a Linux forum helped me to write a BASH script that played a random episode of Scrubs in the Totem player, and I ran this with KAlarm in lieu of an alarm clock. (In fact, I might want to set that up again with a Raspberry Pi or something.)

These days, I live in the command line, whether BASH at home or PowerShell at work. I couldn't go without it. My primary tools at work, apart from Firefox to test the code, of course, are PowerShell, Vim, and MySQL Monitor. All three of these are CLI tools. (Also, notably, all three of these are scriptable.) At home, it's the same, except for BASH instead of PowerShell (and having the PERL engine for the grep command is really nice).

My office's database is separated into almost 50 separate ports, so it's not uncommon for an inconsistency to appear. For example, one dev may add a column to one database and none of the others, then update the code in SVN to match. This causes mysqli to freak out when it can't find a column. It's not uncommon to see emails to the entire dev team saying "Could whoever is in charge of column XYZ add it to all levels?"

I guess the people sending out these emails must refuse to use the command line. I wrote a simple PowerShell function to run a command on all databases. With that function, it's a two-command process to run "SELECT table_name,column_type FROM information_schema.columns where column_name='XYZ'" across all databases to find what table to add the column to and what type to use, and then add the column to that particular table across all of the databases. This takes about 60 seconds, so I've never felt the need to send out a mass email to every dev to add a column. I just find what needs to be added and add it myself.

I won't go so far as to say something obnoxious like "You're an idiot if you're not using CLI tools," of course, but I do think that these tools offer advantages that a lot of people seem to miss. The most useful tools that I've built for myself have been PowerShell, BASH, and VimScript functions.

What I find is that having a strong grasp (or even a mediocre grasp) of scripting languages like command line languages can really help you to complete a lot of tasks that GUI tools just aren't designed to handle. This is because these tasks are too nuanced for the designers of the GUI tools to have anticipated. The example that I have above with the British History Podcast is a decent one. I did that because my podcatcher program on my phone is great for listening to the most recent podcast, but not so great for binge-listening archived podcasts. I decided to get all of the mp3s so that I could put them in my audiobook reader instead. This could be done in a better scripting language like Python or Ruby, of course, but that's just using different scripting language-- The process remains the same.

Tip 21 from The Pragmatic Programmer by Hunt and Thomas is "Use the Power of Command Shells." "Gain familiarity with the shell, and you'll find your productivity soaring," they say. I would extend that to other scripting languages, and I think Hunt and Thomas would, too, because, in the same chapter, they recommend using a text editor that's programmable. I say, if you're interested, give it a shot. You don't really need to buy a book to learn them (the one PowerShell book that I bought turned out to be a complete waste). Just read a few brief tutorials online, and then you can Google everything else you need to learn as you go. It's been really beneficial to me.