shaun.

It's been a while...

Sun, 22 Sep 2024 00:00:00 GMT

Oh my god, it's been a while.

I'll be honest, I kind of forgot that I had this blog.

A few things have been happened since then. Covid, for one. The 2021 insurrection. Divorce, etc.

Enough things have happened since then that the people who knew about this blog are long and gone.

This blog has always been tech related stuff. I'll keep it that way, for now. But I may go on different tangents.

I actually started this back up because I wanted to put a pgp key on my root domain.

Here's what's changed technologically:

obsidian is the note taking app I wish I knew about years ago
I've left a job, got laid off from the next, then found a new one I am ABSOLUTELY in love with
REALLy digging AI/copilot
dropped python for the most part
I dropped all my ansible stuff for chezmoi
I navigated to a proper pure LUA neovim setup
kitty terminal is my new go-to
lazygit
tailscale is fucking awesome
got a new laptop
bought a Supernote, love it
using mealie for my family recipe server
my mom has a podcast
SWAG is great!
lately I've been setting up a bunch of new docker containers from awesome-selfhosted.net

Okay well, let's see if this even will post a new one.

Mr. Robot's Bootstraps

Mon, 13 Jan 2020 00:00:00 GMT

Watching Mr. Robot, it occurred to me that Elliot is almost always on a new system. He's either using a live system (Kali Linux), on someone else's computer, or he's just finished microwaving his RAM. But there's no way that he's always using default set-ups. Right? He wouldn't use nano and bash for his late night coding. He'd have custom python modules, elaborate nmap scripts, and a million zsh aliases. So how does he have time to set up his environments?

I think that he probably has some automated scripts sitting out on the internet somewhere. Scripts that will install everything he needs and configure all the dotfiles to his liking. Sitting on some AWS server paid for anonymously, whose IP address he has memorized. His own personal bootstrap kit floating in the cloud...I want one!

My Bootstraps

While this was a fun thought experiment, it's also been on my to-do list for a while. I spend too much time setting up new systems or VMs. Plus, I can never remember how I got the directory colors just right so it's readable in dark theme terminals. It'd be nice to stop fighting that.

Design

Needs to be:

Fast. Mr. Robot needs to go from zero to writing exploits in less than a minute.
Versatile. I could probably stuff it all in a bash script, but that'd be a pain for more complex stuff like updating dotfiles based on programs installed or if it's a work system or personal.
Easy. No long commands to memorize or type. No flags I can't remember.
Fully headless. I don't know, just because I like not being tied to a GUI.

I went with ansible because it seemed better suited for provisioning a localhost. You can do the same with salt or chef, but you have to make some config changes before you can. Ansible just worked out of the box for what I needed.

All the ansible "playbooks" are obviously in a git repo. But I didn't want to type out a long git clone command, so I wrote a bootstrap for my bootstrap! Just a quick bash script that would install git and ansible, then clone my repo full of the playbooks. This saves me time and about 5 lines of commands.

So this part I'm really pleased with myself about: I stored this bootstrap bootstrap script in the root directory of this site!

shaungarwood.com/bs.sh

Here is the github link if you just want to check out my bash scripting skillz.

This means that all I need to type to begin setting up a new linux host is:

wget shaungarwood.com/bs.sh

Solution

In fact the full solution is a total 3 lines long!

wget shaungarwood.com/bs.sh
bash bs.sh
ansible-playbook my_bootstraps/tasks/*.yml

Gorgeous, isn't it? This simple beauty probably saves me half a day of manually installing/configuring things.

Yeah, yeah - I could pipe the result of wget into bash and have the script run the ansible-playbook command which would mean the final solution is just one command. But that would require a few wget flags and I don't always want to run ALL the playbooks. Three simple commands is fine with me.

Full repo: https://github.com/shaungarwood/my_bootstraps

Other concerns

I just want to address the pedantic nerd voice in my head.

This isn't secure! You're blindly running a bash script that could compromise your whole computer!

Okay, yes. Someone could theoretically hack into my github account or man-in-the-middle my wget. That's a weird amount of work just so they can own my raspberry pi. If I were Mr. Robot - I'd probably memorize the checksum of the bash script so I'd know if someone messed with it. I'm not going to do that.

You're downloading the script from your personal website? Goodbye anonymity.

Yeah, it doesn't get more personal on a fresh system than having the first command contain my full name. If I'm trying to stay mysterious, I won't use this project. Elliot would probably just host the script on a public e-corp server he hacked.

Demo

Here's an asciinema recording if you want to see it in action.

Want to try it out yourself? Here's a full run Vagrant demo using ubuntu and centos:

Vagrant.configure("2") do |config|
  $script = <<-SCRIPT
  wget shaungarwood.com/bs.sh
  bash bs.sh
  ansible-playbook ~/my_bootstraps/tasks/basic.yml
  SCRIPT

  config.vm.provision "shell", inline: $script, privileged: false

  config.vm.define "centos" do |centos|
    centos.vm.box = "bento/centos-7.2"
    centos.vm.hostname = "centos"
  end

  config.vm.define "ubuntu" do |ubuntu|
    ubuntu.vm.box = "bento/ubuntu-18.04"
    ubuntu.vm.hostname = "ubuntu"
  end
end

Even if you don't know ansible, you should check it out. All the magic is in the "tasks" directory. It's very human readable and easy to start hacking up to make your very own bootstrap kit in the cloud.

Edit (2020/01/14): Fixed the vagrant demo code. It was running everything as root, fixed with privileged: false.

Not So QuickBooks

Mon, 02 Dec 2019 00:00:00 GMT

Not every story on here will be a triumphant victory. There are plenty of things to be learned from falling flat on our faces. Or spending days spinning our wheels THEN falling flat on our faces.

So a friend was doing some work with QuickBooks online and she said that it was slow-going, repetitive, and going to take forever.

"I can automate that!" said an overly confident voice at the table.

"Great!", said the relieved friend.

All she needed was any transaction on the books before 9/1/2019 to be deleted. How hard could it be?

Hurtle #1

I had to register as developer.

Not a big hurtle, but more than I had planned on for writing a quick script. I couldn't find anything on the internet for one-shot client scripts to interact with QuickBooks online, but a ton about developing apps to interact with QuickBooks online. I guess I'm an app developer then.

Intuit (the company who makes QuickBooks) actually has a pretty great development setup. Once I signed up (free), they redirected me to a nice little dashboard with the keys I needed AND they create a sandbox company to play around with, complete with 50+ small business transactions. Pretty sweet.

Hurtle #2

I had to write a full app, just to get logged in.

So I was getting this python package running with my new shiny oauth keys and I keep seeing this "redirect URI". Come to find out, it is where Intuit's oauth login page redirects your user WITH the necessary refresh token. Confusing? Yeah.

The assumption is you're writing an app or developing some online tool to be used by strangers. The user clicks "login to QuickBooks" on your app/site, you send the user to Intuit's login page, they authorize your app, Intuit sends the user AND the tokens back to your app.

My flow, just to get all the ids and tokens to login:

                   (1)                                             
  +----------+  generates a  +------------+                        
  |          |  link to      |            |                        
  |  run.py  |-------------->| intuit's   |                        
  |          |               | auth page  |                        
  +----------+               |            |                        
       ^                     +------------+                        
       |    (4)                     |            (2)               
       | which feeds                | sends tokens and users to    
       | back into                  | "redirect uri" (my localhost)
       |                            V                              
+-------------+               +----------+                         
|             |               |          |                         
| config.toml |<--------------|  api.py  |                         
|             |   (3) writes  |          |                         
+-------------+   tokens to   +----------+

run.py is the main script and api.py was my little flask API to retrieve the tokens that the Intuit auth page sent. Write them to my config.toml (I LOVE toml files for configs) and have run.py use them. Phew! I have to be getting close...

Hurtle #3

I am not an accountant.

I'm in! Aaaaand I have no idea what I'm looking at. The API doesn't flow the same way as the dashboard. It's probably minor differences in how things are worded, but it's enough to throw a non-accountant, like me, off.

A few emails back and forth with my friend and I learn that the "view register" part of the dashboard does not have an API equivalent. Opening a register on the dashboard would show you all the bills, payments, credit card transactions, etc. for an account. In the API, you can pull all bills, all payments, all credit cards, etc - but you're doing so for only that transaction type. I code around it.

What a pain. Well, it's almost over...

Hurtle #4

Out of the sandbox, into production.

I've sunk more than a day on this "easy solution" now, but if I can just finish this - I'll still be saving her days, if not weeks, of work. My code works on the sandbox company, I deleted a few transactions, now it's time to turn this loose on the actual target.

In order for Intuit to let me touch a production company, I have to:

Finish filling out my profile
Link to my app's privacy policy
Link to my app's EULA
Supply my app's redirect URIs

UGH! Finish my profile: no problem. A privacy policy and a EULA? This is just ridiculous. So I find some online privacy policy and EULA generators. Could I have just linked to a text file for the lyrics to a Sir Mix-a-Lot song? Probably, but the generators were fast enough and they hosted the end result.

More redirect URIs? I got this, I'll just use my localhost address and...

Error: All Production URI requests must use HTTPS.

Noooooooo!!!!

Okay, maybe I can just use a self signed cert...

Error: Please enter a unique valid redirect URI

It doesn't like localhost. It needs an FQDN (something with a ".com").

Game Over

At this point I took a good hard look at what I had become and the lengths I have gone to "save time and energy". I admitted defeat. In hindsight, I should have called it quits at hurtle #2. But I learned a lot about QuickBooks, some things about OAuth2, and a little about accounting. So it wasn't a total loss.

Called my friend and I told her to contact someone at Intuit. They could do it for her in minutes.

Here is the repo of all my hard work, in case some poor soul needs it

Darn You Splunk! Darn You to HEC!

Sat, 16 Nov 2019 00:00:00 GMT

So, the goal was to get Splunk running to monitor/alert/graph several docker containers and the physical hosts they're running on. I ended up hitting a few roadblocks getting there, while finding surprisingly little help in my Google travels. So I'm writing out a simple top-to-bottom proof-of-concept to get docker logs flowing into a Splunk docker instance.

# quick splunk server, with HTTP Event Collector running and pre-determined token:
docker run -d \
  --name "splunk" \
  -p '8000:8000' \
  -p '8088:8088' \
  -e "SPLUNK_START_ARGS=--accept-license" \
  -e "SPLUNK_PASSWORD=password" \
  -e "SPLUNK_HEC_TOKEN=secret-token" \
splunk/splunk:latest

# manually insert an event
curl -k  https://localhost:8088/services/collector/event -H "Authorization: Splunk secret-token" -d '{"event": "hello world"}'

# using docker's "splunk" log driver
docker run -d \
  --name "nginx" \
  --publish "80:80" \
  --log-driver=splunk \
  --log-opt splunk-token=secret-token \
  --log-opt splunk-url=https://127.0.0.1:8088 \
  --log-opt splunk-insecureskipverify=true \
nginx

# inspect nginx to see logging driver (you'll need jq if you want pretty print)
docker inspect --format='{{json .HostConfig.LogConfig}}' nginx | jq .

# use above nginx, generating a log event
curl http://localhost:80

http://127.0.0.1:8000/en-US/app/search/search?q=search source%3D"http%3Asplunk_hec_token"

If you want to set a system wide docker logging setting, edit your /etc/docker/daemon.json like so:

{
  "log-driver": "splunk",
  "log-opts": {
    "splunk-token": "secret-token",
    "splunk-url": "https://127.0.0.1:8088",
    "splunk-verify-connection": "false",
    "splunk-insecureskipverify": "true"
  }
}

I didn't want it to verify the connection first, because it'll error out starting the docker containers if it can't reach Splunk first. Best to just throw logs that direction and hope for the best.

Gotchas

You would be a FOOL to think that just because your Mac is running on BSD code base that it's the same as Linux. One MIGHT spend HOURS re-configuring a completely useless /etc/docker/daemon.json on Mac, only to find out you need to configure the daemon.json through the taskbar (Preferences > Daemon > Advanced).
Not being familiar enough with Splunk, I kept watching the main search page to see if my new changes were making it to Splunk. APPARENTLY the "What to Search" section and data summary are only for the default index. If you're putting things in a different index (I had set the index to "logs" in my testing), you need to search for that index specifically to see the results. Even the "last event" counter is only for the default index.

to-do

need to tag docker container names
set-up a universal forwarder
dashboards and alerts

My Set-Up For Blogging

Mon, 03 Jun 2019 00:00:00 GMT

Nothing better to kick this blog off than explaining how I set it up!

Choices

I went with github-pages because I wanted something fast & easy, backed by a CDN, and obviously a platform where I still own the content. A big plus that I can tweak and tinker with everything. Could I write my own site from scratch and host it on the cloud? Yes, and I still might. But this works for now and will be easy to port all the content if I do.

Initial Set-up

jekyll new blog
Edit Gemfile, comment out jekyll, uncomment github-pages.
Update _config.yml with details, comment out theme cause we're importing all the files manually.

git clone https://github.com/poole/hyde.git ../
cp -r ../hyde/_layouts/ ./
cp ../hyde/*.html ./
cp ../hyde/atom.xml ./
cp -r ../hyde/public/ ./
cp -r ../hyde/_includes ./

Currently

As always, I'm using Vim and will until the day I die. I'll probably use it to finalize my will.

In order to try things out locally, I needed to run a local web server for github pages. Dockerhub user starefossen wrote a nice little Dockerfile for exactly this: https://hub.docker.com/r/starefossen/github-pages/

So to see live updates of my changes I just run:

docker run -t --rm --name ghp -v "$PWD":/usr/src/app -p "4000:4000" starefossen/github-pages

Future Improvements

I'll definitely write a quick ruby script to generate a skeleton markdown post with the current date/time and proper front matter.