Grand Canyon/Zion Trip

I’ve just returned from a short, but very enjoyable trip down to the Grand Canyon and Zion National Parks.

A few thoughts for anyone organising something similar…

Camping

In the Grand Canyon I stayed at the huge NPS Mather Campground which was nice enough. However it sells out quickly and for those who don’t want to plan so far ahead, or would rather not spend the $$$ I’ve since discovered that there is dispersed camping available in Kaibab National Forest, just outside the park. Details here.

Food

On the drive from the Grand Canyon to Zion, Kanab is a good place to stop for food. I would particularly recommend the Kanab Creek Bakery:
http://www.kanabcreekbakery.com/.

Yellowstone Logistics

Here’s a quick summary of how I organised my six day trip to Yellowstone.

  1. Flew Vancouver -> Salt Lake City. Picked up rental car, bought gas + groceries and drove up to near the West entrance of Yellowstone. Slept in the back of the rental off a forest service road.
  2. Entered via West Yellowstone and drove anti-clockwise around the loop, sorting out my backcountry permits in Grant Village. I then continued around the loop to the Cascade Lake 4K5 Trailhead just north of Canyon Village and hiked into my campsite at Wolf Lake.
  3. Hiked back out, check out the canyon and then drove to Mammoth HotSprings via Tower Junction. After walking around the hot springs I drove down to the Biscuit Basin parking lot which was the trailhead for my hike to tonight’s campsite at Firehole Falls.
  4.  Hiked out and visited a bunch more geysers and hot springs in the Old Faithful area. Continued driving to the DeLacy Creek parking lot where I started walking to my next campsite ‘Bluff Top’ on the shores of Shoshone Lake.
  5. Hiked out and drove out the South Yellowstone gate and through Grand Teton National Park. Had an early dinner in Jackson before continuing south to a free campsite I found on Wikicamps, next to the Salt River – Whitetail Lane Recreation Area, just outside Afton.
  6. Drove down to Park City and did a few hours of mountain biking before driving over the hill to Salt Lake City and flying home.

4 states visited – Utah, Idaho, Montana (briefly), Wyoming

Identifying scam websites

I’m writing this post after growing increasingly frustrated with the ‘tips’ that news sites put out each time a prominent scam does the rounds. I find they tend to give a bunch of general, waffly information that misses the point entirely. I’m here to tell you how than you can avoid 99{8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98} of scams with this ‘one simple trick’:

Look At The URL

That’s this thing (aka. the web address): 

The usual tip that gets dished out is to check for ‘https’ and the little padlock, and it’s true you should do that. If a site is asking for your credit card details and doesn’t have those two things then you should definitely get ‘outa there fast. Unfortunately any semi-sophisticated scam site will have no trouble putting those two things in place. Thus their absence is a very likely indicator of a scam but their presence is no guarantee that it isn’t one.

I’m going to use Air New Zealand as an example here, for no other reason than I’ve seen a few scams attempting to fool people into thinking they can get free flights. This same information is relevant for any website.

What you really need to be looking at is the domain and top-level domain of the URL. What is this techno-gibberish I speak of? Here’s a quick lesson:

URL Structure

A URL consists of a few parts, separated by periods:

  1. Protocol – either http or https. The ‘s’ stands for secure and tells you whether communication between your browser and the web server is encrypted.
  2. The sub-domain. This could simply be www or it could be mail in the case of mail.google.com. It could also be blah.something.else. Usually there is one sub-domain, but there can also be none or multiple. The good news is that we don’t care about it when checking for a scam site.
  3. The domain is the unique word(s) that the site developer purchases. It’s a bit like buying a bit of land – as long as you continue paying your rates, you can keep it.
  4. The top-level domain is the top of the hierarchy and is used to segment the internet into smaller chunks. .com is probably the most well known top-level domain. Country specific top-level domains are also common eg. .co.nz. There’s also .govt, .net and a whole swath of more esoteric ones.
  5. The path is everything following the first slash after the top-level domain. This is just how the website developer has organised their site and is also unimportant when identifying a scam site.

A website developer purchases a combination of domain & top-level domain which must be unique across the entire internet.

What is important is the domain and top-level domain and their proximity to each other in the URL. Look closely at the periods. The hyphens don’t count!

Often a scam site will register a longer domain that includes the same word(s) as those in the legitimate site. eg. https://www.airnewzealand-freeflights.co.nz. Or they’ll put the word(s) from the legitimate site into a sub-domain eg. https://airnewzealand.freeflights.co.nz. With your new knowledge of URLs you should see that these are obvious scams. For instance, in the second example the scammer has legitimately purchased the domain freeflights.co.nz and has then added a sub-domain to try and make it look like the real deal.

Here’s some other examples. The key thing is that you know what Air New Zealand’s normal/legit domain is. The best way to double check is just to do a Google search and check the first (non-ad) result:

Once you know that, see if you can identify whether these are safe sites or not. Note that I haven’t made them clickable like normal links (except for the last one).

  • https://www.airnewzealand.co.nz/safe. The domain is airnewzealand which is what we expect.
  • https://www.airnewzealnd.co.nz/ not safeRegistering a domain with a slightly different spelling is a common trick among scammers.
  • https://www.air.newzealand.co.nz/– not safe. The domain here is newzealand and they have added air as a subdomain to make it appear from a quick glance that is the same site.
  • https://www.air.newzealand-online.co.nz/– not safe. For similiar resasons. The domain here is newzealand-online.
  • https://flightbookings.airnewzealand.co.nz/vbook/actions/ext-search?searchLegs[0].originPoint=YVR&searchLegs[0].destinationPoint=&promoCode=&adults=0&children=0&tripType=return&searchType=flexible&internalRevenueSource=cms-book-book-now-button{8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}7Cbook&_ga=2.147022391.1465521394.1536551535-1663654223.1536551535 safe. Don’t be put off by the fact that it looks long and scary. All you need to care about is that airnewzealand is separated by a period in front and is the last part of the URL before the top-level domain .co.nz
  • https://flightbookings-airnewzealand.co.nz/vbook/actions/ext-search?searchLegs[0].originPoint=YVR&searchLegs[0].destinationPoint=&promoCode=&adults=0&children=0&tripType=return&searchType=flexible&internalRevenueSource=cms-book-book-now-button{8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}7Cbook&_ga=2.147022391.1465521394.1536551535-1663654223.1536551535 – not safe. Hopefully you noticed the hyphen before airnewzealand. The registered domain here is flightbookings-airnewzealand which is a completely different address.
  • https://www.airnewzealand.co.nz – not safe. Be careful, the text you read on the page (or text, Facebook, WhatsApp…) may be pointing to a different site. Which URL did you end up on when you clicked this link? In general it’s OK if you follow a link and end up on a dodgy site, as long as you then check the URL and exit. Bad stuff only happens once you interact with the site in some way eg. download a file or enter personal information.

Hopefully this helps someone out there!

Extra for experts: The first section of this post is an awesome introduction to how DNS works https://hacks.mozilla.org/2018/05/a-cartoon-intro-to-dns-over-https/.

Update (22/01/2019): Google just launched a great quiz which can test your knowledge of this stuff. Check it out: https://phishingquiz.withgoogle.com/.

Porting a humongous Perl script to Python

The Problem

The first project I was tasked with at my new job involved porting a large (>18k lines long!) Perl script to Python. I knew from experience that trying to do this in one ‘big bang’ step was sure to result in the new version having a bunch of bugs that had been squashed out of the Perl script over years of development. Instead I sought a more cautious approach which is described in this post.

The Perl script in question is run as a console app. It takes a document id along with various optional arguments. The script locates/generates a number of urls, writes them to a database and exits. It is invoked by another Perl script which reads the results from the database on completion, all in the lifecycle of a FastCGI API request.

Now, many years on from this script being created it seems an obvious fit for a ‘microservice’. Thus the goal is to both port the code to Python (as part of a company-wide push to consolidate languages) and to change it from a console app to a Flask API.

Interfacing Code

Going back to the cautious approach I mentioned above; fortunately the structure of the existing Perl script lent it to being gradually ported over piece by piece. I looked for a way to interface it with the new Flask API and the Python subprocess module looked like it would work nicely.

In terms of data transfer, I made a minor modification to the Perl script to write its output to stdout as JSON, rather than to the existing database (which I did not want the Python API to be coupled to). Writing data to stdout sounds fragile but I rationalised that this is exactly what Linux utilities piped to each other have been doing for years. It just means you have to be careful not to have any stray print statements floating around.

The interfacing Python code looks something like this:

def get_links_from_perl_script(start_process, process_info):
	input_list = [start_process, process_info]
	input_json = json.dumps(input_list)

	p = Popen(perl_script_path, stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=perl_script_dir)
	output, err = p.communicate(input_json.encode())
	rc = p.returncode

	if rc == 0:
		logger.info('Perl script finished successfully for start process: {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s' {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98} (start_process))
		if err:
			logger.warn('But the following errors were reported: {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s' {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98} err)
		links = []
		json_output = json.loads(output)
		return json_output 
	else:
		logger.error('Perl script exited with non-zero error code: {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}d for start process: {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s. Error: {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98}s' {8b0b96060afa46d1ca7c90aeb24f8da085e11471ba2cd3d6bf9cf57471789b98} (rc, start_process, err))

And on the Perl side:

use strict;
use JSON qw(encode_json decode_json);
my $str = do { local $/; <STDIN> };
my $decoded_json = decode_json($str);
# do stuff....
print encode_json(\@some_results);
exit(0);

One extra quirk is that I work on a Windows machine. Whilst options exist to install Perl on Windows, it definitely doesn’t seem to be a first class citizen. However we now have the WSL (Windows Subsystem for Linux)! My Ubuntu WSL already has Perl installed, so I wondered if I could get my Python Flask API to spin up a Perl subprocess on the WSL and pipe data to and from it. It turns out this is fairly easy. In the Python code above, the perl_script_path variable is declared as follows:

perl_script_path = 'wsl perl /mnt/c/Users/nware/Dev/this_project/humongous_script.pl'.split()

Note: a trick for young players is that this won’t work if you have a 32-bit version of Python installed. The WSL is 64-bit so Python won’t know how to find it. Ideally just install 64-bit Python, but you can work around it with this magical incantation:

perl_script_path = os.path.join(os.environ['SystemRoot'], 'SysNative', 'wsl perl /mnt/c/Users/nware/Dev/this_project/humongous_script.pl'.split()

Perl package management

A quick note on Perl package management. I was frustrated at the seemingly manual process of installing Perl packages with cspan. Coming from a Python/C#/Javascript background which all have good(ish) package management solutions, this seemed archaic. I went looking for something similar and found exactly what I was after: Carton.

Containerization

This all worked nicely for dev/test but I wanted the Flask API in a Docker container for production. The tricky thing here is that containers are meant (for good reason) to run a single workload. Thus there are official Python base containers and official Perl base containers but obviously none that have both.

I ended up creating an intermediary container, which is essentially the contents of the official Perl Dockerfile but with the based changed from buildpack-deps:stretch to python:3.6-stretchhttps://hub.docker.com/r/nwareing/perl-python/.

Note: The long term goal of this project is to gradually port all of the Perl code into Python. When this is done, the interfacing code can be removed and we will just use the offical Python docker image as per normal.

I could then create my actual application Dockerfile as follows:

FROM nwareing/perl-python:latest

RUN cpanm Carton && mkdir -p /perl_src    
WORKDIR /perl_src

COPY ./perl_src/cpanfile /perl_src/cpanfile
RUN carton install

WORKDIR /app

RUN set -ex && pip install pipenv uwsgi

COPY ./Pipfile* /app/

RUN pipenv install --system --deploy

COPY ./perl_src/humongous_script.pl /perl_src/humongous_script.pl

COPY ./src /app

# Start uwsgi web server
ENTRYPOINT [ "uwsgi", "--ini", "uwsgi.ini" ]

Summary

In summary, the work of art / monstrosity I’ve created looks something like this: