Porting a humongous Perl script to Python

The Problem

The first project I was tasked with at my new job involved porting a large (>18k lines long!) Perl script to Python. I knew from experience that trying to do this in one ‘big bang’ step was sure to result in the new version having a bunch of bugs that had been squashed out of the Perl script over years of development. Instead I sought a more cautious approach which is described in this post.

The Perl script in question is run as a console app. It takes a document id along with various optional arguments. The script locates/generates a number of urls, writes them to a database and exits. It is invoked by another Perl script which reads the results from the database on completion, all in the lifecycle of a FastCGI API request.

Now, many years on from this script being created it seems an obvious fit for a ‘microservice’. Thus the goal is to both port the code to Python (as part of a company-wide push to consolidate languages) and to change it from a console app to a Flask API.

Interfacing Code

Going back to the cautious approach I mentioned above; fortunately the structure of the existing Perl script lent it to being gradually ported over piece by piece. I looked for a way to interface it with the new Flask API and the Python subprocess module looked like it would work nicely.

In terms of data transfer, I made a minor modification to the Perl script to write its output to stdout as JSON, rather than to the existing database (which I did not want the Python API to be coupled to). Writing data to stdout sounds fragile but I rationalised that this is exactly what Linux utilities piped to each other have been doing for years. It just means you have to be careful not to have any stray print statements floating around.

The interfacing Python code looks something like this:

def get_links_from_perl_script(start_process, process_info):
	input_list = [start_process, process_info]
	input_json = json.dumps(input_list)

	p = Popen(perl_script_path, stdin=PIPE, stdout=PIPE, stderr=PIPE, cwd=perl_script_dir)
	output, err = p.communicate(input_json.encode())
	rc = p.returncode

	if rc == 0:
		logger.info('Perl script finished successfully for start process: %s' % (start_process))
		if err:
			logger.warn('But the following errors were reported: %s' % err)
		links = []
		json_output = json.loads(output)
		return json_output 
	else:
		logger.error('Perl script exited with non-zero error code: %d for start process: %s. Error: %s' % (rc, start_process, err))

And on the Perl side:

use strict;
use JSON qw(encode_json decode_json);
my $str = do { local $/; <STDIN> };
my $decoded_json = decode_json($str);
# do stuff....
print encode_json(\@some_results);
exit(0);

One extra quirk is that I work on a Windows machine. Whilst options exist to install Perl on Windows, it definitely doesn’t seem to be a first class citizen. However we now have the WSL (Windows Subsystem for Linux)! My Ubuntu WSL already has Perl installed, so I wondered if I could get my Python Flask API to spin up a Perl subprocess on the WSL and pipe data to and from it. It turns out this is fairly easy. In the Python code above, the perl_script_path variable is declared as follows:

perl_script_path = 'wsl perl /mnt/c/Users/nware/Dev/this_project/humongous_script.pl'.split()

Note: a trick for young players is that this won’t work if you have a 32-bit version of Python installed. The WSL is 64-bit so Python won’t know how to find it. Ideally just install 64-bit Python, but you can work around it with this magical incantation:

perl_script_path = os.path.join(os.environ['SystemRoot'], 'SysNative', 'wsl perl /mnt/c/Users/nware/Dev/this_project/humongous_script.pl'.split()

Perl package management

A quick note on Perl package management. I was frustrated at the seemingly manual process of installing Perl packages with cspan. Coming from a Python/C#/Javascript background which all have good(ish) package management solutions, this seemed archaic. I went looking for something similar and found exactly what I was after: Carton.

Containerization

This all worked nicely for dev/test but I wanted the Flask API in a Docker container for production. The tricky thing here is that containers are meant (for good reason) to run a single workload. Thus there are official Python base containers and official Perl base containers but obviously none that have both.

I ended up creating an intermediary container, which is essentially the contents of the official Perl Dockerfile but with the based changed from buildpack-deps:stretch to python:3.6-stretchhttps://hub.docker.com/r/nwareing/perl-python/.

Note: The long term goal of this project is to gradually port all of the Perl code into Python. When this is done, the interfacing code can be removed and we will just use the offical Python docker image as per normal.

I could then create my actual application Dockerfile as follows:

FROM nwareing/perl-python:latest

RUN cpanm Carton && mkdir -p /perl_src    
WORKDIR /perl_src

COPY ./perl_src/cpanfile /perl_src/cpanfile
RUN carton install

WORKDIR /app

RUN set -ex && pip install pipenv uwsgi

COPY ./Pipfile* /app/

RUN pipenv install --system --deploy

COPY ./perl_src/humongous_script.pl /perl_src/humongous_script.pl

COPY ./src /app

# Start uwsgi web server
ENTRYPOINT [ "uwsgi", "--ini", "uwsgi.ini" ]

Summary

In summary, the work of art / monstrosity I’ve created looks something like this:

IEC Vancouver – Tips and Tricks

So you’ve got your IEC visa and booked your flights to Vancouver. Now what? I’d like to share a little of what I’ve learnt in the two months I’ve been here. I’m a Kiwi, so the comparisons I draw will be with life back in New Zealand but hopefully my tips will still prove useful for those from other countries.

Unless otherwise noted all prices are in Canadian dollars and do not include tax.

Insurance

Lets get the boring stuff out of the way first. As you probably know, you need medical repatriation insurance as a condition of your IEC visa. I went ahead and spent $$$ on the 2 year Down Under Insurance (DUI) policy. Arriving at the border they only asked for my entry letter and passport. I didn’t need to show proof of funds or insurance.

I’ve since learnt that if you are employed in BC at least 18 hours a week, you can enroll in the government health care plan. This would mean not having to do any paperwork or pay an excess when ending up in emergency. Presumably it also means you’d be treated without payment for accidents resulting from backcountry skiing or climbing which is excluded in the DUI policy. Details here: https://www2.gov.bc.ca/gov/content/health/health-drug-coverage/msp/bc-residents/eligibility-and-enrolment/are-you-eligible/working-holiday-programs.

Take from that what you will – personally if I had my time again, I’d just get the cheapest possible policy to meet the IEC requirements and then enroll in the BC plan once I had a job.

Arrival

I bought my bike over in a big cardboard box and wasn’t sure how I would get it in from the airport along with all my other stuff. I needn’t have worried as it turns out the taxis all have very reasonable fixed prices (around $30-$35). You can pay with a card as well, so there’s really no need to get Canadian dollars out before you fly over.

There’s a fare zone map, with prices on this page: http://www.yvr.ca/en/passengers/transportation/taxis.

Money

Opening a bank account is probably job number one. Most of the major banks have a promo for new residents where you don’t pay monthly fees for the first year (it’s not all free like in NZ). Make sure you get one of these packages.

I ended up going with CIBC, mainly just because there was a branch near my hotel. Made an appointment and had everything sorted in under 30 mins – including getting a debit card on the spot. They’ve been good so far. App and website work well.

A good plan might be to use a major bank for your first year and then switch to Tangerine for the 2nd year. They have no monthly fees but are a little harder to sign-up for initially without a BC driver’s license or other ID.

Transferring money to other people is a little different here. Rather than just using their bank account number, there is this service called Interac where you ‘send’ money to their email address or phone number.

TransferWise is the way to go for transferring money to and from your NZ account. Low fees and super easy to use. If you sign-up with my invite link you’ll get your first transfer free: transferwise.com/u/nicholasw235.

A note on credit cards. It turns out you really do need one here. Deposits on car and ski rentals are one example. Often websites (eg. paying my mobile phone bill) will not accept international credit cards. Also some shops charge a fee for using debit cards, but not for credit cards which is super weird. Some cards also have useful benefits like covering your rental car insurance. Cards take a couple of weeks to arrive, so plan ahead.

Driving

One of the best things about Vancouver is the car-sharing services. Think Uber, but where you’re the driver. You’ll need a BC driver’s license which you can get easily as Canada has a reciprocal arrangement with NZ. You just go to an ICBC and exchange (yes they do take your NZ one) your license and pay a fee (around $30). Details here: http://www.icbc.com/driver-licensing/moving-bc/Pages/Moving-from-another-country.aspx .

The two main car-sharing companies are Evo and car2Go. Both are good. For Evo, use my referral code C000115783 to get 30mins free.

Phone Plans

Phone plans are expensive here. Shop around to find the best deal as the companies seem to have promos that come and go. The big three mobile providers: Rogers, Telus and Bell all have spinoff brands: Fido, Koodo and Virgin respectively. They use the same networks but seem to be rebranded to attract a younger / non-business audience. You’ll probably want to go with one of these spinoff brands.

I’m paying $45 a month with Koodo for 2Gb data and unlimited calls/texts which is considered a good deal here…

Temporary Accomodation

You’ll have a hard time trying to sort out permanent accommodation before you arrive, so best to book something temporary whilst you go flat hunting. It’s the usual suspects here: AirBnB, Hostels, Hotels (or a couch if you’re luck enough to know someone here). I arrived on the 14th of the month and managed to sort out a room to move into for the 1st of the following month. Over those two weeks I stayed in a couple of places:

  1. The Buchan Hotel – near Stanley Park. This appealed as I was able to have a private room and place to spread out with all my stuff. Similar price to a private room at a hostel ($80 ish / night). Would be a particularly good option for couples.
  2. HI Central – right in the middle of everything on Granville St. Good for meeting people, and breakfast is included. Big lockers in the dormitory rooms. Bad WiFi though.

Finding a Room

Most rooms changeover at the start of the month, so in an ideal world, you will have arrived near the middle of the month – long enough to find a place (hopefully) but not so long they you are splashing the cash on temporary accommodation for too long.

I’m going to make your decision making easy and suggest that unless you are particularly cost conscious, you look for a room in one of the following areas: Downtown (West End / Coal Harbour / Yaletown / Gastown), Kits or Main St. These areas all have cool stuff happening nearby and are close to public transport.

As a single person looking for a room in a shared flat/house, expect to pay between $750 and $1100 per month. Sometimes hydro (what they call electricity) and internet is included, sometimes not – don’t worry too much about that as both are much cheaper than in New Zealand.

Craigslist is where most places are listed here. It’s a pretty clunky site, but OK once you get used to it. Definitely make use of the price and postal code filters to help you narrow the list down. I’d suggest avoiding listings with text like “Great for students / internationals” as they are likely just looking for soft targets to rip-off, or those without any details about the other roommates. There are also a bunch of people trying to rent out the living rooms of their apartments, which sounds pretty awful…

Good luck!

Finding a Job

I’m not going to spend too much time on this because I imagine your experience will vary depending on what line of work you’re in (I’m a software developer), but I thought I’d make a few observations for those intending to further their careers whilst in Vancouver:

  1. It was harder than I was expecting. There is definitely a bias against those here on a short term visa (which is probably fair enough), so you will need to work to present yourself as more attractive than the local competition.
  2. I got very few replies submitting applications from back in New Zealand. You really need to be here, have a local phone number and preferably an address, on your resume before recruiters will take you seriously.
  3. Almost exclusively, the companies I heard back from were those where I had applied through some sort of back-channel, such as a referral (even from the newest or most tenuous of connections). This is probably the biggest takeaway I can give you – meet some local people in the industry / companies you want to apply for and submit your resume through them. Obviously forming these connections is easier said than done, but they often pop up in unexpected places and it’s worth following up.

Buying Room / House Stuff

Moving from New Zealand, I was all excited about how much easier it would be to buy stuff on Amazon. Turns out that’s not the case. Vancouver might only be 3hrs drive from Seattle, but we’re still very much second class citizens when it comes to Amazon shipping.

That said, If you’re not too fussed on quality, and just want cheap bedroom linen and the odd kitchen appliance (like a decent toaster for $10), then I’ve found Walmart online shopping to be the easiest. Delivery is free and you can choose to have your package delivered to the nearest post shop, so it doesn’t matter if you’re not home.

Fun Stuff

Whistler

Skiing at Whistler is a good time. Unfortunately it’s also super expensive. A massive American company with the appropriately sinister sounding name of “The Vail Corporation” bought the Whistler/Blackcomb resort in 2016. They’ve since jacked up all the prices which the locals are all very sour about. Basically you get OK prices if you buy a season pass, but day passes are just really expensive.

Here’s a couple of tips to minimize the damage:

  • The Epic Rides bus gets you there and back for $35, and they leave on-time (even on the snowiest day in Vancouver all year).
  • Rent ski gear from one of the private ski shops, rather than from the resort. A particularly good option is Coastal Culture Sports in Creekside. When I’ve been there it’s been super quiet, the staff are helpful and they give you good gear.
  • Creekside is a little village just south of the main Whistler Village. It has a ticket office and a gondola which goes up Whistler mountain, joining up with everything else on the resort, so it’s a good alternative to the busyness of Whistler Village. You can still ski down to Whistler Village for lunch if you want to check it out (and to get better and cheaper lunch than what’s on the mountain).

 

Food for tramping

Having recently done a few multi-day solo tramping trips I’ve gotten reasonably good at throwing together a menu that keeps me going, doesn’t cost the earth and tastes decent. I thought I’d use my recent Travers Sabine trip as an example.

A few other things I try and keep in mind:

  1. If I’m travelling around a bit in the car before starting a trip, I won’t bring stuff that might go off without refrigeration for a few days (eg. cheese).
  2. This menu is for one person. I might plan a similar menu (just with double the quantity) for two people, but for larger groups it would make sense to use fewer pre-packed meals and use more raw ingredients. Lentil-based meals are a great option for larger groups.
  3. To conserve gas, I prefer meals/ingredients that don’t have long cooking times. Couscous requires much less cooking than rice for example (although I think rice tastes better so I’ll do that when in a hut where gas is provided ;).

Here’s the food I took for 5 nights / 5(ish) days on the Travers Sabine:

Here’s how that breaks down:

Breakfasts

  • Oatmeal (1/2 cup per morning) – combine with 1 1/4 cups water and as much milk powder as you like (maybe 1/4 cup).
  • Cinnamon/nutmeg mix and brown sugar to go on oatmeal.
  • Coffee

Snacks

  • Muesli bars
  • Whittakers peanut slabs
  • Scroggin/nut mix – I like the Pams Super Foods range – also good to sprinkle a bit of this on oatmeal at breakfast.
  • Olives
  • Energy balls (often I make these at home)
  • Gingernuts

Lunches

  • 2 days – crackers with chutney and salami
  • 3 days – Sealord tuna sachet and crackers

Pre/post dinner snacks

  • Tea
  • Cup-a-soup
  • Chocolate

Dinners

  1. Indian MTR meal with 1/2 cup couscous – these meals are super tasty and cheap ($3.50 from Pak ‘N Save) but are not dehydrated so are a bit heavy. I tend to have them just on the first night.
  2. Packet pasta with salami
  3. Packet pasta with salami
  4. Packet pasta with dehydrated peas
  5. Absolute wilderness dehydrated meal with couscous
  6. Extra/emergency dehydrated meal

Trail Runs in New Zealand’s South Island

A brain dump (in a routhly north to south order)

Anything and everything on the Abel Tasman Track. North of Totaranui tends to be less crowded and so an especially good choice. Water Taxis can provide transport if needed.

Nydia Track

From the Cobb Valley road – up the up the Lake Peel track, along the ridge track and back down to the road

Lake Rotoiti Circuit (23km)

Charming Creek Walkway (9.5km oneway)

Avalanche Peak – a good loop option from the village is to run up the Mt Bealey Track, scramble over Mt Bealey, Lyell Peak and Avalanche Peak and descend via Scotts Track.

Hooker Valley – go early to avoid crowds of tourists in the summer

Up South Temple Valley to the hut

Upper Clutha tracks (eg. Alexandra to Clyde)

Up the Dart River from Chinamans Bluff carpark (or the full Rees-Dart if you’re really keen)

Routeburn Track (and any of the other great walks in the region)