NodeJS Scaping

Every programmer at some point in their career will need to scrape at least 1 webpage, guaranteed; it’s almost a rite of passage. Recently I’ve started a side-project that required data, data which unfortunately couldn’t be provided in a programmatic manner and needed extracting from 2 completely different websites. An annoyance? Maybe. An opportunity to tryout some different scraping frameworks, definitely! We wanted to do this scraping in a lean manner and we don’t anticipate using the final code in anger, so the search was limited to NodeJS and the following two libraries found: »

Ghost to Middleman

Just recently I’ve been struggling with Ghost. It’s always worked flawlessly, but since I added SSL support its become kind of flakey, the site would be working and then BOOM nothing, Now this might be nothing to do with Ghost, it might be the way I have LetsEncrypts certificate set to automatically renew; it might be… but honestly I didn’t check. Now this blog isn’t really anything more of a sounding board for things I’ve learnt, so missing traffic doesn’t bother me. »

Operation Scan All The Things 1st 1000

So operation “scan all the things” is well and truely under way. I’ve scanned/adjusted just over 1000 photos and wanted to provide some insight into the results and process modification over the last few weeks. Negative Tray Utilisation So initially I was scanning the 35mm negatives as I found them. The scanner can hold 12 at a time, but they’re typically sliced in strips of 4. I quickly realised loading the scanner with 8 negatives when it can take 12 is a waste of my time. »

Selenium testing with Nightwatch.js

We love automated testing over at Futureproofs. Up until recently we’d been using the Selenium IDE to write and execute browser tests. The IDE worked well enough initially, but as the system became ever more complex and the person we were leaning on to maintain them found his time tied up, we needed another solution. Our webapp is written in AngularJS so it made sense to use something written in javascript; we looked at protractor but ultimately decided that the tests should be agnostic and not tied specifically to AngularJS. »

php-amqplib vs PHP AMQP Extension

##The Problem While testing Futureproofs API performance using BlackFire I noticed that we had a bottleneck centred around an fgets method. It wasn’t part of our code (vague I know, I’ve lost the reference). After a bit of digging I found the culprit in the php-amqlib library; a library we use quite extensively from our API to add messages to RabbitMQ. Now php-amqlib is a great library, we’ve used it a lot and its never once let us down. »