Why is that surfcam offline?

I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably. 

However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.

Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test. 

But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down. 

So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through. 

Is it a local power issue?

Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).

Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available. 

Is it a local electrical issue at the property?

This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…

After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.

The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.

But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).

Is the problem related to our Internet Service Provider?

This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).

Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…

We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.

However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on. 

Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost). 

We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered. 

For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again. 

Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.

This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day. 

Anyway, back to troubleshooting.

Is the problem related to our surfcam hardware? 

Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.

Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install). 

Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.

Another example: two weeks ago, our Newcastle surfcam went down.

It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week. 

But sometimes it's not that easy to source the problem. It could be one or more of the following issues:

• Modem is dead and needs to be replaced
• Surfcam is dead and needs to be replaced
• Surfcam power source (also called a midspan) is dead and needs to be replaced
• UPS is dead (or its battery is dead, as per Snapper Rocks last weekend) and needs to be replaced
• Custom cleaning hardware is dead and needs to be replaced
• Remote rebooting hardware is dead and needs to be replaced (the most ironic of all!)
• Surfcam and/or modem firmware is playing up and needs to be factory reset (requiring a specialist technician visit)
• Cabling has been affected and needs to be replaced (once, we had rats chew through the data cable to the surfcam!). Also worth pointing out that this is usually a not-insignificant project; laying cable is often the most time consuming part of any surfcam installation
• Lightning strike and an associated power surge may have damaged cabling through the property (couple of years back, we had four locations affected by lightning in one week) - again, a difficult problem to source, and can only be fixed by an electrician

Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks. 

So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.

Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.