Why is that surfcam offline?
Also, some updated on current surfcam outages:
Wollongong: we're relocating this camera to a better location, however awaiting final Council approval. We have everything ready to go, so once the green light is given, it'll be a couple of weeks until its live.
Thirroul: as per Wollongong.
Bondi: approval for new location given, awaiting final sign off for installation. Hoping this will be up very soon.
Cronulla Point: camera has been intermittently on/offline over the last few weeks (after six years of service), so we're sending a replacement - though it's a little harder as we're putting in a new kind of cam with a different mounting bracket, which requires a bit more planning and work. Will be back ASAP though.
Moffat Beach: ongoing network issues (ADSL services are terrible in the Caloundra region) and a site relocation, expecting this to be back once NBN is made available early next year.
Portsea: SLSC is being rebuilt, ETA this summer (fingers crossed!). Will be back online shortly after the club is occupied.
Would you believe it - just finished posting that, and got an automated update that our Cronulla surfcams are all offline. Already confirmed electricity is no issue. Our ISP is not reporting any issues (yet) but I'll lodge a fault shortly.
And.. the Cronulla surfcams are all back (20 min down). Not sure what caused it, but here's the trace from our offsite monitoring network over the last 3 hours.
Hi Ben,
Doesn't matter which cam I go to all give me message saying they're down due to technical issues.
Something at my end perhaps?
Hmm, this may be browser related. Can you please try Chrome? We swapped our surfcam player for subscribers a few weeks back, and we've had intermittent reports that Internet Explorer doesn't like it very much.
Bingo!! Works in Chrome but obviously not explorer
Thanks Ben
Hi Ben,
Remember last week how I mentioned that I couldn't get any of the cams working using IE with WIndows 7 Pro.
I just checked again.
The funny thing is that the cams work when I'm not logged in (obviously not the Premiums), but none work when I am logged in.
Chrome still works fine either way.
I wonder how many people actually use IE these days anyway.
There's a reason why the cams work for you when not logged in - we're using a different media player (that allows us to run ads). When you log in, it switches to a different media player.
Yeah, IE usage is now tiny so it's not a problem we want to spend too much time working out a solution, as it affects such a small percentage of our audience. Fifteen years ago though, IE was the vast majority!
Another lil' story for the record.
In the first post (above) I mentioned remote rebooters, that allow us to turn the power off and on again. Very useful devices indeed.. they've saved a lot of travel and downtime over the years.
The first rebooter I bought was from the USA, and was a bit patchy - though it was a welcome development. The rebooter worked by using the same PSTN telephone line that supplied the ADSL service, attached via a splitter, so if you dialed the telephone number you could power cycle everything after entering a code.
The device wasn't very reliable though, and eventually I found an Australian manufacturer who had a better product, though it was bigger, much more expensive and consisted of two items that had a lot of seemingly unneccesary external cabling, which got in the way (and also created more points of failure). But, it all seemed to work pretty well once it was installed.
When the NBN upgrade came along, one of the downsides was that the associated PSTN telephone service for each connection was made redudant. So, our rebooters didn't work any more after we upgraded to NBN.
As such, I had to source a third device - connected via 3G mobile and an antenna - that could plug into our cumbersome rebooting setup - so we could replicate the service. Unfortunately, these 3G devices (and I tried a few manufacturers) were another costly expense, were poorly designed and very unreliable, and added further cabling nightmares into each surfcam cabinet.
So, I was stoked just over a year ago to find a new manufacturer who had developed a 3G remote rebooting system inside a small compact device, that worked perfectly using SMS services. Unfortunately, this meant ditching the old system (which had cost a significant sum of money over time), but at least I'd have a reliable rebooting system.
As such, I began the process of upgrading our rebooting system one by one. For each location, I used a 3G Vodafone SIM card, as it had the cheapest plans (remember, the devices didn't need to send SMS messages, only receive them). I visited every surfcam site that was within driving range or close to my interstate travels to change the equipment, otherwise I teed up our local sparkies to swap them over.
A couple of months ago, I was about 2/3 of the way through the national upgrade, when the entire rebooting system stopped working. Initially I thought it was another manufacturing fault, but after several weeks and a great deal of investigation I soon learnt that Vodafone had switched their 3G network without warning from 850Mhz to 900Mhz.
For most people, this doesn't matter - most phones are 'full band' and seamlessly switch between networks (3G, 4G across 850Mhz, 900Mhz, 2100Mhz etc). However, in order to reduce costs, the manufacturer of these devices had kept them to a single network.
So, the rebooters were fine, but just needed to have their internal SIM cards swapped from Vodafone to Telstra, which is the only other telco that operates in the 850Mhz band.
Therefore, over the last two months I've been doing just that - either driving from location to location to swap out a tiny SIM card, or arranging for someone to do it at our interstate locations.
What a pain in the arse!
I have to say that for the price you are asking for a membership to this site I would expect all cameras be working all the time. Currently Victoria only has 5 cameras and of those 5 only 2 work. So I can only presume that you must only have 2 paying subscribers to this site and can't afford to get them fixed or add any more?
I so dearly want to subscribe but I won't until there is consistency in the service being provided. It just isn't worth the money to me right now.
Did you read the explanation above re: Lorne and 13th Beach surfcams?
It is impossible for us to avoid these kinds of incidents occuring as they're completely out of our control. I'm always open to suggestions though.
Portsea has been offline for the last year because the surf club was demolished and is being rebuilt.
misterbrettc's comments above have got me thinking re: surfcam uptime.
What is an acceptable period of downtime for a surfcam, over the course of a year? 15%, 10%, 5%, 1%?
If we were to aim for 95% uptime for each camera - i.e. 5% downtime - that would equate to a little over 18 offline days every year, almost three weeks.
Obviously, there are some instances of downtime that are completely outside of our control, for example ISP network outages and power outages. But, for Swellnet hardware problems - where our equipment has broken down and needs replacing - I think there's merit in having a target.
We have our own internal monitoring system that keeps us notified as to when our surfcams go offline, and for a few years now I've been wanting to output the uptime information for each location. So, we might display something on the website like:
Snapper Rocks uptime
- 100% last day
- 100% last week
- 99.5% last month
- 97% last year
This would give subscribers some level of indication of the reliability of each camera over a period of time.
Because, in the case of Lorne and 13th Beach, although their current offline status is annoying, the fact is that both surfcams have been very reliable over the last ten (or more) years.
@misterbrettc, I reckon if your basing your decision of whether to subscribe or not based upon whether a surf cam is working, then you should probably keep the $2.24 per week and get yourself an extra latte on the weekend.
Or put that little extra cash towards fuel to actually go and have a look.....
As a provider of software as a solution services we no longer support IE. No idea why, its our techos' call.
Whenever we have an outage we publish a planned restoration time / date to manage expectations. If we need to revise we'd publish target history
Publishing up time history helps me know you're aware of issues (but dont include dark hours)
Your fee is worth my time to avoid the adds, enjoy a glimpse of what I'm missing and plan a goof off day
A quick update on a few surfcams:
13th Beach came back online Saturday.
Lorne has a scheduled NBN visit early next week. Hopefully there won't be any problems (of which there could be many) and the surfcam will then come up shortly afterwards.
Newcastle has had intermittent issues since the most recent problem (which was determined to be a loose power cable). The symptoms are the same as before so I'm hoping this wil be rectified by another site visit (rather than an expensive hardware upgrade) but we'll know in the next few days once someone's been around to check the equipment.
Illawarra: made some progress but still no ETA (we have a few going in here).
Update on Newcastle: looks like the camera hardware is broken (perhaps that power plug issue from a few weeks ago fried the circuit board?). I've send down a replacement to our sparky today, will have it swapped over ASAP.
Fascinating (though insanely frustrating) complication in the Lorne surfcam/internet saga.
Because of all of the problems over recent weeks (read above), I decided to fly down to visit the surf club in person, when the NBN technician rocked up.
As soon as he got there, I knew we were in trouble. Turns out our (previous) connection had been listed as having an incorrect address all along. So, even though a new service was ordered (which he was there to install), he couldn't do anything. Old mate lasted three minutes detailing the situation before driving off to the next job.
This explains the problem we had a few weeks ago, when NBN received a churn request etc (see above). But it doesn't explain how we managed to get an NBN connection the first place (it'd been operational for about a year). Or how the previous ADSL service was connected for many, many years prior to that - all with an incorrect street address.
I suspect it's part of the phenomena I've encountered many times with Surf Lifesaving Clubs over the last fourteen years. And that is: each building rarely has an actual street address. Search for Lorne SLSC anywhere, and invariably it'll be listed as "Mountjoy Pde, Lorne". Portsea Surf Club is at "Back Beach Rd, Portsea". Etc etc. As they were all built before Telstra tightened its regulations (around emergency services), most Surf Clubs just didn't have a street address, and going through the process of sourcing Lot numbers from Council added a disproportionate amount of red tape to the supposedly simple task of ordering a telephone line or internet service.
Anyway, we're back to the drawing board with Lorne. I've had to source new paperwork from the Surf Club (who have been fantastic throughout all of this, BTW), and the new order is being placed, which will require another NBN technician visit, and then after that - a seperate visit from my technician after that to get it all operational (lucky I didn't arrange for him to come down today!).
And to add insult into injury, it's dead flat down here. That's one heck of an expensive maintenance visit, without the bonus of a few waves either.
Ben it's a problem that pretty much all infrastructure/building maintenance managers will come across, especially if you are working all over the nation on telecommunications sites. I remember trundling through paddocks looking for coaxial repeaters, finding far flung radio towers by eye, 4wding up mountains with the threat of being snowed in, driving around towns in circles trying to find the site... imagine being ordered 400km back to a site to turn a light off, or having shotguns pulled on one as the site was close to someone's 'crop'. Imagine being flown into offshore islands, reservations, mines, etc. Did so much of the finding of sites on physical maps like CFA maps which were invaluable, and remember all GPS went offline in around March 2003 - as I think it was being used for more nefarious means! Loved that job.
One implication of this is if you are ever calling 000 and they ask for your beach address, nearest street - sometimes the ambo crew can have problems locating the exact site. Always remember the triangular markers on the beach, eg '67W' :)
.... @ben, the other more pressing problem you face is that nothing absolutely happens in Lorne at anything like normal time. Know people that attempted to renovate a house down there and after 3 years of fighting neighbours and council gave up. On the plus side great coffee at Moon Cafe and burgers at The Bottle of Milk.
Update on surfcams stuff:
Lorne: still awaiting confirmation on the new NBN service.
Wollongong: approval has been given (alright!), so once the paperwork is finalised, I'll shortly begin the process get the data lines installed and the camera mounted. We've got two other Illawarra locations confirmed that will also be installed over the coming months, and another three or four that should be online early next year.
Greenmount: was knocked sideways in the weekend's storms. We upgraded it with a brand new camera and mounting bracket yesterday, but the bloody thing is shaking like a leaf - despite there being no wind locally - which the old camera never used to do (even with 30kts+ of wind). As such I am liaising with our camera manufacturer to work out a solution for this.
Newcastle: was upgraded last week with a brand new camera, but has had some problems over the last few days cutting in and out. We've upgraded the firmware which will hopefully fix things.
Middleton Point: hoping to have this installed at the end of the month.
Goolwa: will be installed at the end of the month.
Cronulla Point: was about to send down the replacement but the old camera sprang back to life! We will still replace this with a full PTZ in the near future though.
Bondi: close to sign off for the new location. Can't wait - it's a cracking view.
Lorne: NBN have finally (after eleven days!) advised that the new service will be installed on Nov 6th. Not sure if that will result in a connection (as we may need to send a tech down to connect to the 'new line'), but it's progress.
Newcastle: This surfcam was replaced just two weeks ago with a brand new camera, but it's going offline frequently. We've had a technician visit the site again, and we've ruled out almost everything other than the camera itself. So, I'm sending down yet another today (will be there early next week). Fingers crossed this one works! Kinda annoying too as we actually upgraded Newcastle with a new surfcam only in June of ths year.
Greenmount: We're still unsure what's causing the shaking. The manufacturer doesn't seem to know why either. So, I'll make a visit with some bracket modifications and some additional cameras to see what we can do to improve image quality.
there is something strangely relaxing about watching the lorne cam even though it never really looks surfable
Victorian update: The NBN tech is at Lorne right now, trying to install our new service. Fingers crossed we're up and running shortly.
13th Beach is offline due to a 'planned outage' across Barwon Heads by CitiPower.
ARGH! Why couldn't they choose a day when it's onshore? (surfcam grab from earlier this morning).
OK! just spoke to the NBN technician at Lorne. We now have our new internet service running to the building.
But.. because old mate's "not allowed to touch the internal wiring", we still don't have internet to the surfcam.
Why? Well, all of the telephone lines coming in from the street (also known as a lead-in) terminate at the first boundary point on the property - usually a small box called an MDF (multi distribution frame).
At the MDF, each incoming line is patched to a corresponding internal line. In the past, a surf club may have many different lines (couple of phones, a fax, internet etc). Though these days most clubs can get everything through a single internet service (phones via VOIP etc).
Swellnet has a dedicated line for the surfcam's internet connection - it's been there for many years - which is patched from the surfcam directly to the MDF, but because the NBN tech "can't touch the internal wiring", our internet connection still isn't live.
It's very frustrating, as the two cables are literally centimetres away from each other - but them's the rules.
So now I've had to arrange for an electrician to do the one final connection at the MDF. Fingers crossed the surfcam will then be live shortly afterwards (perhaps late this afternoon, if the sparky can get down there).
What a mission!
So its after Christmas then ben?
Hopefully this afternoon.
13th Beach should be back after 3pm.
Oh, and I forgot Goolwa! In South Oz. We installed the new cam last week, but having NBN issues there too. Hopefully that'll be up and running shortly.
Middleton Point bracket still isn't done yet either, but shouldn't be too far off.
Looks like the focus on our Noosa surfcam gave up the ghost this morning. You can watch it deteriorate rapidly in the replays! About the 2:30min mark in this link, below.
https://www.swellnet.com/surfcams/noosa-heads/replays#/2018-11-09/272984
I've got a replacement cam in the office ready to go, but of course, today is a Friday so even express freight won't get the camera there until Monday - and that's assuming my local sparky is available.
However, the surf outlook isn't amazing for Noosa anyway so there's no immediate rush.
13th Beach is now back.
looking forward to Bondi being back up. wouldn't surf there for quids but it's a very useful indicator for some other breaks
And Lorne is finally back.
Goolwa is live! Just as fat and slow as I always remembered too! :)
https://www.swellnet.com/surfcams/goolwa
Has a view from Goolwa, Cliffs through to Day Street and Middleton Point (I'll set up a proper tour later).
Ah well I remember when just up the road looked like yesterday's WOTD...
MMmmmmmmmmmmmmmmmm
Noosa has been replaced. Better quality cam too, so the image quality is a little crisper.
Few webcams cop it in the storm last night?
Cams are fine but our ISP is having an outage related to the storms. Hopefully back soon.
Turns out both the router and the midspan (power source for the camera) at Byron Bay got fried in the weekend's incredible storms. Swapped 'em both over this arvo.
Sunny Coast.
Moffats: Will site issue be revisited? Or will site's BC & SN be forever held to ransom by vigilante localism?
Noosa: Can Cam focus be pulled back to get more of Nats in the picture please. Log nutter Comp is over, Tourist season is over, & Cyclone season is upon us.
Keep up good work - let's face it cams are an eco boost. Less Time & Fuel wasted in wave hunting & session planning has to be a good thing.
Fact: there are no Locals - just folk that hang around longer than others.
PS. Notes for forecasting. How good has Agnes been - for weeks.
Hi Ben and while your at it Parkies needs attention.......your probably just lazing around anyway drinking cocktails by the pool eh............
Xmas New Year is always a pain when things go down/offline etc as it's impossible to get tech guys/sparkies, and I'm unable to dash up/down the coast to check out things in person.
Noosa: the cam tour is back up and operational.
Moffs: we've had a series of issues here (internet dramas pre-NBN, hardware failures, etc), and we're also sourcing a better location (without pine trees growing through the field of view - they're slow growers but it's surprising how much of an effect it's had after six years!). I'm hopeful we'll have this back up in the next couple of months.
Park Beach: sending down replacement gear this week. Not quite sure what the problem is though.
Christies Beach: had a hardware failure just before Xmas. Sending down a replacement this week.
Bondi: looking forward to getting this online very soon. Also very much looking forward to getting our new Illawarra cams up in the next month or two.
Oh, and Goolwa is offline AGAIN.
Yet another 'planned outage'. I reckon there's been a dozen or more of these in the last year.
Snapper Rocks surfcam broke down last Wednesday arvo, just before I headed out the door to Noosa for a week at the NFOS. It's an old camera and we've got a brand new one ready to go, but it can't be installed until I get back in a few days time. Sorry 'bout the delay (timing wasn't good unfortunately).
And.. Cabarita. What a debacle. The Sunday before last, the internet service went down. Figuring it was a common outage, I went on site to troubleshoot Monday and then Tuesday (ISP asked for a bunch of tests I couldn't do the first day, because I'd left some testing equipment in the office).
Took some time to work out that it was a PSTN line fault (so, not an ISP problem, but a fixed line issue with Telstra instead). So, the problem then got handballed to Telstra. Cue several lengthy phone calls with the service department trying to troubleshoot the issue.
Eventually, Telstra decided an on-site visit was needed, but they couldn't get there to check it out until Friday (just been).
Once they did, they found out that (1) the hardware at the exchange had failed, and needed to be replaced, and (2) there were no free ports available to swap our service over to.
Annoyingly, the Cabarita region has been in a seemingly 'almost ready to switch on NBN' phase for about the last year. But for some reason NBN is still not available at this location. The local Telstra tech is baffled, as he says NBN is all working fine to the node on the street. There is no reason we shouldn't be able to switch over.
So, I'm in a bind: I technically can't upgrade Cabarita to NBN, and Telstra are 'waiting on parts' so they can fix the ADSL line (and I suspect it'll be low priority, because they know that within weeks - or at least months - the ADSL service will be rolled over to NBN, and the replacement equipment they install now won't ever be used again).
Oh man, the Cabarita outage is really starting to grate.
Right, so to recap the previous post (above) from March 5th: surfcam went offline on Sunday 24th Feb. After several days of testing/troubleshooting, it was finally worked out to be a problem with the Telstra telephone service (this cam is still on an older ADSL service, NBN hasn't been switched on in Caba yet).
It then took Telstra a couple of days to check the site (Friday 1st March) and confirm yes: the problem was at their end, they'd had a hardware failure, they'd need to send out for parts, etc etc.
The Telstra technician called me personally and outlined the issues (which I understand very well, having experienced these kinds of problems many times over the last 15 years, right around the country).
On Monday, having not heard anything from Telstra for more than 2.5 weeks (I called a couple of times the week prior, but was on hold for 45 mins and gave up), I finally got through to their support desk, who then told me that the ticket had been closed (!!) They couldn't tell me why - as they could see that the line was still down - so they apologised, and said they'd prioritise someone to head out there and check the line on Wednesday (that was the earliest). If I hadn't have called, the service would be offline indefinitely, and I'd be none the wiser.
On Tuesday, I received a number of calls and SMS (see below) from Telstra, not only confirming their Wednesday visit, but also stating that they would "fix the service" by 7pm Wednesday.
By 9pm Wednesday I hadn't received any notification from Telstra, and the line was still down.
On Thursday, I called Telstra again (another 30-40 min wait), so be advised that no-one managed to get around on Wednesday, because they were too busy. They said they were sorry they didn't let me know about this, but their systems weren't updated until 9pm that night.
So, they have booked it in again, for Monday. And I'm now being sent more SMSs and receiving more calls that the service "will be fixed" by 7pm Monday (see latest SMS below).
Look, I realise no-one really cares - the surfcam is down, and you want it back up again, and don't really care for the reasons why it's offline. But this is a classic example of what goes on behind the scenes to get surfcams back up and running (especially because all of our equipment is working perfectly at Caba). Except, no-one but me (and the Swellnet team) is ever aware. So, I thought I'd document this one in real time because it'll be a useful example for future outages.
Got two SMSs last night in quick succession. Of course, it was always gonna be delayed further...
Any update on the christies cam?
Got the latest replacement here (for Christies) in the office last week, haven't had a chance to send down to the local sparky but will do tomorrow. Looking forward to Craig getting back in a few days, as it'll free me up to get on top of these surfcam/network outages.
any news on Wollongong/Thirroul ?
Illawarra progress has stalled over the last few months due to external committments but we've finally received all of the ordered hardware for the upgrades (surfcam, routers, mounting equipment, cabling, other peripherals) - so I'll be prioritising these locations in the next few weeks (Craig's back tomorrow).
Just got off the phone to the local Telstra technician here on the Tweed, re: Cabarita service
Good news: they've fixed the connection at the exchange.
Bad news: the line is still down.
Good news: he's confirmed that there's nothing wrong with our equipment, or the Telstra lead-in connection to the property. The problem now lies somewhere between the property and the exchange, and he suspects the NBN installation (across the road) has damaged some of the Telstra cabling.
Bad news: he's run out of time today, is off for the rest of the week, and therefore not available to fix it until Monday.
And so we continue to wait...
SMS just received from Telstra, confirming Cabarita will be fixed by 7pm Monday. Fingers crossed...
I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably.
However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.
Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test.
But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down.
So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through.
Is it a local power issue?
Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).
Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available.
Is it a local electrical issue at the property?
This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…
After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.
The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.
But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).
Is the problem related to our Internet Service Provider?
This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).
Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…
We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.
However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on.
Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost).
We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered.
For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again.
Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.
This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day.
Anyway, back to troubleshooting.
Is the problem related to our surfcam hardware?
Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.
Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install).
Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.
Another example: two weeks ago, our Newcastle surfcam went down.
It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week.
But sometimes it's not that easy to source the problem. It could be one or more of the following issues:
Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks.
So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.
Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.