Why is that surfcam offline?
Yeah we had major throughoutput issues beforehand, but the lightning strike seemed to knock everything out completely. We had intermittent line sync for a few days, then it all went blank.
NBN said they visited the site and "got line sync" - except my newly replaced modem still didn't work (after multiple power-cycling too). And they won't send someone down again until I "try (yet) another modem".
So I treked down yesterday with yet another modem, and guess what! Nothing changed.. both still can't get sync. I reckon they never visited in the first place - wouldn't be the first time this has happened either.
So now we're back on the waiting list.
Surf Cam replays don't seem to be working for Qld cams.
Not sure why they're offline.. but we're looking into it (most of our tech support contacts are away for the Xmas break.. always a bad time for things to go down).
Scarborough (Perth) is back online. Thanks for your patience everywhere.. what a mission! Massive thanks to Tully, Howard & Josh at Engie Services for their amazing assistance.
Any update with the Bondi cam? Last update was almost a year ago saying it was getting sorted soon
Unfortunately, Bondi has taken way longer than expected - for a bunch of reasons I won't go into here - but we're almost there. Getting technicians sorted now to finalise the install ('tis a tricky one, requiring custom build mounting frames).
Kings Beach NBN is slated to be activated mid-late next week. Fingers crossed (as we've lost the previous ADSL service.. came back last week for a day, then it went again).
G'day Ben, any reason the Cronulla Point cam looks like its covered in vaseline?
cheers
It's had focus issues for a while. I've got brand new cams for the Point, Island and Beaches here, just trying to schedule a time to get 'em done.
nice, cheers!
Lorne is back! And we've got a second cam too, looking at the beach. Thanks for your patience everyone.
any word on how long the narra cam will be out of action? (another surf website’s closely located cam seemed to go down at the same time).
also, any update on the bondi cam? last word on 31 may 2019 was ‘soon’
Narra is down for a few weeks while they do some maintenance on the building. Bondi was meant to be installed a few months ago (along with an upgrade of many other sites) but we've had travel/access issues. Hoping to get down there in the next month or so.
thanks for the fast response ben. especially on a sunday morning
The crowds immediately dropped here when the Narra cams on both sites went down and that was during school holidays too.
Find that very hard to believe knowing the diminutive traffic the cam saw - also the angle of the camera in relation to NN - compared to the overwhelming numbers in the water since COVID.
Might this hiatus in east coast activity be used to install the mythical Bondi cam?
Actually, Covid-19 is making our planned surfcam upgrades/installs very tricky, as my guys are unable to travel to Covid-19 "hotspots" (we're based in Northern NSW, and are doing quite a bit of work on the Gold Coast at the moment - so travelling outside of the bubble here means a 14 day isolation etc).
But, I'm working on an alternative solution.
Hi Ben. Fairhaven reportedly had whale sightings early this morning - 6.30 - 7.00 am. Checked the cam around 8.00, but it was down for a while. Any chance it was functioning early and there are recordings?
No replays for Fairhaven at the moment, sorry.
No worries.
hey ben. 3 months now for narra and years for bondi. any actual time lines? (i understand covid but still ...)
re the crowds at narra, there have been two cams on that for years (3 including southy) and almost never any crowds. as soon as the banks got good earlier this year it's been crowded whenever it's working regardless of the cams
Major NBN issues at Narrabeen. Despite having an ADSL service here for more than 15 years, NBN won't recognise the location nor allow me to create a new location ID (been through this many times before, just takes a long time to sort).
Both of our Coffs surfcams have been offline since 1pm yesterday - turns out someone cut the main backhaul fibre link near Mayfield (i.e. shutting down the internet across vast areas of the Coffs region, apparently). No ETA from our wholesaler at this point in time.
Argh. Lots of network issues this morning.
Coffs - both cams came back yesterday arvo once the regional fibre service was restored, but Park Beach went back offline around 12:20am. Not sure why, we are investigating.
Also, almost all of our Gold Coast surfcam network is offline this morning.
Unfortunately, all of our Goldy cams - except Narrowneck and Kirra, which are still streaming - are on a different ISP to the rest of our national network, and this ISP seems to have experienced a major outage around 4am (all of our gear is fine). There wasn't any storm related activity overnight, so I'm not sure what the issue is. Currently looking into it as a prriority.
Good news on D'Bah though, the scaffolding came down yesterday so once the network is restored, we'll have a clear view of the southern end of the beach. Apparently they have a little more work to do in late Jan but until then it'll be clear.
All back now.
And now they're down again. Argh.
Over two years to put a camera up at Bondi now
Ain't as easy as it might appear. Fortunately, we now have access to Sydney (tis no longer a Covid hotpot) so the backlog of upgrades/installations are about to kick off.
any actual dates on when these cams will be back up? Bondi cam (down for years). Narra cam (down for months). Avoca cam (getting on to a month now). of the four cams that I actually use regularly, 3 are down - 2 seemingly permanently which doesn't give me much optimism for the 3rd. I know plenty of people in IT and AV companies in the greater sydney region and they're all working and have been all through covid
Avoca has only been down since Friday (not "a month"). NBN visit scheduled for tomorrow.
Other Sydney sites will be reactivated once we I can schedule our new IT team (based in northern NSW) for a Sydney trip. The on-again, off-again Covid situation makes it very difficult, because we have an enormous amount of Qld based work, and can't afford to have to be banned from entering the state from 14 days because we've been to Sydney.
But rest assured, we're trying as hard as we can to find solutions for this.
thanks for the update ben. the avoca cam went down right around xmas day and came back up briefly a few days back before going back down again
Ahh, fun and games with offline surfcams at the moment.
A few days ago Torquay experienced an internet outage. We immediately lodged a fault with NBN; they sent around a technician who replaced the lead-in (main cable from the street to the property), which was in poor condition owing to heavy corrosion, apparently. FWIW, this is actually pretty common in coastal areas.
However, the service never came back online, and NBN have now handballed it back to us - saying the technical problem 'must be inside the building'.
So now I have to send an electrician around to the Surf Club to investigate, though with the Surf Coast currently in lockdown (as of Sunday night) this may be a little difficult to find at short notice.
Anyway, we're working as hard as we can to get the surfcam back. Thanks for everyone's patience.
Oh, and just on 13th Beach - the Surf Club is undergoing some major renovations (they're adding another storey!) and so the surfcam is down until the building works are completed.
Hi, perhaps some kind of refund or extension of subscriptions for those who have paid for the premium service?
monkeypig wrote:Hi, perhaps some kind of refund or extension of subscriptions for those who have paid for the premium service?
More than happy to have a conversation around refunds etc, but I think I'll start another thread for that. I'll try to do that today.
However, re: Torquay and 13th Beach cams being offline - it's worth pointing out that (1) they are free cams (you don't need to sub to view), though subscribers get the added benefit of not seeing any ads, and have longer timeouts etc, and (2) Torquay and 13th Beach cams are down due to factors outside our control.. i.e. it's not our equipment at fault.
I'll elaborate on Torquay in a little more detail in a sec, to outline what's gone on so far.
OK! Here's where we're at with the TQ surfcam being offline. It's a lengthy read, but worth me putting on the record (this is a good example of the kinds of things that goes on quietly behind the scenes at Swellnet, day in and day out).
Last Friday, our monitoring system detected a drop in speeds at Torquay - the camera is at the SLSC - and a deterioration of our internet service. I lodged a fault with our ISP, we did some (remote) tests and they escalated the fault to NBN, who said they'd rock up Monday arvo (1pm onwards) to take a look.
The service remained poor Friday and Saturday but improved Sunday and early Monday.
On Sunday night, the Vic Govt announced a 7 day lockdown for the Surf Coast, starting Monday morning (this is important).
Around 11am Monday, the Torquay surfcam connection dropped right out. I thought NBN may have rocked up early to begin checking the service, however what was unusual was that they hadn't called me to get the building open. And, because there's no way to contact NBN directly, I wasn't abe to follow up... just had to wait until the job was done. Perhaps they hadn't rocked up, due to the lockdown? Communications are considered essential services but NBN often employs local contractors, who may have decided in their own best interests. I've had plenty of NBN no-shows for previous outages, during non-pandemic times.
Anyway, there was no way to know.
The service didn't come back by Tuesday morning (and I received no further contact from NBN), so I contacted my ISP to work out what was going on.
One email (Tuesday morning) and two lengthy telephone conversations later (Tuesday afternoon), they advised me that NBN had been on site, and fixed the 'lead-in' from the street pillar, which was heavily corroded. They also advised that there was an open circuit somewhere inside the building (on our line) and that was the cause of the internet not reconnecting. So, it was now my problem to fix.
Odd, I thought - why did my internal cabling choose to fail at the same time the NBN tech was out there?
Anyway, I then went through the process of finding a licensed technician who could visit the surf club pronto, to take a look. Because of the lockdown, our usual sparky (who doesn't live in TQ) was unavailable. Other sparkies - I tracked down nine in total - couldn't get to the job "for four or five weeks" etc. I have three technicians up here in Northern NSW that I would have happily flown down to get the job done, but they would have had to quarantine at both ends for 14 days, which is obviously impractical.
Eventually I found Liam (what a legend! thanks mate) who popped over straight away (Wednesday afternoon), and then tried to unravel the internal cabling of the Surf Club - a tough job when you didn't do the installation, and haven't been to the building before. And, the Surf Club has their own internal cabling too, so it's a mission to work out what's what.
Nevertheless, after quite a bit of time, and with some assistance from another technician Brad (thanks fella!), they were able to confirm that there was nothing wrong with Swellnet's line from the building's MDF to our data connection.
Interestingly, they couldn't see where the NBN had repaired the lead-in. And, it's unknown how this could have been done, if they couldn't get internal access to the building either.
Furthermore, their fancy monitoring equipment detected a break in the line outside the surf club, between the building and the street pillar. This is on NBN's side of the boundary point. Essentially, there's a 10-pair coming into the building from the street (five pairs are in use, one of which is mine, five pairs are unused) and it just so happened that my pair is broken - probably the result of corrosion.
Remember - NBN acquired all of Telstra's old copper cabling back in 2014, which connects the majority of properties around the country to the internet (via FTTN). So, for coastal locations - especially those right on the beach! - corroded cables are a common cause of internet outages.
Fortunately, the five unused pairs (inside the 10-pair) were tested as being in perfect working order back to the street pillar.
So! After receiving a detailed report with photos late Wednesday night, I sent this off to my ISP, who then (Thursday morning) forwarded it back to NBN. This clearly shows that the problem is at NBN's end, and not Swellnet's. I followed up with my ISP Thursday lunchtime to make sure they had sent on the information (remember, I can't contact NBN direct). However by Thursday evening I hadn't received a response from NBN.
How can it be fixed? Easiest way will be for a tech to swap over our pair. Alternatively, NBN may choose to replace the lead-in from the street pillar, as the other lines are at risk of failing at some point down the track too. However, that's a 350m underground cable replacement that would be several days' trenching work for a couple of contractors, I presume. So, way more expensive.
And so, here we are. Waiting for NBN to fix the service at Torquay.
In the meantime, I'll continue to politely reply to the backlog of angry emails about "why is your fucken Torquay surfcam offline".
So, who wants to know the latest at Margaret River? NBN tech was out there yesterday (took bloody weeks to get a booking).. and guess what? The problem isn't with our gear. In fact, the connection to the property is also fine too. It's an upstream issue inside NBN's infrastructure, and they're "looking into it".
I won't bore you all with the technical details though.
Thanks Ben, NBN "looking into it" does not sound promising though :)
A response from the NBN re: Torquay - next available appointment time is Monday.
Patiently waiting for NBN to call this morning, and begin fixing the cable.
And no, to the several people who sent irate emails over the weekend, the surfcam hasn't been down for 'months', it's been six and a half days. Yes, I'm still having to personally respond - obviously some people can't see the link on the surfcam page outlining the status, and haven't seen this thread either.
There has been issues in margs for a couple weeks with internet on mobile phones also. Interestingly the surf has been noticeably quieter since the cam has been down ,there has been several days that the wind forecast was more onshore than it actually turned out so bonus surfs for those who check it
thermalben wrote:Patiently waiting for NBN to call this morning, and begin fixing the cable.
And no, to the several people who sent irate emails over the weekend, the surfcam hasn't been down for 'months', it's been six and a half days. Yes, I'm still having to personally respond - obviously some people can't see the link on the surfcam page outlining the status, and haven't seen this thread either.
You get irate emails when the cams are down? Sad.
And we're back!
thermalben wrote:Ahh, fun and games with offline surfcams at the moment.
A few days ago Torquay experienced an internet outage. We immediately lodged a fault with NBN; they sent around a technician who replaced the lead-in (main cable from the street to the property), which was in poor condition owing to heavy corrosion, apparently. FWIW, this is actually pretty common in coastal areas.
However, the service never came back online, and NBN have now handballed it back to us - saying the technical problem 'must be inside the building'.
So now I have to send an electrician around to the Surf Club to investigate, though with the Surf Coast currently in lockdown (as of Sunday night) this may be a little difficult to find at short notice.
Anyway, we're working as hard as we can to get the surfcam back. Thanks for everyone's patience.
Oh, and just on 13th Beach - the Surf Club is undergoing some major renovations (they're adding another storey!) and so the surfcam is down until the building works are completed.
Thanks for the update Ben, good that you take the initiative to keep us in the loop :)
And.. the service has been off and on for the last few hours. Looks like the techs are still sorting out the issue.
Back now... fingers crossed.
Looks like we've fixed the problem at Margaret River.. fingers crossed it stays up.
Caloundra/Kings cams fixed last week too.
Currently trying to rectify a problem at Coffs Harbour (was a power outage.. but the gear hasn't come back).
And now NSW's Covid restrictions have eased somewhat (well, soon to be the case for regional travel), we can finalise our Sydney upgrades!
Mark another year passed without a Bondi cam
Bondi needs an artificial pointbreak made out off that northern headland...offshore in NE winds,, gets plenty of swell and would become an even better tourist attraction...that northern corner is offshore in NE winds but ive never seen waves there. Tought my ex gf (whos now left the country and was only here for two years) how to surf at bondi..she was smokin hot so everyone was encouraging her to get waves while it was crowded.no body dropped in on her or gave her shit...but she thought it was rude people yelling at her to GO! ("ill go when i feel like it")
I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably.
However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.
Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test.
But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down.
So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through.
Is it a local power issue?
Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).
Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available.
Is it a local electrical issue at the property?
This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…
After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.
The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.
But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).
Is the problem related to our Internet Service Provider?
This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).
Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…
We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.
However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on.
Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost).
We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered.
For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again.
Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.
This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day.
Anyway, back to troubleshooting.
Is the problem related to our surfcam hardware?
Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.
Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install).
Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.
Another example: two weeks ago, our Newcastle surfcam went down.
It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week.
But sometimes it's not that easy to source the problem. It could be one or more of the following issues:
Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks.
So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.
Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.