Surf forecast model not updating (Jan 2021)

OK, turns out it's related to maintenance at NOAA. As issued 15th Jan:

"On January 21, 2021, the General Services Administration (GSA) will perform emergency maintenance to the power infrastructure at the David Skaggs Research Center in Boulder, CO. This will impact the Boulder IDP data center causing a total outage of the NOAA Operational Model Archive and Distribution System (NOMADS) website for at least nineteen (19) hours from 1200Z to 0700z the following day, January 22."

We received an email overnight (around 1am) saying that the "maintenance window needed to be extended which has prolonged the outage", though it should be restored in the coming hours.

So, hopefully our next model update should have the latest data.

Thanks for your patience. 

Latest update (0800Z, i.e. 2 hours ago): The NOAA data source "remains degraded.  We continue to work on making it 100 percent viable again"

Another update (1752Z, i.e. 90 mins ago):

"NCO continues working to fully restore the Boulder data center so applications can be failed over to it.  Once that is done, the plan is to reboot the firewall in College Park to fix the Internet degradation issue.  Thank you for your patience."

Yet another update (2032Z, i.e. 8 mins ago): The NOAA data source "remains in degraded mode. NCO continues to work to restore it to an optimal state."

Thanks mate. FWIW, the Forecaster Notes still stand as is.

Latest advice - from 45 mins ago - is that NOAA have fixed the problem.

Our next model run is due around 4-5pm today so let's check back around 6pm to see how it's looking.

Actually, our surf model has already picked up the latest data, so we are back up and running. Probably need to give it a day or two to get up to speed with things, as  there's no way to input the missing data (though because of the current synoptic setup, most locations should be Ok).

Argh. Still down.

Don - it's a complex story. As it is, our wave model developers aren't on standby 24/7, because there's very little ongoing work required once the models are up and running (over the last twelve years, there's been less than a dozen incidents requiring immediate intervention at our end). 

When the data feed went down on Friday - it wasn't until the evening that we worked out what had happened - it was confirmed that the problem was a temporary NOAA outage (which does happen every now and then), and was therefore outside of our control. So, we took the view that it would probably come back sooner rather than later, and given the current synoptic setup, one day offline wouldn't greatly affect the data output.

Unfortunately, NOAA's expectations blew way out of proportion and it took another couple of days to fix.

Some (but not all) of the larger websites appear to have switched feeds over the weekend - for example, those with mission critical clients, such as energy, mining etc - though changing feeds is a not-insignificant amount of work, and could have only been possible for companies with full time model devs who were able to come in on a Saturday (unfortunately, this all kicked off on the cusp of the Australia Day Long Weekend.. terrible timing!).

Anyway, we've been unable to do anything until NOAA confirmed the feed was back - and even once that was done, we had to wait for another one of the six-hourly model cycles to pass before we can determine whether everything's OK at our end. 

I checked one of the forecast pages yesterday afternoon and erroneously concluded that our systems were back up and running (not sure how that happened, I think it's because I was viewing it on an iPad, which displays things a little differently). Unfortunately, it wasn't until the evening that we realised that wasn't the case. I then rebooted a few servers, and then had to wait another six-hour cycle to see if it came back. I've woken up this morning to find it's still not updating properly.

FWIW, NOAA have since been in contact with all end users (i.e. weather services) as it appears there are still some websites unable to get back up and running, as is the case with Swellnet.

Anyway, I've now asked our developers to jump in ASAP and try to diagnose the issue. We'll prioritise this today over everything else, though availability could be an issue as a lot of people are on holiday. 

Apologies again for the inconvenience - we're doing the best we can.

OK, a little more progress.

Seems we managed to get a NOAA feed of yesterday's 12Z run overnight (that was the first complete download since the outage). But, our system isn't updating. I suspect one of the processing tasks has hung, and needs a kick in the balls (non-technical description).

Hopefully our devs can source that issue and get things up and running ASAP.

But, I'm confident we're over the NOAA hurdle.

And we're back! Looks like it was a hung process on one of the servers.

BRC - end of the week to be safe. Depends on which coast you are though.

ID - the models are totally out of whack for Vicco for the next few days, no long period energy at all (as a result of the weekend's malarky).

Argh! Another alert from NOAA: "SYSTEM/SERVICE problems started around 1630Z. NCO support is investigating."

That was a couple of hours ago, and we've missed one model run as a result (though, that shouldn't affect the forecasts very much, if at all). We'll keep an eye on it.

Latest update about 90 mins ago:

"As we continue to bring the Boulder data center back to full health, there will be brief interruptions to data feeds today. We will update users again when service has been fully restored. We apologize for the inconvenience."

More outages over the last 24 hours, due to the same problem as last time from NOAA ("Boulder Data Center system issues"). We've just restarted our servers, so hopefully it'll be back to normal for the next model run in a few hours time.

Model run seems to be working OK now.

Looks like we've got another temporary glitch in the wave model, thanks to NOAA implementing a significant upgrade of the service a few days back. They've gone and changed their directory structure (first time I can recall in the last 10+ years), but we're having a few issues at our end for some reason. Will hopefully be sorted this AM.