Update: Voicemail and phone services are back online

The voicemail system and phone systems are back online and working normally except that voicemail notification lights are not active yet. This issue should be resolved this evening.

Voicemail system is down

The district's voicemail system is currently down, which also means that calls to the main number (835-4000) result in a busy signal.

We are working with our support vendor to fix it, but do not have an ETA yet. Check back on this blog for updates. If you need immediate assistance, please call 835-4091 and the receptionist can deliver a message.

I apologize for the inconvenience.

Brief email outage this morning

The brief email outage this morning was due to a domain used by Google to authenticate accounts (accounts.google.com) being categorized by our security filter as a phishing site. This may actually have been accurate, as there has been an increase in phishing activity lately. Alternatively, it could be that the domain was recently added by Google.

Regardless, we have fixed the problem. Sorry for the inconvenience.

Phones and voicemail are back up

Phones and voicemail are back up and running normally. Thanks for your patience.

Phone/voicemail outage

Phones and voicemail are currently down. We are waiting for the vendor support folks to either get it running again, or to give us the all-clear to reboot everything. If we don't wait for that, inevitably there is some glitch which prevents them from getting the diagnostic information they need to try to unravel this mystery.

Phones and voicemail down briefly

The phones and voicemail systems were down briefly this morning. We rebooted, and our support vendor is reviewing the system logs in a continuing effort to track down the elusive cause of these outages.

We apologize for this outage. Thank you to the central contacts at each building who reported the outage via the helpdesk.

Phones down for maintenance tonight (9/29) 6:45 - 7:30 PM

Phones are going down tonight (9/29) at 6:45 PM for emergency maintenance so a new part can be installed.

Fiber to Brooklyn down

Update:
The likely cause has been identified (homeowner digging without calling Digger's Hotline), but that will not be confirmed until more testing occurs. We are working with vendors to find somebody to diagnose and fix the problem as soon as possible, but it will likely not happen until sometime Monday. In the meantime, we are going to bring portable Internet access devices and phones to Brooklyn to provide the office staff with communication capabilities.

>>>
Phones and computer networks in Brooklyn are down due to a fiber optic cable disruption. We are investigating the problem.

Brief internet outage this morning- debrief

There was a brief internet outage for users on campus this morning when the network security server stopped passing traffic. It was resolved after about ten minutes. We apologize for the interruption.

Voicemail issues

During regularly scheduled voicemail system maintenance last night, the system failed to reboot properly. So, if you were trying to access your voicemail between 10 PM and 2 AM, you were not able to. Also, you should really consider some sleep remedies.

Many thanks to Tim "Night Owl" D'Antoni and our friends at ESG and Mountain Dew for getting the system running again. After the system came back up, it reverted back to Dec. 8, 2000. So, some voicemail messages left this morning incorrectly say they are over a decade old.

Sorry for the confusion.

Outage debrief 2011-03-02

All systems are now operating normally.

At approximately 8:25 AM, our SAN encountered a "Fibre Subsystem Link Failure" which caused all volumes to go offline. Some users at all schools reporting inability to log in, applications quitting unexpectedly, applications not available, Internet not accessible, and general disruption.

Why this did not occur for all users probably has to do with the servers to which each user was authenticated at the time, and the stored location of the file(s) in use.

The cause of the Fibre Subsystem Link Failure was likely a power fluctuation or power failure. However, this should not occur since it has dual redundant power supplies connected to separate battery backup power supply units.

A service company was contacted immediately, and is researching the problem to identify measures we can take to prevent it from happening again.

We apologize for the disruption this caused to our core mission of educating students. Thank you for your patience and flexibility.

Login server back online, at least for now

The server that processes logins and hosts the Navigator Page seems to be stable, at least for now. (Last time I sent out this good news, it was very short-lived, so cross your fingers.) We have a plan for researching this issue and making long-term fixes depending on the root cause*.

Please note that whenever we encounter problems that prevent people from accessing email, we post updates to the tech blog at http://osdtech.blogspot.com/ That might be an address worth bookmarking.

Many thanks to Chris Brown for his work restoring services and doing detective work on this issue. We'd still be down if it weren't for his expertise.

Thank you all for your patience.

Navigator page is up and down- here are some links you need.

Update: Aaaand, it's down again. (08:20). Apparently we haven't figured out the root cause yet. See below for links you may need. Note that some of these services rely on the same server that is having problems, so they may not work even though you have the link.

###
Update: The OSD Navigator page is working again (08:07 AM).

###

Here are the links it normally contains. (It's being updated on the fly, so check back soon if what you need isn't available yet).

PowerSchool for Teachers- http://ps.oregonsd.org/teachers/
PowerSchool for office staff- http://ps.oregonsd.org/admin/

Moodle- http://moodle.oregonsd.net

Skyward self-service- https://skywardea-win.iscorp.com/cgi-bin10/cgiip.exe/WService=wsermaoregonwi/seplog01.w

Staff mail- http://mail.oregonsd.net
Student mail- http://mail.students.oregonsd.net

Aesop online substitute system http://www.aesopeducation.com/

Google Apps logins restored

Update:
Regular (LDAP) login to Google Apps has been restored as of 3:30 PM. It was the result of a combination of a permissions error and a change made by the IT department. Sorry about that.

###

Many users are unable to log in to Google Apps. This seems to be caused by an issue with the link between Google and our LDAP server. We are working on fixing the problem.

You can still access email using an IMAP connection, such as with a smart phone.

Slow Internet today

UPDATE:
Internet is now back to full speed. If the Internet seems slow now, let us know, because it's not the bandwidth anymore.

===
If the Internet seems slow to you today, it's because it is. Our main Internet connection went down yesterday (Tuesday, 1/4, around 1:15 PM). Fortunately, we installed a backup connection over Thanksgiving and configured it for automatic failover in case something like this should happen.

The primary and secondary connections work together to provide a combined total of 75 Mbps of bandwidth. When one goes down, the total bandwidth is reduced. The result is that right now we are running on 45 Mbps of bandwidth instead of 75, so your Internet access might seem slow.

(What's crazy is that five years ago, we had only 3 Mbps of bandwidth.)

We've been working with Charter to get the main connection working again. Thanks for your patience.

Internet access restored, and scheduled outage tomorrow is cancelled

Internet access has been restored, many thanks to Jim and Chen from Wiscnet! All services are now operating normally. In the course of troubleshooting, we went ahead and migrated the circuits to a load-balancing configuration ahead of schedule, so tomorrow morning's scheduled Internet outage has been canceled.

Internet service problems

The district's Internet connection has had intermittent unreliability today. It is still unresolved, though we are working with WiscNet technicians to solve the problem. The issues are making it difficult for staff to check email, and Internet access from inside the district is largely unavailable. We will continue working tonight to get it resolved, and tomorrow's schedule outage from 8-9 AM is still on schedule, though we may need to use it to continue to resolve this issue.

Thanks for your patience. Hopefully you are cooking Turkey, not checking email.

Internet access is down...

UPDATE: Internet access has been restored.
 >>>
Network:           WiscNet
Type:              Unplanned Outage
Location:          Wisconsin
Ticket Number:     178461
Party:
Start Time:        11/5/2010 11:54 CDT
End Time:          Unknown
Description:       Widespread Charter Outage

Engineers:


Reporters:

Affected:
  Charter Customers

Notices:

Maria Accuardi @ 2010-11-05 12:22:04
  Multiple Charter sites around the state lost connectivity at 1154.
      Network Engineers are currently looking into the issue.

>>>

Internet access is down due to a wide spread charter issue in the Madison area. 

Unplanned Internet outage now resolved

A brief Internet outage has been resolved. Here is the notification from WiscNet.

Network:           WiscNet
Type:              Unplanned Outage
Location:          WiscNet
Ticket Number:     107176
Party:
Start Time:        4/28/2010 9:40 CDT
End Time:          Unknown
Description:       Widespread Outage

Engineers:


Reporters:

Affected:
  WiscNet Members

Notices:

Maria Accuardi @ 2010-04-28 09:59:31
  There is a widespread outage for WiscNet members at this time.
     Network Engineers are currently working to resolve the issue.

Maria Accuardi @ 4/28/2010 9:59 CDT

Maria Accuardi @ 2010-04-28 10:35:56
  The routing process on the ISP router in Madison was restarted to
     restore connectivity.  Network Engineers have opened a case with
     the vendor to investigate the cause of the outage.

Elementary server problems

The server for all three elementary schools has been behaving badly. It experienced numerous freezes and reboots which resulted in users not seeing files, taking 10-20 minutes to log in, and causing programs to crash.

A Novell consultant has been working on the problem with us since this morning. We haven't found a definite problem, but there are some abnormalities with the server. We are currently copying the server to different hard drives in hopes of resolving the problem. Unfortunately, it's going very slowly (which in itself is a bad sign, and means it could very well be a problem with the drives). It took one hour for 25 GB of data to transfer, and the whole data set is over 400 GB in size, which means the transfer will take 16 hours.

We will try to get the server working again by the morning. We know you have lots of important things stored on there, and we are trying our best to give you access again without losing anything. Thanks for your patience.

Further updates as events warrant.