Slow logins at OHS- status, explanation

The helpdesk has fielded a number of complaints about slow logins from OHS. After researching the issue, our current theory is that the FreeNAS array currently holding the PANTHER server is slower than the regular SAN or drive arrays in the servers. PANTHER is being temporarily hosted on FreeNAS until it can be moved to the SAN. However, we can't fully migrate it until other arrays have been moved, which we were doing last weekend, but it took too long and had be be aborted.

In the meantime, the aborted PVE migration has been erased from the FreeNAS array, which increases the free drive space from 20 GB to about 300 GB. This should provide more headroom, which will hopefully result in a (nominal) improvement in login times. The only thing users can do is delete unnecessary files to make both individual profiles load more quickly, and reduce the overall disk load. This will also allow the migration to happen more quickly.

We understand this is frustrating, and we are working to correct the problem. It wasn't planned to be a long-term situation, and we're trying to progress to the final configuration (all volumes on the SAN) as quickly as possible.

Internet back to normal- myriad problems solved

Thanks to lots of investigating, research, and phone calls by Zach and Doug, the district's Internet connection is now operating properly. This appears to have fixed the difficulties people were having with United Streaming, Plato, eSped, YouTube, and pretty much anything else that required web access.

As you can see from the pretty picture below, our inbound traffic has still not peaked.

 

What you cannot see is that all the CRC errors on our router have disappeared now that the duplex mismatch between our firewall and the WiscNet router has been corrected.

Thanks again for your patience.

Slow web access

We've received numerous complaints about slow web access, resulting in problems with Discovery Streaming, Plato, and [gasp!] YouTube. We feel your pain- we are experiencing it as well, and we are working on the problem.

We have enough bandwidth to accommodate our traffic, even with the anticipated increase due to Google Apps Gmail. In the graph below, the green parts are incoming traffic, which is 95% web traffic. As you can see, it used only about half of the 20 Mbps available. (The blue line is an upload of backup files, which stopped this morning at 8 AM, and doesn't significantly affect download speeds.)

So, why is this happening? At this point, we believe the issue is caused by a service configuration in our firewall. This summer we replaced an obsolete network firewall with a new one. Like most things technological, the new one has more capabilities, but is also more complex. Everything seemed to be working fine until Tuesday, when everybody returned to school. (It's hard to simulate the 1,600 simultaneous users for a test, so Tuesday was the first day of a real, full load. )

We are working with a Cisco engineer to figure out the problem. We'll keep you updated on this blog.

Panther really, really slow. 2009-03-12

UPDATE 2009-03-16
One of the hard drives in the SAN died on Friday, 3/14. It is likely that this failure contributed to Panther's slowness earlier in the week as it developed. Reports are that performance is much better today.

++++++++++
Panther is being agonizingly slow. No, it's not just you. It's taking many people 15-20 minutes to log in. We are actively working on this. All diagnostics look fine, which is really strange. We've called in the "big guns" (read: external consultants) to investigate on a Priority 1 status, so we hope to have it resolved soon.

Further updates will be posted here.

OHS logins slow; LDAP errors

UPDATE- Panther has been rebooted. Possible cause was a time synchronization issue. If you are still having errors, please log out and log back in, as this will reset the connection between the workstation and the server. If you had PowerGrade open when the server went down, you will likely have errors. Please call the helpdesk at x4025.

>>>
ALERT-

OHS is experiencing slow logins. We are aware of the problem and are working on it. Panther directory structure is being repaired.

People are experiencing LDAP failures when trying to log in to email, after successfully logging in to Novell. We are investigating the issue.