Slow email

Groupwise abended Monday night and performance gradually degraded through Tuesday morning. Server was rebooted Tuesday morning, and it appears to be working fine now. 

I've heard other complaints of slow logins and slow PowerSchool performance, but haven't gotten any helpdesk tickets on those, so I don't have any specifics to use for troubleshooting.

Barebones crew

The IT department/helpdesk is operating at a reduced staffing level today and tomorrow due to sickness and vacation. If you need help, please submit a ticket online unless it's really urgent.

Monthly review, February 20, 2008

The past month in IT:

• The IT department has closed 393 helpdesk tickets.
• We weathered our longest power outage of the year (seventeen minutes) and kept all systems running at the schools that retained power.
• Developed a plan to migrate all servers off VLAN 1.
• Procured two new HP DL380G5 servers to replace six aging DL380G2 servers. How will two servers replace six? Ah, the wonders of virtualization.
• Completed install of Magellan, the host machine for MAP server. (MAP, Magellan... get it?)
• Completed new online registration system for summer school.
• Began an assessment of our packetshaping device and possible alternatives to consolidate and reduce the cost of network management devices.
• Refined the parent mass-mailer system to streamline approval process.
• Developed an improved backup and disaster recovery plan.
• Evaluated several software packages for use as a shared online facility reservation system.
• Tested a subnotebook/UMPC device, the Asus EeePC, as a possible classroom research/word processing device.
• Identified computers to be replaced in summer 2008 as part of the ongoing five-year computer replacement plan
  

The past month of the Information and Technology Literacy Team:

Developed a NETS*A assessment tool for administrators.

Implemented the NETS*T assessment for OHS and RCI. The OHS data included 100% of the teaching staff, and shows us some amazing things. For example, only 3% of the staff are below one standard deviation from the mean in the area of Technology Operations and Concepts. This correlates to an average score of 1.5 on the assessment tool. The majority of the staff (84%) are within one standard deviation, and 13% are above. This correlates to greater than 3.5 on the assessment tool.
This indicates that professional development in the area of technology operation has been effective, and that we can move on to the real goal of improving instruction.

Participants in the Technology Integration Program finished their curriculum units and developed equipment proposals to enable them to implement. The proposals are being reviewed.

Completed the strategic ITL professional development plan.

How Jon kept busy:

• worked with Jeri and the 8O team at OMS to apply for a grant sponsored by HP and ISTE worth $40,000
• worked with Dave and the math department at OHS to apply for the same grant
• assisted with the A-team Tier 1 Indicator culture subcommittee
• participated in planning the WiscNet Future Technologies Conference
• consulted with EEN staff on best computer for a special needs student
• assisted with design of school culture survey for parents
• revised Board policies on web publication and Internet/email acceptable use
• shoveled tons of snow
  

Power Outage Debrief

The DSO experienced a power outage from 10:59 AM - 11:17 AM today, Wednesday, February 7. Because our IT infrastructure is centralized in the DSO, this could have been quite a problem. Fortunately, we designed our new datacenter with some elements to keep us up and running during such an outage. 

Here's a debriefing of the incident. 

At 10:59 AM, power was lost at the DSO, NKE, and PVE (and possibly others as well). OMS reported a brief flicker.

When the power went out, the UPS units kept all servers, core switches, routers, transceivers, and PBX equipment running until the emergency generator turned on and supplied power. The generator was supplying power within one minute. The datacenter load transferred to the generator power source seamlessly. All VoIP phones in the DSO stayed operational because they use power over ethernet (POE) from the 3750 switch. 

Jon opened the datacenter door with a key because the strike plates did not open when card was swiped. This was fixed a few minutes after the generator power came online. Control Solutions will add UPS backups to their systems soon to insure that card access keeps working in an outage.

Desktop computers in the DSO all turned off (except Zach's because he put his on a UPS). Those in the tech area turned back on when the generator power kicked back in. 

The WAN connection to OMS stayed up (Kiwi Syslog provides notification of dropped links), which is better than the previous two power "blips" that have caused the switches at OMS to reboot, thus interrupting the network connections for the entire school.

The only WAN connections that went down unexpectedly were the switches at PVE. They should have lasted longer. This was not a crucial issue, since they do not use VoIP phones, and all the computers were down anyway. Upon investigation, the UPS powering those switches was too small and old. Zach replaced it with the one from his desk.

When designing the tech area, we decided to include all the electrical outlets on the generator-supplied circuits so that the tech offices can be used as an emergency headquarters. This was the first extended blackout since then, so I informed other DSO staff that if they had urgent work to do, they could use a desktop computer in our area. 

The idea is that district administrators could bring their laptop computers and phones to the tech area and set up a temporary command center. Our wireless access points use POE to remain active during a blackout, and laptops run on battery power for a while, so they can stay running and connected for quite a while. 

Power came back online at 11:17. Generator turned off at 11:27. Desktop computers in the tech area stayed up during the cutover back to regular power.

In the old scenario without an emergency generator, all district phones would have gone dead around 11:05. All server connections would have been lost around 11:10. 

I'm very happy with how everything worked out, especially considering how bad this would have been without the emergency generator.