Here's the truth for all the gossip girls.
Folks,
Most if not all of us are getting calls, texts, and e-mails from folks wondering what the heck is going on with yesterday?s draw results release. The false rumor about a security breach also continues to circulate. In an effort to get us all on the same page:
? Bottom line: The passion of our hunters for their sport overwhelmed the system . . . not just ours but, according to ASET (The Arizona Strategic Enterprise Technology office, i.e., Arizona?s IT agency) the demand on the system temporarily affected every ?az.gov? website in the state.
o Unlike our efforts several years ago, the system did not ?crash?, it did not break.
o The results went on line ~1:30pm yesterday and, initially, the system worked as planned. But once the word got out, the load became huge and folks attempting to get into the site ?timed out? before the system could give them their turn. The phone system continued to work, but that was not going to be a substitute for getting your results on-line. Until late in the afternoon, the contractor believed (incorrectly as it turned out) that there was a ?denial of service? attack on the system.
o Neither the contractor nor ASET (the state IT agency) expected the load . . . no other website in the state draws this kind of load. The contrator?s comment was that they had ?never seen anything like it?. While we do not intend to throw the contractor under the bus, we also won't let them off the hook . . . throughout the course of our preparation for a return to the on-line draw, they were told how important it was that the system be able to handle the customer load on deadline day and draw results release day. We were repeatedly assured they had adequately load tested the system for the high demand we expect on the first couple of days of the release of the results. Obviously, their testing was inadequate.
o Our preparations for this draw included:
? More than a year of pre-launch testing and coordination with the contractor.
? Using the Spring draw to shake out remaining ?bugs? . . . we used the same contractor and system for that draw and things went well. As the smallest of our three draws, this was a good test bed, but we knew . . . and made clear to the contractor . . . that this draw would be a different breed of cat when it came to demand on the system. If they didn't believe us before, THEY DO NOW.
o The contractor should have done better, but I do believe they have learned a necessary lesson from this and it will pay off for our customers during future draws. I hate that our customers (including our folks who put in for the draw) are going thru this ? IT IS UNACCEPTABLE -- but the reliability of this system will be better in the future . . . 30 years in the Infantry taught me that the lessons learned best and longest were the ones that were the most painful.
? The Current situation:
o The draw results page currently is up.
o ASET (Arizona?s IT agency)and the contractor continue to use workarounds (they worked literally all night) to get us through the current situation so our hunters can find out if they were drawn.
o A detailed post-mortem is already planned to shake this out for the next draw and beyond.
o Bonus points are being updated throughout the day and they should be fully updated by close of business today.
? The rumor about a compromise of our customer information data base:
o Bottom line: It didn't happen.
o In March, VISA and Mastercard announced a breach of their databases.
o Shortly thereafter we had a few (I think the total so far is 4, but we should expect more once everyone has checked the draw results) folks who used credit cards for the draw that were denied by their banks as a result of the VISA/Mastercard breach.
contact us to the effect that the credit card they used for the draw had been compromised and they had to cancel their cards.
o For these folks, this occurred after the cut-off date for changing their method of payment (we will advertise that date more widely in the future).
o We are researching options for the Commission and Department for dealing the with loss of bonus points that could result from a denied credit card and are going to work on the rules necessary to enable us to deal with identity-theft-driven bonus point issues in the future.
o Having received one of those customer calls, I told staff to be watching the system to see if this was a significant issue or just a problem for an unfortunate few.
o I suspect that our efforts to be prepared for the impact of the VISA/Mastercard security breach became translated into a breach of OUR system. Again, that did not happen.
That's it for now,
--Gary
Gary R. Hovatter
Deputy Director
AZGFD
There you go. There's the deptpartments answer and the dust hasn't evem settled.
"The deadliest weapon in the world is a Marine and his rifle." General John J. "Black Jack" Pershing, US Army
"Most men go through life wondering if they made a difference, Marines don't have that problem." President Ronald Regan
