We're now in full fledged ship mode trying to get Beta 2 wrapped up. The part of the process we're in now (across the Division) is called "ask mode", which I've mentioned in prior posts. Basically as you move through ask mode you slowly rachet up the bar on which bug fixes you'll take. Early on there is a set of things that you'll take based on a pre-defined bar. Things like accessibility fixes, scenario blockers, performance regressions, etc. As time goes on you continue to consider those things but only if they are super critical (e.g. can result in user's work getting lost or crashing the whole product in a mainline scenario, or are a security issue) and/or have relatively low risk. Things that don't make this bar will be addressed after Beta 2. There is a bunch of process around bringing a bug in Ask Mode, which is by-design as you want to make sure teams are only bringing the ones they really care about. We've been bringing between 1 and 4 bugs a day lately. In general, we're pretty happy with Beta 2.
But when we are in this part of the product cycle and thinking about what we call "exit-criteria" I'm reminded of all the little things we need to make happen to ship software, and I kind of remince back into the old days when the list seemed to be shorter. I think back to debugging in CodeView on Windows 3.1, and pretty much what it came down to was making the functionality work and shipping it. Here's kind an off-the-top-of-my-head list of things our team has to track in order to ship this product:
- Designer has to work with four different languages: C#, VB.NET, Managed C++, and Visual J#. We try to be as language-agnostic as possible but each of these tends to have it's own features or idiosyncracies that the designer needs to handle properly.
- Each of those languages has an "Express" version, and is part of several other SKUs (I've lost track of exactly what we're delivering for this VS 2005, pro, std, enterprise, etc)
- Platforms -
- Runtime parts have to work on Win98, WinME, Windows 2000, Windows Server 2003, Windows XP, IA64, and AMD64, and Longhorn. It's got to look right with and with out theming turned on, etc.
- Designer stuff has to work on Windows 2000 and newer
- Virtual PC and Terminal Server have almost become platforms in their own right. We've seen a few issues and it's got to work there too.
- Globalization/Localization: a set of languages with their own set of issues: complex scripts (GB18030) in Chinese or Hindi, the IMEs in Japanese, Chinese, and Korean, for example.
- Accessibility - we need to be Section 508 compliant with our own Active Accessibilty support, plus support a set of screen readers such as Windows Eyes and JAWS. Oh yeah, we have to make sure we look right in high contrast and large fonts too.
- Right To Left - we need to properly support RTL settings for languages like Hebrew and Arabic. The number of complex issues here are astounding, ranging from varying levels of support in underlying Windows controls (we've been trying to rationalize this) to idiosyncracies of how GDI and GDI+ behave, to general ideas about how this should work across containership, etc.
- FxCop - this is an automated tool that runs across the compiled code looking for all manner of issues, from possible security issues to design guideline violations to dead code to unused parameters. It generates a ton of output. We have a lot of code that existed long before FxCop did (see, FxCop is a Windows Forms app...) which is one issue. We've gotten as far as we can on most of that. But in general FxCop is relatively sensitive to code churn and a change in one place can cause new violations to pop up elsewhere so we're kind of always chasing this goal around.
- Compatibility - We're hard-core about making sure Windows Forms applications from V1.x will run on V2.0, but it is amazing what kinds of changes will result in a compat break. Even when we guess that a fix might be a compat issue, it's hard to know for sure. We try to stay conservative here but it's not unusual for us to discover that some change we made breaks a corner case that an application is expecting.
- Partial Trust - there's a set of scenarios that need to work on apps running in the Intranet and Internet zones. The main thing here is security - for these scenarios, we have to be extra vigilant that they are secure.
- Stress - we've got a bunch of tests that open and close forms or controls thousands of times over and over again, looking for leaks, race conditions, etc. These bugs are difficult to diagnose because usually when they actually fail it's long after the real problem has happened. These often end up being sent over to the CLR team, but they're ours as well.
- 64 Bit - I mentioned this above but it's got it's own set of issues. Mostly there are structures that we've defined in managed code that don't map up exactly right when the word-size goes to 8 bytes. Specifically, 64 bit structures are packed on 8-byte offsets and if we're using structures that are defined in native as a nested structure or an array of structures, sometimes that bites us.
- Performance - we've got a developer that works full time on performance. We have a set of performance scenarios that we monitor (e.g. complex form startup time, open form in designer, etc.) and set fairly aggressive goals for. Fortunately we're in pretty good shape there but my perf dev spends almost 100% of her time just keeping these things in goal. When the stack is as complex and has as many moving parts as ours does, little changes cause little regressions that add up over time. We had the dev team spend a week focused only on performance earlier this month too.
- Side By Side - we need to make sure that you can run V1, V1.1, and V2.0 of the Framework and Visual Studio all on the same machine happily together. And that you can install, uninstall, and reinstall them in any order and it still works. Same goes for those Visual __ Express SKUs as well.
- UI Consistancy - we try to make sure all the dialogs, themes, layouts, and look and feel are consistant across Visual Studio, which is comprised of organizations called Product Units that are fairly autonomous.
- Security - we just spent 5 weeks across the team focused only on security, and that's after we've already had security modeling and reviews for all the new features and we'd done security pushes on the existing code twice. And that was just to find potential issues and some of the investigation spawned other work items and investigation that we still have on our plates.
- Integration issues - our product is so large that we can't have all the developers on it checking into the same spot. So we have several different "labs" that a subset of teams check into and then the changes are "integrated" back and forth across them through a common parent. This is a huge subject on it's own so I won't go farther than to say that this adds all sorts of complexity (but is the best we've come up with so far!) to the process compared to a smaller project. For example, on a really bad-ass machine, building the whole product (Frameworks + Visual Studio) takes about 8 hours and gigs and gigs of hard drive space. So the problem for us is not only having the right bits but getting them in the right place at the right time.
- I'm sure I'm missing a few. I can never keep all of these in my head at the same time.
Plus we need to do things like work with PSS on customer issues and occasionally issue fixes for V1 or V1.1, the former of which tends to take almost one full-time person most of the time. We try to get folks to go out an answer questions in the news groups, make blog entries, stuff like that. It's also nice to have some people off thinking about "what's next", like how we integrate with Avalon and Longhorn, etc.
So we keep pretty busy. From now until we ship, it's really kind of a wresting match with all these various factors to get them lined all lined up, all at once.