One can leave it at that, noting that the program works "and that's what counts." This is an attitude that is fostered by laziness and pressure to deliver value alike. If given in to, this will lead to extra work and less value later, when changes in the system or the environment cause the code to break, and the poor maintenance developer has to slash his way through illogical and brittle code.

So, the smart next step after initial development is to clean up the mess: reorganize and partly rewrite the code so that the functional working stays the same, but it is also pleasant to behold and easy to maintain. This process is called "refactoring," and is recommended if not outright mandated by many (agile) methodologies.

As so often, the business of software has not invented the concept, although it may have given it this particular name. For example, work processes are reviewed and changed to remove inefficiencies or redundant tasks, a couple may receive marital counselling to rebuild part of their relationship, and once in a while I rearrange the content of my shed so I can again move within it without being impaled on misplaced garden tools. All this intends to keep the functional aspect intact, while improving the chance of surviving on the longer term.

We develop our software in teams. Sometimes - regularly - we change team composition. Team members should be located close to each other to facilitate communication. (Being engineers, we need all stimulation we can get to make sure we actually communicate.) Add to this that we have nonstandard workplaces - desks in different sizes and heights, pictures of different spouses or pets on those desks - and you can imagine that swapping team members means moving around a lot off stuff.

Today, this refactoring of the workplace was on a larger scale than usual. Also other departments had decided to move people from the places where they had ended up to places where they ought to be. The difference with moving code around is that you tend to look for people not where they "logically belong," but where you were used to find them previously. The next couple of weeks I'll probably be entering a lot of wrong offices.

So, a lot of desks were carried around today. And because I program with head and hands only, the fact that I dropped a desk on my foot will not functionally change the working of the R&D department.

The Microsoft SQL Server Database Engine Tuning Advisor (DTA) is a very useful tool to help us in our quest for performance. In our development infrastructure, we've implemented automated trace file generation for typical workloads, and we've also managed to run the tuning advisor automatically every night (using the dta.exe command line tool). However, the tool has a lot of quirks, especially when used in conjunction with Microsoft's Entity Framework.

Quirk 1

The sp_executesql construct is not supported by the DTA. And Entity Framework generates pretty much all queries wrapped in sp_executesql calls!

Quirk 2

Statements of the following form are not supported by the tuning advisor:

Quirk 3

Ever since we upgraded from SQL Server 2005 to SQL Server 2008, the tuning advisor kept giving us errors like "59% of consumed workload has syntax errors. Check tuning log for more information.". The tuning log then told us that perfectly fine queries contained an "Invalid object name dbo.Something". And running those queries through the tuning advisor invididually actually went just fine!

We eventually found a workaround, which was to update  the queries to reference [Database name].[dbo].[Something] instead of just [dbo].[Something]. This was in a situation that worked in the 2005 version, we passed only a single database for optimization, and we also passed a database to connect to -- and still it couldn't find the objects.

So, the workaround?

Well, we're programmers, so we wrote a tool. The on-disk trace file format is not documented, but the trace table format is, so we loaded the trace file into a trace table. We then wrote a tool to go through that table, and to replace all sp_executesql calls with expanded versions of the same statements. The tool does something similar for the DECLARE...SELECT... statements (or at least for the limited forms of such statements spewed out by Entity Framework). And finally, it replaces "[dbo]." by "[Database name].[dbo].". And voilà: the number of syntax errors went down into the single digits. There still are some minor queries that the tuning advisor doesn't understand, but at least the majority of our queries are analyzed and included in the tuning advice.

 So, how do we handle all this stuff? Well, some of it is distinctly low-tech: shipped releases are stored on a network drive somewhere, and holiday planning is handled using a simple spreadsheet on the network. There’s also some physical blu-tack involved, or at least some sort of real-world adhesives: one of our sprint teams keeps track of sprint items and progress using post-it notes. On the other hand, another team uses a home-grown administration tool for the same purpose. As time progresses, and Aia grows, our tool set expands, and we are forced to adopt more technologically advanced solutions. Let me give you some examples. These are just some of the things that are happening at the Aia Software R&D department to further optimize and professionalize how we keep track of things. It’s an ongoing effort, and things are continuously improving!

 Let’s start with test results. At first, our automated test results were verified by manual log checking, with totals entered into a spreadsheet. The spreadsheet retained the history of test result totals, but no individual test information. As the test sets expanded, and as the number of test platforms increased, this became increasingly difficult to keep up with. Then one of my colleagues came up with a log parser tool which automatically calculated the totals, written in perl. That was step one, and it saved a lot of work. Still, gathering the totals was still a manual process. Then at some point I wrote a gathering tool in C# that wrote the results into a database, on a test-by-test basis. No more manual intervention was required. A spreadsheet with pivot tables could be used to aggregate error statistics by platform, product, etcetera. Furthermore, it allowed easy access to the failure history of a test – there is a lot of value in knowing the time when a failure first occurred.

 As for documentation, we used to keep track of that in Microsoft Word. Nothing fancy – just the docs please. The documentation was stored in our regular revision control system that was also used for the program source code. Again, at some point this became hard to use: there could easily be multiple people who would want to work on the same manual, and that would be impossible. Therefore, this solution has now been replaced with a proprietary document authoring system, that we can also use to author help files. Multiple users can log into the authoring system at once, and edit different topics in the same manual without interfering with each other. As an added benefit, the new tool allowed us to publish our documentation in HTML format for our Aia ITP Knowledge Center.

 As a final example, consider revision control. This is such an essential part of the software development tool chain that it is easy to get into holy wars. We currently use PVCS, which was chosen a long time ago and which at the time was considered a more stable choice than, say, SourceSafe. But we are now frequently stretching the limits of what PVCS can do. For instance, we simulate “streams” or branches using PVCS labels. That’s nice, but if you rename files frequently your folders will be full of files that are actually no longer in use. Other wishes include better support for assisted porting (merging) of changes between branches (e.g. for back- or forward-porting of hotfixes), better performance, atomic commit, and integration with bug tracking systems. For that reason we are probably going to move to a new revision control system at some point. When? That’s hard to predict. Probably after the holy wars have been settled – and those haven’t even started yet. But I for one can’t wait!

Unfortunately, as we found out later, the Entity Framework also comes at a cost. In particular, performance is an issue. You won’t notice this if you’re only querying or updating a couple of records in an interactive application, but if you’re doing bulk processing, you are definitely going to notice. And what does ITP/CCM do? Right: it does lots of bulk processing, so we noticed. Big time.

Let’s start with some background on ITP/CCM. We aim for the system to scale to extremely large numbers of processed envelopes. Because of this, we use a loosely coupled design centered around a database. Processing agents, which run on multiple servers, bulk fetch a set of objects that are in a certain stage, and process them to produce a set of objects a subsequent stage, which are then written back to the database. Clearly, performance of bulk database access is of the utmost importance in this design. So, where did the Entity Framework fail us and where did it hold up?

Bulk inserting / updating: Frankly, the Entity Framework performance for inserts and updates is dreadful. Why? Well, apparently it takes a huge amount of CPU to figure out what exactly has changed, and what SQL statements have to be executed, in which particular order. And then those statements are executed one… by… very slow… one. We updated one particular processing agent to use plain SQL INSERT statements instead of Entity Framework. The effect? An instant speedup of our benchmark run for this agent from about 90 seconds to about 7 seconds, including all of the reading and processing. We then applied an extra “special sauce” abstraction layer which submitted the SQL queries to SQL Server in batches of several dozens of statements, to reduce the time taken by round trips. This brought the time back to less than 5 seconds.

Data Reading: For data reading, we’re still actually using the Entity Framework quite a lot. In the preceding example, the bottleneck was in write performance, not read performance. We did have some read performance issues though, a couple of which I’d like to share.

 ISSUE 1: Overuse of the Include() function. It turns out that if you use Include() to preload relationships, this works fine for one or two relationships, but once you get past that, you may suddenly get nonlinear decreases in performance. For instance, we had a query which preloaded about 6 relations, and that took about a minute. Removing one relationship – ANY relationship – from the Include() list reduced the run time to about 10 ms.

 So what happens here? Well, it turns out that every Include() adds significant complexity to the query generated by Entity Framework. Especially because of the Relationship Span feature (see below), each Include() may add several tables. And when the number of tables increases, the database query planner’s task becomes intractable (because query planning is VERY hard), and it will make weird decisions. We had one case where SQL Server would produce a plan to first join a 3-million-row table with a 25-million-row table, then join the result of that several times, and then join the end result to a table which had three rows in it. Doing the exact reverse would probably have been wiser, but of course there is no way to tell that to SQL Server. So instead, we removed the Include() which added the 25-million-row table to the query, and loaded that part separately. Poof! Performance problem gone.

 So, what’s the general advice here? Well: do not Include() multiple relationships that are 1:N, because each of these will multiply the result set size by N. Do not Include() more than a couple of relationships, and Load() the remaining relationships.

 ISSUE 2: Relationship Span versus NoTracking. Does the LINQ query (from o in database.MyEntity select o) translate to the SQL query SELECT * FROM MyEntity? The answer is NO. Entity Framework has a feature called Relationship Span, which basically means that it will already pre-fetch the entity keys for all related objects that have an x:1 relationship with the queried object. So, if MyEntity has a relationship with MyOtherEntity modeled by a foreign key in MyOtherEntity, this LINQ query will translate to something like SELECT MyEntity.*, MyOtherEntity.MyID FROM MyEntity LEFT OUTER JOIN MyOtherEntity ON MyOtherEntity.FK = MyEntity.MyID. Nice, no? Well, no indeed. For performance this can have very nasty effects. And this feature interacts with the Include() feature described earlier: it will do this for the objects in the included relationships as well! So, if you use Include(“A.B”), then you will actually get Include(“A and all its x:1 relationships”), Include(“A.B and all its x:1 relationships”). That results in a lot more tables in the generated SQL query, and a greatly increased chance that the query optimizer will no longer know what to do.

 So is there a solution? Well, you can turn on the NoTracking option before querying, and this will turn off relationship span. Unfortunately, it will also turn off some other features which you might want, including “tracking”, which means that the objects are not attached to the object context. This means you can’t modify and save the objects that you retrieved, you can’t use Load() to load additional relationships that you didn’t Include(), and duplicate objects in the result set are not deduplicated. For us, this was not an issue. First of all, we no longer use Entity Framework to perform updates, so that is not a problem. Furthermore, it turned out that when Relationship Span was turned off, we could once again Include() everything without much performance impact, because the number of tables in the generated queries was much lower, and the SQL Server query optimizer once again understood how things were to be done.

 ISSUE 3: Database Tuning. Entity Framework generated SQL queries are hard to tune, because the SQL Server 2005 Database Engine Tuning Advisor does not recognize the queries generated by Entity Framework. This is because the Entity Framework queries use sp_executesql to promote query plan reuse. So, how did we work around this issue? Well, we now use the SQL Server tracing facilities to trace to a table, then run a home-made tool which transforms the queries into something that the tuning advisor understands, and then run the tuning advisor on that. We have automated the entire tuning process so that it runs every night. This works very well and gives us decent query performance, but it must be said that this is not a solution that can be used by mere mortals – we hope that Microsoft will eventually improve the tuning advisor so that it will understand the Entity Framework queries. Perhaps SQL Server 2008 works, but we haven’t had time to try that yet, and we weren’t willing to upgrade our performance test machines because we try to test on the lowest common denominator.

English is the language of Aia Software to the outside world, at least as far as the international market is concerned. Of course. If you want to sell your solutions and services around the globe, you'll talk in the language most potential customers understand.
 
The education system of The Netherlands is eager to give all Dutch a firm grasp of the English language, starting in the childhood. With the usual lack of effectiveness of this system, however, learning English starts after the age that children will learn a language naturally, from being exposed to it, but before they have a proper command of their native language. The effect is, predictably, that in the end they will not be fluent in English, nor be able to use their mother tongue flawlessly. College and university will worsen this situation by insisting to use English for half or all of the lessons. As the teachers are not native speakers either, in the end we're all learning each other some pidgin dialect of the English language. Anglophone readers may have been amused or annoyed in this respect by the documentation shipped with ITP or by this very blog entry itself. (Hey, we're doing our best!)

Talking about English-eagerness: today I saw a Dutch newspaper translate the German word “Müll” for their Dutch readers as “trash”. The translation is correct, but it would have been so much logical to translate it to the Dutch word “vuilnis”…

English is also the language of the R&D department of Aia, since a couple of months. When the development team was reinforced with a new member who was non-Dutch speaking, it was decided that the entire department would switch to English as the day-to-day language. The team was not only expected to talk English to the new member, but also when talking to other team members. The idea is that one can learn a lot from overheard conversations. As a side note I might add that many of us wear headphones playing music to be able not to involuntarily overhear every conversation. We have a lot of them, and team members share a single room, and sometimes it is just necessary to be able to concentrate on the task at hand.

The switch to English was triggered by the new team member's arrival, but not really caused by it, as the possibility to use English internally was already thought about before. Anyhow, the entire organization was informed about the rule of the new language, and funny language signs were posted on the doors to the R&D rooms. (# BEGIN language_code("ENG"), which is ITP-speak for Lasciate ogni speranza, voi ch'entrate).

And we started speaking English. From 8 am. Up to 6 pm. Which appeared much harder than we thought, and especially much more tiring. A new member needs to be brought up to speed with a lot of technical knowledge. Sessions lasting an hour and a half, explaining nontrivial concepts and details in the architecture of ITP is tough already when done in Dutch. Speaking English is much different from writing it. You don't have the time to think for words or look them up, nor do you have time to ponder the correctness of your grammatical constructions. This is only partially compensated for by not having to look up the spelling of the words. Not only do you have to suffer from the awful accent of your colleagues, you also hear your own awful accent the entire day, which is much worse. On the other hand, you do actually learn from each other, because even through the accents you get to hear a lot of nice idiomatic expressions.

In time, our resolve to speak English continually began to erode. In a room where the new team member was not present, people would look cast around a quick suspicious glance before starting some small talk about the children, politics, or sports in Dutch. In some more time, small talk could even be Dutch with the new team member within earshot. Currently, sometimes even R&D topics get a Dutch treatment in presence of the new team member, if we are sure it would not interest her. We enter into temptation: the spirit indeed is willing, but the tongue is weak.

Still I think speaking English in the department is a good idea apart from being able to communicate with all team members. I sincerely hope that it in time will help us raise the standard of language usage in our documentation. And of course, all R&D meetings still are held in English.

Our new team member thinks we hate her guts because we're forced to speak English because of her. Her study of the Dutch language is progressing rapidly. Perhaps these two statements are related.

Here are some examples of business rules that can now be controlled by plug-ins in ITP/CCM:

Communication rules. These rules define what envelopes are sent, to which recipients, and through which channels, containing which documents.

In our view of how ITP/CCM will be used, these rules are based on communication profiles that are taken from CRM. Therefore, we will be providing a default implementation which can integrate with Microsoft’s CRM solution. However, the plug-in architecture allows for any source of such communication profiles. But even if an organization uses our default implementation, their business rules may vary wildly, depending on e.g. agreements with agents.

Combined output streams. These rules determine which documents are combined into output streams for which output device.

Each and every organization has their own very distinct business rules regarding the composition of output streams. For instance, an organization’s envelope inserting machine might require separate streams of 1-page envelopes, 2-page envelopes, and so on. The local postal service may require documents to be sorted by postal code. The organization might want separate streams for full-color and black-and-white output, or for duplex and simplex output.

Envelope bundling. These rules determine whether multiple envelopes for the same recipient can be bundled into a single envelope.

Bundling of output into a single envelope provides many opportunities for savings. However, this is yet another area where business rules are unique for each organization. For instance, large enterprises may not want to bundle correspondence from different business units, even though they share a single document output infrastructure. Some organizations may simply bundle the envelopes and include all cover letters for the separate envelopes, while other organizations may want to replace individual cover letters with a single cover letter that describes the entire bundled envelope contents. Also, the maximum size of bundled envelopes may be different for each recipient: a large agent may gladly receive boxes containing 5000 pages each, while Joe Customer would like envelopes to fit in his mail box. All of these business rules can be implemented using the ITP/CCM plug-in architecture.

Of course, a plug-in architecture introduces some difficulties as well. For instance, in the design of ITP/CCM we have always tried to honor the mantra “configure, don’t program”. The use of plug-ins seems to violate this mantra. However, we think we have found a proper balance, and that we have minimized the amount of programming required for the implementation complex business rules and e.g. integration with CRM systems.  Because:

It’s simple: XML in, XML out. Plug-ins only need to implement the business rules on the input XML, and write out the results to another XML stream. That’s all! All the behind-the-scenes bookkeeping, which includes retrieving all relevant information from the ITP/CCM database, storing the results, and scheduling further processing, is handled by ITP/CCM.

Plug-ins can have configuration: We promote the use of configuration in plug-ins, by allowing PluginConfig XML to be attached to all ITP/CCM configuration objects. These PluginConfig XMLs are free XML fragments which are passed to all plug-ins that require them. For instance, the PluginConfig for a correspondence pack may provide additional information to a plug-in regarding which of the documents in the correspondence pack should be distributed to agents, and which documents should be left out (e.g. Terms and Conditions, cover letters). In our view, a well-written plug-in should implement only the mechanisms for the business rules, but should leave plenty of room for configuration, so that the actual decisions of when-to-apply-which-rule are left to business analysts.

All in all, the ITP/CCM plug-in architecture provides implementers with an unsurpassed level of flexibility without requiring them to do everything themselves, and it provides our customers with reduced time-to-market for business rule changes (since no core ITP/CCM product changes are required for new or updated business rules). In addition, it frees up the ITP/CCM development team to work on the core of ITP/CCM, to the benefit of all ITP/CCM users, free from the distractions of working on specific business rules that benefit only a limited number of customers.  

Often people ask me why we haven’t added any GUIs yet, after all, modern tools provide features for rapid GUI development. Well, the answer is rather simple: we do not want just any other GUIs, we want a good GUI. Let me explain what I mean by that.

With ITP/CCM we try to keep the technical concepts as close as possible to the concepts known by the user. We are continuously spending time on designing, mapping, refining and remapping these technical concepts to high level user identifiable concepts. Having concepts in the product that are aligned with real world concept as close as possible should help us create a good GUI. This is necessary as one of our goals is to make ITP/CCM as business managed as possible.

But even when the high level concepts are right, it isn’t that easy to create good GUIs. For instance one of the common mistakes is to have a programmer design the GUI with his or her assumptions on the users perspective. A good example of this can be seen in our own ITP/MDK Repository. At some point we’ve added a menu option to run a model and named that option “Run Model”. This was done under the assumption that users want to run model. Looking at the problem from a programmer model this makes sense. But in the user’s model “running a model” is almost meaningless. No model developer wants to run a model for the sake of running a model. The only reason for them to run the model is to test it. Therefore the menu option should have better been named “Test” instead.

And that is only one example of what can go wrong with quick GUI design. To be able to design good GUIs we need to know our users. Not only what their regular daily function is but also what their mindset is when working with the product, how often they use the product, etc. We also have to be aware that there are different types of users with different roles involved with ITP/CCM. They all require different GUIs with different requirements. So we can only create a good GUI if we know what user will use that specific GUI.

One decision has already been made about the GUI: it is going to be web based. Parts of the GUI might be integrated with or provided by a CRM system. We are currently working on the design of the GUI and coming up with the proper GUI concepts and metaphors. Once we have that developing the GUI will probably go in a rapid pace, assisted by the modern tools and technologies that are at our disposal. From then on not only the functionality, which we are extending constantly, but also the number of bytes in ITP/CCM will grow rapidly.

Besides setting these properties in the document, macros and scripts need to be developed to set or change these settings ones more when many different hardware components are used within the organization. Some users feel right at home being in full control of the final output. Most however, have limited knowledge of the company infrastructure and advanced techniques in their word processor.

In all its products, Aia Software strives to separate concerns. Key is this is the adagio “Business owned, IT controlled”. This is also true for ITP/CCM. The product will allow all roles involved to concentrate on their field of expertise. The business user can focus on the content. The business manager can configure correspondence packs, defining a set of documents. The model developer creates document templates and the mail room or IT department configures the infrastructure.

Besides the adagio there is also a functional reason to separate these concerns. In ITP/CCM many decision will be made at run-time. Therefore it will not be possible to control them upfront from within the document. Some possible run-time event or decisions: Page numbering may depend on the final correspondence pack. Page numbering might continue over all or part of the set. Third party print streams can be included. Fonts and location might need alignment. OMR codes will only be needed for print, not email. The requirement may even change ones more when document are bundled before distribution.

ITP/CCM will introduce two main objects that allow the configuration of all these properties, The correspondence type and devices. The first defines the document sets and its properties, this includes page numbering properties and default devices. Devices define technical aspects like addresses, duplex settings and spool file size. But also the post processing steps to be done, including OMR settings can be defined on a device level. Only when the content differs per output channel this needs to concern the user defining the document. ITP/CCM will facilitate the optimal use of resources, whether staff, knowledge or devices.

We will call her ITP/CCM for short.

• Date: 30th september 2009
• Time: 16:50 pm
• Weight: 7.730.245 bytes

The parents will be resting from october 1st to october 12th. ;-)