Codewell - Recently Updated Pages

Intro

phomer — Mon, 09 Mar 2009 15:41:31 CDT

We have been working with complex software for well over 5 decades now; that is a long time in developing, distributing and running computer code. Gradually, we have built up our experience and knowledge.

However, even with all of our understanding we still routinely fail to build software that lives up to our expectations. Is this because we expect too much?
Software is simply a tool to manipulate data on a computer, its behavior should and can be dependable, but we need to build it that way. We are not currently doing this.

Our troubles lie with the way we go about developing our projects. We need structure, not the type that provides legalized protection from lawsuits by showing we did everything according to industry standards, but the type of structure that fits over the process to insure that the right work is happening at the right time.
There is nothing complicated about software development, other than the amount of bad technology we build upon and the culture of software developers in the industry.

Few developers push the technological envelope, and even if they do, it is with only a small percentage of the code. Virtually all designs are composed of at least 80% routine engineering, if not more. They need to provide all of the excess functionality to provide a well-rounded program. The only surprises come from the technology not behaving as expected. Bugs and flaky code create unstable foundations. Building on these can be difficult.

We like to make programming harder than it needs to be. The culture of development seeks to accentuate the romantic nature while trying hard to stay away from any aspects of routine engineering, which actually make up the bulk of the work. Software development does involve a great deal of creativity, but it happens in solving the problems of the users, not in deciding whether a ‘for’ or ‘while’ statement is more appropriate as a loop construct. Building the systems is not glamorous, but is must be accomplished quickly and reliably for the project to be successful.

The answer to many of our development problems is to find a simple process that produces better results. One that is easy to follow. A reasonable process will prevent us from wasting our time.

Software development projects fall short of their expectation for all sorts of reasons. They vary from lack of resources, to inexperienced developers, to bad technological designs but they all share one common aspect: most of them were avoidable, and all of them could have been caught earlier. By following a reasonable process, we can identify and deal with the problems early instead of letting them fester until they have become so significant that the entire project is in jeopardy. My experience with both large and small development teams over the years keeps dragging me back to the understanding that we need to simplify the problems; they get too complex, too quickly; not only the code, but also the entire domain of building and releasing the code. The lessons behind all my years of working – which I tried to enshrine into this methodology -- is that less is more; simple is better; consistency is king. A little bit of structure goes a long way and it is all a lot easier if we are doing the right work at the right time.

We have so little time, that our wasting it – even the smallest drop -- because of bad or missing processes is extremely harmful to both our morale and our projects. If we have to sit there and agonized at each stage about all of the little decisions, even if we get each choice right, we still lose a significant amount of time.

Codewell

phomer — Wed, 04 Mar 2009 12:55:50 CST

This site is dedicated towards the exploration of all things programming.

It is open to anyone that is interested in software and how it is developed. New Members are always welcome!

Just interested in updates? You can subscribe to the RSS feed in Web Apps like Goggle reader to get the latest changes. You can find a link on the "What's New" screen at the right-hand side.

Methodologies

phomer — Tue, 03 Mar 2009 16:34:58 CST

There is no abstract available for this page revision.

Overview [SA]

phomer — Mon, 02 Mar 2009 21:12:31 CST

This is a breakdown of the architecture of the system. Simple diagrams are easier here, and often more expressive. The format of the diagram is irrelevant, so long as the diagram is simple and conveys the base information.
Includes lots of casual Charts and Diagrams

WHY: Helps everyone understand the design, keeps it on course. Helps debugging and problem tracing. Should be really

High-level diagrams, that are simple and get the point across.
Broken into components (between 5 and 20)
Broken into levels (between 2 and 6)
Broken in modules
Blueprints

System Architecture

phomer — Mon, 02 Mar 2009 21:10:15 CST

The system must have an overall architecture. This set of documents partitions the system into its base component pieces.

WHY: Architecture explains the general view, but it also provides a important partitioning of the system into both different components (horizontal) and different levels (vertical). This break down should be utilized in all aspects of the project, particularly in testing and support.

This includes:

Overview

Layers, Components and Modules
Dependent Technologies

Project Scope

phomer — Mon, 02 Mar 2009 21:09:38 CST

All projects require an overall scope document that takes a 20,000 view of the project and where it sits. We need to put the development work into perspective.

WHY: We always need to understand the big picture. It drives all of the little decisions. If we have a long-term plan, then the other decisions made during the project are better and we flip-flop a lot less.

DEFINITION: Problem domain – the domain of the problem, including all of the terminology, conventions, etc. This can be an industry, profession, trade or any other classification for which this software tool is specially addressed. There are business problem domains and technical ones as well.

There are a number of important things that need to be known to define the scope of a software development project:

Purpose of Tool
Business Justification (prove that research is true, or point out that it is unknown.
Summary Description
Long Term Goals

WHY: To be effective, we need to know where development is headed in the future. Even if it is wrong, it is better than nothing.

Documentation

phomer — Mon, 02 Mar 2009 21:08:12 CST

Project Administration Documentation. Besides getting everyone on the same page, documentation makes it easier to add in new people, and occasionally remind ones self about why they are building the things that they are…
Documentation serves two alternative purposes:

Communication with the testers
Inform the users

Communication with the testers. The test group is impendent and they are responsible for creating the full round of tests that will be applied against the software. The bridge between the two groups is the documentation created. As such, it needs to be contain enough information so that the programmers can create their code, and the testers can create their tests.
Inform the users. The documentation created by the developers is the basis for the documentation sent to the users. There are some differences in style, but these two things should be linked to save time.

STYLE: Documentation is an endless sinkhole for work. As such, it should be managed carefully. It needs to be simple, fluent and compact. The contents of the documentation is more important than the delivery or presentation. A well-though out document with lots of spelling and grammatical mistakes is better than a brain-dead one with endless useless tables and graphs.

ADVICE: Most developers are hopeless with documentation. Along with technical courses, writing courses are important and will help to grow better developers.

Things that re needed in a documented plan are:

Project Scope
System Architecture
System Specifications
Implementation Details
Schedule

SCALE: Can be added into one great big document, or broken into pieces depending on the size of the project. Less documents is generally better.

Prototypes

phomer — Mon, 02 Mar 2009 21:06:19 CST

Prototypes are small, specific tests to prove or disprove that some feature in an underlying technology does something. Write millions of them.
The code can be re-used, but only if it is refactored.

Research and Prototyping

phomer — Mon, 02 Mar 2009 21:05:46 CST

Technology is flaky; predicating a design on a flaky underpinning means having to take a few precautions to insure that the core pieces will in fact work as expected.
Technical details, standards, conventions, etc should be researched and/or prototyped before implementation.

WHY: Depending on the degree of complexity for a project, many technical details need to be sorted out before the implementation will work. Dealing with this as early into the process as possible helps to ensure that the implementation phase goes smoothly.

Coding Analysis

phomer — Mon, 02 Mar 2009 21:04:56 CST

This may also be called a design review. We all do it, sit around and shoot the shit about how the code should function. If its something that is fun and programmers do it normally, then should it not be part of the process?
Some methodologies have it occurring later in the process, but by them, it is neither interesting nor useful. Too little too late.

Technical Analysis

phomer — Mon, 02 Mar 2009 21:04:29 CST

Most technology sucks. Knowing how it sucks is important to utilizing it. No, you cannot just read parts of the manual. In order of ‘quality’:

Real experience
Prototypes
Specific in-depth documentation
Entire reference manual

For each bit of functionality that the system is ‘dependent’ on, technical analysis should show that the specific technology used for this implementation actually has that feature and it will actually work.

NOTE: In some technologies there are many different ways to skin a cat, so one only needs to be assured that one of them will work.

NOTE: Optimizations, unless they are driven by constraints do not necessarily have to work for the system to be usable. If they are not mandatory, then they can be skipped.

Business Analysis

phomer — Mon, 02 Mar 2009 19:57:22 CST

Software is a tool to manipulate data. The users always use the tool relative to some specific problem domain that contains the data. Even if it is generalized, all functionality and data related to a specific (perhaps general) set of problem domains.
Need to really understand the various problem domains, and all of their little annoying little details. This includes the terminology, standards and conventions, expectations, process, etc.

NOTE: Business analysis is the single most complex part of building software, and generally, this is where all of the problems originate. Most frustrating as well, is that you will NEVER know everything up front. They will not feed it to you initially; only time will weed out the details. Signoffs are a effective weapon where for managing domain experts.

DEFINITION: Domain Expert – sometimes also called the system champion. A non-technical person that is extremely active in the problem domain. It has to be someone who is still in the process of doing this on a daily basis. Upper management does not count and if often out of touch anyways.

DEFINITION: Business Analyst – someone who has the skills and patience to slowly, continuously mine the domain experts for information. This is converted into a format that is suitable for software development. Sometimes a separate occupation, but often it can be multi-talented (older, experienced) developers.

Market Analysis

phomer — Mon, 02 Mar 2009 19:56:25 CST

This is for commercial or service based projects only. Need to know who the potential market is, and how much it will bear.
Need to make a business case for building and selling a product.

Analysis

phomer — Mon, 02 Mar 2009 19:55:39 CST

Analysis is the act of collecting information about something, and then applying some formulization to the data to make sense of it. Analysis is never complete.

Every piece of software is a tool to be used by a user to manipulate some data. Everything else about ‘that’ is a higher-level concept ascribed by users and software developers. Know you audience.
Includes:

Market Analysis
Business Analysis
Technical Analysis
Coding Analysis

Analysis is on going based on the life of the system, so it is ever changing.

Planning

phomer — Mon, 02 Mar 2009 19:54:52 CST

“The best-laid plans of mice and men go oft awry” Robert Burns (1759 – 1796)
Planning is the most important phase in a project. Clearly defining and documenting all of the major details in a project will increase the likelihood of success. Just making a list of all of the work involved is not sufficient. Understanding the dependencies between tasks opens up the ability to reorganize the plan to meet the changing parameters of the project.

WHY: Planning allows for effective use of available resources. Without a plan, things are missed. Work is not consistent and things are not deployed correctly. The project might work, but only accidentally.

Planning Activities include:

Analysis
Research and Prototyping
Documentation
Scheduling

Partial Authorizations

phomer — Mon, 02 Mar 2009 19:53:46 CST

The project is rapidly changing. If someone signed off on 95% of a document, then they do not necessarily need to signoff on the last 5%. This is no need to re-circulate every change to everybody. A single person can signoff a small set of changes. The existing signatures should remain in the document. They are associated with the original version they signed. Large changes or major semantic changes should be re-circulated to the correct people, but only if needed.

NOTE: The more people sign, the less they read, so re-circulating signoffs for no reason waste time and reduce focus.

As such, a document will contain lists of signatures for various older versions: Bob Jones, Project Manager, January 27th 2006, Version 1.29.3 Mike Smith, Lead Developer, August 17th 2005, Version 2.27.12 Mike Smith, Lead Developer, August 1st 2005, Version 1.26.1

Signoffs

phomer — Mon, 02 Mar 2009 19:52:48 CST

People will not commit unless you force them to. A signoff gets a person to be explicit, think about what they are agreeing to and take the process seriously. It does not guarantee that the information is correct, but it does help to keep people from shooting from the hip, or constantly changing their mind.
Signatures should be at the back of the document, and they built up over versions. Multiple signatures can be one the same page, but they can span multiple pages do it that is easier. The entries should have the signature, the date, the name, the position title and the document version.
Documents should have a page of signoffs at the front. Any weirdness should be documented and then signed by someone with the appropriate authority.
Only relevant people should have to signoff a document. For any development document, that is the author, the lead developer, and the lead manager. For domain specific documents, like changes to the text, the document would require the signature of the ‘expert’.

Signoffs follow contract rules, so changes to something need to be initialized by a signatory to authorize them.

Overview

phomer — Mon, 02 Mar 2009 19:52:15 CST

This is an iterative methodology designed to control the process of software development. The focus of this methodology is on insuring that all of the work needed is completed, and that little or no extra work is accidentally done.

DEFINITION: Iterative development is a subset of the methodologies for software development that subdivide the work into an endless series of iterations. Contrast this to the heavyweight methodologies that attempt to complete all of the design and building up front in one big development effort.

WHY: Iterations reduce the risk of the project becoming stuck in design, or getting stuck on some large technical problem. There is some extra overhead associated with iterations, but if that allows other aspects of the project to be more effective it can easily pay for itself.

Dates and Versions

phomer — Mon, 02 Mar 2009 19:51:41 CST

Everything important in a project should have its own independent version number that is keep up-to-date. Everything should be dated. Everything should be assigned an ownership string of some type.
Version numbers should always be composed of three elements:

Major Change Number
Minor Change Number
Small Fix Number

All version numbers start at one and increase (there are no version zeros).
The Major Change Number denotes the current major revision version of the document. This number is incremented when changes are made that are fundamentally incompatible. For a system, this means significant issues when upgrading to the new version, which may not even be the same underlying code. Jumping from one to the other is huge. For documents, this means a rewrite or change in direction.

The Minor Change Number denotes the minor changes to the document. They have semantic meaning, but do not as a rule alter any of the base assumptions. Not keeping up with minor changes should not be a problem. For a system, a bump in the minor number mean that the new version can be installed on top of the old one with no noticeable differences. Things are fix or operate a bit differently, but essentially, it is the same program. For documents, there has been editing changes, but they are not significant to alter any assumptions.

The Small Fix Number denotes a small and trivial fix on behalf of keeping things consistent. For a system, that’s a non-destructive bug fix, for a document that’s a typo, or misstated fact.
Documents are always referenced with a specific version. There is a major and minor number and typo number.

When a document changes, large or small the version number changes as well. The person who changes the document should add their name, the date and a description of the changes to the start of the document.

WHY: Things are constantly changing, so it is important to know how ‘fresh’ the information is. Accurate version numbers are important. Distinguishing the depth of changes between two versions should be obvious without having to examine the documents.

Iterations and Phases

phomer — Mon, 02 Mar 2009 19:50:50 CST

The project life span is broken down into a series of overlapping iterations each of which consists of five discreet phases. Development iterations are composed of:

Planning
Implementation
Testing
Release
Support

WHY: Breaking it down into different phases make it easier for tracking, and to determine the current project status.

OBSERVATION: Well-organized and effective development gets into a sustainable “pace”. If the project is lurching from huge highs to dramatic lows, then the process is broken and is probably hugely inefficient. Software development is “building”, which does not need and should not contain “drama”. For that you might choose a different career, perhaps politician or something. Drama is a bad sign.

TERMINOLOGY: Why call it a phase, and not something like a cycle? Don’t know, will think about it. Phase is an overloaded term that often is applied over a big span of time. I want something that is distinct (can’t overlap within a cycle).

HINDSIGHT: If you examine a final version of a product and carefully understand the work that went into it, you will see lots and lots of wasted effort. Some of that effort is necessary just because that is the path chosen to get there. Some of that effort was wasted because of ping ponging, and some of that effort was wasted because of sloppiness or accidents.

Each one of these phases has its own section in this document that describes it in detail. Each pass through an iteration is called a cycle.
Iterations can:

Overlap
Be Extended