Software Blueprints - Recently Updated Pages

Alternative Description

phomer — Mon, 04 Aug 2008 17:14:52 CDT

An alternative description of the "Pipeline of Transformations" idea, is that it is just the minimalist independent set of "predicate transformers" as described by Harlan Mills (and probably Dijkstra). In Mills' 1977 paper on Software Engineering, he talks about how Dijkstra, as part of structured programming, provides a mathematical basis for modelling software. It comes down to a model: P {S} Q, where P is a pre-condition, S is the sequential action, and Q is the post-condition. Step-wise refinement can be used to break down S.

If one looks at P, as a fixed state of data, ie a set of data-structures, static at the initial time, and Q as another static snapshot, then the "predicate transformers" in the model are just the steps necessary to map P -> Q. Mills states that this can be provable, but in the Pipeline idea, the whole essence of the idea is to provide all of the tiny "transformations" independently and allow the computer itself to reassemble the flow. Thus the underlying idea matches the mathematical model.

You want, as your Q a screen full of data about a specific user in a computer system, and a list of their last ten actions within that system. P is the preceding screen, and all we need now is the full set of transformers that clear the screen and replace it with the correct underlying data. The big S, then is composed of many many small S's, each of which gets data from a specific source, formats it for presentation and then places in on the screen somewhere. The users name for instance may come directly from the data store, while if the user is active, the last ten actions come from an internal system cache. This things are formatted, then wrapped in pretty output formatting and positions as described initial in Q.

The big difference, is that the computer itself, on seeing a request for Q, executed thousands and thousands of underlying predicate transformer primitives. It did this by calculating the path between Q and P. The path is more or less dynamic, but it deterministic. The text for the first name is grabbed from the data store, and then concatenated with the last name and placed at an X,Y position as requested in Q. The last action id is grabbed from the cache and formatted as a clean integer. The time-since is computed from the current time, and the action time, and then display at X, Y.

All the developers have done is supplied lots of predicate transformers like:

(Date current, Date action) -> (timeSinceOccured) {
return current - action;
}

(Name userFirstName, Name userLastName) -> (userFullName) {
return userFirstName + ' ' + userLastName;
}

(Name userFullName) -> (presentFullName) {
return "<b>" + MixedCase(userFullName) + "</b>";
}

etc.

Mostly if you think of each predicate as a simple one-liner that's close enough. The smaller, more atomic the transformations the better the computer system will be able to optimized and parallelized the processing. Note, there are no "function names", just a set of inputs, and an output. The software, searching for the correct output will come across a series of transformers, and choose the correct one (based on internal state?) to get the data (maybe it just chooses all of them? Heuristic model?)

Toss in millions of transformations, and a pick a bunch of Qs, (screens) and the system will be 'definition' do the right thing. The 'proof' of correctness is built into the mechanics. Q is the correct Q by definition (of course that might not be the Q you meant :-) Intrinsically, proving programs to be perfect is meaningless because that doesn't imply they are doing useful things, but certainly in this case, typos at the mid-level won't exist any more. More interestingly, if the visual transformation of the data is wrong, it is fairly trivial to create a new transformation to fix it. Testing and deployment are simple and obvious. The system is stable by virtue of the fact that the assembling of the code is not really done by the programmers, but by the machine.

Graphitext

phomer — Fri, 25 Jul 2008 15:25:10 CDT

A picture is worth a thousand words, yet text succumbs to the power of regular expressions and full text searching. In looking for a format for software blueprints it would be nice to use pictures, but text is also just as useful. So perhaps the only answer is to combine the two:

If you twist your mind, you can see that text is really just a bunch of little pictures (glyhps) all placed in a big sequence. Each character in the font is a primitive, and it doesn't overlap its fellows, but other than that it is just one big set of little images.

PRIMITIVE ELEMENTS

So what we want, is a set of primitives (like a font), that are arbitraty in size, and overlap each other. Position information is trickier, but where characters occur in an array (an implicit ordering), these primitives will have couple of navigation mechinisms: 1) a dynamically structured table (cells are all different sizes, based on their content). 2) explicity 'graph' links between the elements (or not). 3) some storage order.

#1 should be able to allow positional questions, such as what is left of X? #2 should group things together and allow specific traversal of the different groups. As well each primitive should have a name, and an instance_name (although it can be implicit).

NAVIGATION

Then, the cell postion gives enough information for drawing the primitives, (and in the primitive set we know things like size, etc). The cell and graph positions along with some extensions should allow something 'regular-expression-like' to be used to find specific patterns. Like fonts, the primitive sets are stored separately, from the graphitext files. The files can exist as images, or text, but their storage format should be text (display can be either). Any format like XML can be used, as the format is a listing of primitive instances with naming, and navigtional parameters.

SUMMATION

The extensibility comes from being able to create abitrary primitive sets with any level of element decompostion. The sets could be just geometrical, like lines, circles, triangles, etc. They could be font'ish symbols, or they could be some mix. The sets themselves should be 'formalized', and locked into place, similar to handling known fonts (files should allow for primitive set embedding).

SIMPLIFED EXAMPLE:

2 files, the first one is a primitive set for basic network topology, the other is a specific network.

The first one is a listing of the elements, computers, routers, switches, cables, etc. For each primitive, there is a name, default information and a vector image. For example:

NetworkElements.gpe

<version value=1.0 />
<primitive name='computer'>
<width>40</width>
</height>80</height>
<vector>
<line x1=0 y1=0 x2=80 y2=20 />
(enough lines and things to make up a pretty picture of a computer, based on a 40x80 size)
</vector>
<connect name=nextwork type=simple />
<connect name=user type=simple />
....
</primitive>
<primitive name='network cable'>
....
</primitive>

The for a specific instance, mynet.gsi

<instance>
<primitive set='NetworkElements.gpe version=1.0 />
<computer name='mymachine' />
<width value=28 />
<height value=47 />
<position x=12 y=23 />
</computer>
<computer name='anothermachine' />
<width value=28 />
<height value=47 />
<position x=32 y=23 />
</computer>
<router name='local'>
<position x=45 y=23/>
</router>
<cable>
<from>mymachine</from>
<to>local</to>
</cable>
</instance>

which if diplayed would show a pretty little picture of a small network. Also, one could lauch a grep-like client:

graphitext --file mynet.gsi--path .

> find {computer}*

mymachine
anothermachine

> find *{connect}{local}

anothermachine
mymachine

> find {anothermachine}[next]

local

where the primitive tokens, the token names, instance names were all accessabile in a name space (segmented properly). Any unique extension operators (I used [next] for positioning) where also available in the syntax. In that way we entend out Regular expressions (which are really charset symbol indicators and operators) into a large family of applicable syntax that more or less does the same thing (the set of names is infinite, the set of operators is finite).

I used a very simple example, only because I want to show that all of the main elements in the design are 'extensions' on existing concepts, except for the positioning mechnisms. The natural ording one is easy, things are in the order they are definined in. The graph theory one is also fairly easy, there is a link between any two elements, if and only if the link is explicity, other wise it is not there (although I think primitive sets can set up 'default' explicit links, just cause life would be ugly without them). The relative positing stuff is funny:

RELATIVE POSITIONING GRID

1. generate a matrix with a 1x1 square for each square in the grid that covers the total size of the diagram. for the above example assume 100x100 works (can be larger than nessary, but not smaller). use a text layout grid (not a math one) starting with 0,0 in the top,left ,and 100,100 in the bottom right

2. Layout all of the elements into the grid, using their position for the top,left location, and then span their size across rows and columns (as if working in HTML).

3. Collapse all ununsed cells both vertically and horizontally into larger cells. Any row without an element start or end can be removed, the same with any column. This gives us an arbitrary grid with erratic, but relative cell sizes.

4. Used the grid to answer questions like: on top of, to the left of, below, etc. Note that each thing in this table occupies one or more cells, and that the number of cells it occupies is relative to what is around it, not its size.

(ok, strickly speaking, I am not sure if the grid is necessary to answer the questions, but you have to admit its a fun algorithm. I used it long time ago to layout complex tables in HTML, but it was constructed in that case in a top down fashion by splitting the current space into an arbitrary underlying grid.)

Vi

phomer — Fri, 18 Jul 2008 16:41:29 CDT

- text editor
- state-driven interface (command mode, edit mode).
- all commands are a simple syntax: [range] item action (can't remember, but its documented in a book somewhere)
- calls out to sub-editor (ex) via colon syntax for heavy lifting, calls out to shell as well.
- limited configuration (.exrc file?)
- single file editing (new knock-offs like vim are better at this)
- standard 'gap buffer' editor internal structure (?)
-base editor spawned a large number of vi-like clones (vim, vile, etc.)

Systems to Blueprint

JSiegrist — Fri, 11 Jul 2008 00:49:19 CDT

The following systems are suggested as good modelling candidates:

Text Editor: vi, Emacs, pico, ed, TextMate, MS Word, OO Writer
IDE: Eclipse, Netbeans, Visual Studio, IdeaJ
Build Systems: Make, Ant, msbuild, SCons, Rake
Graphics Editor: Photoshop, Gimp, Illustrator, Scribus, Corel Draw
Spreadsheets: Excel, Visicalc, Lotus 1-2-3, OpenOffice Calc
Instant Messaging: AIM, Trillian, ICQ, GoogleTalk, any Jabber client
Email: Eudora, Outlook, Thunderbird, Pine, web-based mail
Web Browser: Mosaic, Firefox, Safari, IE, Opera

Because a wide range is available for each class of utility, how does one look past the features on the comparison checklists to determine what core attributes make up that type of system? One could do detailed analysis of features and find a common subset, or you could do some sort of blackbox analysis and start out with what the goals of the tool user and look for necessary components to meet the usage criteria.

A basic model of any computer program starts with a diagram of: Input --> Process --> Output. Would this picture be suitable as a high level description for any software system? Or are there differences now when you start considering interactive GUI-driven systems?

Alternative ls description

JSiegrist — Fri, 11 Jul 2008 00:31:29 CDT

I attempted to describe the same ls command without resorting to copying the existing description posted. Could the minimal differences between these be reconciled together or are the descriptions close enough together that it is not necessary?

User executes the command: 'ls' + args
The shell parses the command.
The shell invokes some code, passing in any arguments and a file system path (either cwd or a user-specified argument).
The code queries data from the file system based on any applicable arguments.
The code calculates metadata from the returned file system data.
The code formats its results based on any applicable arguments.
The codereturns the formatted results for display.

Common Blueprints

phomer — Thu, 10 Jul 2008 22:32:09 CDT

If we look at common systems, then deconstruct them into their components, we should be able to get some fairly generalized blueprints. Following with the spartan programming philosophy, we want to create spartan blueprints -- only the absolute minimum details necessary to insure that the CORE system works as expected. Thus blueprints for an programming editor should allow for vi or emacs, for an IDE should allow for Eclipse or Netbeans, for build mechanics should allow for 'make' or 'ant', etc.

The usage of the thing is independent from the specific technological implementation. The technology does restrict some of the degrees of freedom, but the software still exists abstractly as a solution to a specific set of problems. Our modern technologies just happen to be the only crude mechanisms we have at the moment for construction, but that does not alter the real-world nature of the problem, it is entirely independent of the computer.

Models

phomer — Mon, 07 Jul 2008 17:24:54 CDT

Various fun ways of looking at software beyond the standard DFA, NDFA, Turing Machine, Pushdown DFA, etc. appraoches.

Pipeline of Transformations

phomer — Mon, 07 Jul 2008 17:23:48 CDT

Another, alternative model of looking at computation is as a series of pipelines of transformations. In this perspective, any simple operator is just a transformation that takes a finite set of heavily typed elements, and provides a 'single' mutually exclusive set of transformations on it. In that way, if you were to watch the data as it exists on various places on disks, it is picked off in pieces. Each piece goes through a separate pipeline of atomic transformations, only joining together when the transformation is not mutually exclusive.

If the system is encoded as a huge collection of all of the usable, simple transformations, then the system itself, in knowing that it needed a specific type of data, could produce a map backwards to get the data it needs, from the data that is available. The system could then parallelize, and optimize any of the pipelines entirely on it's own, without the need for human intelligence. The trick to making this work, is that the transformations, while simple must be exceptionally strongly typed.

transformation(in,out): in -> out

Then in a sense, any datum is the outcome of some atomic operation on some discrete series of datum:

datum <= T(data-structure)

Which means to get the specific data needed to make a screen for example, for each 'discrete' piece of information on that final screen, the computer could trace backwards, following the transformation dependencies to their roots. The data for the clock traces back to the some location in memory were the time is updated. The banner comes from a image stored on disk, the user's name is from some virtual temporary data-store related to a running session, etc. Some small amount of data sources makes up all of the data bound together to form a screen in an application.

Some precedence would exist for the computer to choose between alternative data-stores, and alternative pipe-lines. Once a pipe-line is created, it's path can be cached to speed up performance for the next time through.

The data is strongly-typed, yet the pipeline optimizer can apply 'generalized' sections when there is nothing specific. So, T(int, int) -> int for an 'add' operation, or T(bond price, cash flows, date, accrued interest) -> Yield for something more specific. The transformations are always the 'smallest' possible algorithm that will complete the work (there will always be alternative optimizations, that can be switched by statistics), needing the 'least' amount of data.

In this sense, since everything is atomic, and it all interconnects, down to the specific typing of the data, a large number of systems can be meshed together into a great big sea of transformations, perhaps millions of them, optimized over time by letting the computer keep records about usage.

Functions within a Context

phomer — Mon, 07 Jul 2008 17:06:09 CDT

There are many different ways of modelling computation, including famous ones like finite state machines (DFA) and Turing machines. From another perspective, we can see all behavior in a system as being a series of functions applied against data. The function is the active code, some type of algorithm (or heuristic), while the data is located in many different parts of the program.

Function(context, arguments, data) => (results, side-effects)

For a simple model of the data, we'll definite it to be a) context, b) function arguments and c) some set of data structures. As a result of the function executing, there is some primary data result, and some number of side effects.

The context is important, because all significant computing exists with a context, like an operating system or a program, or some architectural structure, like a command shell. We strive to re-use our work, so we must separate out the 'context' in which it runs, to keep it from being inter-mixed.

The function arguments are also significant because there are the standard degrees-of-freedom within the behavior of the function. The context may provide some global effects (preferences, settings, etc), but the arguments localize the behavior to be what the user specifically intended. So, two executions of the same function at a similar time will likely operate in the same context, but with different arguments.

Software is all about managing piles of data, so anything significant is really just a transformation on some data-structure. We can just assume one in the model to make it easier, as the top-most data-structure could be a list of references to other data-structures. We differentiate between the incoming data-structure and the out-going one because that allows for things like caching. They may be the same, or they may not, it shouldn't matter to the model. The incoming one is the data, and the out-going one is the results. For functions like summaries, the two data-structures are not related at all, except by their computational ties. For functions like create or destroy, the data and the results are identical, yet altered by adding or removing things.

Side-effects are separated because like context, there is much going on in a 'system' that is beyond the standard behavior. For this model to be more usable than a DFA, it needs to more closely reflect the way computation is most often implemented, not it's theoretical minimum. Software is always full of a huge number of side effects, it makes up a significant amount of complexity.

Software Blueprints

phomer — Mon, 07 Jul 2008 14:36:16 CDT

This site is dedicated towards exploring blueprints for software development. This includes any and all issues related to blueprints, and how they can or cannot be used for software development.

It is open to anyone that is interested in software and how it is developed. New Members are always welcome!

Just interested in updates? You can subscribe to the RSS feed in Web Apps like Goggle reader to get the latest changes. You can find a link on the "What's New" screen at the right-hand side.

Attempt #2 - Structured Requirements Definition

JSiegrist — Fri, 20 Jun 2008 00:46:23 CDT

There is no abstract available for this page revision.

Simple Textual Blueprint

phomer — Mon, 09 Jun 2008 11:09:05 CDT

1. Take something simple, and describe it in text.
2. Reduce it to it's essence, and then see if what is missing changes people's intepreation?

The simplest programming interface is a 'command line' tool or program. The simplist one in most systems is the listing of the contents of a specific directory.

Called various names, such as "dir", "ls", "lf", etc.

The underlying data belongs to a 'file-system'. The context is usally some location in the file-system (having navigated their earlier), and some set of options to control the basic display of the function.

Thus, the user issues the command: ls
The code gets any 'command' options.
The code gets the current location in the file-system.
The file-system returns some data for location. (file names, directory names).
The file-system generates some meta-data for the location. (count of items)
The code takes the data and the meta-data and "reformats" it based on the options.
The code displays the presentation version of the data (and meta-data) to the user.

There are probably not any side-effects (although read-access or an audit log could be updated some-else) (earlier, the shell updated it's history (before the command even ran)).

There is an infinitely large number of different presentation options. (sorting, filtering, coloring, screen positioning, etc.)

The underlying data is driven by technical requirements (file-system is a crude database) and user requirements (user's need to move around and manipulate different files and directories).

Attempt #1

JSiegrist — Sat, 07 Jun 2008 00:55:15 CDT

Introduction:
What exactly are software blueprints? So far in discussions both online and in email, two possibilities have presented themselves:

1) A blueprint is an abstracted partial description of a software system that omits everything except for enough information to describe the most important parts of the system and the relationship between these parts, or
2) A blueprint provides a complete description of the system, detailing every component, sub-component and the linkages between components.

Of these two, I believe we have mostly discussed software artifacts of type #1, even though they aren't really blueprints in the proper sense. After studying the problem for the past few days, I now believe that items of this kind are really design aids. As an important part of the software process, system design aids can best be represented as a combination of just two things:

a) A description of the system data structures, and
b) A myth about the software system.

Detailed Description:
Data dictionaries are boring, unglamorous, and yet vitally important. Nothing more need be said about them because these are well known and already a common design document.

Like all good myths, the software system one needs supporting characters who have been beset by evil (or in this case their business problem). The myth requires a hero who is the program to be designed and purpose built for slaying dragons and saving users in distress. A well-defined setting that describes the circumstances under which the hero will operate (OS, programming language, external interfaces, etc.) and the restrictions that will make its task more difficult is also necessary. Finally as the protagonist, the hero must have a fully fleshed out description allowing anyone to understand how it ticks.

Software myths seems a good fit to the design task because they are understandable to programmer and nonprogrammer alike. The story can be expanded in a layered fashion, allowing the addition of detail if and when it's needed. Most stories rely upon a number of literary and cultural conventions to fill in much detail that would otherwise need to be spelled out at length. Software stories can likewise make use of industry and company specific programming guidelines. One additional benefit to this format is that it's fairly easy to analyze a story to see if it's balanced and how the story parts fit together. In short, for a design aid, this seems like it is a good way to go.

Conclusion:
However, as I realized to my dismay last night, this is not really what a blueprint is or should be about. The blueprint needs to be a complete description of what the final system looks like. As a document, it is concise because it leaves out all information about why, for whom, how, or in what sequence the pieces should be built and assembled. This is exactly how the blueprints in the architecture field work. Architectural drawings are concise because they leave how to put the building together to the contractor, only specifying what the finished product needs to look like. This works contractors, tradesmen, and building inspectors are all expected to be trained in knowing the building codes, which their work must conform to.

Lessons Learned:
So in closing, while this first attempt at a usable software blueprint failed, I learned several requirements of any eventual software blueprint. These are as follows:
I) Software blueprints must only describe the parts of the system and how the parts fit together into the solution. Any other information is extraneous.
II) Software blueprints must not specify how the system, system components or sub-components are constructed. The blueprints should describe what parts are used to make up components and sub-components, and what functionality each component (or sub-component) provides, leaving questions of 'how' up to the implementer.
III) Software blueprints must not impose any sequence on the construction process. Attempts at sequencing this work will only place unnecessary constraints on how this work is accomplished. Sequence of construction operations is extraneous to the overall description of the system anyway.

Proposed Blueprints

JSiegrist — Fri, 06 Jun 2008 01:11:32 CDT

This section of the wiki is where any attempts at software blueprints will be placed for public scrutiny and discussion.

Experiments

phomer — Mon, 02 Jun 2008 16:18:13 CDT

1. Expression of low, to medium complex architectural components in some type of simplified textual representation.

If we take a bunch of non-trivial common pieces of software, and experiment with stripping down the irrelevant data and functionality into some minimal (canonical) form, will various different programmers and software developers differ on the significance of the remaining details? We do want to leave some wiggle room in the specifications for intrepetation, technologies, and conventions, but still control it enough that the 'core' attributes of the system as conceptualized are the same ones that are built.

Requirements

JSiegrist — Wed, 21 May 2008 02:03:24 CDT

In an effort to ensure that work on this project is productive, we have determined the following set of operating principles for Software Blueprints:

Everything should be simple and obvious.
It should be as self-documenting as possible.
Consistency reduces complexity.
Every avenue is a possibility (at least for the first 100 years).
Each experiment as part of this project should be fairly focused.
Each investigation as part of this project should have specific validation criteria.

The Software Blueprints project has as its end goal finding a solution to the ongoing 'Software Crisis'. Our avenue of approach to this problem is to find or invent a better way to document software systems. It is believed that if a suitable 'software blueprinting process' can be developed, we could expect to see at least an order of magnitude improvement in reliability, scalability, and software defect rates throughout the system lifecycle. We recognize that software engineering is a young field and that this is a very ambitious goal. However, through study and experimentation, we believe that this project can begin taking the first steps towards the eventual solution to the software problem.

Provisional set of goals for the project:

Identify requirements for the software blueprints themselves.
Validate the software blueprint requirements.
Identify what is necessary to fully specify a software system.
Identify what other fields have related 'blueprint' requirements.
Identify what design artifacts these fields use as their completed specifications.
Identify existing software engineering artifacts and evaluate them for suitability as part of a software blueprint.
Develop any new software engineering artifacts needed for a concise set of software blueprints.
Experimentally validate the usefulness of each potential blueprint artifact.

Links

JSiegrist — Wed, 21 May 2008 02:01:49 CDT

A collection of URLs that are relevant:

The Programmer's Paradox Blog Post that Originally Started this Wiki

Software Design:
What Is Software Design? - By Jack W. Reeves
What Is Software Design: 13 Years Later - By Jack W. Reeves

Background Information:
Wikipedia entry on Software Blueprint
One description of a software blueprinting language

PDFs:
Software Blueprints - By Patrica H Baucom

Books:
Software Blueprints: Lightweight Uses of Logic in Conceptual Modelling - By David Robertson and Jaume Agusti

Companies Offering Software Blueprint Services:
Software Development Blueprinting
Dynamic Answers