I found this. I think it makes fascinating reading 15 years on
at the 25 th aniverasry of the QL.
At the end TT says the future depends on your(our) responses. In
retrospect were they correct?
A BRIEF HISTORY of SMSQ
Le Grand Pressigny, FRANCE - Tony Tebby
"Incompatibilities & Improvements, Bugs & Features"
"Was It All a Terrible Mistake?" I knew right from the
start I should not do it, but so many people were asking for
SMS to come out from under its covers that, in early 1992, I
outlined a strategy (with Miracle Systems and Jochen Men) to
make a "QL compatible" version of SMS available.
The outline was quite simple. A QDOS compatible SMS kernel
existed (and had been in regular use since 1990). A complete
set of SuperBASIC procedures and functions existed. A
complete set of (extended QL style) device drivers for the
Atari ST series existed as well as "portable" disk, serial
and parallel port drives for other hardware. There was an
environment which supported QLiberator compiled programs.
All that was required was the core of a SuperBASIC
interpreter.
Following the success of the Gold Card, Miracle Systems were
looking for a legitimate operating system for their (as yet
undefined) forthcoming computer and Jochen Men needed an
operating system to legitimise the QL emulator for the Atari
ST series. It was clear that it would be best to provide a
version for the Gold Card as well but as Miracle Systems did
not want to get involved in selling software, the Gold Card
question was left.
I embarked on trying to find out what Sinclair's SuperBASIC
interpreter did. It was not difficult defining what it
should do, but Jan Jones had built it on the principles of a
GIGO (garbage in, garbage out) machine. With limited ROM
space, there was no room for real error checking so Jan just
tried to make sure that whatever rubbish SuperBASIC was
asked to deal with, it carried on and did something. The
"something" was not always obvious.
While WE (QJUMP, those wonderful people at QVIEW, Jochem
Merz, Albin Hessler etc.) would never deliberately exploit
"holes" in SuperBASIC, the same was not necessarily true of
other software suppliers or contributors to PD libraries. In
addition, even WE had been known to fall through a hole by
accident.
Compatibility, therefore, meant not only reproducing
SuperBASIC as it was intended, but reproducing as many
oddities as would be necessary to execute most QL software.
The two "compilers" for SuperBASIC programs provided a
starting point. The aim was to provide a BASIC interpreter
which would provide:
1. better compatibility with SuperBASIC than either
compiler,
2. execution at least half the speed of QLiberator,
3. an environment supporting both QLiberator and Turbo
compiled programs.
Clearly, as some software for the QL will not even work on
all QDOS ROMs, total compatibility with a particular QDOS
ROM can only be provided by copying that ROM code. Even
slight re-ordering of the QDOS ROM routines (as in the Thor
XVI) can cause considerable incompatibility. At this stage,
there was no intention of providing improvements. The
"Minerva Experience" had shown the extent to which the
slightest improvements could give rise to extensive
incompatibilities. So much for intentions!
The Birth of the QXL
"They should not have done it either!" In principle, the
implementation of SMSQ on a 680x0 processor embedded in a PC
should have been fairly straightforward. The Gold Card used
the IBM PC disk controller, the IDE hard disk interface is
not very different from the Hardcard used in the Miracle QL
Hard disk and the serial and parallel ports on the PC are
much the same as the serial and parallel ports that you find
anywhere except on the QL.
If the QXL had been designed as a card which plugged into a
standard AT motherboard (no processor or memory) and
provided with drivers to drive a standard keyboard
interface, a standard multi IO / floppy / IDE card and a
standard Super VGA card, it would have been simple.
To have done this, however, Miracle would have had to have
supplied the PC hardware (at a cost of about half the QXL
card for a single floppy / 110 Megabyte HD configuration
this would have seemed, to me, the obvious way to do it).
However, it would have meant that the machine would not have
been usable as a PC (sigh of relief) and would have ruled
out the use of portables and notebooks. In addition, second
hand PCs were widely available either free or for less than
$30.
As one of the most likely reasons for a second hand PC being
available at a very low price was that the hardware was not
quite a perfect clone, then there was no possibility of
having a version of SMSQ which would access the IO devices
directly. Also, if the QXL were to be put into a real
"working" PC, it would not only have to co-exist with its
host but would have to work through whatever low level
software (Stacker, Doublespace, Hypercache, Smartdrive etc.)
was used to improve the IO performance of the PC.
As a result, any direct access from the QXL to the PC was
ruled out and the QXL was to be hosted by a DOS program. A
logical decision, maybe, but, from the point of view of the
operating system software, it was a disaster.
Where is the problem? The PC comes complete with device
drivers for all of its peripherals all that needs to be done
is to pass data from the QXL to the PC device drivers (using
BIOS calls) and vice versa.
There are three problems with this.
1. The design of the PC BIOS does not take account of the
requirements of multitasking (it is, for example, impossible
to write something to disk while you are waiting for input
from a serial port).
2. While the accuracy of the reference manuals about the
QDOS operating system entry points left a lot to be desired,
the (in)accuracy of "reference" manuals for the PC gives a
whole new meaning to the word reference.
3. All the reference manuals (so far examined) for the PC
were written in the days of the PC and PC/XT. The PC BIOS
also dates from this period. The BIOS has been considerably,
and incoherently, changed while the manuals have been
superficially updated to take account of AT keyboards, hard
disk drives larger than 10 MByte and 3.5" and HD floppy
disks.
Take, for example, formatting a floppy disk on the PC. On
QDOS, it is a single operating system call. On the PC,
however, part Of the format operation is performed by the
application. The format routine for the Gold Card floppy
disk driver took a couple of hours to write. As the PC BIOS
does most of the work for you, it should be easy to write a
format routine on the PC.
There are a variety of DOS and BIOS calls to help you do
this: setting the device parameters, formatting and
verifying tracks etc. I have three reference manuals which
give example format programs.
I look at the first one, and think "this is very strange".
There seems to be no way in which you specify the density
and there seem to be no checks for whether the tracks have
been correctly formatted: it appears to be automatic. So I
try it. Fine, it makes all the right noises and tells me
that my DD disk has 1440 sectors. I try an HD disk. Fine, it
makes all the right noises but tells me that my HD disk has
only 1440 sectors. It try without a disk at all. Wonderful,
it formats much more quickly and tells me that I have 1440
sectors! - AU FREE!
I try the second program: this one checks the error return
from the "Format track and verify" call: it even allows me
to specify the density. I try it with a DD disk. Fine! it
tells me that there are 1440 sectors. I try it with an HD
disk. Fine! it tells me that there are 2880 sectors. I try
it without a disk: the format fails, excellent! I try it
with a bad DD disk telling that it is HD. Fine! it tells me
that it has 2880 sectors. Suspicious, I try to copy some
files to it using DOS. DOS refuses to recognise it. I try
the other two disks: neither is readable. Over to QDOS to
look at the disks: there are no sectors 1, 2 or 3 on any of
the tracks I look at. This would explain why the first
program did not bother to check the error return from
"Format track and verify": it does not verify!
On to the third program. This is similar to the other two,
but uses the old "INT 13h" BIOS calls rather than the more
powerful "DOS function 44h" calls (wonderful this DOS
terminology). This requires the use of a separate "verify
sectors" call. The verify sectors call seems to work: this
routine gets an error on every track: it is right, after
formatting, none of the tracks are readable on any type of
disk.
So, I try it myself. After a lot of experimenting with the
BIOS calls described in the various manuals, I am able to
write either DD or HD tracks and verify them. The only
problem is that I write too many sectors to a track: the
last sectors overwrite the first sectors on the track. After
a week of work, I can select the density and I can nearly
format a track.
Thinks! Microsoft can do it, I should be able to as well.
Now we start to see the problem: the DV3 floppy disk format
routine is less than 512 bytes. Microsoft's FORMAT program
is greater than 32 kbytes, almost the size of QDOS, all its
device drivers, SuperBASIC and all its procedures and
functions. No wonder all the manuals are wrong. It would
take about 200 pages just to list the MSDOS FORMAT program,
without trying to explain how it works!
Disassembling all of this program could take months. I
decide to trace the two paths of interest: 720k and 1440k
formats. It turns out that what you need to do is to poke
special values into various undocumented locations in low
memory. I note all the locations to be poked and set up a
format routine. Within this routine I poke all the required
locations, format the disk and restore all the locations to
their previous values.
Success, I can now format DD and HD floppy disks. The only
problem is that, despite my care in restoring all the poked
locations, after a format the PC refuses to recognise any
disk change until you hit the reset button. Two weeks have
passed and I still do not really know how to format a disk
using MSDOS.
Do I spend another 2 weeks finding out how to restore the
BIOS after a format operation?
Even if I succeed in making it work on my PC, will it work
on any other PC? How can anyone succeed in selling an
operating system where it takes two weeks to write a routine
using the operating system calls when it would only take two
hours to write the same routine accessing the hardware
directly?
Now we find the real cost of the QXL in development. Even
though the IO performance of the QXL is well below the
levels that it would be reasonable to expect, the
implementation of the QXL device drivers has cost between 10
and 20 times the cost of equivalent drivers for other 680x0
platforms. As a result, all the time that had been set aside
for the development of the SBASIC interpreter has been
swallowed up. For the first purchasers of the QXL, things
looked grim: poor IO performance, no SBASIC interpreter. Not
a very promising debut for SMSQ.
"You Take the High Road and I'11 Take the Low Road" The QXL
hardware strategy was not the only problem to be faced.
Miracle Systems, for reasons which should be obvious, wanted
the QXL to seem as much like the Trump Card and Gold Card as
possible, while Jochen Merz wanted an operating system which
was not just developed along the same line as the Atari QDOS
extended device drivers but one which went much further.
One man's improvement is another man's incompatibility. Now
we have the problem of developing (and maintaining) two
different variations of SMS: SMSQ, the basic QL-like version
and SMSQ/E, the extended version which is likely to diverge
evermore and more from SMSQ. Jochen Merz, therefore, decided
to supply SMSQ/E for the QXL as well as the Atari and Gold
Cards. Easy for him to decide: it was me that had to do the
work!)
More problems. It seems that computer users are not very
sensitive about how much they have to pay for their
operating system. They are, however, very sensitive about
how much other users pay! Gold Card and Atari users do
not complain about having to pay for SMSQ/E (we told then it
would be necessary back in 1990), but they do object that
QXL users get a "free" version of SMSQ with their QXL. QXL
users do not seem to mind being asked to pay extra for
SMSQ/E (at the moment the differences are fairly small so it
is not usually worth "upgrading") but they do object that
Gold Card and Atari users are not being asked to pay more.
Even worse, there are some QXL users who seem to think that
they are being provided with a specially naff version of
SMSQ to oblige them to cough up a few extra pennies for an
upgrade!
Then to cap it all, Miracle Systems produce a Super Gold
Card which looks like a Gold Card, but turns out to be
rather different. We now have implementations of SMSQ on
three distinct hardware families, seven different hardware
variants, four different display types, with four different
68000 series processors, in three (and sometimes more)
languages. So far, there are more possible combinations than
there are users.
To avoid the necessity of producing a different version of
SMSQ/E for each user, SMSQ now uses a module structure which
has been borrowed from the Stella (Stella????) operating
system. This allows operating system modules to be selected
(or ignored) as the system is booted. In principle, a single
version of SMSQ could be delivered which would autoselect
the right modules for any hardware combination. In practice,
each hardware family (Gold Card, QXL and Atari ST/TT)
requires its own special loader, so that it is not worth
incorporating all the modules in each version.
Just as the number of users starts to take off, so does the
number of variations. Jochen Merz ships a copy of SMSQ/E
to a Gold Card user: the next day there is a message "SMSQ/E
does not work with the XXX keyboard". Not surprising, the
XXX keyboard uses a patched version of the JS ROM. The cure?
Another keyboard driver module for the Gold and Super Gold
Cards and another language module (the keyboard tables). The
net result is one new user and four new variations. Counting
variations is soon going to be like counting marbles in a
cookie jar.
"'Till Death us do Part" In the days of easy divorce for
reasons of mutual incompatibility, it is surprising to find
so many QL users wedded firmly to the old software packages
of the "use it at your own peril" style. Then I started the
evaluation of SBASIC, Miracle Systems sent me a bundle of
diskettes (about 10 Mbytes worth) of the type of software
that they thought might provide a test for the compatibility
of SBASIC.
I started looking at these disks on the Atari ST with JS and
the E level drivers. After resetting the Atari ST for the
tenth time without having found any software which even
started to work, I gave up and tried using a Gold Card.
After a day or so, I found two programs that could be
executed, played with and removed without crashing the
system. All the rest either crashed right at the start,
could not be made to do anything sensible, or could only be
removed by resetting (I began to understand why some users
have been asking for a quick reset). I have been told that a
lot more of the software would have worked if I had set the
memory size to 128 kbytes, but if you are going to reset you
machine to 128 kbytes use one program and then reset again,
there is no point at all in using SMSQ: you might as well
stick with QDOS on your old faithful QL. Seriously, does
anyone use this type of software anymore?
The first compatibility tests were very encouraging: all the
programs which crashed on a JS QL crashed with SMSQ. It
seemed that we had obtained better than 95% compatibility.
Moreover, one of the two programs that worked on the QL
worked with SMSQ: the figure was up to 98% compatibility.
"New Lamps for Old" One of the best ways of checking the
originality of software is to investigate the bugs. If two
items of software perform the same functions correctly, one
could be a copy of the other, or they could both be written
to the same specification. If, however, two items of
software exhibit the same bugs, it can be assumed that one
is copied from the other.
There are very few "first level" bugs (bugs which prevent
the system functioning correctly under "normal" conditions)
in QDOS. Because of the GIGO policy and the desire to limit
error checking to a minimum to maintain efficiency, there
are a much larger number of "second level" bugs (where the
system misbehaves when passed incorrect parameters or data
structures) and even more "holes" (where calling a system
function with deliberately incorrect parameters has a
reproducible if bizarre effect).
During testing of SMSQ and SBASIC, a large number of second
level bugs were uncovered in the JS ROMs. Many of these
showed up also in Minerva, none in SMSQ or SBASIC. From time
to time, users have uncovered a number of second level bugs
in SMSQ and SBASIC. All of these were entirely new and have
no connection with old QL ROM bugs: SMSQ and SBASIC are
entirely original!
Streamlining code has the effect of removing, altering or
introducing holes. It is not surprising, therefore, to find
that many of the holes that are exploited by some common
software, have either disappeared or been altered in Minerva
(giving rise to complaints of compatibility problems).
One such case is the xx.xxxxx SuperBASIC vector which is the
same in all QL ROMs. This vector is intended to be used with
data structures set up by the SuperBASIC interpreter. This
has three defined paths controlled by the value of one byte
(0, 2 or 3). Someone discovered that it could be made to
produce a bizarre effect if the passed a value of 1 in the
control byte. The resulting code fragment (which takes
longer than using a legitimate call) has been incorporated
into a utility, which has found its way into a large number
of programs for the QL. The streamlined Minerva code no
longer had this hole so a large amount of software stopped
working on Minerva. The Wizard did not manage to find the
real villain in the code, but succeeded in restoring
"compatibility" by setting a register to a value which it
would not normally have with SuperBASIC. This, in turn,
altered another hole and introduced different compatibility
problems.
In SBASIC, however, the hole never existed. Once the villain
code had been identified (a week's work) it was, therefore,
a simple matter of detecting the villain case and emulating
the hole directly. It was a waste of time and effort, and it
slows down SBASIC, but that's what it's about, isn't it?
The boundary between a bug and a hole is a very fine one and
if some software relies on a bug in the QL ROM do I need to
reproduce this bug? Unfortunately, the answer is sometimes
GRRRRRR YES.
Twice recently, I have received reports of "bugs" that have
appeared in the string handling in recent versions. These
"bugs" have been introduced into SBASIC to improve
compatibility with SuperBASIC (there are still three "bugs"
in SuperBASIC string handling which are not emulated in
SBASIC). Neither of these users was aware of that the bugs
existed in SuperBASIC: SBASIC is now being used where
SuperBASIC never went before.
"Whose Fault is it Anyway?" One rather tetchy letter
complained that SBASIC was very fragile by comparison with
SuperBASIC: using a well-known piece of commercial software:
"SBASIC crashed". This was misdirecting the blame. As the
software was invoked correctly by SBASIC and as it never
returned to SBASIC, SBASIC could hardly be to blame.
This well-known SuperBASIC extension started off by trying
to identify a fragment of the QL ROM, and, when it could not
find any QL ROM code (there is none in SMSQ), it jumped to a
completely arbitrary location. BANG. The cure: I re-wrote
the extension and incorporated it (with improvements) in
SBASIC.
In fact, SBASIC is more robust in this respect than
SuperBASIC: error trapping is much more thorough (and
forceful). If it had happened in a daughter SBASIC: it could
simply have been removed with no harmful side effects.
(Perhaps I should implement a keyboard "restart" for Job O.)
Another difference between QDOS and SMSQ which might give
the impression that SBASIC is more fragile than. SuperBASIC
is the default error handling: QDOS carries on but SMSQ
stops to allow a debugger to be started. If the job is
already being monitored by a debugger, there is no
difference. In the normal state, however, allowing a job
which has produced illegal instruction or address errors to
continue could easily result in widespread damage to the
system data structures, possibly resulting in the loss of
part or all the data on a hard disk. SMSQ is, therefore,
much safer, even if jobs appear to stop more often.
Many programs compiled with the current version of Turbo are
wonderful examples of this. At the start of these programs
we find some code which sets a location in memory to O.
Several times later on, the value in this location is moved
to register A2 and then there is the instruction to move the
contents of address O (A2) to 04, D2 is compared against the
new value in D4 and then there is a conditional branch.
MOVE.W (A2),D4
....... .......
SUB.W D2,D4
BLT.S ......
On the QL the MOVE sets D4 to 3 so the operation of the code
is dependent on whether D2 is greater or less than 3.
On Atari STs modified for the old QL emulator, the MOVE sets
D4 to 24,622. The behaviour of this Turbo code will,
therefore, be significantly different on these STs as the
operation now depends on whether D2 is greater or less than
24,622.
On unmodified STs with a patched version of the JS ROMs, the
MOVE will cause a "bus error" which QDOS ignores and so
execution will continue without changing the value in 04. 02
is, therefore, compared against an unknown value: this will
give yet different, and rather unpredictable, behaviour of
Turbo programs.
On unmodified STs with SMSQ, the MOVE is trapped and these
Turbo programs just stop. It is possible to set a special
"Turbo mode" (PROT_MEM O) which emulates the QL ROM access
by setting 04 to 3 and continuing. The behaviour of Turbo
programs on these STs is, therefore, the same as on a QL:
SMSQ is more compatible with the QL than QDOS!
"I See no Ships" While SMSQ with SBASIC marks a great
improvement in performance and capability over the old QDOS
ROMs, this has not been achieved without creating a few
problems. The original SMSQ and SBASIC had very few
intrinsic bugs, but many incompatibilities. Unfortunately,
it is one of the facts of life of computing that making any
changes to a existing software borders on vandalism. The
neat structure of the original conception begins to crumble
and soon each little "fix" risks introducing a host of new
problems. Fixing each one of these introduces more. It is a
tribute to Jan Jones original SuperBASIC conception that
during the early days of "active development" when new
features were being added every day, the ratio of changes to
bugs introduced was better than 10:1.
SBASIC is a much more complex piece of software which has
the disadvantage of being required to emulate all the
quirks of someone's first attempt at writing a BASIC
interpreter. It, therefore, starts off being not very neat
and it is prone to degenerate more quickly. SBASIC's ratio
of changes to bugs introduced is closer to 5:1 - small
enough to be convergent, but too large for comfort.
Fortunately, the bugs introduced from version 2.11
(experimental SBASIC) through version 2.25 (the first
"release" version) to version 2.42 (current at time of
writing) have usually been smaller than the ones they
replaced.
Although some compatibility problems with some hardware
variations remain to be resolved and there are one or two
program which still refuse to function with SMSQ, the
original aims for compatibility and performance have been
well exceeded and SMSQ is now establishing itself as a
living replacement for QDOS.
So, is SMSQ/E at last stable? The answer must be no. As more
and more people start using SMSQ/E there are more and more
requests for improvements (i.e. decreasing compatibility).
SMSQ/E has now passed the point of no return: there are more
requests for improved capability than for improved
compatibility. The current versions of SMSQ/E are at least
as reliable as any QL ROM version and are getting as close
to 100% compatibility as is possible while providing better
performance and more facilities.
SMSQ/E is a commercial product and as such needs to meet
users demands. If users require changes, and it is
commercially feasible to provide them, they will get them.
SMSQ/E cannot, however be developed in all directions at
once.
For software development, a 16 MHz 68000 based Mega STE (1
MIP with 2 Megabytes of memory running under SMSQ/E (my
"standard" configuration) is more than a match for a
"standard" 50 MIP 32 Mbyte workstation. A TT or a QXL on a
good 486 machine is more impressive still. The Gold and
Super Gold Cards have the same standards of raw performance,
but suffer from limited display capabilities, poor keyboard
and IO.
How much more is it reasonable to do with SMSQ which, by its
need for compatibility with a 10 year old computer, is
locked into a 10 year old design? Is there enough interest
in the type of operating system concepts pioneered by QDOS
to make it worthwhile producing a completely new system? The
future depends on your response.
