By Charles Jardine
In 1970 The University Computing Service was created. The Mathematical Laboratory [0] was renamed the Computer Laboratory, and split into two sub-departments: Teaching & Research, and the Computing Service. The first director of the Service was Dr David Hartley [1]. The Service's first major task was to commission an IBM 370/165 mainframe as a replacement for the Titan [2], which had been running a time-sharing system for the University since 1966.
[0] The Mathematical Laboratory was founded in 1937. It is most famous for EDSAC (Electronic Delay Storage Automatic Calculator), the world’s first fully operational and practical stored program computer. This was built by Maurice Wilkes and his team between 1946 and 1949. See https://en.wikipedia.org/wiki/Computer_Laboratory,_University_of_Cambridge.
[1] See https://en.wikipedia.org/wiki/David_Hartley_(computer_scientist). Note what he did for his PHD.
[2] The Titan was was Cambridge's third mainframe, following the two EDSACs. It was the prototype of the Atlas 2 computer developed by Ferranti in collaboration with the Mathematical Laboratory. Much of the operating software was designed by a team led by David Wheeler, co-inventor of the subroutine. The Titan was in operation from 1964 to 1973. From 1966 is provided a time sharing facility. See https://en.wikipedia.org/wiki/Titan_(computer) and https://elearn.cs.man.ac.uk/~atlas/docs/The%20Atlas%20story.pdf.
The Titan was a hard act to follow. Preparing the IBM for general purpose use, by the University at large, was a massive task. The software provided by IBM was complex, but did not include much of the needed functionality. Computing Service staff had to write applications software, utilities, resource control systems, and management tools. The hardware had to be stretched very thinly. As delivered, the 370/165 had only 1 Mb of memory! Moreover, there was no virtual memory hardware. The Service needed efficient tools to ration available memory, storage, and CPU cycles between the infeasibly large number of users. One of the required tools was a repository of data about users, resources and allocations. The Service needed a database system. Cambridge had nothing suitable.
Mike Challis was hired to write a database package. This package was called Jackdaw. In 1974, a year after the package was finished, Mike wrote "Computer Laboratory Technical Report No. 1: The Jackdaw Database Package". See First Jackdaw. This report describes a simple entity/relationship system with a procedural API presented as a BCPL [3] library. The package included a CLI interface for manipulating data, which could be used interactively or in batch. There was also a reporting tool which could produce statistical annexes for annual reports. The CLI was called INTERFAC (IBM command names were restricted to eight characters).
[3] BCPL (Basic Combined Programming Language) is a small Algol-like language designed by Martin Richards in 1967. It was intended primarily for compiler writing and systems programming. It was a successor to CPL https://en.wikipedia.org/wiki/CPL_(programming_language) and a precursor of C. See https://en.wikipedia.org/wiki/BCPL. The Computing Service continued to use BCPL for systems programming on IBM mainframes until we stopped using IBMs in 1995.
Jackdaw had no non-procedural query language, but no-one else had one this early. IBM did not ship System-R/SQL until 1977. Ingres had something running Quel at about the same time. Oracle didn't ship anything until 1979.
The Jackdaw report describes two significant algorithms. Section 5.4 describes a copy-on-write (COW) algorithm. Section 5.5 describes a primitive intent-log algorithm. These days, intent logs are used in many file systems and databases. COW algorithms are less common, but are used in some significant file systems, e.g. xfs, the NetApp fs, zfs and btrfs. I don't know if Challis invented copy-on-write, but I am reasonably sure that the Jackdaw report is the first publication of the idea. The existence of the report may have benefited the world in a particular way [4].
[4] Roger Needham told me that he had once dissuaded a company from attempting to patent a copy-on-write algorithm by showing them a copy of the Jackdaw report as evidence of 'prior art'. Sadly, he did not tell me which company it was. It might have been Silicon Graphics, the original proprietors of xfs.
Challis's Jackdaw had its shortcomings. For example, it had no secondary indexes. The entries in a class were indexed by their names. Links were also indexed, but the values of primitive fields were not. We resorted to printing lists of people sorted by surname and keeping copies beside the terminals in the reception office. However, Jackdaw was unbelievably reliable. Moreover, it automatically supported useful integrity constraints [5]. We continued to run Jackdaw, with very little trouble, until we turned off our last IBM mainframe in 1995.
[5] Section 2.3 of the Report describes the constraint that link fields are reciprocal. The Report does not mention the way in which Jackdaw entries and links could be marked cancelled. This facility provided the constraint that a link to or from a cancelled entry must itself be cancelled.
I was taken on to run the Jackdaw database in 1978. I was responsible for the database system and its data, INTERFAC and the reporting tool. I was not responsible for the down-stream processing. Chris Thompson was already maintaining JOB0 (job nought). This batch job scanned the database for changes, reconfiguring the OS appropriately. JOB0 was the ancestor of all future "user-admin" scripts.
I soon discovered that the database essentially ran itself. The reporting tool was a little more tricky, and its customers, the senior management, were more picky. INTERFAC, with its paper secondary indexes, was not easy to use, but, then as now, reception staff made everything look easy. I found I had the time to start a career as a systems programmer. In the early '80s the Service prepared for and commissioned an IBM 3081 (later upgraded to a 3084). This was delivered in 1982 and went live in 1984, replacing the old 370/165. We were all busy during the commissioning period. Jackdaw itself was not directly affected by the upgrade, but almost all the down-stream processed needed replacing. I wrote parts of the replacement for JOB0 and interfaces to IBM's new Resource and Access control Facility (RACF). This, and other similar work continued to occupy much of my time long after go-live.
During the '80s the nature of the Computing Service changed enormously. Our services became more varied as the focus moved away from the mainframe, and personal computers [6] became commonplace. At the same time, the number of users of the mainframe increased greatly. In 1980, is was unusual for an undergraduate to have a mainframe account. By 1990 all undergraduates had mainframe accounts. The driver for this was email. Before 1993, the only email system available to all members of the University was that on the mainframe.
[6] The first IBM PC was introduced in 1981.
During the mid '80s I was mainly occupied with mainframe systems work. I was aware that work was required to provide tools to help the user-admin group to cope with the increasing complexity and volume of their work, but I had very little time to do anything about it. I knew that we would need to to replace Challis's Jackdaw with something else sometime, and I had started to look for an adequate DBMS that we could afford to run on the mainframe. I began to think of Jackdaw as an item of legacy software, that would soon be replaced by something completely different. This led me to think that there was no point in developing Jackdaw any further. With hindsight, I now know that this was the wrong thing to do. The replacement of Jackdaw was neither going to be soon nor completely different.
In 1986, Pat Stewart asked me to find a way of recording which users were undergraduates and when they had started. The right way to have done this would have been to add a Jackdaw class named COHORT with an entry named UG1986, and to link all the new undergraduates to this entry. I baulked at this. Instead, I suggested a botch which avoided having to make a change to Jackdaw's metadata. I advised Pat to misuse the INST tree by polluting it with children of the college nodes with names ending in 86. These were, and still are, the notorious 'year-nodes'. I would not have done this if I had known how the next 6 years would turn out.
I never did find my "adequate DBMS we could afford to run". I now believe that this was because there was no adequate DBMS. The automatic consistency constraints in Jackdaw were way ahead of their time. SQL DBMSs are very hard to implement. To equal Jackdaw you need a DBMS which can enforce foreign key constraints. The concept of referential integrity was in the research literature in the early '70s. Implementations came 20 years later! The first version of Oracle that enforced referential integrity was 7.0, released in June 1992.
At the end of the '80s it, at last, became clear what the platform for the replacement for Jackdaw must be. There were two compelling influences:
The Computing Service decided what platform would be used for 'mainframe replacement' services. It was Unix, specifically SunOS 4 on SPARC hardware.
The Administrative Computer Unit, the predecessor of MISD, obtained an Oracle license which would allow the Computing Service to run Oracle at no cost.
So, the server platform for the replacement for Jackdaw would Oracle over SunOS4.
In 1990/91 the Computing Service obtained five large Sun servers, three for the new Central Unix Service, one for a mail switch and one, called pavo, for odds and ends. Pavo ran a Gopher server. Gopher was a predecessor of the Web.
During 1991/92 I obtained resources on pavo and ran up a couple of Oracle 6 instances [7] for the Jackdaw replacement. I formed the following plan (reconstructed from memory):
[7] Oracle 6 allowed FOREIGN KEY declarations in DDL, but it did not enforce the constraints. This allowed me to start writing the schemas for the Jackdaw replacement while waiting for the release of Oracle 7.
Really freeze the Jackdaw metadata.
Write an Oracle schema (JDAWADM) which includes an emulation of the whole of Jackdaw.
Write a script driven by the Jackdaw metadata which has the effect of synchronising the data in the Oracle schema with Jackdaw.
Write a script driven by the Jackdaw metadata which has the effect synchronising the data in Jackdaw with the Oracle schema.
Start running script (3) as a daily batch job on the mainframe.
Optionally start developing reports and outgoing feeds on the Oracle system.
Using a testing Oracle instance, develop and test a replacement for INTERFAC. Train the user-admin people in the use of this tool.
Disable INTERFAC.
Change the daily batch job on the mainframe to use script (4), thus reversing the data flow.
Enable the use of the INTERFAC replacement against the production Oracle instance.
With Mike Challis's permission, start to use the name 'Jackdaw' to refer to the new, Oracle based system.
Start work on the backlog of requirements for Jackdaw which accumulated in the late '80s. Top priority - automatic registration of students.
This plan actually worked. It left so little mark on the Jackdaw data that it is hard to see traces of the switchover. I think that step 5 occurred in the autumn of 1992. I think step 6 included a new user-admin script for the PWF. The most challenging job, of course, was step 7. There was time pressure. The planned closure of the mainframe was beginning to loom. The INTERFAC replacement was an Oracle*Forms-3 application. I can't remember much about it. My memory has been overlaid by a later Forms-4 application.
Steps 8-10 occurred at the beginning of the Christmas vacation in 1993. Step 11 was in January 1994.
The effect of all this was that Jackdaw was not really replaced. It was ported to a new platform. The Jackdaw data were preserved. The metadata was also preserved, but expressed in a new way. The old Jackdaw remained as a slave copy of the new one. It was not necessary to change the down-stream processing on the mainframe. It continued to work until the mainframe was switched off in September 1995.
The downside of the above is that the mistakes, botches and bungles of the previous 20 years were preserved. The year-nodes are still there a further 20 years later, in 2015:-(
Step 12 started immediately after step 11. I recall a day in August 1994 when I was writing a server-side PL/SQL procedure to do with the automatic registration of MPhil students. I was very happy to have the power of SQL, scripted on the server, to do things I could not have done on the old Jackdaw.
... to be continued? ...