# Makumba Design Issues

versão $Revision: 2.2 $, November 2000, in Stockholm

This document continuously presents the state of the Makumba project. It is intended for Makumba designers and implementors. The idea is to keep all knowledge about Makumba in one place and not to have it spread in all sorts of files. The spec will be maintained in parallel. As different from spec, this document also includes past ideas and future visions.

Recently design issues are kept in Makumba's bugzilla, with designDiscussion milestone.

The only information that is not in this document are the introdutory text, the spec, the API documentation, the tasks, and the supported databases documentation.

## 1. Principles/philosophy

(see Cristi's Philosophy mail)
pre-packaged BEST features are very different from professional
- no fine-grained control of all details
- a feature is available on the most generic sense with no concern for speed or space
- few features are available, only the essentials
- makes it easy to learn and develop
- not simple-to-complex learning because an advanced user might use complex features that a novice won't understand

non-professional
- total control
- feeling of independence
- no general purpose (taught in school) commercial technology exists for
data-driven web apps (hack over database systems to get to the web or vice versa)

next (visionary) system will come by design refactoring of the present one

## 2. Professional Data-driven Web technologies

### 2.1 Lasso (blueworld.com), its LDML language

we didn't manage to run Lasso server on the laptop, but we could look at language examples
LDML looks very much like what we want for Makumba
observations learned from Lasso:
- one can have more than one presentation language
- any language can have an XML description (whatever the use)
- html editors can be used to edit a good language, and if the tags are like <..> the html editors will not rewrite them (see difference between LDML and object LDML
- LDML is very much programming (rather than presentation) oriented
- too many tags
- nice simple logging mechanism, but it shouldn't be at presentation layer, or should it?
- lasso seems to be a one-table search thing
- but this is not a problem since you can have views for each join you can think of

### 2.2 CFML (Cold Fusion Markup Language, allaire.com)

- SQL dialect dependent
- but more search power
- named queries look interesting

### 2.3 Webmacro (webmacro.org)

Looks like JML/ JSP. Just a different syntax and probably different compilation and error messages. But that doesn't make it much better.

### 2.4 General observations

both CFML and LDML miss the generation of the default presentation because they don't describe the data anywhere.

an improvement of CFML and LDML over stuff like PHP or JSP is that there is better integration with the database and less db connection installation overhead

both have server-side includes

Tango (pervasive.com) is a dead product but it makes a nice distinction (presentation, data model and business logic)

## 3. Trainee orientation

Makumba should be oriented in helping trainees to learn it fast, install it easily and start coding with as few problems as possible
- removing installation overhead
- tomcat, jdk, database engine, etc., should be pre installed and packaged
- MySQL as database for windows
- CD or zip with the pre installation and example data (e.g. complete general tables)
- CVS as overhead during trainee startup (then very useful)
- we don't necessarily want/have the skills from open-source. Not all trainees need help there, but it's a good diversification of opportunity.

Igor case: hindrances of the BEST development environment
- a lot of good will
- poor connection
- different locations, slow email communication
- not yet enough experience
- initially a victory (odbc4 driver)
- when the new problem came in (message board), local devel stopped

Student organisation identification with an ideal
- professional software devel (the spec document, framework that was promising to comply with changes in spec, etc)
- open source (Makumba), no release yet because of prestige. Not all of us have to be open source programmers. Some trainees will only get to do Makumba code development and maintenance, some others will get to work on spec and internals in the open-source process.

(kosice) meeting preparation with hands-on examples (low overhead should help this)

## 4. Reflective history

(this is working information for now, but might be later "demoted" as an annex)

### 4.1 first version (March 1999)

- abstract, DB and presentation data description
- only had presentation implemented

### 4.2 second version (Xalara in Athens July 1999)

- re-organised directories in package like manner
- some "redundant" features: templates, one-to-many and many-to-many tables. redundant features complicate the language while they can actually be implemented with the existing ones
- we were still puzzled about search
- we believed that there is a ideal presentation layer and we tried to find it
- what we still like: general idea of record description, replication fields, record title, pointer, set, general type correspondence, FIXED, NOTNULL
- implemented the first version based on these ideas

### 4.3 after Athens

- proposed searches in separate files. still not clear what to do with arguments
- still not clear how to select fields from the search
- attempt a second presentation layer; quite cryptic. See Annex 1
- "new database layer" attempted to model all SQL SELECT concepts in Java objects (Expression and Filter)
- presentation toolkit and other upfront design OOP ideas, for generating any kind of content, etc...
- ODMG database layer proposed OODB-like principles for the API
- JSP taglib-based presentation layer. See Annex 2
- OQL discussion - resulted in Autumn 2000 implementation of OQL
we liked the join in the FROM. it "moves" the join from the WHERE
clause, which is mostly for filtering

        SELECT DISTINCT s FROM /best/student s, s.education e,e.studyfield st
        WHERE st.name="Chemestry"

joins in the implicit relations?
`/best/minerva/student.minerva m(s)`

### 4.4 Lost in Kosice July 2000

- attributes are kind of constraints
- not table, but object-oriented
- lists as a clear difference from sets
- inheritance rather than inclusion
- multilanguage suport in text literals
- title is the first non_pointer field
- clean up of syntax
- business logic
- actors
- authentication mechanism
- "extension"
- administration mechanism
- new field constraints UNIQUE, ENCRYPTED
- minimum and maximum char length [a..b]

### 4.5 after Kosice (ML discussions)

- if we define inline types, we don't have to think of complex sets and 1:1 pointers anymore. this simplifies the language
- Toto's Business Logic sources.
- fieldname: int { default longhish_shit_that_is_not_comfortable_to_repeat, a, b, c }
- now as default for date
- interfaceLanguage in actor profile
- %localize(lang1, lang2) at presentation level
- dot instead of / in the class name. capital for type name as style rule. OQL becomes:
`SELECT DISTINCT s FROM best.Student s, s.education e, e.studyfield st`
`WHERE st.name="Chemistry"`
- data ecryption not one way
- spec shouldn't talk about SQL, or specify that it is an example

### 4.6 Coimbra October 2000

- tried to make sense of all stuff and set short-term and long-term goals
- see design issues below, plus annexes 4 and 5
- we ended in a frustrated note due to too short meeting time. But overal the meeting was very productive.

### 4.7 After Coimbra (Stockholm, ML discussions, stuff done,...)

- ODBC and mSQL (temporarily) not supported
- Implemented OQL (with antlr, fixed their OQL grammar, submitted it, got no reply yet)
- bought makumba.org domain name
- made a database copy utility (a good ground for future replication, transfer between different databases, different SQL servers, cleaning historical table names)
- moved some Toto's own methods (presumably most frequently used by a app developer) to a prominent place (metadata.db.Database) for everybody to use
- webmasters looking for alternatives (PHP, Perl, Apache server side includes...)

### 4.8 Balatonföldvar November 2000

- subselects in MySQL not fully supported
- realised that database-level indexing improves performance significantly
- made OQL work in minerva student search
- compiled and packaged a set of programs (17 megs) needed to run an application (in this case Minerva, working on a non-programers' laptop) No need to set any PATH or CLASSPATH. Only needed change might be to enlarge the memory of the command.com but we can provide proactive assistance for that
- confronted webmasters with jsp --> enthusiasm!
- made BEST realize of Makumba's importance to the organisation / raised the interest by introducing Karamba!
- markeTeam takes over the PR campaign- promotion of Makumba as an open source project
- eliminated 'dot convention' - no more dots in table names

### 4.9 After Balatonföldvar

- eliminated the need to manually translate names of fields and tables to avoid conflicts with SQL keywords on some database engines (perticularly mySql is very sensitive, postgres more tollerant)
- added a feature to shorten table names (Postgres has limit to 32 characters, MySql no problems)
- documentation clean-up
- updating this doc, API spec

- found a nice model for Rapid Application Development

### 4.10 Rome December 2000

- First test of a bigger application with the latest version of Makumba.
- Presentation to webmasters, they love it, we need to make them a distribution for them to play with it.

### 4.11 After Rome

- Writing the Makumba JSP tag specification, finalizing the taglibs.
- New Makumba DD language implementation postponed (and have it done later with antlr)

### 4.12 Weekend in Stockholm, December 2000

The objective was to get to the minimum powerful makumba to be able to implement Karamba

Solved database config files

Decided to implement types in pointers

Replication when DD changes. What happens to the LBG copies? How to detect DD and logical changes?

Dillemas about Archive and Document. We didn’t know how to organize the DDs, and how to set the access rights.

Dillemas about static (database registered) and dynamic (application dictated) access rights

Application template, components (maybe configurable by webmaster), maybe new tags, skins around them

DD-parallel code, express there all we wanted to express in the DD Business Language for now. Such classes will be called by the taglib directly (no more JSP parallel code). If a class doesn’t exist, a default one is used. If it exists, it is a sublcass of a powerful toolbox (refactored from needs and experience) class that knows

- rigts set (static, or by OQL)

- decorated (pointer) field attribute

- unique field attribute

- !unique constraint OQL expression

- computed fields

- unique and notnull to implement directly in SQL

(cristi toilet) 3 tier application: servlet container, DD-parallel code, database. Will improve development speed

- JSP calls DD-parallel methods of the kind: “do(type, operation, data) in the main tables

- DD-parallel code calls methods of the same kind in the tables and subtables, after security checks, etc.

We realised that insertUnique just comes from the unimplemented unique constraint

We have a clear idea of what an application is. Application come in bean-like components (maybe will be implemented as java beans?)

- DDs, with a field that indicate which instance of the app do we talk about (name)

- DD-parallel files, written in general

- JSPs, maybe with an argument

- Maybe a special tag, say <karamba:todolist specificArguments../>

### 4.13 After Stockholm

package name metadata –> org.makumba

## 5. Design issues

Issues are divided in current problems (5.1), short term objectives (5.2) and long term guidelines (5.3). Every such timeframe is divided in General principles, Data description, BEST (organisational/Business) Logic (BL), Java API, Presentation and Development support. Development support describes objectives for fast development and change by experts, trainees, webmasters and normal users.

The new spec will be written by Stefan and comply to 5.1 and 5.2
The long term spec will be developped based on 5.3

### 5.1 Current problems

#### 5.1.0 administrative stuff

as a testbed for new taglibs and API we can use few pages from minerva, see how it feels, what webmasters feel about it.

ANT distribution

#### 5.1.1 general principles

get the business logic in DD-parallel code

finish makumba!!!

#### 5.1.2 Data description

unique, unique constraints, list, fixed, default values…

#### 5.1.3 BL (BEST/Business Logic)

specify and implement DD-parallel code

- make use of strong context provided in the data description files and in the presentation. Making use of context leads to less initialization code, less installation overhead
- API growth by pattern discovery. When typical features are identified, they are included in the toolbox

- decoration will come by refactoring

##### Actors, administration security models

- authentication: !auth=field_name. the class of the field has to have at least one authenticate* method

- shouldn't the authentication field be always UNIQUE, FIXED, NOTNULL?

- shouldn't !admin be a method, like !title and maybe other commands... ?

- an actor has an authentication mechanism or is a decoration of it

- authenticating the extension just needs to authenticate the base
!base=class_name

- how to express in the language that 2 objects of different classes have the same auth?

#### 5.1.4 Java API

implement the Database config files with names like

localhost_mysql_makumba

malaka.best.eu.org_pgsql_minerva2

table alteration for the whole DB

There is no way of setting a field to null. There should be a Null value defined at the abstract level.

implement database catalog and full database copying

subtables (ptrOne, set*) should accept data in insert and return data in read. Data should be part of the Dictionary for ptrOne and Enumeration for sets. This looks like it can be written at the metadata.db.level

delete with deletion of internal sets and pointers...

list, set, bag API

profiling of the code once everything is implemented

change /best/student -> best.Student, relative:type from the same package in reference and include

transaction

text type

notification services

DBSV autogeneration

keep configuration in a special table

database copying starting from a type

implement title as first non-pointer field

implement full-text search

parse default values

implement LIST, FIXED, NOTNULL, UNIQUE

Database bugs:

- some db engines return Float at getObject when they read ints... Should be a cast to Number (rather then Integer), then a intValue()
- msql.Database does not create a connection pool, which will make concurrent accesses fail

Other db problems
- find other Windows development DB engine (recompile Postgres with cygwin32 ?)

metadata.abstr.printer.subtablePrinter prints the description after all the fields of the subtable. should be before

#### 5.1.5 Presentation

finish taglib

#### 5.1.6 support for development

implement RAD interface development

existing: support for pre-Makumba existing tables: changes of table and field names. nice stuff

### 5.2 Short term

#### 5.2.1 General principles

- the present Makumba code can be kept (taglibs, tables and searches can coexist or be based on it)
- maybe logging

#### 5.2.2 Data description

##### extension, include?

- non-pointer inclusion of types

##### DD language

- a well-formatted metadata file should somehow mention its name for easy citation

- classes rather than tables. objects rather than table records. this needs discussions... instances of applications, etc. See OODB ideas (Annex 6)

- Constraints, field constraints, sets are a constrained bag

- field seperator in DD files (newline too strict?)

- default values

- title as expression rather than field

##### Decoration

- if you mention an decorator object (say minerva.student) and the base object exists, the mentioned object is automatically created.

- if the base object doesn't exist, it is created first.

- editing/deleting the base object can be a default right?

- should based types be returned together with the base ?

- should the base object be somehow notified when a wrapper is created?

- Toto: Can it be recursive (i.e. decoration of decoration) ? With the current definition, I don't think so. Moreover we don't really need it, and it may be more difficult to handle...

- Toto: Can an extension be authenticated in a different way than its base ? (i.e. !authenticate)

- Toto: (see annex 3) Does /best/minerva/student inherit the !admin from /best/student implicitely (i.e. Helpdesk) ? I would say yes. And in case !admin is defined in /best/minerva/student ? Should it be the union of admins of /best/student and /best/minerva/student or only admins of /best/minerva/student ?

#### 5.2.3 BL (DD parallel)

##### Virtual sets

- virtual sets with arguments

- virtual set with arguments == search?

- virtual set called from another virtual set

- subsets of (virtual) sets, lists of (virtual) sets, etc. introduce constraints beautifully

- virtual set inside the virtual set (see johnny student)

- static virtual set

#### 5.2.4 Java API

##### API

- wrapper managers are a more general concept, should move up from database. Should be metadata.abstr.WrapperManager or so

Abstract level bugs
- if there is a redirection to a type which is redefined on a lower family, the lower family type should be obtained. Instead, the same family type is obtained now

Save OQL searches for future use (with different parameters). Maybe keep them as stored procedures

Database bugs

- at boot time, the ptrIndexJava computes the max index. This means that the index values of deleted records might be reused, which is bad for replication
- - unique id generation should use more the power of the SQL engine (now it's java level). CREATE SEQUENCE seems to be the standard way. This would also allow remembering the index value from one DB boot to another
- generation of creation date and last modification date should use TIMESTAMP facilities of the SQL engine
- before altering, tables should be backed up
- there should be some factory method that produces a pointer for each type of database

OQL

- if just a label is mentioned in where/select then the primary key field of that table is used

- parameters $number

- NPE bug if no label and no type exists, a "best" table is mistakenly considered

- best.Student is considered, should be error... or is this windows?

- unmentioned labels should not generate joins

- multiple objects &expressions in SELECT

- expressions in select and where are equivalent, so it's easy to have joins generated there

- subselects can be implemented, inheriting the labels of their containing selects. how about the SQLs that don't support subselects?

- SQL verbosity level

#### 5.2.5 presentation

- make sure that webmaster can't change the access rights of a page
- support modification in html editors?

#### 5.2.6 support for development

- not much added for programmers (maybe the new generated classes best.Student().insert(data))

### 5.3 Long term

Design will continue from here.

#### 5.3.1 general principles

- data description modification: try to detect and signal all presentation files and java files that might be affected
- java code in the data-description file, or modification of generated java file out of the description?
- problems with modification of original data description when default code generation is used at presentation and data level (and then we let the programmer/webmaster alter the code). Toto proposes class inheritance for the generated Java files (you extend the generated file, which stays untouched)
- dealing with data description changes: there will be extensive change detection (i.e. analysis that takes into account the previous version) and automatic fix of database tables, database code and presentation code (this supports automatic preservation of existing data and code)
- types of data structure change: field name change, field type change, field move in another table, ptr -> set transformation (and back?)
- transactions, table and record locking
- logging. what level? actor? db?
- XML dump and import. DTD
- data travels in XML rather than Hashtables
- allow the server to receive data described by a certain DD via email
- support for backup and versioning at db level
- multilingual support tables, enumerated type string, default values (mb also presentation layer)
- attachement conversion... what level? presentation? preprocessing and save as text or html??
- replication... keep date of last replication. transitivity. how does the data structure change support affect replication? quite much... replication of data descriptions between sites
- selective replication: a location could get replicated locally data that is more related to its people
- support again more DBs (ODBC, msql, etc)
- looks like we can't count on ODBC (LIKE, close-on-exit, different versions...). on the other hand, it's cool to be able to generate excel views of the data. we could support it partially.
- remote database??? Old description: to support remote access to a database, we need:

- some remote wrappers: metadata.remotedb.Database, metadata.remotedb.Table, etc. Such wrappers will be remote objects and their implementation will keep a reference to the normal objects (metadata.db.Database, metadata.db.Table) and call their methods when needed
- at the abstract level, the data has to be serializable
- some sites might not have the metadata files. The remote tables should be able to pass their RecordInfo (which is serializable)
- the remote and local Db and especially tables should implment some common interface so they can be used interchangeably

- need notification services for insert and update. different actors may choose to be notified when a new record is inserted or a certain record is changed. only some such records may be relevant for the respective actor, so filters may need be installed. access rights problems exist here (i.e. I may be notified only of records I am allowed to see)
- should pointers be represented as 1 or as 2 SQL longs (java ints)
- save search parameters for future refinement and use…
- relations with other buzz technologies: Java Beans, WAP

#### 5.3.2 Data description

- change parser to understand ENCRYPTED. field encryption not one way
- surely implement on-the-spot types (to replace ptrOne and setComplex)
- omit ptr keyword
- minimum and maximum char length [a..b]
- fieldname: int { default longhish_shit_that_is_not_comfortable_to_repeat, a, b, c }
- now as default for date
- database-platform independence (compilation up to stored procedures for higher performance)

#### 5.3.3 BL (BEST/Business Logic)

- actors, rights and other constraints, erros if constraints are not respected, actions (field sets, etc)
who : action (fields) precondition, operations, postcondition

- actions (understood as method adjustments) are actually expressions of DB redundancies? Toto: constraints can also be expressed in actions

- the meaning of set in argument lists and method results seems to be different from the meaning of set in sets and virtual sets

- implicit assigments might be difficult to "read"

- methods with no actor cannot be called from presentation

- static methods

- look again into best/johnny/student (see annex 2)

- looks like constraints don't have to sit in the method. but then, how do you ensure that a consistent state is kept?

- we could say "these 2 op form an atom. don't check consistency between"

- how to name decorator base when used in argument lists?

#### 5.3.4 Java API

- should loose importance but should still exist
- ODMG, OODB ideas

- optimize current drivers

- consider also introducing relationships on the database level
- Stored procedures from OQL or CREATE VIEW ?
- evaluation of the select() should be lazy. That is, the SELECT should be executed only when the hasMoreElements() or nextElement() methods of the returned enum are called
- there could be a form of Table.insert() that gets a record set (not just a record) as argument. At SQL level this could generate an INSERT INTO table SELECT ... if the recordset comes from a search _in_the_same_database_

DBSV auto generation?

support for existing tables: besides changes of table and field names, import structure from SQL tables, changes of type, automatic change of #alter file, facilities of changing types (such as a abstract-level int{..=1 ...=0} that might actually be a BIT in the existing pre-Makumba DB)

it's not clear what will happen with DBSV for an existing database. A possible solution is to consider it 0 for start (replication is not on anyway). Before the first replication, a pointer transformer can be executed on each table, to adjust all pointer fields

- redirection property files could be eliminated. each record handler should have an associated redirector class, that has methods for each redirected type, and a getFamillyName method. Due to the getFieldMethods(), each handler family has to have a FieldHandler (i.e. cannot inherit it). That's wrong (e.g. metadata.sql.odbc and metadata.sql.msql don't need FieldManagers, they are just empty extensions of metadata.sql.FieldManager)

#### 5.3.5 presentation

- XSL is not webmaster-friendly, but XML+XSL= presentation. Maybe XSL should be generated out of our own presentation language
- we don't want access rights (field level and actor) in the presentation
- database-to-presentation tools that provide default views can be found in:
- IDEs
- database management applications (Access, Oracle 8i)
we should look at those, just the same as we looked at LDML and CFML
BUT none of these are easily accessible to the BEST webmaster and BEST programmer on the BEST platform and server in the distributed BEST environment. any of them can change and the system should survive
- need to check enhydra.org http://www.enhydra.org/software/enhydra/index.html
- support modification in html editors
- interfaceLanguage in actor profile. how should it work?
- how should the content in the web pages be localized?
- %localize(lang1, lang2) at pres level
- think of other kinds of presentations: AWT, RTF, XML, WAP, etc

#### 5.3.6 support for development

- the moment you have the profile of an action, you can make a default JSP page that uses it. that page already has access rights on it
- installation: one can automatically install ODBC data sources with odbcconf.exe
- (html and view source) is an epitome of low installation and initialisation overhead
- provide for trial and error experimentation
- maybe we need to look a bit of how webmasters work (see the alttabing stuff in the cfml tutorial)
- pay attention to errors, where should they come up and how. should not be hidden in the source
- manual, reference manual, functional sample applications (FAQ, support forum, discussion, to-do, calendar, system db, bug db, see mail about small app ideas)

- wizard (sounds too professional)
1 new metadata file
2 links to generated presentation and business logic file
3 the generated files contain comments with explanations, suggestions and links to tutorial. the form and comments of generated code can be adjusted by setting a "level of expertise"
4 the programmer can edit the default presentation and business logic
- in the general case, altering should be done by ALTER TABLE, though a convenience java alter should be provided

### 5.4 Administrative to-do

- GPL needs be checked in connection with the antlr OQL grammmar we use

## Acknowledgements

Thanks to BEST for trusting us to come up with something better than Lotus Notes.

Thanks to LBGs in Athens (summer 1999), Chania (autumn 1999), Kosice (summer 2000) and Coimbra (autumn 2000) for their kind hosting support.

Thanks to Yngve for supporting our web-programming ideas (since JML in 1996) which were not on the research agenda and even employing supplementary personnel for their development.

## Annexes

Annexes present code snippets that might still be useful for future development

### Annex: Makumba philosophy mail

Date: Thu, 31 Aug 2000 19:05:26 +0200
From: Cristian Bogdan <cristi AT nada.kth.se>

1. common technology:
- our data is fragmented across more technologies. In Notes, data is fragmented across more databases, because we don't master the technology or simply because Notes sucks.
- we need total control over the technology or a very good customer service from whoever provides it
- in order to achieve a low learning (and maintenance, if we do it) treshold, it has to be simple, i.e. have as few features as possible, the minimum necessary for us
- it should be portable to any platform our low-budget needs will get us to

2. what is the nature of our applications?
- we have web database applications.
- thus the technology should have ready-made building blocks oriented to fast generation of such applications
- it should also provide the power for webmasters to adjust the pages to the lowest detail
- same goes for programmers

3. how do our applications evolve?
- we rarely do ready-made applications. normally we have the minimum necessary to gather some data. this ensures early availability of a working application, without much efforts from or pressure on the very-specialised-people (this corresponds to data description files)
- then (when programmers have time or pressure) we add more and more logic, or provide ways to enter more data. (this corresponds to business logic level)
- in the meantime, webmasters (when they have time or when they are pressed) add more and more layout (this corresponds to presentation level)

4. problems with commercial technologies
- they usually contain loads of functions to make for the needs of all big customers
- ... and to obtain lots of speed for very specialised tasks
- if the technology is commercial we should be able to obtain not only the technology itself but also its customer support and make sure we can provide for its maintenance overhead
- commercial would usually tie us to a vendor. BEST breaking relations with the vendor results in problems. ITC starting to hate the vendor gives other problems
- commercial might tie to a platform or a set of platforms and porting to a new platform is independent on us

5. "external" technologies (open source or commercial):
- they are used on more than one place so they can be more reliable
- there is a dedicated maintenance crew
- if we can't find any external, we can get to both these advantages by
i) making a good design
ii) implementing it
iii) open-sourcing and publishing it in good places (i.e. places we looked for one and didn't find :)

### Annex: Presentation language as of autumn 1999

    <h1>All students:</h1>
    $/best/student
    [
        Make a <a href=$newLink>new</a> student or check this list:<p>
         {
                this is a student:<br>
                name is $(name)[$value] <b>$(surname)[$value]</b><br>
                <font size=-2>
                <a href=$editLink>edit</a>
                <a href=$deleteLink>delete</a>
                </font><br>
                $(birthdate) [<font size=-2>$value</font><p>]
                $$[<b>$description</b>:$value <br>]
        }(<br>)
    ]

### Annex: First proposal of JSP taglib presentation. Will go to API Spec

    <% // give a name to the data to be formatted (there might be better ways to do this...) %>
    <%pageContext.setAttribute("std1", someDictionaryFromTheDatabase);%>

    <%// I want to format a student for editing, this should open a <form> %>
    <mtd:record
       structure="/best/student"
       data="std1"
       submitMethod="editMyself"
       afterSubmit="index.jsp"
       success="<font color=green>Changes done!</font>"
       failure="<font color=red>Changes not done:  $error</font>"
    >
         <%// I want to format some fields, separated by <br> %>
         <mtd:fields names="name,surname,birthdate" separator="<br>">
              <%// field description in bold, then :, then value in a form input%>
              <b><mtd:fieldDescription/></b>:
                 <mtd:fieldEditValue/>
         </mtd:fields>

         <%// now I format other fields, a bit shorter but with the same result, the fields are displayed for reading ($value), not as html form fields ($editValue) %>

        <mtd:fields names="military, etc" pattern="<b>$descr</b>:$value" separator="<br>" />

        <%//format the addresses for reading %>
        <h2>Addresses</h2>
        <dl>
        <mtd:set name="languages" separator="">

           <%// the address kind as a <dt> %>
           <dt><b><mtd:fields name="kind" pattern="$value" /></b>

           <%// all unmentioned fields, i.e. all but "kind" , as <dd>%>
           <dd>
           <mtd:fields names="$rest" pattern="$name:$value" separator="<br>"/>

        </mtd:set>
        </dl>

    <%// ok, that's it for this student, this should close the <form> %>
    </mtd:record>

### Annex: Toto's BL sources after Kosice

    -----------/best/auth------------------
    username : UNIQUE FIXED char[1..40] ; EN=username
    password : ENCRYPTED char[4..40] ; EN=password
    code1 : int ; first 5 digits
    code2 : int ; last 5 digits

    <authenticate>
    <arguments>
    username
    password
    </arguments>
    <result>
    result : /best/auth (
    result .username==username
    result .password==password
    )
    </result>
    </authenticate>

    <authenticateByCode>
    <arguments>
    code1
    code2
    </arguments>
    <result>
    result : /best/auth (
    result .code1==code1
    result .code2==code2
    )
    </result>
    </authenticate>

    <new>
    <arguments>
    username
    password
    </arguments>
    <body>
    </body>
    </new>

    <edit>
    <actor>
    this
    </actor>
    <arguments>
    password
    password2 : char[4..40] ; confirm password
    </arguments>
    <body>
    password==password2 ; wrong password
    </body>
    </edit>

    ---------/best/student----------
    !authenticate=auth
    !title=auth.username

    auth : UNIQUE FIXED NOTNULL /best/auth

    firstname : char[1..40] ; EN=firstname
    lastname : char[1..40] ; EN=lastname

    address : set
    {
    kind : UNIQUE char EN={"home addres","study-time address"}
    street : char[80]
    city : char[80]
    counbtry : /general/country
    }

    <new>
    <actor>
    anonymous
    </actor>
    <arguments>
    firstname
    lastname
    password
    password2 : char[4..40] ; confirm password
    </arguments>
    <body>
    password==password2 ; wrong password
    auth.new(firstname+" "+lastname,password)
    </body>
    </new>
    -------------/best/minerva/student-------------
    !base=/best/student
    !admin=set /best/minerva/admin

    restriction=set /best/minerva/company ; companies not allowed to see my data

    <edit>
    <actor>
    this
    </actor>
    <arguments>
    restriction
    base
    </arguments>
    <body>
    </body>
    </edit>

    <view>
    <actor>
    set /best/minerva/company
    </actor>
    <arguments>
    base
    </arguments>
    <body>
    !restriction.contains(actor)
    </body>
    </view>

    -------------/best/minerva/admin-------------
    !authenticate=auth

    auth : UNIQUE FIXED NOTNULL /best/auth

    -------------/best/minerva/company-----------
    !authenticate=set /best/minerva/company.representative (authenticate.company==this)

    name : UNIQUE FIXED char[1..40]

    <search>
    <actor>
    this
    </actor>
    <arguments>
    languages : set /general/language
    interests : set /general/interest
    universities : set /general/university
    </arguments>
    <body>
    actor.paid() ; You are not allowed to search for student : pay the entrance fee first...
    </body>
    <result>
    result : set /best/minerva/student (
    !(restriction.contains(this)
    result.student.interest.intersection(interests).size>0
    result.student.studies.university.inter(universities).size>0
    )
    </result>
    </search>

    ----------------/best/minerva/company.representative-------------
    !authenticate=auth
    !admin=set /best/minerva/admin

    auth : UNIQUE FIXED NOTNULL /best/auth
    company : FIXED NOTNULL /best/minerva/company

    ----------------/best/johnny/admin-------------------------------
    !authenticate=auth

    auth : UNIQUE FIXED NOTNULL /best/auth

    ---------------/best/johnny/student-----------------------------
    !admin=/best/johnny/admin
    !base=/best/student

    season : set /best/johnny/season (
      applic : set /best/johnny/application (
      applic.student==this AND applic.season==season
      )
      applic.size()>0
    )

    applications(season : /best/johnny/season) : set /best/student/application (
      applic.student==this AND applic.season==season
    )

    -------------------/best/johnny/season----------------------
    activities : set /best/johnny/activity (activities.season==this)
    announcementDL : date
    applicationDL : date
    rankingDL : date

    optimise()

    ----------------/best/johnny/activity----------------------
    applications : set /best/johnny/application (applications.activity==this)
    ranking : list /best/johnny/application
    rankingPlaces : int ; number of accepted people
    season : FIXED NOTNULL /best/johnny/season

    ----------------/best/johnny/application------------------
    activity : FIXED NOTNULL /best/johnny/activity
    student : FIXED NOTNULL /best/johnny/student
    extraAccepted : date

    season : /best/johnny/season (season==activity.season)

    <new>
    <actor>
    set /best/johnny/student
    </actor>
    <arguments>
    activity
    </arguments>
    <body>
    season.applicationDL.after.now ; Too early to apply
    season.rankingDL.before.now ; Too late to apply
    student=actor
    student.applications.append(this)
    </body>
    </new>

    ---------------------minerva.student.register.jsp------------
    <metadata:new type="/best/student" response="registerResponse.jsp">
    <! register creates the student ; it has a response form...>

    --------------------minerva.student.login.jsp-----------------
    <metadata:login type="best/minerva/student">

    <! we could have different kinds of login :
    <metadata:login type="/best/johnny/student" type="http" method="authenticateByCode">

    --------------------minerva.student.index.jsp----------------
    <metadata:actor type="/best/minerva/student" login="login.jsp">
    <! by default, login="login.jsp">
    <! if the /best/student exists but not /best/minerva/student, there should be a "base-extension" notification (specified in login.jsp)>

    <metadata:view>

    ------------------minerva.company.student.jsp--------------
    <metadata:actor type="/best/minerva/company">

    <metadata:retrieveObject type="/best/minerva/student" name="student">
    <! retrieve the student from the http arguments>

    <metadata:view object="student">

### Annex 4. Thoughts about the generated stuff and its dynamix (Coimbra)

DataDescription generates and compiles JavaFile
ModifiedJavaFile extends JavaFile
Presentation calls ModifiedJavaFile

DataDescription generates Presentation

GeneratedPresentation is insert, edit, delete, view-all pages

GeneratedPresentation is generic -> the only reference to DataDescription is the name

OR: GeneratedPresentation is not generic, it refers DataDescription fields, especially title, pointer and set fields. it is fully commented

Problems:
1. what if somebody wants to use another structure than IEDVA? what if somebody wants different filenames?

URL is part of the presentation so it can be changed creative
two different pages might do essentially the same IEDVA operation

immediate answers
- tutorial and scrap areas
- default area (if stuff is to be changed it is copied in other areas)

2. What happens to the presentation when DD changes?
professional solution: semantic analysis of Makumba tags in Presentation

low profile solution: identify the pages that refer to the DD and list them

### Annex 5. Thoughts about how to present suggestions to novices in generated code

`/* name is the title fields, so we format it differently */`
`<b> <mtd:field name="name"></b>`

`/* we put the title first, it is the field "name", the first non-pointer field in the data description, be aware that if you change the DD, you might change the tilte field*/`
`<b> <mtd:title> </b>`

novice-expert modes for comment level

`/* we could have done all this by saying <mtd:record>. this would use all the above defaults */`

### Annex 6. OODB proposal (cristi, Feb 2000)

needs review

#### Object DB update

Unlike classical relational DB operations (based on insertion, update and deletion into tables), object repositories are based on writing down in the database all changes that occur to the fields of the objects involved in a transaction. Instead of thinking in terms of tables, the programmer can then only consider the objects involved, and the transactions necessary for their state to be consistent.

Metadata already has concepts from object databases, including pointers, sets, etc, and already defines a database scheme, which makes it easy for it to support a more object-like update mechanism.

Metadata can actually be brought closer to Object database management and ODMG standards by:

- supporting inheritance (easy, not urgent)
- supporting method declaration (and mapping it to ODL) (difficult, not urgent)
- specifying how the metadata definition maps to ODL (object definition language) (not urgent)
- using ODMG's OQL or a mapping of it to specify queries (quite urgent)

If APIs look like below, metadata/db can be implemented using any ODMG (or later JDO) implementation (e.g. Sun's Java Blend already implementsODMG, we could try to see if we can't implement metadata/db with it already, there is a 30 day trial).

To get faster access to fields, classes can be generated automatically out of the metadata files. In order not to need JVM changes or class file postprocession, field changes/reads can be accomodated with methods that follow the Java Beans property name patterns: setXXX(Object) and Object getXXX(), where XXX is the field name. This patten can co-exist with the present object-Dictionary approach.

#### Proposal

All records are represented in memory as DBObjects (can extend Dictionary). To pass a pointer to an object, one just has to pass that DBObject. Such DBObjects can be registered into the Database (which will lead to an INSERT), and some of them can be given special names. Queries made in only one table return collections of DBObjects, and pointer and set fields will come as DBObjects.

DBObject methods for reading :

- Object Object get(field)
- Dictionary get() returns a clone, without the special fields
- Timestamp getLastModificationDate()
- Timestamp getCreationDate()

Fields are returned as:

- their normal data for int, char, text, date
- other DBObjects for pointers
- DBSets (Collection?) for sets, with specific operations for the respective type of set

Modifications (incl insertion) and destructions are performed inside transactions. Every thread is part of at most one transaction. A thread can join an already existing transaction. If a write operation is invoked, but no transaction is associated with the thread, a transaction consisting of only that operation will be performed.

Write operations will cause one INSERT or UPDATE for each affected object inside a transaction. INSERT is performed on an object with no pointer (and leads to setting that pointer), and UPDATE otherwise.

- put(field, value)
- put(Dictionary) updates only the mentioned fields

Object destruction: DBObject has a delete() method. All ptrOnes, setInt, setChar, setComplex will be deleted (in the same transaction).

A transaction can lock certain DBObjects for reading or writing. A transaction can be cancelled (abort, nothing is written to the database), or declared finished (commit, all changes are written). Also, the transaction can finish and re-start immediately, without releasing any locks.

#### Implementation considerations

At JDBC level, the Database will employ one Connection per transaction. Connections can of course be recycled.

The database has a cache, having the composite key [RecordInfo, ptr]. The cache contains data from the respective table (eventually in an array Object[] for each record).

The DBObject contains just a reference to the cache entry where its data stays. When the DBObject goes out of scope, the cache is notified that the respective entry has one less usage. A cache cleaner passes regularly and deletes cache entries which are not used.

To achieve this cache cleaning, based on the DBObjects going out of scope, one has to build a DBObject instance whenever a query result is returned, or pointer de-reference is made. That means that two DBObjects can represent the same thing, so changing a DBObject might affect another DBObject. To test if two DBObjects represent the same record in the database we can have a method such as

- boolean isIndenticalWith(DBObject)

To avoid this, one has to use Java 2 (JDK 1.2) weak references. Using java 2 will also bring more standardization with the Collection classes. But for that, we need to upgrade to Java Web Server 2.0 (which might solve other problems we have now, such as page recompilation), currently Johnny and Minerva use Java Web Server 1.1.

Locking can be implemented at Java level or at database level.

For many of the classes (Database, Transaction) we could directly implement the ODMG interfaces. This will create a ODMG-like "metadata binding", which will give a standard value to metadata.

OODB discussion (Cristi and Stefan in Technico). This discussion probably came from the data-code relationship... In normal DBs, all objects of a type are accessible by a simple select. In OOP, objects are accessed thru references. This looks strange. Proposal to implement data inheritance with 2 extra columns (one table name and one pointer) in each table that points to a record containing the extension columns in separate extension tables. For example, a committee might request an extension to their archive, which in principle will behave like all other archives. This leads to thoughts about application inheritance, etc. In an OODB approach, viewing a table actually would view one single virtual set.

### Annex 7. JDBC type coding

useful for development

BIGINT BINARY BIT CHAR DATE DECIMAL DOUBLE FLOAT INTEGER LONGVARBINARY LONGVARCHAR NULL NUMERIC OTHER REAL SMALLINT TIME TIMESTAMP TINYINT VARBINARY VARCHAR | -5 -2 -7 1 91 3 8 6 4 -4 -1 0 2 1111 7 5 92 93 -6 -3 12

### Annex 10: initial presentation layer

constructor RecordFormatter(RecordInfo) | initializes the formatter from a given metadata
void writeInPage(Writer stream, Dictionary data) | writes a view of the given data to the given stream
void writeInForm(Writer stream, Dictionary data) | writes an edit form of the given data to the given stream
void writeInFormNew(Writer stream) | writes an empty form (with default values) to the given stream
Dictionary readFromQueryString(Dictionary query) | reads the field from the http query The query object can always be constructed as a Dic from any query string (even with multiple values per field)

adjustment methods (less urgent):

- formatting: adjustField(fieldname, "<b>$description:</b> $value") etc
- field order change: moveBefore(field1, field2), etc
- field hiding/keeping: hideField(fieldname)

later:

void writeInFormSearch(Writer stream) | writes a search form Still to be thought out (depending on how searches will be represented internally)
void writeInFormFormat(Writer stream) | opens the formatter for editing to a non-tech user, as a edit form
void writeInPageFormat(Writer stream) | pens the formatter for editing to a non-tech user, as a page

- according to the RecordHandler conventions, the writeIn* methods can be implemented as RecordHandler.writeAll (which RecordFormater will inherit) if each field type is implemented as a metadata.html.FieldFormatter (having more or less the same methods as above). Thus, there will be intFormatter, intEnumFormatter, charFormatter, charEnumFormatter, dateFormatter, etc
- we should think of cool demos

#### issues and dilemmas

- fast development: besides having html.RecordFormatter, one should be able to get html very fast once metadata files URLs links like

/best/student/$edit
/best/student/$view
/best/student/$new
/best/student/$list

are a tempting way to go. Then one just has to pass a pointer (for $view and $edit) to get results, which should be the defaul for $list views. A generic servlet interpreting this (reading the path to the metadata from its url path, reading the pointer and the form data from the query string, etc) can handle all cases needed, generating automatically some jhtml/jsp source for each case, which will then be adjusted by the webmaster.

Problems:

- one has to provide several such addresses for one metadata, one for each actor or task. Probably something like

/organiser$/best/student/$view
then organiser could be a search in the auth and student tables that would give a non-empty result if the respective is an organiser

- how does a non-technical user adjust the formatter (change field order, hide fields, etc)?
- a easy-to-do way, but quite technical, is a fo rmat() method in the servlet (jhtml, jsp, etc), that would be called once, at servlet initialization. That method would call various methods of the default formatter (for the respective RecordInfo) to adjust it. Then the servlet service code (or the jsp/jhtml page) will just call

getFormatter().format(getData())

where getFormatter returns the formatter adjusted in the format() method, and format() returns the data identified by the pointer (for $edit and $view) or the bunch of data selected by $list.

- a not that easy to-do, but very user friendly is a html form that would open for URLs like

/best/student/$view$format

- a more webmaster-like technique: we can generate jhtml/jsp containing formatting calls and let the site owner format it. That has some problems:
- how/where to format new fields?
- how to format fields uniformly?

- search forms can of course be generated from the metadata info. In the structure above, one would have /best/student/$search
- passing pointers via CGI: sv=xx&uid=yy can be too obvious, a pointer registry can exist, where all Records are registered by hashCode, then the hashCode is passed and used as a password to get the Record back
- in general, one can think about encoding the URL-passed data, at a later stage
- logging at http level?

### Annex: JSP-Parallel java code

This proposal was made for a fast implementation process and to give a well-known place for the java code.

Every JSP page that uses the Makumba taglibs will create a parallel Java class per operation invoked. The first page execution will call default implementations of the methods in the class and may announce the developer where to go to edit the code. The "pointer" to the code can stay in the generated HTML as a comment. It is intended that when the parallel code changes, the JSP is touched and re-compiled automatically. The JSP will call dedicated callbacks in the parallel class at different moments in the execution as illustrated below. References to the page and to the "brother classes" generated by the page will be available.

For example, a typical edit form will invoke the operations "writeInForm" and "edit" (since there are no response pages) So it will generate two parallel classes. Still, in such a case (presentation-response), one might have a single class dealing with the whole transaction, since the two are logically linked.

Still need to consider
- the webmaster-security of the solution
- the mechanisms by which the objects from the parallel class are created and referenced
- our last dilemma in Coimbra: should the access rights and field access rights be expressed in JSP-parallel or in DD-parallel (one java class per Data description, i.e. Makumba file) java code?

- The argument for the DD-parallel is that access rights should be changed in only one place. One counter-argument is that if the access rights changes, the JSP pages that count on them have to be changed anyway
- The JSP-parallel pro is that they seem more practical. Only one (rather than two) set of java classes is needed, which makes clear where the application code is. On the other hand, it's not clear what happens with doing the operation on a sub-table (set or pointer). One way to do it is to generate a parallel-class for that operation as well.
- the DD-parallel can be replaced by a more powerful DD grammar
- instead of changing the access rights as such, one can change the definition of the actor who has the respective rights.
- We should probably play with both and see what's more practical for BEST and Karamba purposes. We should define the Karamba applications (the present state and what we suspect for the new state) and see what we need.

- the clear definition of the methods. Below are some guidlines, but the final interface will come out per operation (view, view in form, edit, delete, insert) after some practice.

method signature | semantics | default implementation
Object getInitialState() | returns the initial state of an operation implementation. the state info will be passed to all methods of the template method | returns null
boolean isAuthorised(Pointer auth, Dictionary args, Object state) | gets a pointer in an auth table detected by some http way returns whether the authenticated principal can be authorized as an actor that can do the respective operation on the data | if you are in the JSP root, return true otherwise try to authenticate an actor with the type indicated by the first directory. how that dir indicates the type is left open for now
Object getData(Pointer auth, Dictionary args, Object state) | gets the data to view, view in form (pre-edit, pre-insert), insert, or edit This is the typical place to do an OQL search. Having it here will also allow for ordering and other presentation-related issues. | one http arg is a pointer, data is retrieved from the main table using it
boolean beforeExcecute(Object state) | Executes code before the operation (insert, delete, edit) is executed. Returns whether the operation should be executed or not. | empty
void afterExecute(Object state) | similar to above | empty
boolean needTransaction(Object state) | tells whether the operation and the rest should be wrapped in a transaction | returns false
boolean canView( String field) boolean canEdit(String field) | field access level. not clear if it will be here or not. | returns true

### Annex: Former explanation of folder structure... Should go to some tutorial

    general/general_table1 (ex all countries, all languages)
           /general_table2 ...
    organization1/major_table1  (ex best/student)
                 /major_table2 ...
                 /application1/table1 ...  (ex best/minerva/company)
                              /table2 ...
                 /application2/table1 ...
    organization2/major_table1 ...
    common_application/table1 ...  (ex eventcalendar/event)

### Annex: BEST Data Definitions hierarchy

This is what the Stockholm meeting had in mind for BEST hierarchy. Object orientation and decoration were assumed

    Auth
    Student
    best.Student decorates Student

    Group
    best.Lbg extends Group
    best.Committee extends Group
    best.vivaldi.Student decorates Student
    best.vivaldi.Season
    best.vivaldi.Activity
    best.vivaldi.Application
    best.minerva.Student decorates Student

    Company
    best.minerva.Company decorates Company
    best.minerva.CompanyRepresentative
    best.karamba.Document
    best.karamba.Message
    best.karamba.Repository
    best.karamba.OfficialArchive extends Repository
    best.karamba.CommitteeArchive extends Repository
    best.karamba.TigroBlabla extends CommitteeArchive