Makumba design

Statement of purpose

What do we design makumba for?

(should be put in sentences and posted on www.makumba.org)

fast member involvement
rapid development
based on members' HTML and SQL knowledge
unlike popular scripting languages, it still enforces a clean model-view-controller (MVC) architecture, leading to a healthy structure of the applications even as members come and go
db-driven web applications (clear delimitation of the technology)
simple (few concepts)
source code should be clear and easy readable to ensure efficient learning by example for new members
easy to install (few commands/directories/files to work with)
independent of any commercial technology or data format (easy to migrate to other technologies that can access databases through SQL)
as much as possible "compiled" rather than interpreted (tell the developer at first page execution that somthing is rotten, rather than wait for it to blow in production)
personal member development towards java, MVC, http, JSP and other technologies involved

The issue of design

the biggest issue with makumba today is design.
reasons:

design is originality: if makumba design weren't original, there would be no need for makumba, sth else could be used
there was a lot of pressure to stop design and start implementation, so some parts of makumba are still poorly designed (while others give its strength and originality)
a lot of stuff had to be implemented from scratch as the standard or 3rd party technologies at the time (1999-2001) were non-existent, in bad shape, or uncertain. Even the ubiquitous Tomcat did not exist at the time, and JSP was in its conceiving stages

Objectives of makumba design work

we should adjust the makumba design so that:

keeps its strength and originality as expressed in the statement of purpose above
makes better use of 3rd party technology (especially java-endorsed standards), taking advantage of their healthy design, reliability and 'standardness'.
in components where makumba is still considered more original, or 3rd party technologies are still unsatisfactory, publish the respective part of makumba as a separate, original product
this will lead to a smaller, easier to maintain, and more reliable makumba.
eliminating sluggish design will make us more proud and ready to better promote makumba in the open-source world

as such, what we need to do now, is search literature and read, not write code, except where design is straight-forward and experience from karamba and other applications clearly shows that a feature is needed, and can't wait much for redesign.

3rd party technologies

how to look at a technology:

is it used by makumba already? if so, does makumba use it to its full potential (e.g. it may be that makumba adopted the technology early in its development)
if not, what would it do for makumba, in which area?
try out the technology. install it and do an example
is it a standard? would it be good for best members' personal development to learn it?
would the overhead (in installation/configuration, data conversion, etc) of using that technology be acceptable? would the development process (makumba usage) be easier? even if it would be a bit more difficult, are the other advantages important enough?
in general, how will makumba+that technology respect the statement of purpose?

technologies used by makumba today, and their versions

Java of course. the latest 1.5 version brings a lot of useful stuff, e.g. for business logic code (e.g. using int and Integer interchangeably i.e. autoboxing)
antlr: used to parse OQL. transparent for the makumba user. We went through antlr 2.7.1 and 2.7.2. only 2.7.2 works with Jikes 1.18 (see below)
Junit: used to test makumba. transparent for the makumba user
JSP: used to implement the taglib. mostly 1.1 is used (though 2.0 brings lots of useful stuff for the makumba user)
servlet: used to filter requests (the mak controller), display errors, etc. servlet 2.2 was not supporting filters, we introduced that with servlet 2.3
both JSP and servlet need a servlet container, we use Tomcat for that. it's not always transparent for users. We went through Tomcat 3, 4 and 4.1. Tomcat 5 is now up, supporting JSP 2.0 and servlet 2.4

Jasper (the JSP compiler) is an important part of Tomcat. Jasper 2 was introduced with Tomcat 4.1. generally transparent for the user as long as makumba is not producing errors

Jikes is a fast java compiler, used to compile Java code generated by Jasper (and to compile makumba code itself). Jikes is fully interchangeable with Java's normal javac compiler, but it's faster. only Jikes 1.18 works with Jasper 2 due to the UTF character coding. transparent for the user, only needs installation

JDBC: to connect to databases. makumba hides a lot of its details. JDBC's original goal (version 1) was to be able to connect to a database at all (i.e. try to support as many as possible, not necessarily in an uniform manner). Database neutrality is more important for JDBC 2 and 3. These versions also provide features that may speed up makumba internals.
DB engines. in general transparent for the user, as part of DB neutrality

MySQL is the main DB engine supported by makumba. we went from version 3.23 to version 4.0, then we started using InnoDB tables instead of MyISAM tables. MaxDB tables seem also to be interesting for some fatures. MySQL 4 (4.1?) brings interesting features like subqueries, multi-table updates.
other DB engines that makumba was tested with: Informix, DB2, Cloudscape (recently open-sourced by IBM), partly ODBCJet on windows
mysql-connectorJ, the JDBC driver used by makumba, presently used version is 3.0.7, which is compatible with JDBC 3.0, there are more advanced versions of it (3.0.14, 3.1), supporting pretty recent JDBC features.

Ant: used to build and publish makumba. mostly transparent for the user as far as makumba is concerned (Ant is also used in karamba, but that's another story). We went from Ant 1.3 to 1.6, increasingly using its new features

Makumba components and their design status

mak:list engine

design status:

this is clearly the most original and powerful part of makumba.
familiarity with SQL within HTML is well encouraged. groups of mak:lists are nicely combined in a high-performance original way
$attributes are transformed into $1 OQL attributes in an easy transparent manner
easy nesting of mak:lists and composing queries from them. this seems to be very original to makumba

design still not finished for

parametrized query fragments (especially in orderBy, etc). what should be allowed? should such parameters (attributes) be expressed differently than nomral $attributes which become normal OQL query $1 parameters? should they be declared?
an important application of the parametrization mentioned above is search. there is little support for search forms today

implementation status:

the JSP taglib API is still not enough for makumba's needs. makumba-specific JSP page parsing needed to be introduced in the last taglib version
there are small features missing but the new architecture of org.makumba.view.jsptaglib allows for easy extensions
the current implementation/architecture is very complex. maybe JSP design should be actively lobbied? or should the ambitions be reduced? maybe more research on JSP 2.0 (especially "simple tags") can help solving this?

3rd-party tools and standards:

of course many tag libraries (incl the standard JSTL) know how to do SQL queries or display tables (JSF) but none combine the two like mak:list does (for better or worse)
the taglib was designed for JSP 1.1, since then JSP 1.2 and especially 2.0 added more features that, integrated in makumba, may simplify makumba pages
especially the Expression Language (EL) of JSP 2.0 makes it much easier to display values by using ${expression } notation
JSTL helps replace java scriptlets. makumba should be integrated more with it

forms

design status:

keep nice familiarity with HTML, while taking good advantage of the MDD information. this is a strength of makumba design
multiple forms are powerful and original

design still not finished for

form annotations
expanding form interaction over more pages
chooser flexibility

implementation status

see unfinished design above
form responders are lost at webapp reload

3rd-party tools and standards

JSF (Java server faces) has excellent design towards transforming web interaction into component-based interaction. however, that goes too far into breaking the familiarity with HTML forms, while bringing in a much less spread familiarity (GUI component programming)

MDD (data model)

design status

a simple, easy to read, database-neutral data description.
a small number of types, that cover most needs. more types would bring better performance, but create more confusion
some concepts could have better names (more related to db theory): set, internal set/pointer, etc

design still not finished for

the initial ambition was to be object-oriented: encapsulation is more or less done, but inheritance and polymorphism are still not addressed. is this really needed?
subtyping (a database theory concept) is supported but can be made more explicit
business rules. a data model should not only describe data, but also specify rules on how it can be created and changed

implementation status

the parser is a bit messy but it works
multiple-field uniqueness still not done

3rd-party tools and standards

there is acute need for database theory reading (to decide whether to have it object-oriented or keep it relational, etc)
JDO supports (in Java standard, there are other tools like it) an object-oriented way to describe data and deal with it, and make objects persistent

Database layer

design status

a pretty straightforward API for accessing MDD-generated tables in a database-neutral manner
the Dictionary usage for data arguments is a bit heavy (needs casts, is not type-safe, etc)
maybe Database should be really called Transaction, or Connection.
maps MDDs to DB tables. when reading a non-makumba DB this is especially useful
maps makumba types on their best representation in the host DB engine
the API for sending arguments to queries and retrieving their results using Dictionaries seems to be pretty heavy. Besides, it is not type-safe.

design still not finished for

uncertainty about object-orientation (see MDD)

implementation status

old, trusted code, using JDBC 1
there is a discussion on whether to use foreign keys or not
presently indexes are created for almost every field. this results in the SQL engine not knowing which to use

3rd-party tools and standards

database-neutrality made lots of progress in the past years. JDO, JDBC Connector, even JDBC itself do much better then they used to. dropping the db layer for one of these technologies could be very beneficial
the JDBC API (currently at version 3 and going towards version 4) made lots of progress since JDBC 1, most probably there are loads of features that makumba does not take advantage of, including e.g. Rowsets

OQL (object query language)

design status

specify queries in a database-neutral manner. but this can increasingly be done in simple, standard SQL.
OQL is also used to translate from MDD names to DB table names in queries (by calling the DB layer)
we borrowed OQL from the now defunct ODMG. the initial version of their OQL grammar is still on the makumba CVS.
the ambition was to be object-oriented (see MDD)
an important feature of makumba OQL that distinguishes it from SQL is the ability to refer sets easily (FROM general.Person p, p.addresses a ... instead of FROM general.Person p, general.Person->addresses a WHERE a.person=p, though the second form should also work)
another important feature is the ability to analyze queries, including type checking, useful during page analysis. even if we adopt a thrid party technology, some form of query parsing will still be needed.

design still not finished for

functions. they should probably keep the object oriented manner (object.call(arguments) )

implementation status

subqueries are badly missed
patchy, hard to add new features. one should start fresh from the original OQL grammar and reimplement it to add functions and subqueries

3rd-party tools and standards

database-neutrality made lots of progress in the past years. JDO (which has its own form of OQL), JDBC Connector, even JDBC itself do much better then they used to. dropping or adapting OQL for one of these technologies could be very beneficial. JDO is explicitely recommended by ODMG as the natural place to go

Login

design status

based on "secret attributes".
the design was so poor that it had to be patched with "checkAttributes".
the design led to a number of problems, including secret attributes with identical names in different parts of the site
the design also hides the concept of session, which is important to learn (and use) for developers. sessions can be used for more than login

3rd-party tools

the servlet speficification itself has a good login design, and that standard login (and the design of its integration in makumba) should be looked at first
there are other tools like jSai that seem to work in this department

"Business Logic"

design status

bad design, mixture between logic and controller.
thus the "clean MVC" part of the statement of purpose is still not well addressed
it's clear that business rules should live with the data, maybe in MDD files, maybe near them.
parts that are more dependent on the interface (on the way the form was shown, etc) should be part of the controller

design still not finished for

i think this part should be completely restructured.
stuff that does pure data processing should move to "data model"
stuff that does data adaptation from and to the dialogs (forms) should go to the controller
sessions should be visible to the model, the controller, or both

3rd-party tools

Jakarta Struts, JSF, EJB (e.g. for session design)

Controller

design status:

transparent for now. executes at every page access, via a servlet filter

design still not finished for:

see "business logic" above: stuff that's specific to the form should be specified in/near the JSP and executed by the controller
not able to show different pages after form submission ("view selection")
in general, not enough flexibility in treating the errors

3rd-party tools:

Jakarta Struts, JSF

Developer tools

design status:

pretty original, building on the originality of MDD and JSP parts

design still not finished for:

interacting more with the programming during development

implementation status:

"page summary" is still not implemented, although the recent page analysis code makes it much easier
"pages and BL classes affected by an MDD" still not implemented, although most of the tools are in place (page analysis, etc)

3rd-party tools

Ant
Eclipse

Templates/Components

design status:

none done
templating was really desired from the very begining of makumba.
it becomes more and more important as it seems that "dynamic website components" become pretty important on the net today, e.g. in CMS systems
included mechanisms can be a way to support templates. the current support for include mechanisms (jsp:include and @include) do not cross mak:list contexts. you can't have a mak:list and include something within it which would

be dynamic in name (i.e. decide what to include at runtime)
know its context (i.e. know that it's included in a certain type of mak:list)

JSP 2.0 supports the definition of new tags in JSP files (tag files) instead of Java classes. This can also support templating

3rd-party tools

JSP 2.0 tag files
PHPNuke as an example

Internal architecture design

design status:

this is not a very important part of design (is not affecting the user) but it is important for sustaining makumba
handler families seem to be problematic to understand and maintain
there are automatic tests for the most critical part (database layer) and for others (MDD, etc) but they are badly missed for the taglib.