Makumba design

• Statement of purpose
• The issue of design
• Objectives of design work
• 3rd party technologies

• The mak:list engine
• Forms
• MDD
• Database layer
• OQL
• Login/Access control
  
• Business logic
  
• Controller
• Templates
• Developer tools and documentation
• Internal architecture

Statement of purpose

(Should be put in sentences and posted on www.makumba.org. See also the whitepaper. The previous review of makumba design dates from 2000, look at it to get more historical context)

What do we design Makumba for?

• fast member involvement
• rapid development
• based on members' HTML and SQL knowledge
• unlike popular scripting languages, it still enforces a clean model-view-controller (MVC) architecture, leading to a healthy structure of the applications even as members come and go
• db-driven web applications (clear delimitation of the technology)
• simple (few concepts)
• source code should be clear and easy readable to ensure efficient learning by example for new members
  
• easy to install (few commands/directories/files to work with)
• independent of any commercial technology or data format (easy to migrate to other technologies that can access databases through SQL)
  
• as much as possible "compiled" rather than interpreted (tell the developer at first page execution that somthing is rotten, rather than wait for it to blow in production)
  
• personal member development towards Java, MVC, http, JSP and other technologies involved
  

The issue of design

• design is originality: if Makumba design weren't original, there would be no need for Makumba, sth else could be used
  
• there was a lot of pressure to stop design and start implementation, so some parts of Makumba are still poorly designed (while others give its strength and originality)
  
• a lot of stuff had to be implemented from scratch as the standard or 3rd party technologies at the time (1999-2001) were non-existent, in bad shape, or uncertain. Even the ubiquitous Tomcat did not exist at the time, and JSP was in its conceiving stages

Objectives of Makumba design work

• keeps its strength and originality as expressed in the statement of purpose above
• makes better use of 3rd party technology (especially java-endorsed standards), taking advantage of their healthy design, reliability and 'standardness'.
  
• in components where Makumba is still considered more original, or 3rd party technologies are still unsatisfactory, publish the respective part of Makumba as a separate, original product
• this will lead to a smaller, easier to maintain, and more reliable Makumba.
  
• eliminating sluggish design will make us more proud and ready to better promote Makumba in the open-source world

3rd party technologies

how to look at a technology:

• is it used by Makumba already? If so, does Makumba use it to its full potential (e.g. it may be that Makumba adopted the technology early in its development)
• if not, what would it do for Makumba, in which area?
• try out the technology. Install it and do an example
  
• is it a standard? Would it be good for BEST members' personal development to learn it?
  
• would the overhead (in installation/configuration, data conversion, etc) of using that technology be acceptable? Would the development process (Makumba usage) be easier? Even if it would be a bit more difficult, are the other advantages important enough?
  
• in general, how will Makumba+that technology respect the statement of purpose?

Makumba components and their design status

mak:list engine

design status:

• Lately we call this "query fragments". This is clearly the most original and powerful part of Makumba.
• familiarity with SQL within HTML is well encouraged. Groups of mak:lists are nicely combined in a high-performance original way
• $attributes are transformed into $1 OQL attributes in an easy transparent manner
• easy nesting of mak:lists and composing queries from them. This seems to be very original to Makumba
• number of queries generated is independent on the number of results of any enclosing mak:list.  This is the crux to makumba performace.
• recently there are ideas of displaying data from a generic database, e.g. from an XML file

design still not finished for

• the query-fragments concept is currently used only in the view level. A the last meeting we started to experiment with using it at the BL level for authentication, but also more use at view level for e.g. not showing inaccessible links. using query-fragments for both BL and view would allow the query engine to optimize its queries much better so e.g. the number of queries per access would be minimal. this is a cool design challenge (and if we can get it right, it can become yet another makumba strong point, along with the existing three).
• search. There is little support for search forms and their response (which will include mak:lists)  although mak:list query fragments are pretty flexible now
• counting  (25)
  
• default value  (should it be constant or OQL?)
• mak:value() as a EL function ${mak:value("someOQLExpr") }
• text type formatting: there seems to be confusion around html=auto, html=true, maxlinelength, problems with line separators,  etc
• paging
  

implementation status:

• the JSP taglib API is still not enough for Makumba's needs. Makumba-specific JSP page parsing needed to be introduced in the latest taglib version.
• there are small features missing but the new architecture of org.makumba.view.jsptaglib allows for easy extensions
  
• distinct is still needed for lists.
• the current implementation/architecture is pretty complex (see a diagram). Maybe JSP design should be actively lobbied at the Java Community Process (JCP)? Or should the ambitions be reduced? Maybe more research on JSP 2.0 (especially "simple tags") can help solving this?
• automatic $parameter conversion from string to int
• server hangup by wrong queries

3rd-party tools and standards:

• of course many tag libraries (incl the standard JSTL) know how to do SQL queries or display tables (JSF) but none combine the two like mak:list does (for better or worse)
• the taglib was designed for JSP 1.1, since then JSP 1.2 and especially 2.0 added more features that, integrated in Makumba, may simplify Makumba pages. We already use some 2.0 features, such as taglib functions for mak:count() etc.
• especially the Expression Language (EL) of JSP 2.0 makes it much easier to display values by using ${expression } notation
• JSTL helps replace java scriptlets. Makumba should be integrated more with it
  

forms

design status:

• think for example that you could use forms to build e.g. java objects instead of makumba ones:
  * say you have a separate taglib with prefix "form", then you write <form:new type="mypackage.myType" ...> i.e. you indicate a Java type instead of a Makumba one. most of the current mak:form philosophy would stay the same, but the concept would be more general. with the same tags you would be able to create mak:objects and (e.g.) java objects.
  * in other words the (increasingly powerful) form engine does not have to be tied to makumba databases. i also think viewing them separately will lead to a much better structure by reducing dependencies between them. this is not very far, the current structure is pretty healthy in this respect.
• keep nice familiarity with HTML, while taking good advantage of the MDD information. This is a strength of Makumba design
• multiple forms are powerful and original. You can use any number of forms inside of a form, enclose forms in mak:lists, etc

design still not finished for

• form annotations
• expanding form interaction over more pages (wizard)
  
• extra fields (783)
  
• autoforms (781)
• non-ptr non-set options (784)
  
• defaultValue (501) not clear whether it should be OQL or constant expr
  
• empty forms as links (796), delete as button (16)
  
• guards that enable forms (685)
  
• date input control needs improvement (510)

implementation status

• multiple submission of new and add forms should be detected
• hide the button on click via javascript (won't solve refresh)
  
• see unfinished design above
• show form/input whose response fails (790)
• multiple forms: use a result in a field of a subsequent form  (687)
• mak:attribute (somewhat connected to forms) and findXXX methods should be deprecated  (787)
  
• form responders are lost at webapp reload. The user needs to go back and resubmit the form if the webapp is reloaded while they submit it.
• the page analyzer should understand jsp:attribute, especially for mak:action
• set editors should offer a two-choser variant
• set int{} should allow submitting empty sets (788)
• checkbox/radio button options still not closed (585)
  
• mak:option fails with subrecords (797)
  

3rd-party tools and standards

• JSF has excellent design towards transforming web interaction into component-based interaction. However, that goes too far into breaking the familiarity with HTML forms, while bringing in a much less spread familiarity (GUI component programming)
  

MDD (data model)

design status

• a simple, easy to read, database-neutral data description.
• a small number of types, that cover most needs. More types would bring better performance, but create more confusion
• some concepts could have better names (more related to db theory): set, internal set/pointer, etc
  

design still not finished for

• a cleaner MVC was discussed at the first congress.  Basically a part of the BL should be integrated with the MDDs (and remain as model), and another part should stay close (and maybe be integrated more) with the JSPs and be part of the controller
• May be dropped or adapted for simple java classes and EJB/Hibernate.
  

• the initial ambition was to be object-oriented: encapsulation is more or less done, but inheritance and polymorphism are still not addressed. Is this really needed?
• subtyping (a database theory concept) is supported but can be made more explicit
• business rules (see also business objects). A data model should not only describe data, but also specify rules on how it can be created and changed
• session objects,  caching, dirty caching
  
• bidirectional sets is a new idea. Easy to implement, may still need a bit of MDD language design
  

implementation status

• inheritance (811) is pretty well planned
  
• the parser is a bit messy but it works
• multiple-field uniqueness still not done
  

3rd-party tools and standards

• there is acute need for database theory reading (to decide whether to have it object-oriented or keep it relational, etc)
• JDO supports (in Java standard, there are other tools like it) an object-oriented way to describe data and deal with it, and make objects persistent
  

Database layer

design status

• a pretty straightforward API for accessing MDD-generated tables in a database-neutral manner
• maybe Database should be really called Transaction, or Connection.
• maps MDDs to DB tables. When reading a non-Makumba DB this is especially useful
• maps Makumba types on their best representation in the host DB engine
• the API for sending arguments to queries and updates and for retrieving query results using java.util.Dictionary seems to be pretty heavy. Besides, it is not type-safe.
  

design still not finished for

• uncertainty about object-orientation (see MDD)
  
• case insensitivity is not provided
  

implementation status

• old, trusted code, using JDBC 1
• there is a discussion on whether to use foreign keys or not
• presently indexes are created for almost every field. This results in the SQL engine not knowing which to use
• maybe do the indexes only for foreign keys and some selected fields
  
• timestamps should be created in the db server
  
• Error when no DBSV is specified
• subrecord TS_modify update
  

3rd-party tools and standards

• database-neutrality made lots of progress in the past years. JDO, JDBC Connector, even JDBC itself do much better then they used to. Dropping the db layer for one of these technologies could be very beneficial
• the JDBC API (currently at version 3 and going towards version 4) made lots of progress since JDBC 1, most probably there are loads of features that Makumba does not take advantage of, including e.g. Rowsets
  

OQL (object query language)

design status

• May be dropped for EJB QL or Hibernate QL
  
• specify queries in a database-neutral manner. But this can increasingly be done in simple, standard SQL.
• OQL is also used to translate from MDD names to DB table names in queries (by calling the DB layer)
• we borrowed OQL from the now defunct ODMG. The initial version of their OQL grammar is still on the Makumba CVS.
• the ambition was to be object-oriented (see MDD)
• an important feature of Makumba OQL that distinguishes it from SQL is the ability to refer sets easily
  FROM general.Person p, p.addresses a
  instead of
  FROM general.Person p, general.Person->addresses a WHERE a.person=p
  though the second form should also work
• another important feature is the ability to analyze queries, including type checking, useful during page analysis. Even if we adopt a thrid party technology or simply decide to use SQL, some form of query parsing will still be needed.
  

design still not finished for

• functions. They should probably keep the object oriented manner (object.call(arguments) )
• comparisons with current date

implementation status

• set selection in original order is a new idea, rather nice to provide
  
• subqueries are badly missed
• patchy, hard to add new features. One should probably start fresh from the original OQL grammar and reimplement it to add functions and subqueries
• date arithmethic, a new idea
  

3rd-party tools and standards

• database-neutrality made lots of progress in the past years. JDO (which has its own form of OQL), JDBC Connector, even JDBC itself do much better then they used to. Dropping or adapting OQL for one of these technologies could be very beneficial. JDO is explicitely recommended by ODMG as the natural place to go
  

Login/Access control

design status

• based on "secret attributes". The design was so poor that it had to be patched with "checkAttributes".
• secret attribute $x in JSP less used
  
• bad support for authorization, separation of authorization logic
• knowing the rights of the user in the view
• acegisecurity.sourceforge.net
  
• the design led to a number of problems, including secret attributes with identical names in different parts of the site
• the design also hides the concept of session, which is important to learn (and use) for developers. Sessions can be used for more than login
• there is limited support for the servlet specification standard login. If R is a role and the user is in that role, then $R will evaluate to the user name.
• there were design discussions about roles, <mak:checkRole>. Also this could allow to show some content (links) only if the user has the right to see them.
• how to deal with replicas? Database replicas should be encoded, otherwise the replica administrator can see everything. Even so, the JSP writer should not have the right to execute just any query! There was an idea based on "tickets" to limit that.
  

3rd-party tools and standards

• the servlet speficification itself has a good login design, and that standard login (and the design of its integration in Makumba) should be looked at more
• there are other tools like jSai that seem to work in this department

"Business Logic"

design status

• bad design, mixture between logic and controller.
• thus the "clean MVC" part of the statement of purpose is still not well addressed
• it's clear that business rules should live with the data, maybe in MDD files, maybe near them. Currently they tend to follow the JSP page structure rather than the MDD structure
• parts that are more dependent on the interface (on the way the form was shown, etc) should be part of the controller
• non-java BL
  

design still not finished for

• maybe this part should be completely restructured.
• stuff that does pure data processing should move to "data model"
• stuff that does data adaptation from and to the dialogs (forms) should go to the controller
• sessions should be visible to the model, the controller, or both
  

3rd-party tools

• Jakarta Struts, JSF, EJB (e.g. for session design)

Controller

design status:

• transparent for now. Executes at every page access, via a servlet filter

design still not finished for:

• see "business logic" above: stuff that's specific to the form should be specified in/near the JSP and executed by the controller
• not able to show different pages after form submission ("view selection", bugs 29, 443)
• in general, not enough flexibility in treating the errors
• attribute debugging (800)
• after_add doesn't have access to base pointer (617)
  

3rd-party tools:

• Jakarta Struts, JSF
  

Templates/Components

design status:

• none done
  
• templating was really desired from the very begining of Makumba.
• it becomes more and more important as it seems that "dynamic website components" become pretty important on the net today, e.g. in CMS systems
• included mechanisms can be a way to support templates. The current support for include mechanisms (jsp:include and @include) do not cross mak:list contexts. You can't have a mak:list and include something within it which would
• be dynamic in name (i.e. decide what to include at runtime)
• know its context (i.e. know that it's included in a certain type of mak:list)
• JSP 2.0 supports the definition of new tags in JSP files (tag files) instead of Java classes. This can also support templating
• Jakarta Tiles
• OpenSymphony SiteMesh
  

3rd-party tools

• JSP 2.0 tag files
• PHPNuke as an example

Developer tools & Documentation

design status:

• pretty original, building on the originality of MDD and JSP parts
  

design still not finished for:

• interacting more with the programmer during development
• more Eclipse tools (MDD plugin)
• Parade editors

implementation status:

• API documentation
• Mak specification is too big
• Confluence with Bugzilla
  
• expand includes in JSP source viewer (814)
  
• "pages and BL classes affected by an MDD" (782) still not implemented, although most of the tools are in place (page analysis, etc)
  
• splitting specification into more documents
• linking to lines of sources rather than just to the file (816)

3rd-party tools

• Ant
• Eclipse

Internal architecture design

design status:

• this is not a very important part of design (is not affecting the user) but it is important for sustaining Makumba
• handler families seem to be problematic to understand and maintain
• there are automatic tests for the most critical part (database layer) and for others (MDD, etc) but they are badly missed for the taglib.
  

design still not finished for:

• we were not sure how handler families could be redesigned but now we got around that
  

  implementation status:

3rd-party tools

• Jakarta Cactus can help in making automatic tests for the Makumba JSP taglib layer
• Eclipse can help refactoring
  

technologies used by Makumba today, and their versions

• 
  
Java of course. The latest 1.5 version brings a lot of useful stuff, e.g. for business logic code (e.g. using int and Integer interchangeably i.e. autoboxing)
• JSP: used to implement the taglib. Mostly 1.1 is used (though JSP 2.0 brings lots of useful stuff for the Makumba user)
• servlet: used to filter requests (the mak controller), display errors, etc. Servlet 2.2 was not supporting filters, we introduced that with servlet 2.3
  

technologies transparent for the Makumba programmer

• ANTLR: used to parse OQL. We went through antlr 2.7.1 and 2.7.2. only 2.7.2 works with Jikes 1.18 (see below)
  
• Junit: used to test Makumba
• both JSP and servlet need a servlet container, we use Tomcat for that. It's not always transparent for users. We went through Tomcat 3, 4 and 4.1. Tomcat 5 is now up, supporting JSP 2.0 and servlet 2.4
• Jasper (the JSP compiler) is an important part of Tomcat. Jasper 2 was introduced with Tomcat 4.1.
• Jikes is a fast java compiler, used to compile Java code generated by Jasper (and to compile Makumba code itself). Jikes is fully interchangeable with Java's normal javac compiler, but it's faster. Only Jikes 1.18 works with Jasper 2 due to the UTF character coding.
  
• JDBC: to connect to databases. Makumba hides a lot of its details. JDBC's original goal (version 1) was to be able to connect to a database at all (i.e. try to support as many as possible, not necessarily in an uniform manner). Database neutrality is more important for JDBC 2 and 3. These versions also provide features that may speed up Makumba internals.
• DB engines. In general transparent for the user, as part of DB neutrality
  
• MySQL is the main DB engine supported by Makumba. We went from version 3.23 to version 4.0, then we started using InnoDB tables instead of MyISAM tables. MaxDB tables seem also to be interesting for some fatures. MySQL 4 (4.1?) brings interesting features like subqueries, multi-table updates.
  
• other DB engines that Makumba was tested with: Informix, DB2, Cloudscape (recently open-sourced by IBM), partly ODBCJet on windows
  
• mysql-connectorJ, the JDBC driver used by Makumba, presently used version is 3.0.7, which is compatible with JDBC 3.0, there are more advanced versions of it (3.0.14, 3.1), supporting pretty recent JDBC features.
  
• Ant: used to build and publish Makumba. Mostly transparent for the user as far as Makumba is concerned (Ant is also used in karamba, but that's another story). We went from Ant 1.3 to 1.6, increasingly using its new features

Links

Approaches

• CMS: content management system
• MVC: model-view-controller

Standards and standard candidates

• SQL
• HTML
• http


• ODMG (Object Database Management Group) defined OQL


• Java
• Java servlets
  • JSP: Java Server Pages
    • JSP 2.0
    • JSTL: the standard tag library
    • JSF: Java Server Faces
• JDBC
• JDBC Connector
• EJB: Enterprise Java Beans
• JDO: Java Data Objects
• Struts proposes an MVC standard over servlets
• jSai proposes a critique of servlet authentication

Tools

• Eclipse: a popular and flexible IDE
• Cactus: good for automatic testing of servlets/JSP/tag libraries
• Junit: automatic testing
• PHPNuke, a CMS
• Tomcat implements servlet and JSP
• Jasper, a JSP compiler in Tomcat
• Ant: a make-like tool in Java
• Jikes: fast Java compiler
• ANTLR: parser generator for Java
• Database engines (SQL implementations)
  • Cloudscape
  • Informix
  • ODBCJet
  • DB2
  • MySQL, table formats:
    • MyISAM
    • InnoDB
    • MaxDB
  • Mysql Connector/J