Makumba design
Statement of purpose
(Should be put in sentences and posted on www.makumba.org. See also the whitepaper. The previous review of makumba design dates from 2000, look at it to get more historical context)
What do we design Makumba for?
- fast member involvement
- rapid development
- based on members' HTML and SQL knowledge
- unlike popular scripting languages, it still enforces a clean
model-view-controller (MVC)
architecture, leading to a healthy structure of the applications even
as members come and go
- db-driven web applications (clear delimitation of the technology)
- simple (few concepts)
- source code should be clear and easy readable to ensure efficient
learning by example for new members
- easy to install (few commands/directories/files to work with)
- independent of any commercial technology or data format (easy to
migrate to other technologies that can access databases through SQL)
- as much as possible "compiled" rather than interpreted (tell the
developer at first page execution that somthing is rotten, rather than
wait for it to blow in production)
- personal member development towards Java, MVC, http, JSP and
other technologies involved
The issue of design
The biggest issue with Makumba today is design.
Reasons:
- design is originality: if Makumba design weren't original, there
would be no need for Makumba, sth else could be used
- there was a lot of pressure to stop design and start
implementation, so some parts of Makumba are still poorly designed
(while others give its strength and originality)
- a lot of stuff had to be implemented from scratch as the standard
or 3rd party technologies at the time (1999-2001) were non-existent, in
bad shape, or uncertain. Even the ubiquitous Tomcat did not exist at
the time, and JSP was in its conceiving stages
Objectives of Makumba design work
We should adjust the Makumba design so that:
- keeps its strength and originality as expressed in the statement
of purpose above
- makes better use of 3rd party technology (especially
java-endorsed standards), taking advantage of their healthy design,
reliability and 'standardness'.
- in components where Makumba is still considered more original, or
3rd party technologies are still unsatisfactory, publish the respective
part of Makumba as a separate, original product
- this will lead to a smaller, easier to maintain, and more
reliable Makumba.
- eliminating sluggish design will make us more proud and ready to
better promote Makumba in the open-source world
As such, what we need to do now, is search literature and read, not
write code, except where design is straight-forward and experience from
Karamba and other applications clearly shows that a feature is needed,
and can't wait much for redesign.
3rd party technologies
how to look at a technology:
- is it used by Makumba already? If so, does Makumba use it to its
full potential (e.g. it may be that Makumba adopted the technology
early in its development)
- if not, what would it do for Makumba, in which area?
- try out the technology. Install it and do an example
- is it a standard? Would it be good for BEST members' personal
development to learn it?
- would the overhead (in installation/configuration, data
conversion, etc) of using that technology be acceptable? Would the
development process (Makumba usage) be easier? Even if it would be a
bit more difficult, are the other advantages important enough?
- in general, how will Makumba+that technology respect the
statement of purpose?
See also technlogies used by Makumba today
Makumba components and their design status
For reference on each component, see a detailed description of Makumba internals. See also bugzilla especially the makumba target versions 0.6 and 0.7
mak:list engine
design status:
- This is clearly the most original and
powerful part of Makumba.
- familiarity with SQL within HTML is well encouraged. Groups of
mak:lists are nicely combined in a high-performance original way
- $attributes are transformed into $1 OQL attributes in an easy
transparent manner
- easy nesting of mak:lists and composing queries from them. This
seems to be very original to Makumba
design still not finished for
- parametrized query fragments
(especially in orderBy, etc). What should be allowed? Should such
parameters (attributes) be expressed differently than nomral
$attributes which become normal OQL query $1 parameters? Should they be
declared?
- an important application of the parametrization mentioned above
is search. There is little support for search forms and their response (which will include mak:lists) today
implementation status:
- the JSP taglib API is still not enough
for Makumba's needs. Makumba-specific JSP page parsing needed to be
introduced in the last taglib version.
- there are small features missing but the new architecture of
org.makumba.view.jsptaglib allows for easy extensions
-
start and limit is still needed for lists.
-
distinct is still needed for lists.
- the current implementation/architecture is pretty complex (see a diagram). Maybe
JSP design should be actively lobbied at the Java Community Process (JCP)? Or should the ambitions be
reduced? Maybe more research on JSP 2.0 (especially "simple tags") can
help solving this?
3rd-party tools and standards:
- of course many tag libraries (incl the
standard JSTL) know how to do
SQL queries or display tables (JSF) but none combine the two like
mak:list does (for better or worse)
- the taglib was designed for JSP 1.1, since then JSP 1.2 and
especially
2.0 added more features that, integrated in Makumba, may simplify
Makumba pages. We already use some 2.0 features, such as taglib functions for mak:count() etc.
- especially the Expression Language (EL) of JSP 2.0 makes it much
easier to display values by using ${expression } notation
- JSTL helps replace java scriptlets. Makumba should be integrated
more with it
design status:
- keep nice familiarity with HTML, while
taking good advantage of the MDD information. This is a strength of
Makumba design
- multiple forms are powerful and original. You can use any number of forms inside of a form, enclose forms in mak:lists, etc
design still not finished for
implementation status
- see unfinished design above
- form responders are lost at webapp reload. The user needs to go back and resubmit the form if the webapp is reloaded while they submit it.
- multiple submission of new and add forms should be detected
3rd-party tools and standards
- JSF has excellent
design towards transforming web interaction into component-based
interaction. However, that goes too far into breaking the familiarity
with HTML forms, while bringing in a much less spread familiarity (GUI
component programming)
MDD (data model)
design status
- a simple, easy to read,
database-neutral data description.
- a small number of types, that cover
most needs. More types would bring better performance, but create more
confusion
- some concepts could have better names (more related to db
theory): set,
internal set/pointer, etc
design still not finished for
- the initial ambition was to be
object-oriented: encapsulation is more or less done, but inheritance
and polymorphism are still not addressed. Is this really needed?
- subtyping (a database theory concept) is supported but can be
made more
explicit
- business rules (see also business objects). A data model should not only describe data, but
also specify rules on how it can be created and changed
implementation status
3rd-party tools and standards
- there is acute need for database theory
reading (to decide whether to have it object-oriented or keep it
relational, etc)
- JDO supports (in Java standard, there are other tools like
it) an object-oriented way to describe data and deal with it, and make objects
persistent
Database layer
design status
- a pretty straightforward API for accessing MDD-generated tables in a database-neutral manner
- maybe Database should be really called Transaction, or Connection.
- maps MDDs to DB tables. When reading a non-Makumba DB this is especially useful
- maps Makumba types on their best representation in the host DB engine
- the API for sending arguments to queries and updates and for retrieving query results using java.util.Dictionary seems to be pretty heavy. Besides, it is not type-safe.
design still not finished for
- uncertainty about object-orientation (see MDD)
implementation status
3rd-party tools and standards
- database-neutrality made lots of
progress in the past years. JDO, JDBC Connector, even JDBC itself
do much better then they used to. Dropping the db layer for one of
these technologies could be very beneficial
- the JDBC API (currently at version 3 and going towards version 4)
made
lots of progress since JDBC 1, most probably there are loads of
features that Makumba does not take advantage of, including e.g. Rowsets
OQL (object query language)
design status
- specify queries in a database-neutral
manner. But this can increasingly be done in simple, standard SQL.
- OQL is also used to translate from MDD names to DB table names in
queries (by calling the DB layer)
- we borrowed OQL
from the now defunct ODMG. The initial version of their OQL grammar is
still on the Makumba CVS.
- the ambition was to be object-oriented (see MDD)
- an important feature of Makumba OQL that distinguishes it from
SQL is
the ability to refer sets easily
FROM general.Person p, p.addresses a
instead of
FROM general.Person p, general.Person->addresses a
WHERE a.person=p
though the second form should also work
- another important feature is the ability to analyze queries,
including
type checking, useful during page analysis. Even if we adopt a thrid
party technology or simply decide to use SQL, some form of query parsing will still be needed.
design still not finished for
- functions. They should probably keep
the object oriented manner (object.call(arguments) )
implementation status
- subqueries are badly missed
- patchy, hard to add new features. One should probably start fresh from the
original OQL grammar and reimplement it to add functions and subqueries
3rd-party tools and standards
- database-neutrality made lots of
progress in the past years. JDO (which has its own form of OQL), JDBC Connector, even JDBC itself do
much better then they used to. Dropping or adapting OQL for one of
these
technologies could be very beneficial. JDO is explicitely recommended
by ODMG as the natural place to go
Login
design status
- based on "secret attributes"
- the design was so poor that it had to be patched with
"checkAttributes".
- the design led to a number of problems, including secret
attributes
with identical names in different parts of the site
- the design also hides the concept of session, which is important
to
learn (and use) for developers. Sessions can be used for more than login
- there is limited support for the servlet specification standard login. If R is a role and the user is in that role, then $R will evaluate to the user name.
3rd-party tools and standards
- the servlet speficification itself has a
good login design, and that standard login (and the design of its
integration in Makumba) should be looked at more
- there are other tools like jSai that seem to work in this
department
"Business Logic"
design status
- bad design, mixture between logic and
controller.
- thus the "clean MVC" part of the statement of purpose is still
not well
addressed
- it's clear that business rules should live with the data, maybe
in MDD
files, maybe near them. Currently they tend to follow the JSP page structure rather than the MDD structure
- parts that are more dependent on the interface (on the way the
form was
shown, etc) should be part of the controller
design still not finished for
- maybe this part should be completely
restructured.
- stuff that does pure data processing should move to "data model"
- stuff that does data adaptation from and to the dialogs (forms)
should
go to the controller
- sessions should be visible to the model, the controller, or both
3rd-party tools
Controller
design status:
- transparent for now. Executes at every
page access, via a servlet filter
design still not finished for:
- see "business logic" above: stuff
that's
specific to the form should be specified in/near the JSP and executed
by the controller
- not able to show different pages after form submission ("view
selection")
- in general, not enough flexibility in treating the errors
3rd-party tools:
Templates/Components
design status:
- none done
- templating was really desired from the very begining of Makumba.
- it becomes more and more important as it seems that "dynamic
website components" become pretty important on the net today, e.g. in
CMS systems
- included mechanisms can be a way to support templates. The
current support for include mechanisms (jsp:include and @include) do
not cross mak:list contexts. You can't have a mak:list and include
something within it which would
- be dynamic in name (i.e. decide what to include at runtime)
- know its context (i.e. know that it's included in a certain
type of mak:list)
- JSP 2.0 supports the definition of new tags in JSP files (tag
files) instead of Java classes. This can also support templating
3rd-party tools
Developer tools
(the jspx, mdd viewers, Ant files like commons.xml, etc)
design status:
- pretty original, building on the
originality of MDD and JSP parts
design still not finished for:
implementation status:
- "page summary" from the JSPX views is still not
implemented, although the recent page analysis code makes it much easier
- "pages and BL classes affected by an MDD" still not implemented,
although most of the tools are in place (page analysis, etc)
3rd-party tools
Internal architecture design
For reference on the internal design, see a detailed description of Makumba internals.
design status:
- this is not a very important part of design (is not affecting the
user) but it is important for sustaining Makumba
- handler families seem to be problematic to understand and maintain
- there are automatic tests for the most critical part (database
layer) and for others (MDD, etc) but they are badly missed for the
taglib.
design still not finished for:
implementation status:
3rd-party tools
- Jakarta Cactus can help in making automatic tests for the Makumba JSP taglib layer
- Eclipse can help refactoring
technologies used by Makumba today, and their versions
- Java of course. The latest 1.5 version brings a lot of useful
stuff, e.g. for business logic code (e.g. using int and Integer
interchangeably i.e. autoboxing)
- JSP: used to implement the taglib. Mostly 1.1 is used (though JSP 2.0
brings lots of useful stuff for the Makumba user)
- servlet: used to filter requests (the mak controller), display
errors, etc. Servlet 2.2 was not supporting filters, we introduced that
with servlet 2.3
technologies transparent for the Makumba programmer
- ANTLR: used to parse OQL. We
went through antlr 2.7.1 and 2.7.2. only 2.7.2 works with Jikes 1.18
(see below)
- Junit: used to test Makumba
- both JSP and servlet need a servlet container, we use Tomcat for
that. It's not always transparent for users. We went through Tomcat 3,
4 and 4.1. Tomcat 5 is now up, supporting JSP 2.0 and servlet 2.4
- Jasper (the JSP compiler) is an important part of Tomcat.
Jasper 2 was introduced with
Tomcat 4.1.
- Jikes is a fast java compiler, used to compile Java code
generated by Jasper (and to compile Makumba code itself). Jikes is
fully interchangeable with Java's normal javac compiler, but it's
faster. Only Jikes 1.18 works with Jasper 2 due to the UTF character coding.
- JDBC: to connect to databases. Makumba hides a lot of its
details. JDBC's original goal (version 1) was to be able to connect to
a database at all (i.e. try to support as many as possible, not
necessarily in an uniform manner). Database neutrality is more
important for JDBC 2 and 3. These versions also provide features that
may speed up Makumba internals.
- DB engines. In general transparent for the user, as part of DB
neutrality
- MySQL is the main DB engine supported by Makumba. We went from
version 3.23 to version 4.0, then we started using InnoDB tables
instead of MyISAM tables. MaxDB tables seem also to be interesting for
some fatures. MySQL 4 (4.1?) brings interesting features like
subqueries, multi-table updates.
- other DB engines that Makumba was tested with: Informix, DB2,
Cloudscape (recently open-sourced by IBM), partly ODBCJet on windows
- mysql-connectorJ, the JDBC driver used by Makumba, presently
used version is 3.0.7, which is compatible with JDBC 3.0, there are
more advanced versions of it (3.0.14, 3.1), supporting pretty recent
JDBC features.
- Ant: used to build and publish Makumba. Mostly transparent for
the user as far as Makumba is concerned (Ant is also used in karamba,
but that's another story). We went from Ant 1.3 to 1.6, increasingly
using its new features
Links
Approaches
- CMS: content management system
- MVC: model-view-controller
Standards and standard candidates
Tools
- Eclipse: a popular and flexible IDE
- Cactus: good for automatic testing of servlets/JSP/tag libraries
- Junit: automatic testing
- PHPNuke, a CMS
- Tomcat implements servlet and JSP
- Jasper, a JSP compiler in Tomcat
- Ant: a make-like tool in Java
- Jikes: fast Java compiler
- ANTLR: parser generator for Java
- Database engines (SQL implementations)