JeoFTP

Web Development for the fun of it

2012-01-05T17:37:00.000-05:00

Right now I'm co-leading Product Design for IBM BPM. That means I create all the detailed specs for how a product feature is going to work, but I don't get the luxury of building it myself. An army of software engineers does that part.

Although I don't get to write production code these days, I still like to do prototypes to prove the feasibility of my ideas. This has opened my eyes to an array of incredible new libraries for web developers.

Bourbon makes it reeeeally easy to write modern CSS3, hiding all the convoluted vendor-specific repetitive ugliness from you. It's built on sass, whose mission was basically to fix CSS's stuttering problem. Finally, writing CSS is actually fun again!

"But Jeoff, what about IE, I thought it can't do CSS3?" No problem, even Internet Elephant (IE) can be made to dance with CSS3 gradients, rounded corners, and shadows: it's as easy as PIE! Also, check out flexie, which brings the CSS3 flexible box model to IE and other browsers. Create some layouts using flexbox and style some stuff using CSS3 fx and you'll never go back again. I guarantee it.

Now that you're ready to write some CSS3, check out what's possible!

p.s. Yes, I'm still using jquery. I agree with the SproutCore folks that jQuery has become the standard library of the web.

Long live the quirks-mode box model!

2010-10-20T17:00:00.000-04:00

There are lots of things to hate about Internet Explorer, but I've always preferred their "broken" box model. Well sometimes bugs turn out to be loveable, soft, furry critters, and so now you can have it even in standards mode, thanks to CSS3:

* {
  -moz-box-sizing: border-box;
  -webkit-box-sizing: border-box;
  box-sizing: border-box;
}

From wikipedia:

Web designer Doug Bowman has said that the original Internet Explorer box model represents a better, more logical approach.^[18] Peter-Paul Koch gives the example of a physical box, whose dimensions always refer to the box itself, including potential padding, but never its content.^[10] He says that this box model is more useful for graphic designers, who create designs based on the visible width of boxes rather than the width of their content.^[19] Bernie Zimmermann says that the Internet Explorer box model is closer to the definition of cell dimensions and padding used in the HTML table model.^[20]
The W3C has included a "box-sizing" property in CSS3. When box-sizing: border-box; is specified for an element, any padding or border of the element is drawn inside the specified width and height, "as commonly implemented by legacy HTML user agents".^[21] Internet Explorer 8, Opera 7.0 and later, and Konqueror 3.3.2 and later support the CSS3 box-sizing property. Gecko-based browsers such as Mozilla Firefox support the same functionality using a proprietary "-moz-box-sizing" property,^[22] and WebKit browsers such as Apple Safari and Google Chrome support it as a proprietary "-webkit-box-sizing" property.^[23]

jquery poised to gain on GWT's compiler, thanks to, um ... Google

2010-01-15T12:00:00.009-05:00

Buried in the jquery 1.4 release announcement, under the Miscellaneous section, is the following tidbit:

Now use Closure Compiler instead of YUI Min (commit)

So what is Closure Compiler? According to the project page at Google Labs,

"The Closure Compiler is a tool for making JavaScript download and run faster. It is a true compiler for JavaScript. Instead of compiling from a source language to machine code, it compiles from JavaScript to better JavaScript. It parses your JavaScript, analyzes it, removes dead code and rewrites and minimizes what's left. It also checks syntax, variable references, and types, and warns about common JavaScript pitfalls."

According to the commit message, switching from YUI Compressor to Google Closure Compiler reduced the filesize by 13% with no other changes.

But wait, there's more!

A 13% bump is not bad, and certainly no one is surprised these days when Google one-ups Yahoo. But it gets better: jquery is so far only using Closure Compiler in Standard Optimizations mode (the default setting). It is not yet using the more aggressive Advanced Optimizations. With advanced optimizations enabled, Closure Compiler also:

renames global variables and function names to make them shorter,
removes unused functions, and
inlines references where possible, including function calls, constants, and variables.

The catch is...

In order to use Advanced Optimizations safely, you need to tell Closure Compiler which symbols are actually calls to external code (externs) and which symbols need to be preserved as public API so that other libraries can reference them (exports). In the case of exports, it still shortens the symbol names everywhere they are used, but it sets up a single alias with the public name.

Setting up externs and exports takes time, effort, and testing, and it's probably the reason why jquery isn't using advanced optimizations just yet. Still, the source code is out there in plain sight and anyone can do it. So it's only a matter of time before jquery gains some of the same compiler optimizations that lend GWT its performance advantage.

Of course, for the code you write on top of jquery to also leverage Closure Compiler, you will have to add it to your build step and use it on your own javascript code as well.

The best Javascript toolkit is...

2010-01-05T17:04:00.007-05:00

The contenders are dojo, yui, jquery, mootools, gwt, scriptaculous. The votes are in. The people have spoken. The best Javascript toolkit, by popularity, is....

jquery.

Why I left GWT and came crawling back to jquery

2010-01-05T15:11:00.009-05:00

jquery, I'm sorry I said those mean things about you. After all the good times we had together, I walked away the moment GWT batted its eyes at me.

We got along okay at first, GWT and me. But then I started running into problems with the simplest things due to GWT's convoluted overuse of container markup for things like text and links. Trudging on, I encountered vexing UI layout bugs that could not easily be solved without dissecting that very-same convoluted container markup. When I cracked open the Firebug Inspector, I was shocked to see just how many containers were being created for even the simplest layouts, and that same complexity made it mind-numbingly tedious to comprehend the effects of the CSS on all that markup.

Of course, using Eclipse's superior source-code navigation capabilities, I was able to find the offending source code. It was one of those weird decisions made in a private method 4 superclasses up. (To fork, or not to fork? That is the question.) With the offending code safely barricaded, er, encapsulated, I turned back to hacking the CSS rules on that seemingly unending hierarchy of containers. That was when I first doubted my decision to leave you, jquery... But I trudged on, in search of that beautiful, statically-typed, generic, polymorphic sunrise.

The two faces of GWT-RPC

Okay, so the approach to UI layout is needlessly complex and tedious, but at least there is the beauty of GWT-RPC, right? That's what I thought, until I learned firsthand of the nasty RPC problems that arise when you make assumptions about what objects you can transfer. There are JavaScriptObjects and pojos and a bloodbath of DTOs trying to hold things together.

There was the time I naively assumed that I could read a JavaScriptObject from the "wild" and then pass it upstream in a GWT-RPC call. BZZZTT! Please convert to a pojo and try again.

Then there was the dreaded RPC Types Whitelist thingie that couldn't deal with my clever use of generics. I couldn't find this mythical whitelist file, so I just added dummy methods to my RPC interface referencing all the different concrete types I would need to be able to handle. (Bleh.)

Closures

There was also the misery of recreating in tedious boilerplate Java what is so succint and beautiful in Javascript. Closures become anonymous callback classes with several lines of boilerplate for the interface instantiation and the method declaration. I would agonize about whether to convert to a top-level class, which would then require me to pass in all my closure variables to a constructor and hold them in member variables, and pretty soon the once-elegant javascript closure has exploded into a 100-line .java file of its very own...

Java has a degree of elegance when compared to C++, but not next to Javascript: closures save time. Hip companies like Apple know this. Maybe someday Sun will catch up.

Here's your host.... (loading)

Then there's GWT hosted mode with its slow launch time and its generated code that I can't follow in Firebug because, you know, some-really-smart-guy-that's-probably-smarter-than-me wrote it, and so what right do I have to debug code generated by said-smart-guy's compiler? But then there's this little problem called, I have no idea why my RPC is failing, and why does it have to be this hard? and maybe the GWT-RPC mechanism is just flawed, and maybe I should just dump it and rewrite my own RPC mechanism?

Dump GWT-RPC and rewrite my own RPC mechanism? Not a bad idea....

And so I did.

Using jquery.

JSON, JSOFF

The funny thing is, it was so easy. All I needed was some JSON conversion on the server-side and then your getJSON() method took care of everything else for me. And then finally, I had my Firebug back. And I had my closures back. And no more private methods 4 superclasses up; I had my easy mixins back. And suddenly event-handling was easy again.

I miss a few things from that fling with GWT. The CSS sprites were good (ImageBundle, in GWT parlance). The idea of deferred binding is nice. The focus on performance in general, will be missed. The new native hosted mode is definitely a step up from GWT 1.x. I will miss refactoring, sort of. (It's absolutely essential when dealing with the verbosity of Java, but many of the types of code changes that make refactoring necessary in Java just aren't even necessary in Javascript.)

Still, it's good to be back. It's amazing what you can do in just a few lines with a javascript closure. And I don't mind writing "throwaway javascript code" when it feels good every time.

IBM Acquires Lombardi

2009-12-16T16:28:00.001-05:00

Lombardi, my employer, announced this morning that they have been acquired by IBM.

Tweaks to Zach's humane Date script

2009-07-06T15:47:00.004-04:00

I like Zach's humane Date script (based on work by Dean Landolt, John Resig, and henrah), but I wanted to be able to pass in an actual Date object in addition to just an ISO8601 date string. I also wanted it to be able to handle future dates (e.g. "10 minutes from now"). Here's the result:

 
/*
 * Javascript Humane Dates
 * Copyright (c) 2008 Dean Landolt (deanlandolt.com)
 * Re-write by Zach Leatherman (zachleat.com)
 * 
 * Adopted from the John Resig's pretty.js
 * at http://ejohn.org/blog/javascript-pretty-date
 * and henrah's proposed modification 
 * at http://ejohn.org/blog/javascript-pretty-date/#comment-297458
 * 
 * Licensed under the MIT license.
 */

 /**
  * @param {String or Date} date_str either an ISO8601 date string or a Date object.
  *          Note: ISO8601 dates are always formatted using UTC/GMT timezone.
  *          Example: 2009-06-03T20:06:44Z
  */
function humane_date(date_str){
 var time_formats = [
  [1, '1 second'],
  [60, 'seconds', 1],
  [90, '1 Minute'], // 60*1.5
  [3600, 'Minutes', 60], // 60*60, 60
  [5400, '1 Hour'], // 60*60*1.5
  [86400, 'Hours', 3600], // 60*60*24, 60*60
  [129600, '1 Day'], // 60*60*24*1.5
  [604800, 'Days', 86400], // 60*60*24*7, 60*60*24
  [907200, '1 Week'], // 60*60*24*7*1.5
  [2628000, 'Weeks', 604800], // 60*60*24*(365/12), 60*60*24*7
  [3942000, '1 Month'], // 60*60*24*(365/12)*1.5
  [31536000, 'Months', 2628000], // 60*60*24*365, 60*60*24*(365/12)
  [47304000, '1 Year'], // 60*60*24*365*1.5
  [3153600000, 'Years', 31536000], // 60*60*24*365*100, 60*60*24*365
  [4730400000, '1 Century'], // 60*60*24*365*100*1.5
 ];

 var time = ('' + date_str).replace(/-/g,"/").replace(/[TZ]/g," "),
  dt = new Date,
  seconds = (date_str instanceof Date ? (dt - date_str)
              : (dt - new Date(time) + (dt.getTimezoneOffset() * 60000))) / 1000,
  token = ' ago',
  i = 0,
  format;

 if (seconds < 0) {
  seconds = Math.abs(seconds);
  token = ' from now';
 }

 while (format = time_formats[i++]) {
  if (seconds < format[0]) {
   if (format.length == 2) {
    return format[1] + token
   } else {
    return Math.round(seconds / format[2]) + ' ' + format[1] + token;
   }
  }
 }

 // overflow for centuries
 if(seconds > 4730400000)
  return Math.round(seconds / 4730400000) + ' Centuries' + token;

 return date_str;
};

if(typeof jQuery != 'undefined') {
 jQuery.fn.humane_dates = function(){
  return this.each(function(){
   var date = humane_date(this.title);
   if(date && jQuery(this).text() != date) // don't modify the dom if we don't have to
    jQuery(this).text(date);
  });
 };
}

JSON Annotations

2009-04-17T21:44:00.007-04:00

Suppose you want a generic parser that will turn JSON into a strongly-typed Java object graph. For certain kinds of object graphs you either have to parse into generic objects and then convert, or hint the parser at every step -- a tedious process. What if those type hints were embedded within the JSON itself?

Option 1. Processing Instructions

This approach specifies a reserved prefix for meta-attributes, of which @type is one. Borrowing an idea from sed's variable syntax for find-and-replace separators, you can wrap the entire payload in one outer object that specifies what the metadata prefix is. That way if a given payload would clash with the default metadata prefix, it can be changed easily.

{ "meta": "@"
  "data": {
    "@type": "java.util.HashMap",
    "Joe" : {
      "@type": "jeoftp.Person"
      "firstName": "Joseph",
      "lastName": "Robinson",
      "age": 45,
      "birthday": new Date(23493822934)
    },
    "Sue" : {
      "@type": "jeoftp.Person"
      ...
    }
  }
}

Option 2. Wrap Typed Objects with Annotations

Another way to annotate objects with their types is to wrap each object that needs specific type information within a meta-object that tells us its type. The parser would need to know how to recognize a meta-object vice a regular one.

{
  "@class" : "java.util.HashMap",
  "@data" : {
    "Joe" : {
      "@class" : "jeoftp.Person",
      "@data" : {
        "firstName": "Joseph",
        "lastName": "Robinson",
        "age": 45,
        "birthday": new Date(23493822934)
      }
    },
    "Sue" : {
      "@class" : "jeoftp.Person",
      "@data" : { ... }
    }
  }
}

Option 3. Embed Annotations in Comments

You could also embed type information in comments. This approach borrows from Javadoc, Python docstrings, etc. This would certainly break some JSON parsers, but the advantage is that it strictly demarcates metadata from data.

{
  /* @type java.util.HashMap */
  "Joe" : {
    /* @type jeoftp.Person */
    "firstName": "Joseph",
    "lastName": "Robinson",
    "age": 45,
    "birthday": new Date(23493822934)
    },
  "Sue" : {
    /* @type jeoftp.Person */
    ...
    }
}

If your type parser was smart enough, perhaps you could even handle generics:

{
  /* @type java.util.HashMap */
  "Joe" : {
    "firstName": "Joseph",
    "lastName": "Robinson",
    "age": 45,
    "birthday": new Date(23493822934)
    },
  "Sue" : {
    ...
    }
}

I haven't settled on one approach yet, but after reading this I'm leaning toward option 1 -- specifying a metadata prefix -- because it is valid in curent JSON syntax, easy to parse, and relatively compact.

Determine the Java TimeZone of your web visitors using client-side javascript

2009-03-23T11:10:00.008-04:00

You want your Java web application to know what TimeZone your visitors are in. You can use client-side javascript code like the following to produce a timezone string that Java can understand:

function getTimezone() {
  var tzo = new Date().getTimezoneOffset();  //returns timezone offset in minutes
  function pad(num, digits) {
    num = String(num); while (num.length < digits) { num="0"+num; }; return num;
  }
  return "GMT" + (tzo > 0 ? "-" : "+") + pad(Math.floor(tzo/60), 2) + ":" + pad(tzo%60, 2);
}

This will produce strings like GMT-04:00 and GMT-07:00 which you can then pass up to the server and into TimeZone.getTimeZone() to produce a TimeZone object.

Timing Oracle SQL queries with Glassbox

2009-03-19T23:04:00.012-04:00

Glassbox, the slick AOP tool for automatically diagnosing your Java webapp's performance bottlenecks, first caught my eye nearly a year ago. In consulting activities, where off-the-shelf products and custom solution code mix, it can be very difficult to determine whose code is at fault when web pages load slowly.

Glassbox comes to the rescue by tracking each web request through every layer of the J2EE stack. It identifies slow requests and groups them by URL, then identifies the controllers, methods, database queries, network operations, etc. that are causing the operation to be slow. It comes packaged in a war file: just drop it into the app server, hit the webapp root from a browser, and let it autoconfigure itself.

The hitch I ran into was that Glassbox had trouble instrumenting our Oracle JDBC driver, so it couldn't identify the precise SQL queries that were causing slowness. I went to the forums initially, then forgot about it for a while when an answer wasn't readily available.

I became interested again over the past couple days, so I contacted Ron Bodkin directly: he suggested the AspectJ load-time weaving is probably at issue. He gave me some quick pointers on how to use the ajc tool to create an instrumented version of the Oracle JDBC driver so that it wouldn't need to be instrumented on the fly at load-time. I gave it a shot and it worked!

In testing, I purposely crafted a slow query by using a ridiculous number of pointless joins, to verify whether Glassbox would call me out on it.

The cookbook

First, grab Glassbox 2.0 and AspectJ 1.5.3. (To do offline weaving you'll need aspectjtools.jar, which doesn't ship with Glassbox).

Create a project directory and drop in the jar file you want to weave. Then create a lib/ subdirectory. Add aspectjtools.jar from the AspectJ download, and then explode glassbox.war into it. Find ./lib/glassbox.war/install/glassboxMonitor.jar and extract the file META-INF/Xlint.properties from it; place it in the root directory of the project.

Finally, run the following ant script:

Note: Adjust your path to AspectJ as needed.

Things to study when I get a minute

2009-03-19T21:02:00.003-04:00

Some of the things I'd like to spend more time with:

AspectJ, AJDT, Glassbox - AOP supertools
Jaxer, Axiom Stack, Helma, AppJet - web platforms that leverage the symmetry of using javascript on both the server and client side
Lucene, Berkeley DB - faster than a database, given the right use case
Terracotta - enforces java memory model across a grid of servers

Reason #82734 why I hate Oracle JDBC drivers

2009-03-17T15:34:00.006-04:00

Are you wondering why the date/time values you read from Oracle 10g's jdbc driver are off by several hours? It's because the 9i/10g drivers don't properly account for the jdbc client's time zone when writing the date value.

So what's the fix?

Suppose you know your server date/time values are stored in GMT. You can call getTimestamp() with a GMT calendar, and then you'll get the right values.

public class OracleJdbcDriverSucks {
  public static Calendar GMT_CALENDAR = Calendar.getInstance(TimeZone.getTimeZone("GMT"));

  public static Date getGMTDate(ResultSet rs, String column) throws SQLException {
    return new Date(rs.getTimestamp(column, GMT_CALENDAR).getTime());
  }
}

jquery rocks! now back to GWT

2009-02-12T09:28:00.005-05:00

Update: read the followup to this post, Why I left GWT and came crawling back to jquery.

I just finished a project that includes a rich, complex web interface built on jquery, and now it's time to report the good, the bad, and the ugly.

The Good

Certain things can be expressed with incredible succinctness in jquery. In a finished project there are snippets of code sprinkled throughout that really evoke a smile. Certainly other projects have a lot to learn from the expressive power of jquery.

The Bad

One core principle behind jquery is the idea of progressive enhancement; that is, you start with HTML markup that would render reasonably well in a browser even with no javascript enabled. Then you add in some well placed jquery "enhancements" via selectors, plugins, etc. and the page literally comes alive.

The problem is, projects have deadlines. When a business person funds a project, they don't really care if the site runs without javascript enabled. So, in the push to launch the project, the ideals of progressive enhancement quickly fall by the wayside.

The Ugly

In reaching for the ideal of progressive enhancement, jquery performs weakly in one important feature of traditional software engineering: encapsulation. Certainly the code within plugins is encapsulated, but the markup on which those plugins operate is completely exposed. The very power of selectors and plugins for progressively enhancing existing markup are also their downfall: when the markup needs to change for some reason, you must carefully review/debug/bang-your-head-over all the selectors and plugins and the assumptions they made about the markup that are no longer true.

Back to GWT

I'm starting a new project now, and I'll be doing this one on GWT. Is it because GWT is better than jquery? No, that is certainly not true in the general case. To breathe new life into an existing web site, there is perhaps nothing better than jquery. No, I'll be doing this project on GWT because progressive enhancement is not relevant. The ability to encapsulate DOM structures and behavior together, and then refactor wildly (with no fear of breaking carefully crafted selectors and plugins) will be essential to this project. And so it's back to GWT.

Fear not, jquery, my friend. We have built great things together, and there will be other times.

Review and annotate PDF files without Adobe messiness

2009-02-12T09:21:00.004-05:00

A technical writer at work sent me a document for review, and told me to use Adobe's commenting tools to annotate it. I open it up in Reader and -- surprise -- there aren't any commenting tools!

I expected this to happen, because it wasn't the first time this had happened to me. Adobe Reader can only annotate PDF documents when the author has enabled specific usage rights. Apparently most technical writers either don't understand this, or they all have the full Adobe Acrobat and simply have never encountered the problem.

The quick solution: grab PDF-XChange Viewer (which allows mere mortals to annotate any PDF document), and leave the Adobe mess for someone else.

p.s. the small "portable" version includes all the commenting tools (second Download button in the link above, as of this writing) ... not sure why the full version is so much bigger.

Beware jquery's :nth-child selector

2009-01-22T14:43:00.004-05:00

The past day or two I've been using selectors like this to manipulate tabular data:

$("table#summary tr:nth-child("+k+")")

I was using the selectors to pump a table of audit data into a summary table, and the whole process was 5-6 seconds for under 50 rows. There were lots of ":contains" statements in there, so naturally I attacked those first, caching the results in .data() fields. Yet most of the lag time remained.

So I cranked on the Firebug profiler and profiled the function that does the heavy lifting. Calls to nth-child were taking up 4700ms out of about 5000ms total!

After staring at it for a minute, I recalled that :nth-child looks for all elements that are the nth child of their parent -- that's what I want, right? -- but it does not assume that they are direct descendants of the preceding element in the selection chain.

"Matches all elements that are the nth-child of their parent ... While :eq(index) matches only a single element, this matches more than one ..."

This is certainly what you need in some cases, but if your jquery consists of a set of siblings and you just want to grab a particular element by index, it's far quicker to do one of the following:

$("table#summary tr:eq("+k+")")
$("table#summary tr").eq(k)

Once I made that change, Firebug's Profiler timing went from 5000ms+ down to the 300-400ms range. Lesson learned.

Searching tables with jquery

2009-01-19T15:22:00.010-05:00

Suppose you have an html table like the one below (name, favorite color) and you want to grab all the favorite colors.

Name	Favorite Color
Joe	blue
Sally	green
Mike	yellow
Ziggy	orange

Here are some jquery plugin functions that let you do it:

/** find the index of this node among its siblings **/
jQuery.fn.childn = function() { return 1 + jQuery(this).parent().find("> *").index(this); }

/** given a table row, find the cell value under the supplied heading **/
jQuery.fn.column = function(heading) {
    var idx = jQuery(this).parents("table:first").find("th:contains("+heading+")").childn();
    return jQuery(this).find("> *:nth-child("+idx+")");
}

Example:

$("table tbody tr").column("Favorite Color");

Update: use :eq instead of :nth-child! See my follow-up post for details.

How to construct immutable objects using reflection

2008-08-16T11:24:00.003-04:00

John O'Hanley argues that Javabeans should not be used with database frameworks. The reason various ORM frameworks use Javabeans is because they need to be able to construct objects using reflection. Javabeans provided this capability, whereas constructors were hard to introspect (at least as of Java 1.4) because the compiler saves only the types of method arguments, and not their names. The need to reflectively set properties by name is a legitimate one. Javabeans, by virtue of their extreme mutability, are certainly a problematic approach. But rather than discarding Javabeans and ORM entirely, John, you should propose a solution for those who want ORM. Then you will be taken more seriously. Perhaps Java 5 annotations finally give us enough capability to construct immutable objects reflectively. A constructor could be given an annotation that tells the ORM framework how its arguments map to entity fields.

RulePoint and RTAM

2007-12-26T16:38:00.000-05:00

Here is a good description of the products I worked on while at Agent Logic: Agent Logic’s RulePoint and RTAM
http://www.column2.com/2007/12/agent-logics-rulepoint-and-rtam/ I did the designs for both products and was the lead developer on RulePoint through the first 2 releases.

Column-oriented databases. Yawn.

2007-09-14T17:23:00.000-04:00

WOW, just rotate your thinking 90 degrees and everything gets faster! An inline explanation of what it means to be a column-oriented database would have been nice. For that I referred to http://en.wikipedia.org/wiki/Column-oriented_DBMS My conclusion is that your run-of-the-mill RDBMS can easily provide a new table type that uses column-oriented storage, without varying anything else (same SQL, same drivers, same integrations). That leaves Vertica and its column-oriented friends in a dicey position: as soon as money starts flowing towards columns instead of rows, the big boys wake up and add a column-oriented storage option. Then everyone sighs and upgrades to Oracle 12p or whatever they end up calling it.

This irony brought to you by Google

2007-08-14T13:24:00.000-04:00

Google informed me today that java.exe is a process belonging to Microsoft Office! See for yourself in the screenshot at left. (hint: the ad with the red box outline).

Oh and by the way, Google needs Java experts! Be sure to remember during your Google interview that java.exe is a process belonging to Microsoft Office...

A simple LRU cache

2007-07-26T14:05:00.001-04:00

Here it is. (Note: apply external locking when thread-safety is required).

int maxEntries = 50;
java.util.Map lruCache =
  new java.util.LinkedHashMap(maxEntries, .75F, true) {
    protected boolean removeEldestEntry(Map.Entry eldest) {
      return size() > maxEntries
    }
  };

jwhich - Java classes can run but they can no longer hide

2007-06-27T13:09:00.005-04:00

This bash command will search all .jar files in the current directory and all subdirectories for MyClass:

for f in `find . -name '*.jar'`; do jar tf $f | grep "MyClass" | xargs -n 1 echo "$f:"; done

Here is a complete shell script version that accepts a search argument, optionally followed by a start path. I keep this script at /usr/bin/jwhich on my box.

Sample output:

$ jwhich MyClass
./myapp/my/hard/to/find/subdirectory/mymissing.jar: com/Missing/MyClass.class
$

The script itself:

#!/bin/bash

whichclass=$1
path=$2

if [ -z $whichclass ]; then
echo "usage: $0 searchstring [path]"
echo "\tsearchstring - search for class names that contain this substring"
echo "\tpath (optional) - start at this path (default: current directory)"
echo "\nSearch for a Java Class, scanning all .jar files in the current directory and all subdirectories"
exit 1;
fi

if [ -z $path ]; then
path="."
fi

for f in `find $path -name '*.jar'`; do
jar tf $f | grep $whichclass | xargs -n 1 -r echo "$f:";
done

A simple servlet Filter for JDBC Connections

2007-06-21T10:15:00.001-04:00

Here is an example of a servlet Filter that could be used in a web application to check out a JDBC Connection from a connection pool, keep it open for the duration of a web request, and then insure that it gets closed at the end even if an exception occurs. The implementation is similar to the open-session-in-view pattern, but in this case it is more correctly described as open-connection-in-view. Note: Because connection pool initialization varies depending on the pool provider, the initialization of the pool DataSource is left as an exercise for the reader.

package jeoftp;

import javax.servlet.Filter;
import javax.servlet.FilterConfig;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.FilterChain;
import javax.sql.DataSource;
import java.sql.Connection;

/**
 * ConnectionProviderFilter
 *
 * Provide JDBC Connections to downstream servlets and insure they
 * are cleaned up before the web request completes. Servlets can use
 * request.getAttribute(ConnectionProviderFilter.CONNECTION_ATTR)
 * to access the Connection.
 *
 * @author Jeoff Wilks
 */
public class ConnectionProviderFilter implements Filter {

  public static final String CONNECTION_ATTR = "jeoftp.ConnectionProviderFilter";

  //most connection pools provide a standard JDBC DataSource facade
  private DataSource connectionPool;

  public void init(FilterConfig filterConfig) {
    //TODO: initialize your connectionPool here
  }

  public void destroy() {
    //TODO: clean up connectionPool here
  }

  public void doFilter(ServletRequest request,
                       ServletResponse response,
                       FilterChain chain) {

    Connection connection;

    try {
      //check out a connection for this web request
      connection = connectionPool.getConnection();

      //tie it to a request attribute so other servlets can grab it
      request.setAttribute(CONNECTION_ATTR, connection);

      //let other servlets do their work
      chain.doFilter(request, response);

    //optional: roll back transaction if an exception occurs
    } catch (Exception e) {
      connection.rollback();
      throw e;

    //mandatory: insure the connection is closed
    } finally {
      if (connection != null) {
        connection.close();
      }

    }      
  }
}

Google Spamtop (er, Desktop)

2007-03-12T11:37:00.000-04:00

Kudos to Google Desktop for making it even easier for me to "save on the Med$".

Programming is easy...

2007-02-03T16:45:00.000-05:00

...which is exactly why it's so hard.

Business Guy asks for some feature X. Programmer starts implementing X. But before he's done, Business Guy comes back and says, "Hey, why don't you throw Y in there too. I mean, it won't be that hard, right?"

Later during the demo of features X and Y, manager watches and then says, "That's not exactly what I had in mind for Y. Also, we really need Z in there too."

If you could freeze the requirements imposed on a piece of software, like you can for a bridge or a power plant, then the programmers could finish those features. But the temptation to slip in new features is too great, because "Hey, it's easy, right?"

The best defense against this "feature creep" phenomenon is to have a visual metric that shows not just the progress toward the release goal, but also how the release goal grows in size with time. (A scrum burn-down chart is one good example of this). The chart communicates to all involved that new features (or revisions to existing features) don't come for free.

Transparent JVM clustering solutions will improve

2006-12-06T12:11:00.000-05:00

Originally posted in the comments to Terracotta open sources JVM clustering, in response to Cameron Purdy's comment:

Terracotta can cluster two servers (one hot, one standby) while Tangosol can run a couple thousand servers as "hot" (active + active + active + etc) in an n-way fully-connected mesh (virtual channels). Our server throughput in a 100-server system is 50x ...

Ari made a great point in his Google Tech Talk presentation about how when Java first came out it felt like a step backward in terms of memory management, but with HotSpot and some bake time it became much better than a developer can manage explicitly at code-time. He went on to say that VM-level clustering will eventually be that way. One can argue about how much scale-up work the Terracotta hub needs today, but it's hard to dispute Ari's basic premise: that (1) the JVM can transparently replicate objects by watching the GET_FIELD and PUT_FIELD bytecodes, and (2) that you can insure thread safety across a cluster the same way you would in a single VM, as long as you can meet the contract of the MONITOR_ENTRY and MONITOR_EXIT bytecodes. If, in 5-10 years, it becomes as automatic as garbage collection is today, that is a huge ideological victory for Ari, Terracotta, and the KISS principle. I think they're on to something very big. For example, from Martin Fowler OOPSLA2005:

"While I'm on the topic of concurrency I should mention my far too brief chat with Doug Lea. He commented that multi-threaded Java these days far outperforms C, due to the memory management and a garbage collector. If I recall correctly he said 'only 12 times faster than C means you haven't started optimizing'."

Beware of connection pools when swapping in HSQL DB in a Hibernate setting

2006-08-15T16:59:00.000-04:00

I was trying to add support for HSQL DB to a project that already uses Hibernate. The app makes use of cascading inserts, but with HSQL in place it kept throwing a foreign key constraint violation. So I turned on log4j debug output for:

org.hibernate.sql (shows prepared statements)
org.hibernate.type (shows the values bound to and retrieved from the statements)

The debug output demonstrated that Hibernate was correctly inserting the referenced row before the one with the foreign key association. So why the failure? The debug output also showed that the value bound to the foreign key column was "0". That value is obtained by doing the INSERT and then, in HSQL's case, issuing a "call identity()" statement, which retrieves the value of the last identity generated in the current connection. I remembered that I'm also using c3p0 connection pooling, so I changed the max pool size to 1 for the HSQL DB configuration, and that fixed the problem of "call identity()" returning 0. It turns out HSQL DB does not benefit from connection pooling anyway. It remains to be seen what other problems that will cause...

Load a Spring Resource relative to your context file

2006-06-04T21:20:00.001-04:00

In MemeStorm's Loading resources from files or the classpath, I left this comment:

I was surprised how hard it was to get XmlWebApplicationContext to use a resource root *other than* the root of the webapp. For security reasons you usually want your configuration files under /WEB-INF/, not the actual webapp root. But if you place them there, your app context XML file has to reference “/WEB-INF/db/init.sql” — which means that configuration file is no longer useful outside of a web container context.

In solving this problem, I initially tried to subclass XmlWebApplicationContext to remember the directory it was loaded from and use that as its base when loading other resources. However, the loading of the actual context file was buried deep within the bowels of its code, making it difficult to retain knowledge of the directory it was loaded from without slicing through a deep chain of method signatures.

I settled on a less elegant but quicker solution. I added a new bean property "contextRoot" that is prepended to every path passed into the getResourceByPath method.

public class ChrootXmlWebApplicationContext
 extends XmlWebApplicationContext {

 protected Resource getResourceByPath(String path) {
   return super.getResourceByPath( getContextRoot()
     + path.startsWith("/") ? path : "/" + path);
 }
 ...
}

The contextRoot property is pulled from a servlet context init parameter the first time it is requested; trailing slashes are trimmed. When prepending it to a resource path, a slash is reapplied between the contextRoot and the requested path, if necessary, as shown above.

By placing the context file in WEB-INF/ and setting contextRoot to "WEB-INF/" as well, you can mimic the behavior that would occur if the context was loading files relative to itself.

Connect once to a Dojo Topic

2006-05-19T12:10:00.001-04:00

Scenario: a script that is loaded/evaluated inside a ContentPane needs to subscribe to a topic that is publishing URLs, receive exactly one URL, and then reload the pane. Option 1. Look up the topic and connectOnce to its "sendMessage" method. (Note: scripts evaluated inside a ContentPane use the "_container_" variable to access it).

if(_container_)
  dojo.event.connectOnce(dojo.event.topic.getTopic("treeurl"), "sendMessage",
                         _container_, "setUrl");

Option 2. Use kwConnect with once:true.

if(_container_)
  dojo.event.kwConnect({
    once: true,
    srcObj: dojo.event.topic.getTopic("treeurl"),
    srcFunc: "sendMessage",
    adviceObj: _container_,
    adviceFunc: "setUrl"
  });

Tested with Dojo 0.3 release.

The AJAX-ifier's toolkit

2006-04-20T14:09:00.000-04:00

Here are my favorite tools for AJAX development and debugging.

Internet Explorer

Microsoft Script Editor – contains a full debugger for javascript, not to be underestimated – it even lets you set breakpoints (comes with MS Office, but you may need to rerun setup in order to get it installed; it will be at c:\program files\microsoft office\office11\MSE7.EXE if you already have it)
Fiddler Tool – sniffs your HTTP requests so you can really see what’s going on (download free at www.fiddlertool.com)
IE Web Developer Toolbar – get it for your sanity

Firefox

Firebug – shows you Javascript errors, XMLHttpRequests; lets you filter out CSS requests and other junk; also includes a DOM inspector with tag-search filter and a command console
Web Developer Extension – lets you toggle JS/CSS, change forms on the fly, clear cache, outline DOM elements, resize the window to emulate various resolutions, etc. (go to to get it)
Live HTTP Headers - don't trust your intuition: use this to sniff requests and responses so you can see what URLs are really getting hit, what the cache and auth settings really are, etc.

Eclipse

JSEclipse plugin – syntax highlighting, code completion, etc. (add it to your Eclipse Update list by following the instructions)

IE will cache CSS background images, if you give it a chance

2006-04-13T13:59:00.000-04:00

You're using the latest CSS background-image trickery to get lots of cool effects on your web pages. But when you load the pages up in IE, the images seem to flicker when you move the mouse around over them. You never noticed it before when developing on your local machine, but it's painfully apparent when operating over a slow connection. The reason for this is that Internet Explorer 6 takes a conservative approach to caching, that is: When in doubt, do not cache. But IE will in fact cache your CSS background images -- even when it is configured to Check for newer versions of stored pages [x] Automatically -- if you give it a chance. The trick is to configure the web server to properly set the cache-control header. In order for images to get cached in Automatic mode, IE needs to see an HTTP header like this in the response:

Cache-Control: max-age=86400

The number is the # of seconds to cache (24 hours in this case). If you're using a Java server like Tomcat you will not get cache-control automatically, but you can do it with a servlet filter. If you're using Apache httpd then you can just turn on a configuration directive. WARNING: If anything else on the server is setting the "no-cache" directive, then you may end up with a response header like this:

Cache-Control: max-age=86400, no-cache

That header says "cache me for 24 hours, but actually, don't cache me after all." So IE will panic and simply not cache it. To verify you're getting the right cache-control header, inspect the response headers using Fiddler Tool in IE (fiddlertool.com) or the livehttpheaders extension in Firefox.

Spring MVC and lost man-months

2006-04-12T17:02:00.000-04:00

Like many others, I hopped on the Spring MVC bandwagon with sparkling eyes. Many futile developer-months later (both for me and the small team of developers I lead), I can step back and say definitively: Spring MVC is not fit for prime-time. It purports to be a replacement for Struts, and it is. But Struts has many replacements, and the Spring MVC team has done the Java community a disservice by muddying the waters with their mildly evolutionary offering. I suspect it is because the Spring folks want to be a "one-stop shop" for J2EE developers: the more of their stack you use, the more time you will spend reading their books and attending their trainings -- the more valuable their brains become. The problem is, Spring MVC has a lot of problems under the hood. I detail a partial list of gripes below. I will add to this list if/when I think of others.

Binding you down

The form-binding mechanism provided in Spring MVC has proven to be a constant headache. A little searching through the Spring Web Forum will convince you that no one really knows how to bind complex form elements to Javabean collections (List, Set, etc.). This one flaw will constantly bite you if you dare to use Spring MVC. Someone may tell you to just use an array and save yourself some hassle (it's typed, etc.). Don't be fooled so easily: that may work for Spring, but not for Hibernate, which needs your collections to be interfaces, so that it can work its behind-the-scenes magic on them. (And chances are, if you're using Spring, you're probably using Hibernate too). I have spent many hours sifting through the source code in the Eclipse debugger trying to understand what's going on during various binding snafus. The first strangeness came onscreen almost immediately after I attached the Eclipse debugger to Tomcat. The debugger kept halting at a curious line where an exception was being created for no apparent reason. It was the BindException, and it gets created every time the binding process occurs (hint: every single web request). It turns out the BindException is central to the design of MVC. Every request causes a BindException. Hmm. All sorts of wonderful stuff gets stuffed inside the BindException. Assuming you can get binding to work at all, you'll have the pleasure of using <spring:bind> in your JSP views. One real nicety is that it runs your property values through property editors so that you get an appropriately-formatted string value. This works great for viewing objects and presenting their properties as editable values. Like every other web application in existence, mine also needed to be able to display objects in tables (the list view). And I needed those to display the same neatly-formatted values as you would see in the detail view. No problem, right? Just drop your <spring:bind> tags inside a <jsp:foreach> loop so that each row would have nicely-formatted values. For a simple collection where the concrete class of every element is identical and matches your controller's commandClass property, that works fine. But the collection I needed to display was based on an interface type, where the concrete class of each element may vary. The <spring:bind> tag does not like that, because it is hard-wired to deal with only one command class in any given request, invariable at view-time. So we quickly hit a wall where we could not utilize property-editor-based display formatting for something so simple as a table.

Custom property editors, sort of

Spring allows you to register custom property editors. It can also use the system property editors provided by the java.beans.Introspector. What it does not do is leverage BeanInfo classes very well. For example, if you provided BeanInfo classes (easily generated using doclets) then a PropertyDescriptor can have a propertyEditorClass. But Spring makes no attempt to use that class. This flaw cannot be understated: Spring promises to treat your command objects as JavaBeans, but then it ignores the BeanInfo classes that make JavaBeans somewhat plausible as a component framework. Without the ability to leverage BeanInfo classes your JavaBeans are just a fiesta of getters and setters. So how do you get custom property editors for your beans? The Spring Way, of course: by registering them manually, in the controller, when you handle the request. That, combined with the single-class binding limitations described earlier, conspire to insure that a Spring controller has zero reusability -- you can make it work for exactly one concrete class, but don't get your hopes up about writing a controller that can be reused easily.

Resource "/WEB-INF/myconfig.xml" not found

Let's say one of my beans wants to load a resource: "myconfig.xml". If I include it in my dispatcher servlet's web application context as an import, it will be loaded relative to the directory the spring context file is in (usually /WEB-INF). Now what if I want beans in my context to be able to load resources themselves, from that same directory? Sorry, no luck. The web application context resource loader pins its root at the webapp root, NOT the /WEB-INF subdirectory where the context and other configuration files are typically located. Furthermore, there is no easy way to modify this: it's buried deep within the helper classes that spring app contexts use to do the initial resource load. That means you have to specify your resources as "/WEB-INF/myconfig.xml" -- which then breaks when you try to run tests outside of a webapp container. Because when you're running tests, you're using a different application context class, and chances are it's loading resources relative to its own location. The alternative is to place all your configuration files in your webapp root, which may be accessible to end users. Probably not a good idea... I cannot easily communicate the pain and frustration this has caused---that something so simple as a root resource path cannot be modified easily. I eventually subclassed web application context so that I could override the way it determines the resource root. The pain has subsided somewhat since then...

Spring.htm

Okay, back to square one: let's learn Spring MVC by starting in the step-by-step tutorial. Throw some code here, paste some XML there, and voila, you have a controller available to you at /hello.htm. Wait a second: /hello.htm? Does anyone but me find that strange that all your controllers are mapped via *.htm? Perhaps in some strange way it's a step up from the *.do typical of Struts lore. Or perhaps it's how we trick our users into thinking we're using static files created in Front Page. But what if we're outputting XML or JSON from the controller? What if we want to support content negotiation, or alternate representations, or worse, what if we have an honest-to-goodness static file that ends in *.htm? "Houston, we have a problem." Or at least a very misleading URL. In adopting Spring MVC for my company's project, I first set out to wrest control over URLs by providing the capability to map to clean ones. Luckily, Spring MVC promises a lot of flexibility in mapping. You can write your own Handler classes and your own HandlerMapper classes. So I created one that maps regular expressions to controllers. Each controller can select a different view template based on the extension provided in the URL: so /myapp.xml and /myapp.html both go to the same controller, but get routed to a different view template. Things got worse when I realized that URL mappings in web.xml are very limited. To get the flexibility I needed, the Spring dispatcher would have to capture "/*" in web.xml. That meant that static files would no longer be handled by the servlet container's default servlet. DANGER WILL ROBINSON! I ended up writing a Servlet Filter to wrap the Dispatcher Servlet. This presented its own set of problems, as the Spring Dispatcher does not think it's running from within a servlet -- so if you're not careful you can end up with infinite forwarding loops when trying to forward to a view. I was able to eliminate that problem by having the servlet filter set a request attribute that basically says, "I was here already" and then ignore requests that it had already seen before. In the end I was relatively happy with the way I can map URLs now. Unfortunately, I realize in retrospect that I pretty much rewrote the mapping system entirely. There are a lot of things I could still improve, but when all is said and done I would basically have my own web subframework written inside of Spring MVC.

Dog Food, and the eating of it

There is a saying Joel Spoelsky of joelonsoftware.com fame really likes: "Eat your own dog food." And Spring simply does not do this. You can probably read about the reasons why somewhere on their site:

http://www.springframework.org/index.php

"No!" you say. "Not PHP! They probably just have their Spring dispatcher mapped to *.php!" No, they really are using PHP. Somewhere on the internet I remember reading their list of apologetic reasons for doing this: "use the right tool for the job" and all that. So which job is right for the Spring MVC tool? It's obviously not the right tool for "the leading full-stack Java/J2EE application framework" and perhaps it's not a good idea for you either.

Janino in 30 seconds, using Jython

2005-07-15T22:41:00.001-04:00

Janino looks powerful. It compiles expressions, scripts, classes, etc. either from source code or programmatically. But the code samples elicited my moan reflex.

The "Use Janino as an Expression Evaluator" code snippets look simple enough, but to really try them yourself, you need to: wrap in boilerplate, compile, tweak, execute, scratch head, add System.out.println() statements, remix ... keep that cycle up and you'll know what's going on after a while.
Or, just run the example classes with their command-line syntax. Simple enough... but we're trying to learn an API here, right? Command-line switches insulate us from the API, which is the opposite of what we want.

Enter Jython. The same code from the Expression Evaluator code snippets, remixed here in shot-glass form:

D:\>jython
Jython 2.1 on java1.4.2_06 (JIT: null)
Type "copyright", "credits" or "license" for more information.
>>> from org.codehaus.janino import ExpressionEvaluator
>>> from java.lang import Integer
>>> ee = ExpressionEvaluator("c > d ? c : d", Integer.TYPE, ["c","d"], [Integer.TYPE,Integer.TYPE] )
>>> ee.evaluate( [10,11] )
11
>>>

Nice. I typed as many Janino lines as boilerplate lines. Four lines in, I'm already seeing useful results. That actually makes me want to, I don't know, evaluate another expression:

>>> ee.evaluate( [5,2] )
5
>>>

It's going to be great fun exploring this API. ScriptEvaluator, ClassBodyEvaluator, Compiler, AST ... Jython users, start your engines.

script language="python" - in a web browser

2005-06-24T12:24:00.001-04:00

An interesting discussion came up on comp.lang.python about adding python script support to web browsers. Some people were saying it can't be done because Python can't provide an adequate security sandbox. I wrote this post to clarify how the browser provides the sandbox for the scripting language.

Newsgroups: comp.lang.python
Date: 24 Jun 2005 09:15:37 -0700
Local: Fri,Jun 24 2005 12:15 pm
Subject: Re: Python as client-side browser script language

Sorry to resurrect a slightly older topic, but I want to clarify some points about how the browser DOM and the script language interact.

Paul Rubin wrote:
> Huh?  The language itself has to provide the sandbox.
> Remember that scripts have to be able to see
> certain DOM elements but not others, and some of them have to be
> read-only, etc.

Greg Ewing wrote:
> If the DOM objects are implemented as built-in Python
> types, there shouldn't be any difficulty with that.
> Python objects have complete control over which attributes
> can be read or written by Python code.

In web browsers, Javascript does not provide the sandbox. The browser provides scripts with certain objects like "document" or "window" but these are not native script objects at all. They're wrappers for browser-native objects, usually written in C/C++ and exposed via an ActiveX or XPCOM interface. (IE uses ActiveX, Mozilla uses XPCOM). The interfaces to these browser-native objects enforce the security model by restricting what the scripting language is allowed to do within the browser context.

If you attempt any foul play with a browser-native object, it can simply feed an exception back to the script wrapper, and your script code fails. That's the sandbox.

Following are some relevant links for those interested in further details:

DOM Intro (especially the section, DOM vs Javascript): 
"That is to say, [the script is] *written* in JavaScript, but it *uses* the DOM to access the web page and its elements."
http://www.mozilla.org/docs/dom/domref/dom_intro.html

"Mozilla's DOM is coded almost entirely in C++. ... When, in JavaScript, a client tries to access a DOM object or a DOM method on a DOM object, the JS engine asks XPConnect to search for the relevant C++ method to call."
http://www.mozilla.org/docs/dom/mozilla/hacking.html

"The DOM makes extensive use of XPCOM. In fact, to do anything with the DOM implementation you need XPCOM."
http://www.mozilla.org/docs/dom/mozilla/xpcomintro.html

Talking to XPCOM in Python 
"The techniques for using XPCOM in Python have been borrowed from JavaScript - thus, the model described here should be quite familiar to existing JavaScript XPCOM programmers."
http://public.activestate.com/pyxpcom/tutorial.html

So in theory you should be able to create a python script interpreter, at least for Mozilla. In practice you'd either need to be an expert with the Mozilla source code, XPCOM, and Python... or you'd find yourself becoming an expert by necessity.

---

Incidentally when people don't understand the difference between DOM objects and Javascript objects, they end up with lots of memory leaks: http://jgwebber.blogspot.com/2005_01_01_jgwebber_archive.html

Installing Debian Sarge on Dell PowerEdge SC 1425

2005-04-14T13:30:00.001-04:00

I ran into problems, first with lack of hardware support for SATA drives, then with a real-time clock bug. Here are the steps that eventually worked.

1. Boot from Sarge Installation CD-ROM

2. At boot prompt, type linux26 to load the boot-installer using the 2.6 kernel.

The default installer kernel, 2.4, doesn't include the ata_piix module, which is required to detect SATA hard drives. (It's probably missing other modules as well).

3. Complete the first installation phase normally (partitioning, etc.).

4. After the Debian Installer reboots, the console will freeze each time it tries to read the real-time hardware clock (RTC). Press CTRL+C each time to skip it. (For me it froze 3 times).

This is a confirmed Intel bug. There are different fixes (patch the kernel, maybe patch the BIOS, use a 2.4 kernel with driver patches), but I present what I believe to be the simplest workaround. For details, continue on.

5. Open a second boot console as soon as you can by pressing ALT+F2. Log in as root with no password. (Debian Installer hasn't set a root password yet).

6. Add the --directisa option to every hwclock call by creating a shell script in its place.

# mv /sbin/hwclock /sbin/hwclock.bin
# vi /sbin/hwclock
----- contents -----
#! /bin/sh
/sbin/hwclock.bin --directisa $*
----- end -----
# chmod a+x /sbin/hwclock

(Lifted from a comment by phg here, with one minor correction -- two dashes before directisa instead of just one).

7. Type reboot at the prompt. After restart, the system should boot normally and continue into the Sarge boot-installer.