p2: how I install plugins in extension locations...

The following procedure explains how I install plugins into different extension locations and share those extension locations between different eclipse installations using the p2 update manager and link files. For a lengthly discussion see bug bug 224145

I keep a set of different extensions locations in C:\eclipse\extensions (each extension location is in a subdirectory of that directory). Suppose I want to install foo into the new extension location C:\eclipse\extensions\foo.

1. I start eclipse with:

   -configuration C:\eclipse\extensions\foo\eclipse\configuration

   
   
2. I install foo using the update manager.
   
   
3. Now I can use the extension by creating a foo.link file inside the links directory of an eclipse installation (you might have to create the links directory) containing:

   path=C:/eclipse/extensions/foobar

   
   

That's it :-)


Note: You have to the forward slashes on windows in the links file.

Note: This procedure works for eclipse 3.4 and 3.5.

Note: You don't have to create any directory specified by the -configuration parameter. Eclipse will do that for you.

Note: It is important to follow the pattern EXTENSION_NAME/eclipse/configuration for the -configuration parameter because p2 will put the plugins one directory above the configuration directory and link files require that the directory that contains the plugins is called eclipse.

Note: With this structure I can update plugins into the extension locations by running eclipse with the -configuration and then do the update.

Advanced use: If I want to install bar based on my foo extension, I create a link file to foo in the bar extension location links directory:

C:\eclipse\extensions\bar\eclipse\links\foo.link

and follow my standard procedure described above (the links directory can be populated before the first run of eclipse)...

Note for eclipse 3.4: If you are using eclipse 3.4 and you want to use the eclipse default update sites, you have to run eclipse without -configuration and export them (Help->Software Updates->Available Software (tab)->Manage Sites->Export), so you can import them into the -configuration eclipse.

Note for eclipse 3.5M6: You have to add an (empty) .eclipseextension file next to the plugins and features directory of the extension location.

When the runtime Application uses the wrong plugins...

Today I run into a problem that can cause a lot of frustration: a runtime eclipse launch that runs with he wrong plugins. The frustration comes form the fact that your application behaves strangely and it is hard to guess that this is caused by loading some worng plugins. Those plugins might even be closed or deleted from your workspace!

Here is what happens: PDE (eclipse 3.4) does some hidden magic when you run an eclipse application (lets call the launch configuration foo): it creates a directory in .metadata/.plugins/org.eclipse.pde.core/foo. This directory is used with the -configuration flag when running the application to store the runtime configuration of your application. But, there can be cases when this directory is not updated correctly, like when you switch between "launch with all enabled and workspace plug-ins" and "plug-ins selected below". It can happen that the runtime application still uses all plugins although you have selected to run with some specific plugins. I have not tested the exact conditions when this happens, but it happened several times to me in the last months.

Unfortunately the "Clear workspace" flag does not help, because it clears the workspace but not the configuration area! It took me several hours to figure that out....

As Chris points out in the first comment, here is the solution: On the "Configuration" tab of the launch configuration check the option "Clear configuration area before launching".

Chris also recommends to use -clean -consoleLog -debug -console as arguments for plugin development launch configurations. In this case, the -clean option takes care of cleaning the configuration area.

Tip: you can set -clean -consoleLog -debug -console as default in Preferences->Plug-In Development->Target Platform->Launching Arguments->Program Arguments

I hope this tip saves someone some wasted time...

.

Finally, I found a good book on JavaScript!

For some years I have been looking for a book on JavaScript that would explain the language to someone who already knows programming. But all I have seen until recently are books for beginners or books that talk a lot about web programming, web browsers, DOM, XML, HTML etc, or books that explain all the different versions of JavaScript implemented in different browsers (many of these books have 1000 pages or more).

The book I read is a 180 page book called JavaScript: The Good Parts by Douglas Crockford. The idea of the book is: JavaScript is a messy language but here is a subset and some best practices how to use JavaScript in a manageable way. And that is done on 100 pages (the rest are interesting appendices). I read it last week on a flight and it was a kind of eye opener for me.

I finally understood that the object system of JavaScript is not class based like in C++/Java/Python/Ruby but it is prototype based like in Self/Perl/Lua. This prototype based approach is not supported by a special syntax. There are many ways of creating and dealing with objects. But be book makes some suggestions and explains some best practices for creating objects.

The other eye opener was how closures can be used to build objects with private information.

An interesting design decision of JavaScript is that there is essentially one data structure, called Object, with string keys and any object as value. It can be used as hash map, array or object. Even functions are objects and you can associate information with them or add methods to one or all functions.

If an object is used as array, the index is converted to a string key -- I find this very strange -- but it seems to work. The array.lenght attribute is defined as the "largest integer property name plus one", which means it looks for string keys that look like positive integers and takes biggest one, provided it is less than 4294967295.

If you want to learn JavaScript as a language I would highly suggest to read JavaScript: The Good Parts. After you finished the book, you might not be able to read any bad JavaScript programm, but you might be able to write your own "clean" scripts. I really like that approach.

If you are interested in E4, knowing JavaScript might be useful, because E4 will be scriptable with JavaScript.

Eclipse summit europe -- no more hotels

OK, I admit, it is a bit late to look for a hotel for Eclipse Summit Europe 2008. Today I tried about 30 hotels. It seems all hotels in Ludwigsburg are booked out. Well, I thought no big deal: I would follow the S-Train line and book a hotel close to any S-train station. Well, all hotels are booked out along the S-train lines, including the hotel close to the Stuttgart main station!? I finally found a hotel not far from Ludwigsburg, next to the S-Train station in Asperg, but it turned out that the address was listed wrongly in google maps and the hotel is not even close to any train station....

It's not that bad -- with public traffic it is 1h30 to 1h45 from my home in Heidelberg to the "Forum am Schlosspark". The only bad thing is that my latest train is around 9:30PM, and the good discussions are usually late in the bar....

That means waking up early (it's really early for me, since I "virtually" live in the EST time zone -- I go to bed at 6AM and wake up at 1:30PM)...

If you found a hotel in <30 min distance to summit, let me know.

ConcurrentModificationException: Why do Java collections not have robust iterators?

Ever got a ConcurrentModificationException? It just hit me. How does this happen? It happens when one method iterates over a collection while another method (that bis recursively called from the for loop) modifies the collection.

Note: ConcurrentModificationException has nothing to do with threading! (Well, this is a bit too strong statement, as Rafael points point out in a comment: it might occur due to a threading race condition (but in such cases you probably have other problems as well)). A ConcurrentModificationException may have nothing to do with threading!. Here I am talking about about the non threading related case.

Here's a simple example to show the problem. We have a class that can register new listeners and when fireChange is called it calls changed on the listeners. So the test listener here removes itself on the change call and boom, we get the ConcurrentModificationException:

public class IteratorTest {
 final Collection fListeners;
 public IteratorTest(Collection listeners) {
  fListeners = listeners;
 }
 static class Listener {
  public void changed(IteratorTest subject) {
   subject.removeListener(this);
  }
 }
 public void addListener(Listener listener) {
  fListeners.add(listener);
 }
 public void removeListener(Listener listener) {
  fListeners.remove(listener);
 }
 public void fireChange() {
  for (Listener listener : fListeners) {
   listener.changed(this);
  }
 }
 static void test(Collection coll) {
  IteratorTest t = new IteratorTest(coll);
  t.addListener(new Listener());
  t.fireChange();
 }
 public static void main(String[] args) {
  test(new ArrayList());
 }
}

The problem can happen if you call out to "other code" (code someone else has written) and "other code" can change the collection while you are iterating. One solution is to iterate over a copy of the collection:

 public void fireChange() {
  Listener[] listeners=(Listener[]) fListeners.toArray();
  for (Listener listener : listeners) {
   listener.changed(this);
  }
 }

That helps. But there is a lot of code out there that iterates over a collection and calls "other code" and there is always a chance that the "other code" calls back to modify your collection and you get a ConcurrentModificationException....

The good news is: Unlike threading race conditions it happens deterministically. The bad news is: if you are the client it is often not easy to find a way out.

15 years ago, ET++ (the cool framework created by Erich Gamma and Andree Weinand in the 80ies) suffered form missing robust iterators. "Robust iterators" means robust to changes of the underlying collection. At that time, making a copy of a collection seemed an unacceptable overhead. So, Thomas Kofler added robust iterators to ET++ (the PDF has the pages on reverse order -- here is a readable version). The implementations are efficient and robust.

Robust iterators are so fundamental, I am really surprised that Java does not have them....

.

The bridge between Interaction Design (IxD) and Domain Driven Design (DDD)

Some time ago I read Alan Cooper's book on About Faces 3. I am currently reading his book "The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity". He makes a strong point that interaction design has to be based on research and that is has to be made by interaction designers and not engineers. Engineers think too technical and therefore will create too complicated solutions. Engineers don't know how "normal users" think. And he is right! But I think, it's not enough to have some "interaction designers" designing the interactions and then let the programmers write the code. That might be important. But it is equally important if not more important that the the developers deeply understands the problems domain and goals the users have.

There is a great one hour talk I really enjoyed: Martin Fowler and Dan North Point Out a Yawning Crevasse of Doom (I had to look up the words Yawning, Crevasse and Doom -- those British guys speak hard to understand English ;-)). Their point is that there has to be a bridge between developers and users in order to communicate as apposed to a ferry, where information is transported from one side to the other by someone like the interaction designer, marketing person or analyst.

The solution requires what Eric Evens describes in his book as Domain Driven Design (DDD) (BTW: Domain Driven Design Quickly is a 100 page book available online describing the essentials). One of the central ideas is to have a dialog between the developers and users to come up with a "ubiquitous language" to describe the "core domain". That is: use the same language when talking to the users as when talking about the code. Create classes and methods that model the domain using the same terminology used by the users when they talk about the problem domain.

To get better software we have apply the techniques of interaction design and we have to have a dialog with our users to be able to create models that match their mind sets.

Yes, I am guilty myself of not doing this. So, this post is a reminder for myself....

P.S.: After writing this post, I did some search on ("DDD and IxD") and I figured that about a year ago I started a discussion on this topic on the IxD mailing list. I wish I would have a better memory....

Emfatic downlowad and update site available...

I like Emfatic a lot. It is a very cool textual representation of EMF ecore files. The Emfatic has a nice text editor for .emf files. Much easier to edit than the graphical ecore editor. It was originally created by Chris Daly and made available at IBM alphaworks under a restrictive license. Some time ago it became an eclipse project.

Unfortunately, I could not find a download or update site for the open source emfatic (which has quite some enhancements over the alphaworks version). Only some build instruction. I build it following the "instructions" (which is a simple screen-shot of the projects in cvs, enough for "experts"). Then I realized it also needs org.antlr-2.7.7, which cannot be hosted at eclipse.org because of IP (copyright) issues. I could not find the org.antlr-2.7.7 plugin. I had to create it.

I am sure I am not the only one who simply wants to download emfatic. So, I decided to make my build available.

Here is my emfatic download site it has a zip with emfatic and antlr file that can be dropped into an p2 drop-in folder or used as extension location (it already contains the .eclipseextension file).

And for those who like to use the update manager, I created an update site with emfatic and antlr: http://scharf.gr/eclipse/emfatic/update/

Note the emfatic plugins it require Java 1.5!

Looking for a JavaVM with the weakest possible memory model

The Java Memory Model gives some minimal guarantees about what happens if two threads are accessing the same variables. Brian Goetz describes this in chapter 7 of Java Concurrency in Practice. And here is a good summary of the new (1.5) Memory Model. Doug Lea describes the old memory model. If you really want to dig into it read some more formal specs.

The essence of the java memory models is: If one thread modifies a variable another thread may not see the change unless "some synchronization" happens (a synchronized block, a shared lock etc). But most real java VMs implement a much stricter memory model, that means if one thread modifies a variable the other thread sees it even without synchronization. And that is the problem. It is almost impossible to find those threading problems without a java VM that implements only the minimal memory model guarantees.

My favorite "theoretical threading bug" is NullProgressMonitor.cancelled should be volatile. If a java VM would implement only the minimal memory model the code would not work, but as John points out: "This is true in theory, but never happens in practice. In practice, the thread calling isCanceled may obtain a stale result for a short period of time, but the thread cache is soon synchronized. It is common practice to omit synchronization in cases where obtaining a stale value is acceptable.". This is unfortunately true. At least I could not construct an example where one thread would not see the changes made by another thread.

Getting threading right is extremely hard. Deadlocks often occur only if you have a bad day. You can get away with obviously wrong code just because Java VMs are so gracious.

I wonder if there is java VM that implements only the minimal memory model? It would be cool if it would be possible to force the VM to behave like the worst possible memory model. For example: it would not make changes visible to other threads unless the data is synchronized. This would be extremely helpful for testing and debugging purposes. I wonder how well eclipse would behave on such a "minimal memory model VM"....

Is OSGi the enemy of JUnit tests?

I wrote a set of JUnit(3.8.1) tests for the terminal. Originally those were normal unit-tests. To be able to test non public methods and non public classes, I put the tests into the same package but into a separate test plugin. I also added the test plugin as friend plugin of the packages I want to test. This works fine if I run the tests as normal JUnit tests. But if I run the same tests as JUnit Plug-in Tests I get java.lang.IllegalAccessError. The reason is simple: each OSGi bundle runs it's own class loader and therefore the classes appear not to be in the same package.

There are different solutions:

• Make all methods and classes you want to test public (really bad idea)
  
• Put the unit tests into the same plugin as your code and make the dependency to JUnit optional (not a good separation of concerns).
  
• Only test public classes and methods (I think this is to restrictive and often to coarse grain)
  
• Make your test plugin a fragment. One problem is that other plugins cannot access classes defined in fragments (as Patrick Paulin points out in a more detailed discussion about fragments in unit tests). Another problem is that plugin.xml in a fragment is ignored. And therefore you test plugin cannot contribute
  

I tried out turning a plugin into a fragment. It is as simple as adding adding a line following line to your MANIFEST.MF

Fragment-Host: org.eclipse.the.plugin.you.want.to.test

and removing the plugin you want to test from the required plugins. As long as your test plugin is not part of a bigger test case and it does not need to contribute extensions Patrick Paulin describes a solution for that using reflection, fragments are a good solution.

A good way to avoid having to use extensions (plugin.xml) in your test plugin is to use dependency injection for your classes.

But I think there should be a better way to write a test plugin that can access non public classes and members. I understand why the security concept of OSGi introduces those problems, but I am still looking for a solution for my JUnit tests.

Any ideas?

Databinding Tutorial and Sample Projects

I promised I'd add the sample projects and the slides add our "Understanding JFace Data Binding" tutorial. I added them yesterday to the eclipscon web page, but it took a day to show up. I also added a zip file with the presentation and the sample projects here.

As Wayne suggested, the projects also show it the "hard way". The best way to follow the tutorial is to use "compare with each other" between the projects. The projects are numbered and the "nodatabinging" projects show it the "hard way". But I gave up at the end doing it the "hard way" because it simply was not trivial. I'd be happy if someone could do the master detail the hard way (without binding), so we could add it to the sample projects...