Thursday, May 29, 2014

How to make eclipse attractive to new communities?

Recently, I had some discussions with different people on how to make eclipse attractive for non-java communities, in particular for web and mobile developers. This is also based on my experience doing web development. Here are some of my thoughts:

The Java mind-set in eclipse

The eclipse IDE is for many people (like me) a super powerful Java IDE, and in fact, it killed pretty much all its competitors (except for IntelliJ). Java is a strongly typed language and this allows for a lot of analysis that can be done statically and we got used to refactorings and an IDE that fully "understands" our code and its dependencies. Eclipse encouraged us to structure our projects in a way that make dependencies much more explicit by having a set of interdependent projects in the workspace. The plug-in concept enforces a modular structure and is a way of organising the workspace. That's all great.

But there is another reality: all the systems and languages that are much less statically defined and that are structured differently than java. It starts with C. CDT essentially started by copying JDT, it still suffers form that, because C and C++ is a different languages with very different mind-sets than java, although there are some syntactic similarities

Similarly, trying to impose the concepts and mind-set of Java on all those scripting languages and setup-files is probably doomed to fail. I love strongly typed languages, I love modelling. In that area eclipse has so much to offer! But there is the other text based "unstructured" world with a different mind-set, where you "simply add a line to a script to get that cool feature going" (I have to say a lot about frustrations and benefits this "it's really simple" approach can create).

Is eclipse missing this huge opportunity that has opened in the last years with the shift in direction of web technologies?

The "force of intelligence" applied to eclipse

A few weeks ago, I posted a strange and probably hard to understand article about an equation that describes the physical force of intelligence. The equation essentially says, that intelligent systems try to move into directions where they are not trapped. If we apply the equation of intelligence to this problem, we have look at eclipse and at Jetbrains (the creators of IntelliJ) as two distinct "intelligent systems". Both organisations are "intelligent systems" with a "will to survive". The way those organisations act intelligently, is by moving into a direction that "maximises their future freedom of action", as I explained in the blog mentioned above.

In eclipse, nobody has real power to "direct" what is done, instead, the eclipse eco system has to surf the waves that are coming. Its intelligence is a kind of swarm intelligence. And, I think, eclipse has done a pretty good job on this. If the common interest of eclipse (as an "intelligent" system) happens to be in sync with the interest of one or more of its contributors, that is great. But just seeing where eclipse should go, without the power to steer it in that direction, is not "intelligence" according to the equation of intelligence (intelligence requites anticipating possible futures and the power to move into a promising direction) [by that definition this blog entry may not be intelligent, because I have no power to move eclipse].

Jetbrains, on the other hand, has a pretty clear gradient to follow: it has to satisfy its customers in order to make them to pay for the product, which will give them money to maintain and increase their power of action in the future. I assume the JetBrains has more control over its employees than "eclipse" has over the community to drive the development into a "desired" direction. If they continue to make clever moves they may outperform eclipse in the long run.

If I had superpower and could drive eclipse....

What would I do, if I could influence the direction of eclipse and work with a team of people that would have the manpower to make some changes to eclipse?

I think, with a few relative simple changes or additions, eclipse could increase its future freedom of action dramatically by embracing the reality of many developers. The key factors are: 
  • simplify, simplify, simplify
  • install a minimal set of plugins and suggest additions
  • embrace the file system -- don't impose projects on directories
  • don't try to build everything in java -- integrate external tools
  • embrace the command line
  • do clever search and code completion using heuristics instead of complicated models
Or in one sentence: eclipse could be an editor that talks to its environment that can grow to full blown IDE if the user needs and wants it. Instead of being an all-inclusive self-contained IDE that can be abused as editor.

Here are some more concrete ideas to focus on:
  • Create a simple extensible editor with pluggable syntax highlight (something like LiClipse by Fabio Zadrozny)
    • leverage existing (external) syntax highlighting tools to do the job (tools that are not necessarily written in java)
    • or allow to use existing syntax highlighters like the ones form TextMate
  • allow to simply create projects that are whole directory trees including different git/subversion repositories
    • Essentially, allow to point to one one or more root directories and with some clever filtering, allow to exclude parts of the file-tree that are relevant. 
    • Make it simple to assign properties, like 'generated', 'test-code', 'external-library' to subtrees (using patterns or by some UI) to be used for filtering
  • Allow to easily integrate external tools and builders. There is a strong tendency to rewrite tools in java to integrate them. Instead, provide an environment that allows to run, manage and communicate with processes that run externally on the local or on remote machines.
  • Create a clever non-modal search tool, that understands the basic syntax of languages and that can be configured to know where to search, given a context.
    • When I am in file X, I know that file Y is relevant but file Z is not connected to where I am. The problem with search is often how to limit the search scope cleverly. 
    • Typical filters on string searches would be: Do or don't search generated files, test files or external libraries; do or don't search in comments or strings; do or don't search local variables. With some basic understanding of scopes and different areas (which could come for the syntax highlighter), search can be dramatically improved.
  • A plug-in recommender
    • Provide a mechanism that suggests to install  plug-ins based on the content of the project 
  • Provide a generic heuristics based code completion tool (similar to CodeRecommenders). 
  • Heuristics based code navigation. Instead of trying to do full static code analysis of like JDT, use clever heuristics for code navigation and "learn" form users choices.
  • Provide a simple integration with command line debuggers
    • Debugging scripting languages is often much simpler than e.g. C, C++, or hardware debugging
    • It might be enough to provide a simple integration for a command line debugger. (In the 90ies I wrote an extension to the python command-line debugger that made the commands act like gdb, so I could use the gdb integration in emacs and SNiFF+). 
These are just bullet points and if there is interest, I could expand on each of them. The underlying idea is to embrace the text based "unstructured" nature of lots of systems and go back to a simple editor that can progressively be more clever as the user needs it. Do context-based search and navigation and try to syntax highlight as many languages as possible. Instead of being the super-capable IDE that knows everything and contains lots of feature, the user never uses, be the editor that progressively becomes an IDE as the user needs it.

If eclipse cannot be used as a strong editor, the chances are low that we will attract all those developers that simply use and need a capable editor. The world is changing fast, and I wonder how many projects today can ignore the web and mobile apps that use all those dynamically typed "languages" and set-up files.

Promising future for eclipse

I think, eclipse has a bright future! I see some good attempts to go in a promising direction. The risks is not lack of ideas. The risks are lack of resources and money to fund the work that needs to be done. There are a lot of freelancers and small companies, that form the eclipse eco system, making most contributions and working very hard to keep eclipse successful. They all act very intelligently, which means they have to balance their own survival and freedom of future action with the eco system they live in.

Is that enough? How to increase the "power" to act to make eclipse as a whole more "intelligent"? What do you think?

Tuesday, March 11, 2014

Evolution: form Emacs to Eclipse to IntelliJ and back to Emacs?

A few days ago, Dinis Cruz asked "Why doesn't Eclipse community stand-up more to IntelliJ?". As a long term eclipse fan, let me explain why I am currently not using eclipse but IntelliJ (actually PyCharms) and why I almost feel like going back to emacs. It is not because I think eclipse is bad or it misses features. It is because I have to get my job done and I don't really need an IDE, all I need is a good editor with CM integration and some search and debugging capabilities.

From emacs to sniff to eclipse...

About 30 years ago I started using emacs on a VT100 terminal as my first "IDE"     for programming modula-2. When I started using C++ the first graphical IDE was sniff. About 20 years ago I became part of the team commercialising SNiFF+ and 10 years ago the team switched to Java and eclipse. Each of the development environments was better than its predecessor... Clearly,  eclipse was and is the ultimate IDE.

...then entering the wild wild web....

Since almost a year, I am working on a startup web project. The project is based on open source projects which use coffeescript, python, javascript, jquery, angularjs, json, xml, css, less, elasticsearch, sqlite, flask, grunt, nodejs, git, make, sh, markdown, reStructuredText etc. The projects are linked and nested in some "strange" ways. Not necessarily the choices I would have made. But I have little control over how the open source projects I use are organised and structured.

...with unknown territory...

When I started the project, I did not know much about the inner details and structure of the components in my workspace. It was just a tree of nested directories. I did not know much about the filetypes inside the directory tree. Nor did I understand the relationship between the files and directories. What I needed at this point was a tool that would help me explore the unknown territory. This is not exactly what eclipse is made for. Eclipse wants me to know a lot about the code in order to install the necessary plug-ins and in order to 'flatten' the nested tree into a list of projects.

...using eclipse....

So, given that 'reality', the question is how to make eclipse happy. I really tried hard for a few weeks to get this running with eclipse. But it was a constant battle on multiple front lines. How to find and install the plugins I need to support the languages I use. Another battle is that eclipse has a very strong idea on how projects should be structured. That is great if you start form scratch and build java projects. It turns out to be very difficult if you cannot control or fully understand the structure of the projects. Another front line is that egit requires a certain project layout to work properly (it expects a 1-to-1 relationship between projects and git repositories). In essence, I spend a lot of time trying to install and configure eclipse to make it work with the code, but with limited success.

On one hand, eclipse is very opinionated and that can get in the way if the "reality" does not fit the eclipse model. On the other hand eclipse offers too many choices and little advice on what to install. I want the opposite: give me the freedom to keep the files and directories the way they are and give me good advice on plug-ins I should use and install.

...stop fighting windmills and trying something else...

After a long battle fighting with my inability to force eclipse to do what I want, I gave IntelliJ (in my case PyCharms) a try. After relatively quick setup and learning phase, I came to the conclusion that it can deal better with the my reality than eclipse. It supports most of the languages out of the box well enough for me to be able to work. I could just point to the root directory with all my sources and it found the git repositories inside. I had no problem finding and installing the plug-ins I needed (in fact it suggests to install plug-ins when I open a new file type instead of opening the file with an "arbitrary" application on my system like XCode for .sh files). To borrow a term the python community often uses: "it comes with batteries included". I don't think it is the better IDE, it just works for me with much less pain. Maybe I am to opinionated on what eclipse should do and therefore I struggle. With IntelliJ I am much more humble. I am happy if I get things done.

...(ab)using IntelliJ as editor with syntax-highlight...

It turns out that I use PyCharms essentially as editor. I probably use less than 1% of the features of the IDE. I am not really happy with that. It still feels unfamiliar. And I miss mylyn very much.

What I actually use/need is very simple
  • browsing code with syntax highlight (because a big part of the code is coffeescript, and there is no usable support for coffeescript eclipse, that is one of the killer features for using PyCharms)
  • navigation (not as good as it could be)
  • search
  • editing with code completion (for me it is much worse than it could be)
  • debugging of python (javascript and coffeescript I debug in the browser)
I choose PyCharms, because I thought I need an IDE, but, if I think about it now, I could have gone back to my first love, emacs or I could have chosen one of those many clever editors out there. At the time when I made the switch I thought it has to be a real IDE. But then I learned a surprising lesson:

...the revival of simple text editors and command-line tools...

I am always interested in what kind of tools other people use when working on similar projects. I have to say, I was really shocked at the beginning, when I saw what tools really talented developers use: vi, emacs, notepad++, TextMate and a few other more or less advanced text editors. I have not seen much use of anything I would call an "IDE".

The other observation is that a lot is done on the command-line. The projects use command-line tools to install, setup, build and deploy the code. I am really glad, I got a MacBook, because most projects run out of the box on a mac. With windows this might be much more difficult.

It seems there is a new generation of developers that have never used a VT100 terminal. But they make their environment look like what was the reality 30 years ago when I started programming: simple text editors, green colors on black background and running command-line tools.

What is my conclusion of this? I think there is a different world that needs much simpler tools than the ones we have build over the last 25 years when we created super powerful IDEs. It is a kind of down-sizing when it comes to development environments. When the famous great libraries you use (jQuery, angular, node, backbone...) are just a few thousand lines of code, why should your IDE be tens of millions of lines of code?

In fact, I don't think that IntelliJ is the biggest competitor of eclipse, its all the simple text editors and command line tools that compete with both eclipse and IntelliJ.

...but eclipse can provide solutions...

Hmm, should I really try emacs again (I still use it every day, but not as IDE), but that feels so wrong. Maybe there are solutions in the diverse eclipse community.... Maybe something like a Light Eclipse or an Easy Eclipse could emerge for the eclipse eco system...

...therefore let's contribute money and resources!

...and yes that is just happening! There are some promising attempts to go in the right direction like LiClipse and EasyEclipse. Both are crowd funded. I think, eclipse has a bright future! The risks is not lack of ideas.The risks is lack of money to fund the work that needs to be done. You can help too....

Wednesday, February 12, 2014

A new Equation for Intelligence F = T ∇ Sτ - a Force that Maximises the Future Freedom of Action

Intelligence is a Force with the Power to Change the World

Describing intelligence as a physical force that maximises the future freedom of action, adds a new aspect to intelligence that is often forgotten: the power to change the world. This, I think, was the biggest revelation for me, when I started thinking about the the new equation for intelligence. The second revelation was, that intelligent systems are survival engines, that increase their chances of survival by maximising a single quantity: the freedom of action. Both insights may sound trivial or obvious, but I don't think they are.

A few days ago a saw the TED talk "A new equation for intelligence" by Alex Wissner-Gross. He presents an equation he published in April 2013 in a physics journal. It may not be the most impressive talk I have ever seen. And I had to watch it twice to fully understand it. But the message excites me so much, that I don't sleep well since a few days. I thought everybody must be excited about this equation. But, it seems that this is not the case. Either I am not understanding it correctly or others don't get it. Or maybe it resonates with me, because I am physicist, with a strong background in computing, who has done research in computational biology. To find this out, let me explain my understanding of the equation. Please tell what your think and what's wrong with my excitement (I need sleep)....

So, why did the equation blow me away? Because this very simple physical equation can guide us in our decisions and it makes intelligent behaviour measurable and observable. It adds a new real physical force to the world, the force of intelligence. From the equation we can deduce algorithms to act intelligently, as individuals, as societies and as mankind. And we can build intelligent machines using the equation. Yes, I know, you may ask: "How can the simple equation F = T  Sτ do all of that?"

Intelligence is a Force that Maximises the Future Freedom of Action

Before we look at the equation in more detail, let me describe its essence in every day terms. Like many physical laws or equations the idea behind it is simple:
  • Intelligence is a force that maximises the future freedom of action.
  • It is a force to keeps options open.
  • Intelligence doesn't like to be trapped.
But what is necessary to keep options open and not to be trapped? Intelligence has to to predict the future and change the world in a direction that leads to the "best possible future".  In order to predict the future, an intelligent system has to observe the world and create a model of the world. Since the future is not deterministic the prediction has to be based on some heuristics. Prediction is a kind of statistical process. In order to change the world, the intelligence has to interact with the world. Just thinking about the world, without acting, is not intelligence, because it produces no measurable force (well, sometimes it is intelligent not to act, because the physical forces drive you already in the right direction, but that is a way of optimising resources). The better in can predicted the future and the better a it can change the world in the desired direction, the more intelligent the system is.

The new Equation for Intelligence F = T ∇ Sτ

Note: skip this section, if you are not interested in understanding the mathematics of the equation!

This is the equation:

   F = T

Where F is the force, a directed force (therefore it is bold), T is a system temperature, Sτ is the entropy field of all states reachable in the time horizon τ (tau). Finally, ∇ is the nabla operator. This is the gradient operator that "points" into the direction of the state with the most freedom of action. If you are not a physicist this might sound like nonsense. Before I try to explain the equation in more detail, let's look at a another physical equation of force.

The intelligence equation very similar to the equation for potential energy F = ∇ Wpot. Wpot is the potential energy at each point is space. The force F pulls into the direction of lower energy. This is why gravitation pulls us in direction of the center of the earth. Or think of a landscape. At each point the force points downhill. The direction is the direction a ball would roll starting at that point. The strength of the force is determined by the steepness of the slope. The steeper the slope, the stronger the force. Like the ball is pulled downhill by the gravitational force to reach the state with the lowest energy, an intelligent system is pulled by the force of intelligence into a future with lowest number of limitations. In physics we use the  Nabla operator or gradient to turn a "landscape" into a directed force (a force field).

Back to our equation F = T  Sτ. What it says is that intelligence is a directed force F that pulls into the direction of states with more freedom of action. T is a kind of temperature, that defines the overall strength (available resources) the intelligent system has (heat can do work, think of a steam engine: the more heat the more power).   is the "freedom of action" of each state that can be reached by the intelligence within a time horizon τ (tau).  The time horizon is how far into future the intelligence can predict. Alex Wissner-Gross uses the notion of entropy S to express the freedom of action in the future. The force of intelligence is pointing into that direction. As we have seen, in physics the direction of the force at each state is calculated by a gradient operation ∇ (think of the direction the ball is pulled). The Nabla operator ∇ is used to assign a directional vector (the direction of the force of intelligence) to each state (in our case: all possible future states). The more freedom of action a state provides the stronger the force is pulling in that direction. So, Sτ is the pointing into the direction with the most freedom of action. The multiplication with T means the more power we have to act, the stronger the force can be.

Note: the optimal future state is the optimal state form the viewpoint of the intelligent system. It might not the optimal state for other systems or for the entire system.

If you want to understand the equation in more detail read the original paper 'Causal Entropic Forces - by A. D. Wissner-Gross and C. E. Freer'.

An Algorithmic Explanation of the Equation

As an intelligent system you want to get the best possible future. The best possible future is a future where you are not trapped. To get there, you try to predict possible futures. You adjust the time horizon  τ of your prediction, so that your prediction is reliable enough to make a decision. Look at all possible states and you assign a freedom of action value   to each state. In that map of states, you choose the state that has the highest potential for future actions, the state that gives you the most freedom and power. This requires that you have a model of how the world works and you have to be able determine the current state of the system. Now you move in that direction.

If the sate you are heading is not reachable effortless, you have imposed a force F on the world in that direction. This is the force of intelligence. The temperature T represents the power or "resources" you have to reach the desired state. The more power you have, the more force you can impose. As you move (in time and space), you have to constantly validate desired state and adjust the direction you move. This adjustment is needed, because your model of the future was only a vague prediction and, as you go, your internal models of the world have to be updated with the reality. 

The Physical Force of Intelligence

It is Really a New Force?

This force is a real physical force. This force is not explainable with the other physical laws. If you read this, on a computer or on paper, the letters are in an order that would not exist without an intelligence imposing forces on the physical world. Intelligence influence the world. This influence is directed. The direction can be understood by looking at the formula F = T ∇ Sτ. This means some intelligent actor came to the conclusion that this is a better state to be in. Some state changes requite little force (if I type a or s does not make much of a difference). Other state changes require a lot of directed force, e.g. building a new house.

How to Measure the Force of Intelligence?

If intelligence is a physical force, it has to be observable, right? In the TED talk, Wissner-Gross gives a nice example of how alien observers could detect intelligence on earth with a telescope: Suppose alines would measure the number of asteroids hitting the earth for millions and billions of years.  But suddenly a "magic force" appears, and asteroid do not hit the earth anymore. Instead they detonate or get deflected before they hit the earth. This is when humans have learned to to prevent the impact of asteroids with some advanced technology. For the aliens, none of the "classical" forces of physics would explain this. But the Equation of Intelligence would explain this otherwise mysterious force as the Force of Intelligence...

Is it a Break-Thru Like Chaos Theory?

I believe, this simple equation will change the way we see and understand the world. It may lead to a better future for mankind. I remember in the 80ies, when I studied physics, Chaos Theory emerged into physics. It provided a scientific explanation for the behaviour of complex systems. Once you understand that small differences in the initial conditions of non linear systems can yield to wildly diverging outcomes (butterfly effect),  chaos theory is simple and obvious. The Equation of Intelligence has similar qualities. Once you understand it, it is so obvious that you wonder why it was not formulated much earlier. As physicist, I really like the simplicity, beauty and generality of the equation. I also like the idea that we now can describe intelligence as a force that changes the physical world.

The Problem of Ambiguity of the Future

The biggest problem with this equations is to determine the entropic field Sτ and the accessible Temperature T. One might argue that this equation is useless because it is not well defined. I would agree with this critique, but the equation is a kind of approximation. The intelligence cannot know all possible future states nor can it know the freedom of action of each of the states. So, does it mean this equation is useless? I don't think so, because it as a set of interesting implications.


Let me point out a few implications of the equation.

Intelligence Needs a Dynamic Model of the World

In order to estimate the quality (in terms of actability) of future states, the system needs a model of the world. This model is used to simulate future states and to estimate risks, cost and and freedom of action for those states. This also means that, as the intelligence moves in a certain direction in option space, it has to re-evaluate the future states and correct the predictions and the direction. Our brain is a prediction engine. It makes predictions by associations. For example, we can predict the slope of a moving body without much reflection. But sometimes we have to question the automatic prediction (as Daniel Kahneman points out) and use slow analytical thinking. For example, our intuition for probability is often wrong and depending on how a problem is described we may have a bias in our predictions. Simple intelligent systems, like a single cell following the light, have very simple 'models' of the world. Many systems have no explicit model of the world. Instead they have a "good enough" build-in heuristics that they find good states to move to.

Intelligence Needs Power to Act

Intelligence has to maximise the "temperature" T in the equation to increase the force (the impact to change the sate of the world). When humans learned to utilise energy (like fire) and turn it into physical forces (like with a steam engine), they increased their influence on the world. So, just observing future states is not enough. The intelligence needs to have the power to turn the knowledge about the desired future state into a force to move there. 

The power needed to reach the desired state is a resource that has to be taken into account when we predict the freedom of action in the future. Let's consider an example: a cat is hunting a mouse. Instead of wasting a lot of energy running after the mouse, the cat sits still, and observes the mouse. It follows its movements and makes a prediction when there is a good moment to catch the mouse. This is a very good use of resources.

What is an Intelligent System?

Any system that creates a force into the direction of more freedom to act, is an intelligent system. This means a single cell or a single human is an intelligent system. But also communities can be seen as an intelligent systems. Animal colonies, nations, armies, religions, mankind can form intelligent systems. These systems are hold together by a set of rules that make them interpret the world in a specific way and they act (consciously or unconsciously) to give their system more (or less) freedom of action.  Some of those system do a kind of random walk in time and space, but others have internal models and rules that guides them into a direction of more freedom of action. True intelligence is revealed when the state changes and the old rules do not work anymore. In that situation, the ability to anticipate the new situation and find new solutions distinguishes really intelligent systems form just well adapted systems.

Hierarchy of Intelligent Systems

Let me point out, that intelligent systems are often hierarchical. Cells form plants, animals and humans. Humans form families, tribes, nations. The entire biotope of the earth is an intelligent system. Think about mankind. Each individual person has a model of the world, and the interaction of all humans form an implicit model of the world of mankind. Evolution, with survival of the fittest, is a form of intelligent system. Any believe system or memes, like a religion, atheism, capitalism or communism have a build-in set of models of the world and create forces in the world. It would be really interesting to analyse how good different believe systems are in terms of maximising the future freedom of the system. In terms of force to change the world and in terms of direction. There could be memes that have a lot of force but are misguided and die out.

Competition of Intelligent Systems

Another implication of hierarchical intelligent systems is, that intelligence at one level (a single human, company, religion, nation) may undermine the intelligence of the higher level system. Or vice versa, a higher level system has to reduce freedom of actions of its subsystem in order to maximise its freedom of action. If we see mankind as the more important system that a single human, this might have implication on how we should act as individuals, companies, or nations. If mankind is pulling in too many directions at the same time, then the overall force might be zero and therefore the intelligence is reduced to the intelligence of a bare physical system.

Brute Force Versus the Force of Intelligence

If you know what the best future would be, but you cannot or do not act, brute force systems may determine your future.  It's that simple. Thinking about a better future is not enough. The ability to act is integral part of intelligence. And that is what intelligence is actually optimising, the ability to act. The greatest idea, if not actionable, is not intelligence. Classical IQ tests measure only the future prediction part, but miss out on the power to act. Sometimes simple but strong systems rule the world and "intelligent" systems fail because they are not able to act.

Why is the Force Directed to Future Freedom of Action?

Not having any freedom of action means you are dead. Future freedom of action, essentially means to increase the probability of survival of the intelligent system. Therefore, intelligent systems increase their chance of survival by moving to states with the highest probability of survival. Because it is impossible to predict the future correctly, the best heuristic is to put yourself into a state where you are not trapped. Freedom of action means survival, which is a simple and general goal. From that, any sub goal is magically derived.

Evolution on Steroids

Because intelligent systems actively increase their chance of survival, intelligent systems are systems with an accelerated evolution. "Blind" evolution depends on random variations and the power of selection. Intelligent systems can boost the chances of survival by using force to move to states with higher chance of survival.

How to increase the force of intelligence?

There are tow ways to increase to force of intelligence:

  1. Make better predictions of the future, which means detect sates of maximum freedom of action
  2. Increase the power to move in the desired direction. By adding more energy or being clever in finding paths that require less resources.

Other Implications

The last few days, I did not sleep well, because my mind was overwhelmed by thinking about the implications of that simple equation. Here are just a few thoughts and each of them needs further exploration and refinement:
  • I read about claims that say, artificial intelligence needs senses and a body to interact with the world. I think this is true, because in oder to change the future, intelligence has to interact with the world (apply forces). In order to make predictions it has to sense the world. So, AI without a body and senses is no intelligence because it cannot understand and interact with the world and therefore cannot apply any directed force.
  • Intelligence based on that definition might not have a "moral" in the sense we use moral. The only goal it has is to maximise its future ability to act. The moral emerges from what has to be done in order to get to the best future. E.g. if you own a company and you treat you employees badly, it might fire back in the future because they may leave or they revolt against you.
  • Intelligence may flow with the laws of physics but occasionally use force to change the world in a way that opens changes in the future. 
  • As Wissner-Gross shows in his video on the Entropica webpage (which is also shown in the TED talk), a AI system that is build to maximise future options automatically chooses "intelligent" goals. For example a trading robot that maximises the portfolio, because it gives the system more freedom of action in the future within that game.
  • Politics is often not very good at maximising the future freedom of action of the nation. Instead they are focused on maximising their own power.
  • It would be interesting to analyse the way we manage software development with the question of how we can  maximising the future freedom of action....


The simple formula 

    F = T

explains how intelligence applies forces to the world that maximises its future freedom action. If we understand and apply it, we might be able to act more intelligently, which means we may put us in a better state for the future. It might provides us with cues on how to act more intelligently as individuals, as families, as communities, as companies, as countries and as mankind. 

Friday, October 11, 2013

Imagine the eclipse IDE would cost $300...


Eclipse should have a price tag. Reason: for many customers, it is much easier to pay than to provide a committer. There should be a discount system that allows for different discounts including getting eclipse for free. But the point is users have to explicitly apply for a discount and give reasons why they think they cannot pay the full amount. This is psychologically very different from donating voluntarily.

The tragedy of the commons

Recently, there have been some blogs about the tragedy of the commons [1],[2]. The tragedy is that there are not enough resources to work on some core parts of eclipse. There are many contributions but many of them are in the interest of some companies not in the general interest of eclipse. My "road construction analogy" blog four years ago explains the problem. I cannot say if things have changed since then (I don't want to make a judgement here). But I want to throw out an idea that I have been thinking about for a while...

The solution: the eclipse IDE costs $300

Imagine the eclipse foundation would sell eclipse for $300 (see below) with lots of discount options down to $0 (see below). The foundation would use the money to hire developers to work on common infrastructure to prevent the tragedy of the commons. Customers who pay could assign half of the money to specific projects or bugs. The other half is used to maintain the eclipse common part. 

Who should pay?

The payment is for anyone who uses eclipse, but is not contributing back. If you are a committer or making contributions to eclipse, you have already "payed". 

Why pay instead of contributing?

For many companies payment for software is a standard procedure but contributing is an exceptional case that requires a much more complicated approval process. Contributing back by writing code is limited. Even if a company decides to give let's say 80 hours of work back to eclipse, how efficient would that be if the contributor has to learn all the process and the code etc? 80 hours for a full time committer are probably much more productive than 80 hours for a project newbie or for someone who has never written java code. If a company wants to spent the equivalent of N hours it might be more efficient to pay for licenses instead of appointing a person within the company to contribute.

Why would anyone pay?

If you can get it for free anyway, why would anyone pay? I think there is a huge psychological difference between voluntarily paying (friends of eclipse) and choosing an option like "I believe others should pay for eclipse". The key is, if you wan to get eclipse for free you have to acknowledge that you are not paying for something that otherwise costs money. This is very different form making a donation. To donate you really have to love eclipse. To refuse to pay, you really have to hate it. 

The problem with donation

I wonder how many of the friends of eclipse ask the company they work for to pay the fee? I didn't and my guess is that many donations are private. With an official price, it should be very easy to ask your company to pay the fee, much easier than to say "I want to make a donation to eclipse, can you pay this for me...".

Payment allows you to determine the direction of eclipse

Half of the payment goes to development determined by the eclipse foundation. The other half the paying customer can distribute on bugs, features op specific plugins or projects (a kind of crowdfunding). 

Lots of complaints

No doubt, there will be lots of complains when eclipse would suddenly start adding a price tag to the eclipse IDE. Well, if you thing this is wrong, you can apply for a discount or refuse to pay. This is totally legitimate. But if you feel guilty not paying then this may say something about you and your attitude.... So, there is no reason to complain, if you think it has to be free, you can get it for free (period).

Another reason for payment: give the competition a chance

Here is another reason why eclipse should cost: Do you remember how many IDEs existed 10-15 years ago? Do you remember how expensive they were? Who survived? How can a company create a commercial IDE and make money when they have to compete with free tools like eclipse? I think it is not good for the future of IDEs in general that eclipse is free. What happens if you offer something for free until all competitors run out of business? You create a monopoly. But is that the goal of eclipse, to kill all commercial IDEs and become a monopoly? A monopoly that eventually will die because of the tragedy of the commons...


I just made an assumption of $300 but it could be $50, $100, $500 or $1000. This has to be determined. The amount of discounts is also open but there must be a 100% discount to get eclipse for free. It is important that there is a discount option that allows employees to choose a price that it is easy approvable within their company. If the internal limit for easy approval is $200, you might get a "special" discount to reduce the price to that amount.

One time or subscription price

Extending the licence to new versions of eclipse should be discounted and maybe there is a subscription licence (the exact conditions have to to be determined) 


Here are some possible reasons for discounts the user can choose form. Yes, some of the reasons are a bit cynical, but I think they reflect reality (I'm not sure if all the reasons end up in a real implementation of the idea, and maybe there are better ideas for good discount reasons)
  • I am a committer (discount of 100%)
  • I made contributions (10%-100%)
  • I reported bugs (5%-100%)
  • I use eclipse only for open source projects (up to 100%)
  • It is too expensive (you choose the price)
  • The price is above the limit I get approval for (you choose the price)
  • I cannot afford it (you choose the price)
  • I come form a poor country (10-90%)
  • Eclipse member companies should pay (100%)
  • In my country/company we never pay software licences (100%)
  • Eclipse sucks, but I have to use it (100%)
  • Others should pay, not me (100%)
  • It is my right to get it for free (because it used to be free) (100%)
To get eclipse for free, you have to acknowledge that others are paying or contributing. It is only a moral difference. If you use eclipse for free today, you do the same: you somehow mentally acknowledge that other should "pay" for it, in terms of contributions or money ....


The eclipse webpage should allow to pay directly or get a free (trial) download. After using the trial version for while (30 days of usage) a licence dialog will pop up and ask you to enter a licence or pay (like many commercial applications). Sure, you can remove the plugin or apply for a 100% discount.


The intention is to create a constant revenue stream to found eclipse ground work. The payment should be worded that accountants in companies pay the amount without much questioning. As employee to make your company to donate or asking for time to contribute might be difficult. Getting approval for a software purchase might be simple.

What do you think? Does any of that make sense?

Saturday, May 11, 2013

Support fundraising for LiClipse and PyDev

The last days I have been struggling setting up a development environment to browse and understand some open source projects that contain all kinds of different (programming) languages, like python, javascript, coffeescript, ruby, css, HTML, HTML5, sh, csh, markdown, reStructuredText etc. Unfortunately, eclipse is not really good for that use case.

I have been looking for alternatives to do the code browsing, including the JetBrains products (which are really cool) and emacs (which I use since 30 years). But I don't want to learn a new IDE (JetBrains) nor do I want to go back to the 80ies using emacs.

The real pain with eclipse is to find and install the plug-ins needed to browse and the code. Even if you have the right plug-ins installed it is very painful to set up projects correctly to parse and understand your code.

In fact if you don't have the right plug-in installed, double clicking on an unknown file-type it opens any external tool you operating system has associated with that extension which is really annoying (see bug142228).

Anyway, since eclipse is not usable for "random github projects", I was looking for alternatives and I stumbled over LiClipse by Fabio Zadrozny (the maintainer of PyDev) and a his fund raising for LiClipse at indiegogo which I think is a great idea!

LiClipse wants to bridge the gap between heavy weight IDEs and having a plain text editor. I think this is a fantastic idea and therefore I support it with a donation, and if you think it is a good idea, you should do the same... But the fund raising ends Monday 13 May 2013, so you have to hurry and add your contribution now!

I think it is important to support projects like PyDev and LiClipse because eclipse lives form contributions and many of us would do open source work for the fun of it, but somehow we have families and a life that needs funding. The choices are

  • that you do it in your spare time
  • you find a company that pays you for doing the project 
  • you create your own startup in the hope to get somehow money in the future (might be very risky and does not pay your bills now)
  • you use crowd-funding to kick off your business...
I really hope that Fabio gets enough money to do LiClipse!

Wednesday, March 20, 2013

Coming to EclipseCon 2013 as freelancer...

This year I will come as independent freelancer to EclipseCon2013. After 19 years working for the same company it is time for me to move on and get into new adventures. I had a really great time. In April 1994, I started as a freelancer working for a small startup company called TakeFive from my home office in Heidelberg (and I still work from my home office). TakeFive was located in Salzburg about 500km from where I live. Working from a home office was tough back then. I had only an Modem at 9600bits/sec (about 1kb/s) and calling to Austria was expensive at that time. So, I was regularly traveling with my sun workstation to Salzburg to synchronise my data. If I compare this with today it is hard to believe that I worked that way. The name of the company changed but it was the same old TakeFive for me:
TakeFive then was acquired by ISI in 1995. In 2000 ISI merged with WindRiver and 2009 WindRiver was acquired by Intel. Since 2009 I worked as employee at Wind River...

TakeFive was selling an IDE, called SNIFF+, originally created by Walter Bischofberger based on ET++, a very cool framework created by Erich Gamma and André Weinand. Around 2002/2003 we switched using eclipse as basis of our IDE... But it was in 1999 when I have seen the first versions of eclipse. In 2000 (before eclipse became open source) I started using it. I have been to all EclipseCons (and EclipseCons Europe) until 2010... But now it is time to meet all the nice eclipse folks again....

I am currently working on a start-up project that looks really promising but it is top secret and I can't talk much about it. Working for start-ups is a gamble. My gamble in 1994 worked for 19 years... Let's see if this one works out... I really enjoy working for a small start-up from my home office. It feels very much like TakeFive in the early days: it is challenging -- it also feels free -- free to win and free to fail, but that is the destiny of a freelancer ...

Sunday, December 20, 2009

Contradicting myself with my eclipsecon submissions....

If you look at my EclipseCon 2010 submissions:

  • 1443 Declarative UI sucks? Create your own!
  • 1480 Extensible EMF Switches
  • 1483 Annoyed by all those similar looking interfaces: IMemento, IConfigurationElement, IPreferenceStore, IDialogSettings, ILaunchConfiguration...
  • 1484 JFace/SWT Tables turned upside down

It seems that I am contradicting myself, because in 1443 I propose to create your own (copy) of a declarative UI and in 1483 I whine about all those similar looking interfaces. How does this go together?

Well, in the case of all the different property stores, they all do the same and having 10 different classes/interfaces is indeed annoying. On the other hand. I simply do not believe that there is "the" declarative UI that solves everybody's problems. Whenever you try to create the ultimate declarative UI you are doomed to fail because it is either simple but too restrictive or powerful but too complicated (I'd be very happy if someone could prove me wrong).

The idea of a domains specific language is to be Domain Specific! One could argue that UI is a domain and a (e.g. xml based) declarative UI is a domain specific language for creating UI. But, in reality UI is not UI. It's like creating a domain specific language for programming. You end up creating a Turing complete general purpose language. OTOH, if you want to create a domain specific language for form based UI on top of EMF you have a good chance it becomes a good abstractions. But do not expect this language to be used to create something like JDT or PDE.

I do not believe there is the ultimate declarative UI language. But I believe that saving properties is so universal (and simple) that one 'language' should be sufficient.

Saturday, October 24, 2009

Future of eclipse: The Road Construction Analogy

Frank Gerhardt will host a session at ESE called Symposium on Eclipse Foundation 2.0. Half a year ago I burned my fingers with my "Eclipse Freeloader Award" blog entry. I got lots of negative comments and mails. I also got a few positive reactions. I decided not to continue talking about the future of eclipse.

Today, half a year later I look back and one interesting mail I wrote at that time is an email press interview. I decided to post it as I wrote it and I use it as my position paper for the symposium.

If this blog is to long just read the "The road construction analogy" section. This is the part of the interview I am most proud of....

The last months I have been heavily involved in an internal Wind River project based on eclipse and modeling, especially Xtext and I have not had too much time to work on eclipse except bug reporting and asking stupid questions on the tmf newsgroup.

I have no idea how the community will react on that (very long) post.

Disclaimer: The text below was written half a year ago. It expresses my personal opinion at that time. I have not followed the recent developments and I might change my mind at any time... ;-)

My reply to an e-mail interview on April 18. 2009:

Q: If you were ready to give the freeloader award, who would be the three finalists for the "honor." Why these guys?

A: I had no particular company in mind. However, it is the general mentality of the industry that frustrates me: the attitude to take advantage of something like open source and not give back anything to the system. This is also known as the "Tragedy of the Commons": Scott Lewis pointed out that the bigger a community is the less people participate: "The Logic of Collective Action"

From an architectural perspective, there are things to be done in eclipse that should not been driven by direct interests of some companies but in the common interest of the community. IBM somehow took this role for a while, when they put some of their best people on the core of eclipse. At that time they were interested in the overall success of eclipse. Since some years they started removing quite some developers from eclipse (and put them on the Jazz/RTC project).

I believe there should be an independent group of developers driving eclipse. In my opinion it would be best if they would be paid by the community but act independently in the best interest of eclipse.

Now the questions is: why should companies put money into something that is free? If they put money in the system and their competitors do not, they have a competitive disadvantage. They support the community but they have no direct advantage. And in fact the company I work for (Wind River) just ended their membership (which is a sponsorship of the foundation) just recently.

That is why I was brainstorming about ideas how to stimulate companies to contribute. I am really afraid that eclipse will suffer in the future because the architecture degenerates over time and there is not enough manpower to keep modernize the architecture. Yes, there is e4 (the next generation of eclipse), but I am not 100% convinced that this is the solution.

To stimulate companies to give money/resources to support the commons there has to be a benefit. If there is no positive benefit why would a company do that? So, my idea was to create peer pressure. One way to create peer pressure for companies to make them avoid negative press. A freeloader award would create negative pressure. No company would like to win the freeloader award.

Q: Other people I've spoken to, include Mike M. at Eclipse are much less concerned about the problem. Why do you see it as so serious?

A: Well, I probably have a different perspective than Mike. Mike sees the eco system and the new companies joining eclipse. My focus is more on the architecture of eclipse. So, the irony is that although eclipse is widely successful the underlying architecture ages and dissolves slowly. Partly because eclipse is stretched into so many different directions. It is never good for an architecture to go into too many directions at the same time. When eclipse started it was a platform for IDE like applications. It was later retro fitted to be the basis for rich client applications. The people ported eclipse to embedded systems. Then the it was used for the web and for servers.... All this is good for the short term success but I am afraid that the long term impact on the architecture could be disastrous. The architecture council is not really focusing on architecture. Partly because the members have never been the ones that created and drove the original architecture. And partly because architecture by committee is doomed to fail.

The road construction analogy

One analogy for the general eclipse community problem I have been thinking about recently is road infrastructure in a country. Suppose a big company (=IBM) created a basic road infrastructure and decided to make this available to the public for free (=original eclipse).

In the beginning, the big company somehow drove the journey and continued building the free infrastructure. They also had some interest in this infrastructure because they wanted to have a well accepted public road system to deliver their products. But over time they gave the roads to the public and let the other companies continue building the roads. The hope is that the market is self regulating and it would continue maintaining the road infrastructure and new roads would be build by other companies.

In the beginning the companies understood the plan of the big company and continued building in that spirit. But some companies realized that it is not in their interest to build roads for their competitors. And it makes no sense for them to maintain the highways in general. They might build junctions to their private road network but hope that others maintain the net of highways created by the big company long ago. Sure they would fix some obvious holes on the roads. One could argue that roads build like that really reflect the needs of the community. But some of companies make sure the competitors do not get access to their key roads. What happens is: many private roads are be build in parallel, instead of some new highways. Companies would build "public roads" that connect their private roads. If the companies would understand that they all would benefit form good public roads, then they could come together and hire an independent road building organization and give them money (a kind of tax) to build the roads that are in the interest of the entire eco system. But at the moment there is no authority to enforce the tax payment. Lots of the roads are public and some companies donate "public roads" to the system. Who could blame those companies? They give something free to the community but also act in their own interest. They are much better than companies that use the public roads without contributing back. But something is wrong here. The nicely designed infrastructure degenerates over time. Can we blame the ones who contribute for that? That is the key to the problem: companies are contributing, but the contributions are not really in the best interest of the eco system.

How to solve the problem? How to make sure that roads are build that are in the common interest? Would it make sense to have a "tax paid" independent organization build and maintain the infra structure? How can we motivate the community to pay the taxes? Can we blame the companies that fix the roads and add now ones?

Q: Any suggestions, besides public disgrace, to ease the problem?

A: The questions boils down to: how can we motivate companies to maintain and enhance the common infra structure?

Positive motivation: all participants understand that they have to invest into the system beyond direct interests. But do all participants understand that? And how to deal with community members that do not want to pay their taxes? If someone can get away not paying the taxes the others who pay the taxes are the stupid ones. So, positive motivation works if all participants understand that they depend on the system. It essentially requires a certain level of moral. The bigger the community (and the more anonymous) the easier it is to get away without paying the tax. And in a capitalist world companies act in their own interest. Who can blame them for doing so? How else can they survive?

Negative motivation: have a police, a pillory, public opinion (press), a freeloader award. Something that puts pressure on members that do not want to pay their "taxes"...

Q: I always prefer to quote people by name, but if you like, I can withhold yours.

A: You can quote me. Those who know me know that I can change my mind very quickly. Partly because I simply forget what I say and partly because I like to look at problems from very different angles. Therefore, I could never ever become a politician because you could find lots of statements where I contradict myself.

If I look into the eclipse world, I see that the number of people warning about the future problems is underrepresented. That is why I raised my voice. If everybody would scream and warn about the future of eclipse, I would certainly find a lot of good arguments why eclipse has a bright future and why the current eclipse eco system works so exceptionally well and why eclipse is a great example of a successful open source project.

But at the moment I think there is not enough awareness of the problems of the tragedy of the commons. And that companies think they act in their own interest, but in the long term and from an higher level, they do not act in their own interest because they "ultimately destroy a shared limited resource".

[sorry for the long reply, but this is a topic that upsets me at the moment because I can see the disaster coming]

Thursday, September 17, 2009

Never put things above a user specified path

If you tell a program to use a path to store something you normally expect that the program puts its data inside that path and not into the parent. Sounds simple, but in the last 24 hours I ran two times into violations of this basic principle.

1. For the next release of our Wind River Workbench product we will use p2. We have thought our customers to use the -configuration option to redirect the place where eclipse puts its "stuff" if they use a shared installation. If they have different installations or different versions installed, that worked nicely (before p2): for version 3.0 they would use -configuration /someplace/wrwb3.0 and for version 3.1 they would use -configuration /someplace/wrwb3.1. Unfortunately since version 3.4 eclipse violates the principle "never put things above a user specified path" (bug 230384) and puts things like artifacts.xml and the p2 directory above the directory specified. If customers mix different versions of our product the old way, then they get a mess because two configurations that are supposed to live side by side suddenly share the potentially incompatible information in the parent. We are thinking to change the behavior of -configuration to add implicitly one more directory (to turn a user specified -configuration /someplace/wrwb3.0 into -configuration /someplace/wrwb3.0/configuration) so that the two configurations -configuration /someplace/wrwb3.0 and -configuration /someplace/wrwb3.1 do not infer with each other.

2. I am really excited about Xtext! Yesterday I created a new Xtext language project (let's call it gr.scharf.mylanguage). The wizard suggests to call my language gr.scharf.MyLanguage. Sounds good, but it puts its files into the package gr.scharf which is not such a good idea, because it pollutes the parent name space (see bug 289780) and therefore violates the principle "never put things above a user specified path".

Conclusion: if the user gives you a path put things underneath the path and do not touch the parent!

...think twice to make sure not to violate this simple principle....

Saturday, June 13, 2009

Every pixel counts! The new eclipse fullscreen plugin...

A few days ago Kevin McGuire blogged with the title Every pixel is sacred (not any more!) and a follow up Eclipse UI Real Estate Wasters!. I still believe that every pixel counts and I often wanted to have eclipse in full screen mode to save a few pixel. I use full screen mode in the browser and with pdf files a lot and I really love it. I remembered that there is a new Shell.setFullScreen method in SWT. During a boring meeting, I decided to create a small plugin that would add a toolbar button and a menu entry to toggle eclipse in to fullscreen. I started with the "New Plugin wizard" and choose the Hello, World Command example and replace the handleCommand line with one line:

I changed the labels in the plugin.xml and I was done, in less than 10 munites.....

I should know that even "simple" things can be quite complicated.
I wanted to make the plugin nice and share it but that was the start of a nightmare..... but that is another story.


You can use this site as update site or download the plugin from here:


The plugin has been written for eclipse 3.5 but it works also for eclipse 3.4 although there are some problems with the handling of the toggle state of the commands...

The plugin adds a Fullscreen command to the Windows menu and a Keybinding M3+F11 which translates on windows to Alt+F11:

When fullscreen mode is active all (main) windows are in full screen mode:

There is also a Toolbar button that can be enabled. Because the plugin uses the Commands Framework there are some unfortunate restrictions (bug 280033):

  • When you go to Window->Customize Perspective->Tool Bar Visibility, you will not find the Fullscreen tool bar unless you have enabled it in Window->Customize Perspective->Command Groups Availability

  • The Toolbar details are empty, although the toolbar contains a command:

Once you have succeeded enabling the toolbar button, it will look like this:


I decided to hide the button by default, because in most cases the shortcut and the windows menu command should be enough.