avoiding bootclasspath

[Jason Kinzer]

Greetings,

I noticed the scalac, scala scripts place scala.jar on the bootclasspath (i.e. -Xbootclasspath...). Unfortunately for me, altering the bootclasspath makes Scala a no-go for deployment environments I'm interested in (e.g. Webstart). As a clueless hack, I simply moved it to the classpath and was delighted to see that the couple examples I tried (a parser, swing app) still worked correctly.

So my question: is this approach merely for convenience (perhaps so that a user can change normal classpath settings at will) or does the Scala runtime rely on it and my results are a fluke? And in either case, will Scala2 share the same behavior?

Thank you much for your time.

Jason

[Judson, Ross]

I put scala.jar on the regular classpath all the time and haven't had a
problem.  Give it a try.

RJ
 

-----Original Message-----
From: Jason Kinzer [mailto:[hidden email]]
Sent: Friday, February 03, 2006 6:20 PM
To: [hidden email]
Subject: avoiding bootclasspath

...

So my question: is this approach merely for convenience (perhaps so that
a user can change normal classpath settings at will) or does the Scala
runtime rely on it and my results are a fluke? And in either case, will
Scala2 share the same behavior?

...

[Lex Spoon]

"Judson, Ross" <[hidden email]> writes:
> I put scala.jar on the regular classpath all the time and haven't had a
> problem.  Give it a try.

Right -- the scripts are for convenience.  If you are doing something
unusual, it's probably easiest to add the scala library's jar onto the
classpath of your system.


That said, we have been chatting the last few weeks about what exactly
the "scala" script should do.  The current behavior of the beta is
that the "scala2" script includes all jars in your classpath that it
finds in your sbaz directory.  This way, you can install jars using
sbaz, compile against them using scalac2 with no extra options, and
run linking against them using scala2 with no extra options.  This is
very convenient in practice.


It looks like the ideal solution, though, would be to go down a design
path like ELF on Unix.  With that kind of design, every jar would have
linkage information: what it provides, what it needs, what breadth of
versions it is compatible with.  When you run a program, what actually
runs first is a little linker program -- /lib/ld-linux.so on Linux --
that takes all the available hints into account and chooses which jars
to include on the classpath.

But, no one has put any time into putting such a thing together, so
the current situation is that the script is convenient for most
people, but you have to manage classpaths for ultimate control.

-Lex

[Judson, Ross]

Seems like we need some sort of "runner" program.  The dependency
information is already in the sbaz description file.  If we extended the
sbaz description file to include a notion of "launches", sbaz could
create run-files (scripts) for each launch and place them in the bin
directory.  For each installed component, the recursive set of
dependencies can be generated and set into the run-files.  

RJ
 

[Jamie Webb-3]

On Mon, Feb 06, 2006 at 10:40:51AM -0500, Judson, Ross wrote:
> Seems like we need some sort of "runner" program.  The dependency
> information is already in the sbaz description file.  If we extended the
> sbaz description file to include a notion of "launches", sbaz could
> create run-files (scripts) for each launch and place them in the bin
> directory.  For each installed component, the recursive set of
> dependencies can be generated and set into the run-files.  

We took the approach of loading in the dependencies from within Java,
and configuring a ClassLoader appropriately:

    http://oss.sygneca.com/javarun

This has been far more reliable than trying to specify the classpath
externally. Particularly so in awkward environments such as Cygwin,
but even on Linux we were previously getting strange problems with
classes apparently randomly failing to load, which we can only
attribute to JVM bugs.

Javarun expects a list of dependencies to be provided in a file one
per line, but it could easily be modified to read sbaz descriptions.

-- Jamie Webb

[Lex Spoon]

Actually, there is already an ant task for creating runner scripts.
Here's a use of that task which sbaz uses to create its own runner
script:

  <scalascript
      file="build/toolScripts/sbaz"
      name="Scala Bazaars command-line client"
      class="sbaz.clui.CommandLine"
      version="${sbaz.version}"
      classpath="#SCALA_HOME#/lib/sbaz.jar:#SCALA_HOME#/misc/sbaz/scala2-library.jar"
      javaFlags="-Xmx256M -Xms16M"/>



As you can see, this script is built with an explicit classpath.
Alternatively, you can leave out the "classpath" specification, and
the generated script will use every jar that is in the target
machine's sbaz/lib directory.  If you are happy with either of these
scenarios, then the current Scala toolset (in the beta) is enough.


It would be nice to take it further, though, and handle transitive
dependencies.  If there are transitive dependencies of your libraries,
it would be great to pick up on those, and it would be especially nice
if the dependency resolution happened at link time, not at compile
time.  That way, people can install new versions of libraries that
depend on other libraries.  A classic case is the compiler itself; if
you depend on scala2-library.jar right now, then you need to also link
against fjbg.jar and msil.jar.


I am familiar with the javarun script, by the way.  In fact, it was
part of the inspiration behind the way this script hackery works right
now!  The question it leaves open, though, is how the contents of the
classpath config file get created.  If I read correctly, right now
each user must set up their own file.  Maybe that's the best one can
hope for in Java-land, but I was hoping to get sbaz to the point where
you could "sbaz install eggtimer" and then have "eggtimer" as a new
command available at the shell.

The best design I see so far seems to be to have a custom dependency
entries in the jar files (or possibly elsewhere in the sbaz
directory), and then to write a little runner analogous to
/lib/ld-linux which processes all the dependency information to figure
out what classpath to use.  The main questions remaining are then what
kind of dependency information exists (depends-on, provides,
versions?), and where all this information is stored (sbaz jars,
external jars, environment variables, config files...).  It is easy to
get carried away :( , but hopefully there is a reasonably small set of
features in there which will get library dependencies working as
nicely as they do in Debian and other Linux distros.

A side benefit would be that if you have a big /usr/share/java
directory full of libraries from a 3rd party, then you could link
against those even though (a) they aren't managed by sbaz and (b) they
might be in a different location on each person's machine.  The
approach would be to have a ld.path file somewhere that lists all
directories to scan for jars....


-Lex

[Jamie Webb-3]

On Tue, Feb 07, 2006 at 12:29:20PM +0100, Lex Spoon wrote:
> The best design I see so far seems to be to have a custom dependency
> entries in the jar files (or possibly elsewhere in the sbaz
> directory), and then to write a little runner analogous to
> /lib/ld-linux which processes all the dependency information to figure
> out what classpath to use.  The main questions remaining are then what
> kind of dependency information exists (depends-on, provides,
> versions?), and where all this information is stored (sbaz jars,
> external jars, environment variables, config files...).  It is easy to
> get carried away :( ,

As it happens, I am just now having to address this same problem for
the next version of Kitten (it's morphed into a more general component
container, and the components have to interact).

At present, I'm thinking that there are two types of dependencies,
which I'll call internal and external: internal dependencies do not
define classes that are part of the exposed API of the depending
package, whereas external ones do.

The difference is important where there are multiple incompatible
versions of a package in use by different components (depressingly
common: Log4j is a prime example). If the dependency is internal, we
can load the different versions in different classloaders and they can
happily coexist. If the dependencies are external, we need to maintain
class identities and thus only one copy of the dependency can be
loaded. Two packages which externally depend on different versions of
the same package are incompatible and cannot coexist inside a
classloader (but it's okay if they are themselves internal
dependencies of different packages...).

> but hopefully there is a reasonably small set of
> features in there which will get library dependencies working as
> nicely as they do in Debian and other Linux distros.

Linux distributions have the advantage that they always deal with
source code, tweaked as necessary, either directly as in Gentoo, or
packaged by a centralised group as in Debian, and they have a notion
of a version for the whole distribution. They typically disallow
library version conflicts and settle upon some 'latest' version of
each package, which all other packages must conform to.

Since the Java world seems to insist that Java binaries are portable,
despite all the evidence to the contrary, and since AIUI sbaz is
intended to be much more distributed than a Linux distribution, I
think things get a little more tricky...

> A side benefit would be that if you have a big /usr/share/java
> directory full of libraries from a 3rd party, then you could link
> against those even though (a) they aren't managed by sbaz and (b) they
> might be in a different location on each person's machine.  The
> approach would be to have a ld.path file somewhere that lists all
> directories to scan for jars....

That's what we were hoping for with Javarun, but because jars aren't
properly versioned, we found that in practice we had to explicitly
specify the dependencies in a per-program configuration file. I think
this at least means forcing all jars to have canonical
'package-version.jar' names.

I think this is what you're getting at with your 'package universes'
business, which I have to say I'm a little sceptical about (it may
make life easier for the developer, but what about the user?)

Ok, I'll stop there. This is a can of worms indeed... I hope some of
it makes sense and is at least vaguely relevant.

-- Jamie Webb

[Lex Spoon]

I plan to read your email more closely tomorrow, Jamie, but on one
quick note:


Jamie Webb <[hidden email]> writes:

> Linux distributions have the advantage that they always deal with
> source code, tweaked as necessary, either directly as in Gentoo, or
> packaged by a centralised group as in Debian, and they have a notion
> of a version for the whole distribution. They typically disallow
> library version conflicts and settle upon some 'latest' version of
> each package, which all other packages must conform to.
>
> Since the Java world seems to insist that Java binaries are portable,
> despite all the evidence to the contrary, and since AIUI sbaz is
> intended to be much more distributed than a Linux distribution, I
> think things get a little more tricky...

I would say Scala Bazaars is actually very close to a Linux
distribution.  It's a cleaned up APT which is somewhat specialized for
Scala's and Java's needs.  So if you know Linux distributions, I would
hope sbaz is familiar.


We seem to agree that it is not sensible to try for packages to be so
portable that they can coexist with packages developed completely
independently.  Or more specifically, maybe they can coexist, but you
have to run them in such isolation that their interaction is close to
zero.  I can say a lot here, but will leave it at that for now.

Once we abandon this idea of completely portable packages, you end up
admitting that someone needs to do all the little cleanups necessary
to make one component work with some other set of components.  Those
little cleanups are *packaging*, and the resulting sets of packages
are distributions-- or, as I talk about it abstractly, "package
universes".  To be precise, a package universe corresponds to one
stream of development within a Linux distribution, e.g. Debian/sarge.

Individual package universes have rules, but they tend to be more
informal than what a package tool would insist on.  Importantly, the
rules are different for different groups.  The needs of a 6-person
collaboration with 10 packages are different than those of a
Debian-sized collaboration with 10,000 packages.



> I think this is what you're getting at with your 'package universes'
> business, which I have to say I'm a little sceptical about (it may
> make life easier for the developer, but what about the user?)

Yes, exactly, it's the main idea of package universes.

How does it cause more work for users?  Users just say "sbaz install
finger", which is easy.  Developers (and package maintainers) have to
actually make this work, which is, well, work.  I don't know a
plausible way to completely eliminate that work, though better systems
can reduce it....


-Lex

[Jamie Webb-3]

On Tue, Feb 07, 2006 at 05:38:20PM +0100, Lex Spoon wrote:
> We seem to agree that it is not sensible to try for packages to be so
> portable that they can coexist with packages developed completely
> independently.

No, I agree that it is difficult, but if we are ever to reach that
fabled utopia that is component-oriented software development, I think
we have to try as hard as we can. And the stuff I mumbled about
dependencies is the approach I'm looking at taking for Kitten.

> Once we abandon this idea of completely portable packages, you end up
> admitting that someone needs to do all the little cleanups necessary
> to make one component work with some other set of components.  Those
> little cleanups are *packaging*, and the resulting sets of packages
> are distributions-- or, as I talk about it abstractly, "package
> universes".  To be precise, a package universe corresponds to one
> stream of development within a Linux distribution, e.g. Debian/sarge.

I recognise that, but the existence of multiple 'universes' in Debian
is one of the reasons I switched to Gentoo. The canonical problem is
simple: I run Testing or Stable, but I want to run a program that is
only available in Unstable. The only way I can use the packaged
program version is to upgrade my entire system to Unstable (apt can be
persuaded to do partial upgrades, but they generally don't work). Why
should I compromise the stability of my entire system just to run a
single possibly-unstable program?

And that's just within one centrally-managed project. Add a couple of
forked projects with their own universes (e.g. Ubuntu), and unless a
herculean effort is made by packagers to support every universe going,
end users are bound to be disappointed.

In practice, I'd skip the package manager and compile my program from
source, and it probably worked, once I'd manually figured out
dependencies. Gentoo avoids the problem by always compiling from
source, so package versions can pretty much be mixed and matched as
you see fit. It has a notion of Slots, which allow multiple versions
of the same package to coexist where necessary (e.g. GTK1 and GTK2).
Conflicts can occur, but they are fairly rare.

Going back to Java, we don't have to be anywhere near as strict as
Debian is, because we have much better ABI compatibility than for
native code. For the most part we /can/ just use the latest binary
version of software and there's a reasonable chance it will work. The
Log4j example I gave is certainly far from unique, but I also don't
think it's so common that it rules out even attempting to have
independent packages coexist. It only means that you can't just stick
everything on the classpath and expect it to work.

> Individual package universes have rules, but they tend to be more
> informal than what a package tool would insist on.  Importantly, the
> rules are different for different groups.  The needs of a 6-person
> collaboration with 10 packages are different than those of a
> Debian-sized collaboration with 10,000 packages.

But is the 10-package universe useful? Won't it be forever expanding,
duplicating the work done in some larger universe? Either that, or
each user ends up needing their own custom universe, with the
particular packages they need, at which point the whole exercise has
gained nothing.

Incidentally, is there more than one sbaz universe at present?

-- Jamie Webb

[Lex Spoon]

Hey, Jamie, I will probably just have to say that I find the strong
form of components utopia to be an unobtainable grail.  Arguing about
it seems about as effective as arguing about mandantory static typing.
I don't understand them, and they don't understand me.  But let me
take one shot at it.


The basic argument is that no components system completely prevents
components fouling each other up even when they ought to work
together.  Therefore, any practical components system must have a way
to cope with this situation, some way for humans to get into the loop
and modify one component or the other.  If you leave this out, your
system is hopeless.  If you put it in first, then a lot of problems
that used to be show stoppers instead turn into opportunities for
improvement.

Component utopians seem to believe that a sufficently advanced
component system can allow all components to work together when they
are intended to work together.  Some start down the path of isolating
components from each other.  That's a good path to start down, I agree
-- having some kind of skin between components help prevent a variety
of problems.  The extreme stance is incorrect, though: components are
not supposed to be *completely* isolated.  They are *supposed* to
interact with each other.  If I install a new version of a library,
it's because I want some new behavior on my system.  That system is a
result of components interacting with each other.

You have to leave up some channels of interaction, and every such
channel is a possibility for components to foul each other up.  Even
speaking through a perfectly type-checked API leaves open questions
like, is 0 a valid argument for this or that method?  Even if you use
a theorem prover on all of your components, you merely defer the
problems to problems of specifying what properties, exactly, have been
proven.  Components that should work together sometimes simply are not
going to.  You have to have some mechanism for dealing with this
situation and tweaking one package or the other so that they work
together.


If you leave out the override mechanism, every possible form of
conflict becomes a show stopper.  Different components were compiled
against different versions of each other?  Then you *have* to allow
different versions to coexist.  Different components put stuff in
different parts of the namespace?  Then you *can't* have any global
namespace at all.  Etc.  You end up giving yourself ultimatum after
ultimatum.  As I argued above, the ultimatums never run out.
Eventually one component or the other has to be tweaked.  To make a
human analogy, if two people want to cooperate, then one or the other
has to be willing to compromise a little; it is nonsensical for
people to cooperate while being completely unwilling to reach out a
little for each others' needs.



Thus, include the override mechanism from the beginning!  If you do
so, you immediately get a working system.  Maybe overrides are needed
frequently, but at least they are always possible.  From here, you can
take all of the cool ideas that were ultimatums in the above
paragraph, and consider them as possible ways to improve the already
working system.  For example, maybe it helps to have multiple versions
of the same jar coexisting.  Or, maybe the hassle caused by the extra
complexity is worse than the hassle caused by not having it.  It's a
rich design space, now that the design space is non-empty.


I don't think that source recompilation at install time changes any of
this.  So, I don't expect that Gentoo has managed to avoid the above
fundamental problems.  But this email is too long already, so I'll
leave that for another post.

One thing I'll mention is that the multitude of universes you mention
is only a problem if you overlook repackaging.  Debian/unstable has
10,000 packages in it.  That's a finite universe, but an awefully
large one.  The reason it has so much stuff in it is that anyone can
contribute a new package.  Fundamentally, someone has to do the work
to check that a package works nicely with others, and that work might
require a number of trivial little changes to the package.  In
Debian-like systems (and package universes in general), that work is
done by packagers.



> Going back to Java, we don't have to be anywhere near as strict as
> Debian is, because we have much better ABI compatibility than for
> native code. For the most part we /can/ just use the latest binary
> version of software and there's a reasonable chance it will work. The
> Log4j example I gave is certainly far from unique, but I also don't
> think it's so common that it rules out even attempting to have
> independent packages coexist. It only means that you can't just stick
> everything on the classpath and expect it to work.

Just by the way, you can perfectly well statically link in sbaz if you
want.  In fact, the sbaz package itself does that.  In that case,
though, you should probably not put the libraries in /lib.


>
> > Individual package universes have rules, but they tend to be more
> > informal than what a package tool would insist on.  Importantly, the
> > rules are different for different groups.  The needs of a 6-person
> > collaboration with 10 packages are different than those of a
> > Debian-sized collaboration with 10,000 packages.
>
> But is the 10-package universe useful? Won't it be forever expanding,
> duplicating the work done in some larger universe?


One concrete example would be a 3-person team developing the software
infrastructure for a small business.  They work with 10-ish packages
that are specific to the business.  Scala bazaars supports that, just
as Debian's APT does (though, I think, more clumsily).

Now, yes, those 10 packages do not do everything they need.  They
would not include data structure libraries, GUI frameworks, CORBA
integration, etc.  Thus, the 3-person team would work within the
*union* of the 10-package universe plus a larger one that includes all
the reusable stuff.  The rules for the 10-package universe could be
essentially nil, because it's just 3 people who know each other well.



> Incidentally, is there more than one sbaz universe at present?

There are 3 that I know of.  One on lexspoon.org (scala-dev), one is
on a server at EPFL that is planned to be the new scala-dev, and one
on my laptop (sbaztest) used for testing new versions.

I'm not being flippant.  Anyone can create their own bazaar, just like
any group can make their own wiki or mailing list, and just like in
meatspace where anyone can create their own club.  I don't think that
making a mailing list should require registering with a central
authority.  Nor should having a Wednesday-night bridge game with the
Joneses require registering your meeting somewhere.

Likewise for package sharing.  Every sbaz directory includes a file
meta/universe that specifies the universe of packages visible to that
particular sbaz directory.


Less philosophically, we host scala-dev right now, but I would imagine
after 0.5-2 years or so that someone freezes off something like an
Ubuntu release or a Debian/stable release, thus providing a stable set
of standard libraries for Scala hackers to use.  Exactly who does that
and when and what the rules would be and where the result is posted
are all completely up in the air right now.  Enthusiasts are welcome
to create their own distributions if they like....


-Lex

[Judson, Ross]

Every type system I've read about deals with _right now_.  Having a type
system that allows claims to be made about a program at a certain point
in time doesn't quite fit with the real world, where we want to be able
to make the same sorts of claims _over time_, and _over version_.  

I have no idea how to do that, although I suspect that recent efforts to
generalize type systems into constraint logic can lead to
constraint-logic based dependent types that support a notion of order.
From there we can get to bitemporal constraints on type systems.  If we
have a good notion of type conformance we can test for conformance
between types at different points in time (between different versions).
At that point it is a compiler error to incorrectly use a nonconformant
type.

Note that this is a place where Scala's implicit function calls can
help, automatically inserting _evolution_ functions.  Or by using a
wrapper based on Future[T], we can specify the _version set_ of a given
T we can handle.  Scala's path-dependent type system seems like _the
answer_ for handling structured and nested types.  I wonder if, at the
type system level, a path-based type static identifier can be extended
into an expression including a constraint on time/version.  And how
similar is such a constraint to GADT support?  A set of GADT
"possibilities" bears significant resemblance to a discrete set
constraint in a finite domain.  

There are a lot of variables, and _proving_ behavior about such a type
system seems like a tough problem.

But...the nice thing about what we do is that we don't have to prove
things in order to use them.  We can just build them anyway and hope for
the best ;)  

Scala looks like a very good language to construct a constraint
programming library.  With constraint-based type systems moving towards
generality I should think Prof. Odersky and the Scala team could benefit
from a strong constraint system "in the toolbox", both for use in the
compilation system and for use in the embedded query language area.

RJ

P.S. On the runner issue I like the idea of a small program that
assembles the "right" classpath for itself, then executes.  If we blend
the notion of sbaz with JNLP we get to a nice spot.  By extending sbaz
syntax to include "launch" information, our launcher can also
potentially interact with sbaz to retrieve components not yet installed,
merge them into the local system, then continue execution.

"sbaz run test" could install test, the packages test is dependent on,
then run the result.  

[Jamie Webb-3]

On Thu, Feb 09, 2006 at 01:05:44PM +0100, Lex Spoon wrote:
> Hey, Jamie, I will probably just have to say that I find the strong
> form of components utopia to be an unobtainable grail.  Arguing about
> it seems about as effective as arguing about mandantory static typing.

Certainly similar. Arguing that components are useless because
sometimes they break each other seems to me rather like arguing that
static type systems are useless because they can't prevent all runtime
errors. :-)

In both cases, I'd say that while they can't be relied on, they are
still very helpful as far as they go.

[ Snip long bit about components needing to interact, and needing a way
to override that interaction. ]

That method of overriding is exactly what I started by arguing that
sbaz doesn't have! It's what I'm trying to design for Kitten. I'm also
trying the minimise the need for it.

Suppose we have a universe of packages that all play nicely together.
Now we want introduce a problem package that has dependencies which
conflict with those of some other packages.

AIUI, in sbaz there are two options: modify the source of the package
to work in the universe, or create a new universe. The first is of
course the ideal solution, but it may not be feasible, either because
the source is not available, or because there is not development time
spare to do the conversion. The second option I consider unworkable:
what if I need one of the packages that can't fit into this new
universe?

The package system for Kitten should add some more options: if the
conflicting dependency is internal (i.e. other packages don't need to
talk to it), we can just isolate it in its own classloader. Problem
solved. If it's external, we'll have to do more work: we can create a
new component (probably corresponding to the GoFs Adaptor, Mediator,
or Facade patterns), behind which the problem package can be isolated.
That's an undesirable route, but it may be necessary in some
circumstances.

So, the key difference between Kitten and sbaz is that Kitten has
configurable levels of isolation, whereas AFAICT sbaz does not have
any.

> I don't think that source recompilation at install time changes any of
> this.  So, I don't expect that Gentoo has managed to avoid the above
> fundamental problems.  But this email is too long already, so I'll
> leave that for another post.

In Gentoo, compiling from source has several advantages:
- Conditional compilation based on detected dependencies, i.e.
  typically autoconf.
- Conditional patching by the Gentoo packager if required.
- Avoidance of ABI breakage.

For Scala, the (smaller) advantages would be:
- Correct mixin versions.
- Possibly improved robustness against changes to Scala.
- Possibility of conditional compilation (most likely of whole files,
  since we don't have a preprocessor).

I'm not really suggesting that sbaz should be source-based (though the
mixin problem could bite). My main point about Gentoo is that it
assumes by default that arbitrary versions of packages will just get
along (and takes a few steps to encourage that), and for the most part
they do. Debian (primarily due to C ABI issues) has to assume that
mismatched versions of software will not work together, and thus has
to define separate universes.

Put another way, I think sbaz can afford to act more like Gentoo than
Debian, because Java ABIs are more robust. But, it needs to deal
gracefully with conflicts when they do occur.

> Just by the way, you can perfectly well statically link in sbaz if you
> want.  In fact, the sbaz package itself does that.  In that case,
> though, you should probably not put the libraries in /lib.

I'm not sure what you mean by static linking. Java doesn't have such a
concept. Unless you explicitly use separate ClassLoaders (which I'm
arguing is necessary), everything goes into the same namespace,
regardless if where it's physically located. Unless you mean defining
the whole thing as a mixin?

One trick I understand some people use is to rename the packages of
internal dependencies, so if I use Log4J and want to avoid conflicts,
I can go through the Log4J sources and change all the package
declarations to e.g. com.sygneca.org.apache.log4j. That's pretty evil
though.

> One concrete example would be a 3-person team developing the software
> infrastructure for a small business.  They work with 10-ish packages
> that are specific to the business.  Scala bazaars supports that, just
> as Debian's APT does (though, I think, more clumsily).
>
> Now, yes, those 10 packages do not do everything they need.  They
> would not include data structure libraries, GUI frameworks, CORBA
> integration, etc.  Thus, the 3-person team would work within the
> *union* of the 10-package universe plus a larger one that includes all
> the reusable stuff.  The rules for the 10-package universe could be
> essentially nil, because it's just 3 people who know each other well.

But in that case, how is the little universe a separate universe at
all? Isn't it just a private corner of the larger one? And if it needs
to avoid conflicts with the larger universe, isn't it going to have to
follow the same rules?

-- Jamie Webb

[Lex Spoon]

Hey Jamie,

I see no conflict between sbaz and Kitten.  Sbaz itself is agnostic
about what kind of components are shipped around and the mechanism by
which they are linked into a running program.  So, by all means feel
free to post Kitten tools and Kitten-packaged components to the
scala-dev bazaar!

It is true that, in the lack of any fancy linker, scala-dev has a
*culture* of accumulating jars into /lib that can all be
simultaneously loaded.  That strikes me as a good thing to push even
if there is a linker available that does not need it.  For example,
there may as well be a standard log4j in scala-dev's /lib, and
programs that don't care should use that version when possible, and
the linker can then make that case less hassle than the case where you
use a custom version...

But there is no reason that a better component system could not be
posted to scala-dev as well....  The only real "problem" would be if
we get so overloaded with different component systems that no ones
components can be used with anyone elses.  That's a bit of a
theoretical problem at the moment!

> Suppose we have a universe of packages that all play nicely together.
> Now we want introduce a problem package that has dependencies which
> conflict with those of some other packages.
>
> AIUI, in sbaz there are two options: modify the source of the package
> to work in the universe, or create a new universe. The first is of
> course the ideal solution, but it may not be feasible, either because
> the source is not available, or because there is not development time
> spare to do the conversion. The second option I consider unworkable:
> what if I need one of the packages that can't fit into this new
> universe?

With sbaz, the straightforward approach would be to use a custom
CLASSPATH.  For politeness, you should not, in that case, put the
conflicting jar's into /lib.


Notice that it's hard for a completely new package to conflict with
other stuff except with filenames.  For a conflict to occur, there
must be two packages that try to work but fail.  Thus, we should
probably talk about incompatibility unless its about filenames.



> The package system for Kitten should add some more options: if the
> conflicting dependency is internal (i.e. other packages don't need to
> talk to it), we can just isolate it in its own classloader.

Cool!

Have you looked at "Jiazzi" at all?





> > I don't think that source recompilation at install time changes any of
> > this.  So, I don't expect that Gentoo has managed to avoid the above
> > fundamental problems.  But this email is too long already, so I'll
> > leave that for another post.
>
> In Gentoo, compiling from source has several advantages:
[...]

Yes, it has advantages.  It does, however, not help you with the
classic problems from RPM hell:

 - API's change between versions

 - Packages are split, merged, and renamed, thus invalidating
   declared dependencies (I depend-on "apache", but now the
   package is called "apache2")

 - Packages put files into different places in the filesystem
   (I install into /home/apache/modules, but the new version
   wants me to put them in /www/modules)

 - Packages push the limits of API's, e.g. using NULL instead
   of the empty string



This said, it may well be that Scala Bazaars should rely on source
distribution instead of distribution of jar's.  That, however, is
independent of the basic mechanism for dealing with conflicts like the
above.



> One trick I understand some people use is to rename the packages of
> internal dependencies, so if I use Log4J and want to avoid conflicts,
> I can go through the Log4J sources and change all the package
> declarations to e.g. com.sygneca.org.apache.log4j. That's pretty evil
> though.

Yes.  This can also be done with binary-rewrites, which I am under the
impression that Jiazzi would sometimes use.

> > One concrete example would be a 3-person team developing the software
> > infrastructure for a small business.  They work with 10-ish packages
> > that are specific to the business.  Scala bazaars supports that, just
> > as Debian's APT does (though, I think, more clumsily).
> >
> > Now, yes, those 10 packages do not do everything they need.  They
> > would not include data structure libraries, GUI frameworks, CORBA
> > integration, etc.  Thus, the 3-person team would work within the
> > *union* of the 10-package universe plus a larger one that includes all
> > the reusable stuff.  The rules for the 10-package universe could be
> > essentially nil, because it's just 3 people who know each other well.
>
> But in that case, how is the little universe a separate universe at
> all? Isn't it just a private corner of the larger one? And if it needs
> to avoid conflicts with the larger universe, isn't it going to have to
> follow the same rules?

On the first question: not necessarily.  While for many people it is
possible to work in a corner of an existing universe, for others
obviously it is not.  Google is not going to post their code to a
public Gentoo server even if they use the Gentoo tools.

On the second question, they do indeed have to stay consistent with
the bigger universe.  However, they do not have to follow all of the
bureaucracy and rules of the bigger universe in their own.  They do
not, for example, have to name their packages with DNS names.  They do
not have to follow all the details of the filesystem policy.  They do
not have to subject each other to whatever new-committer process is
used by PUB (probably not *everyone* can post without being screened).

That these questions are hard to answer is precisely the reason I
don't want the system to impose any particular answer.  I'd like to
allow people to organize themselves in a wide variety of ways.  Sbaz
commits to the package universes approach.  That's a large design
space, but it already kicks out some possibilities.  For example,
Eclipse's plugin-updating system is not a package-universes solution.



-Lex

[Jamie Webb-3]

On Mon, Feb 13, 2006 at 03:33:17PM +0100, Lex Spoon wrote:
> I see no conflict between sbaz and Kitten.

There's none at all. They have quite different purposes. Kitten is
only interested in loading components together and managing them at
runtime.

> Sbaz itself is agnostic
> about what kind of components are shipped around and the mechanism by
> which they are linked into a running program.  So, by all means feel
> free to post Kitten tools and Kitten-packaged components to the
> scala-dev bazaar!

I'll hopefully be able to put some sort of standalone 'quick start'
version of Kitten in sbaz, but a production installation will need to
be packaged for the OS. I'm not sure if sbaz makes sense for Kitten
components, since they are just single zip files and the user will
want to manually deploy them into a given kitten instance.

That's a little way off though; it's going to be a few weeks at least
before we get some breathing space to migrate to Scala2 (we're still
on 1.3.0.10; we could never get our code to compile on the 1.4
series), and I have a fair amount of work to do before the next
release.

> > AIUI, in sbaz there are two options: modify the source of the package
> > to work in the universe, or create a new universe. The first is of
> > course the ideal solution, but it may not be feasible, either because
> > the source is not available, or because there is not development time
> > spare to do the conversion. The second option I consider unworkable:
> > what if I need one of the packages that can't fit into this new
> > universe?
>
> With sbaz, the straightforward approach would be to use a custom
> CLASSPATH.  For politeness, you should not, in that case, put the
> conflicting jar's into /lib.

Unfortunately that's not enough. A classpath will always only let you
load one version of a given package. If you need more than one
version, you need separate classloaders.

> Have you looked at "Jiazzi" at all?

I looked at it briefly a couple of years ago. I get the impression
that it accomplishes much the same thing as Scala's built-in
mechanisms, though less invasively.

My focus is much more on managing components at runtime, handling
situations such as hot redeployment. The dependencies issue is more of
an annoyance than a design focus. I'm hoping to avoid dealing with
anything more fine-grained than whole packages.

> On the first question: not necessarily.  While for many people it is
> possible to work in a corner of an existing universe, for others
> obviously it is not.  Google is not going to post their code to a
> public Gentoo server even if they use the Gentoo tools.

I'm not arguing against universes from the point of view of
package distribution or release policies. Going back to Debian, it
makes perfect sense to have separate unstable and stable branches, and
of course many people run their own public or private repositories.

The issue I have is that the unstable and stable branches are
incompatible. You can't mix them. That's pretty much unavoidable for
Debian because it uses native binaries, but it /can/ be avoided for
Java and Scala /provided/ there's code in place to deal with
occasional incompatibilities when they do occur.

-- Jamie Webb

[Lex Spoon]

Jamie Webb <[hidden email]> writes:
> The issue I have is that the unstable and stable branches are
> incompatible. You can't mix them. That's pretty much unavoidable for
> Debian because it uses native binaries, but it /can/ be avoided for
> Java and Scala /provided/ there's code in place to deal with
> occasional incompatibilities when they do occur.

Well, I disagree.  In the case of Linux distributions, as I posted
earlier, recompiling at install time does not save you.  If I take a
Debian/stable package and recompile it, the recompiled package will
still depend on "apache" instead of "apache2", will still place files
in "/home/apache/modules" instead of "/www/modules", and will still
pass in an empty string where it was supposed to pass in NULL.
Sometimes code is just incompatible.


The same sorts of examples occur with Scala and Java packages.  In the
end, someone has to do the work to make components be compatible with
each other.  It isn't going to just happen.  Ideally, the work is
minor, but the minor changes needed to make a package work in
Debian/sid are different from the minor changes needed to make a
package work in Gentoo.  Trying to eliminate that last, tiny bit of
work tosses you right into components utopianism.

Here's an open challenge: No matter the components system, I can come
up with a counterexample where components from different distributions
do not work together with zero modification.  I started to post an
example based on your good ideas from previous posts, but I'll leave
off for now -- it's pedantic and unconvincing to attack a system that
you are still developing.  Nevertheless, the challenge is out there:
where's the perfect linkage system?  The imperfection of them strikes
me as fundamental.


I don't see this as a bad thing.  On the contrary, it provides a
firmer foundation for the research area.  Once you assume a substrate
system that has some human override mechanism, you can start
*evaluating* component systems, *comparing* them to each other.  You
have a basis for saying one is better than another--not to mention,
that some ideas are losers.  Without such a substrate, all you have is
an unending sequence of ultimatums.


-Lex

[Lex Spoon]

Jamie Webb <[hidden email]> writes:
> [...] it /can/ be avoided for
> Java and Scala /provided/ there's code in place to deal with
> occasional incompatibilities when they do occur.

Do you mean, the person *installing* the components writes some code,
as a last resort?  That would indeed be something I hadn't considered.


-Lex

[Jamie Webb-3]

On Thu, Feb 16, 2006 at 09:31:50AM -0500, Lex Spoon wrote:
> Jamie Webb <[hidden email]> writes:
> > [...] it /can/ be avoided for
> > Java and Scala /provided/ there's code in place to deal with
> > occasional incompatibilities when they do occur.
>
> Do you mean, the person *installing* the components writes some code,
> as a last resort?  That would indeed be something I hadn't considered.

Sort of. Writing code would be a last resort solution to deal with
conflicting external dependencies. Bear in mind that conflicts
primarily occur between libraries, not whole applicatons. And just
installing two incompatible libraries isn't a problem either (provided
they aren't so impolite as to stomp on each other in the filesystem).
You actually have to try to load both of them simultaneously to
encounter a problem. This means that the developer of a package
that depends on two conflicting libraries is in a position write code
to resolve the conflict.

Often though, conflicts involve internal dependencies. If package
writers declare which of their dependencies are internal just as a
matter of course, and a linker acts appropriately, no-one need ever
notice that those conflicts exist.

I should emphasise that I'm not suggesting this scheme can prevent all
conflicts, but it may prevent or at least provide a route around most
of them. Here's the sort of situation I'm looking to avoid:

'stable' universe contains:
libA-1.0 (depends on libZ)
libZ-1.0

'libB-unstable' universe:
libB-1.0 (depends on libZ)
libZ-2.0

I want to use both libA and libB, so I create a union universe. sbaz
will have to install both versions of libZ, because it can't know
whether libA is able to use libZ-2.0 (can it do this? if not, that's a
separate issue). But I still can't load both libA and libB, because
whichever version of libZ appears first on the classpath will clobber
the other.

-- Jamie Webb

[Judson, Ross]

Have you guys seen classworlds?

http://classworlds.codehaus.org/index.html

On first glance it looks like it can solve the libA/libB problem Jamie
outlined.  Classworlds creates a directed graph of loaders, where each
"realm" (universe equivalent?) can import from other realms.  I am
pretty sure it can handle multiple versions of a library at the same
time.

We could either use it as is or (since it's relatively small) create the
equivalent functionality in Scala.

RJ

[Lex Spoon]

Okay, I see.

Keep in mind that once you are willing to do some modification of the
code somewhere, then repackaging is a serious option.  You can either
repackage stuff for a public universe like one of Debian's streams, or
you can create a little private universe of your own to hold your
hacks.  This mechanism is always available, and thus the dependency
system and the linker should be designed with that in mind.

For the case you mention of of wanting two different libraries, it
could simply be a matter of posting one library or both to the local
universe, with one of the packages renamed and all the necessary files
renamed so that they do not conflict.  Given a fancy linker, that's
all you need, because the linker can take care of the rest.  Given the
status quo, you'd probably have more work ahead of you.



> I want to use both libA and libB, so I create a union universe. sbaz
> will have to install both versions of libZ, because it can't know
> whether libA is able to use libZ-2.0 (can it do this? if not, that's a
> separate issue).

You have to rename one of the two packages, e.g. "libZ" and "libZ2".

Letting people install multiple versions of the same package does not
strike me as a good deal.  For one thing, it invalidates using the
filesystem as a single global namespace.  Instead, each package must
install its files with versions in their paths, just in case someone
tries to install a different version of the same package.  And each
package must *refer* to filenames using a version in addition to the
filename.  You end up needing some other way to reference files than
filenames.

For another problem, it complicates dependencies and linking.  If A
depends on B, but there are five different B's installed, which one
does it end up using?  Do we then back away from the simple "depends
on B" dependencies?  That would be a pity.

To contrast, the overwhelmingly common intention when there is both
"libZ-1.0" and "libZ-1.1" in the universe is that they mutually
conflict.  Instead of explicitly writing that very common dependency
on almost every package, it seems better to use renaming to handle the
uncommon (but occuring) situation where you do want to let both of
them load simultaneously.

I played with FreeBSD for a while, and it would let you install
multiple versions of the same package.  It became a mess if you
"tried" it, and usually when you "try" it was just an accident.

Overall, the cost of a little more repackaging seems much lower than
the cost of every package having to deal with a system where there are
multiple versions of the same package installed.



-Lex

[Jamie Webb-3]

On Fri, Feb 17, 2006 at 09:47:57AM -0500, Judson, Ross wrote:
> Have you guys seen classworlds?

Yes. It's exactly the sort of thing I'm talking about. I evaluated it
for Kitten, but it didn't quite fit my needs because of a few rather
minor details about the way 'realms' interact. Since as you say it's
not much code at all, I decided to reimplement it in Scala.

The tricky bit is going to be actually using the thing...

-- Jamie Webb