Introducing the Aperture project

[Christiaan Fluit-2]

Hello Tika developers (cc Aperture list),

My name is Christiaan Fluit, I am one of the admins of the Aperture
project. We have recently become aware of the existence of the Tika
project, as several Tika developers/users brought it to our attention.
It seems that you are trying to solve the same problems as we do. This
mail is intended to give you an introduction to Aperture and the areas
in which our projects overlap, so that you know we exist, what we do and
how it relates to Tika. If you are interested, we can also explore
various modes of cooperation.

Aperture is a Java framework for extracting and querying full-text
content and metadata from various information systems (e.g. file
systems, web sites, mail boxes) and the file formats (e.g. documents,
images) occurring in these systems.

You could have a look at the homepage [1], sourceforge page [2] and Wiki
[3].

The project started two years ago, when two organizations (DFKI, a
German research institute [4] and Aduna, a Dutch software firm [5, 6])
recognized they had a common need for a data extraction framework. The
core requirements were to crawl various data sources and to extract data
from the objects that occur in these sources, using RDF [7] as a means
to communicate and store information throughout this framework.

Since its inception the project benefited from contributions from
developers affiliated with both founding partners as well as a group of
external open source enthusiasts. It has been successfully embedded in
various software projects - see [8] for details.

Aperture comes from the Semantic Web community. We firmly believe that
storing data in RDF triples, making it conform to a well-defined model
and making it searchable (Lucene) and structurally queryable (SPARQL)),
allows for very powerful applications. Many aspects of data integration
that plague the users of relational databases or XML schemas become
manageable or non-existent when using RDF technologies.

As far as we can see, the scope of Aperture is currently broader than
Tika, perhaps you can comment on this. We provide seven kinds of
services. Two of them have direct equivalents in Tika, namely Extractor
(processes a stream to extract text and metadata) and MimeTypeIdentifier
(determines the MIME type of a stream using heuristics such as magic
numbers, strings, file extensions, etc.). The Aperture Extractors
correspond directly to Tika Parsers, see e.g. [11] and [12]. As for MIME
type identification, please compare [9] and [10]. The Documentation page
on the Wiki [13] can also provide you with more details.

We believe that cooperation is better than competition. We could both
benefit from our combined experience and ideas. We are looking forward
to hear your view on this.

This mail has been sent to both the tika-dev and aperture-devel list so
that both communities are kept informed of the progress of this discussion.


Kind regards,

Christiaan Fluit,
Leo Sauermann,
Antoni Mylka,

Admins of the Aperture Project.


[1] http://aperture.sourceforge.net
[2] http://sf.net/projects/aperture
[3] http://aperture.wiki.sourceforge.net
[4] http://www.dfki.de
[5] http://www.aduna-software.com
[6] http://www.openrdf.org
[7] http://www.w3.org/RDF/
[8] http://aperture.wiki.sourceforge.net/ProjectsUsingAperture
[9]
http://aperture.svn.sourceforge.net/viewvc/aperture/trunk/aperture/src/java/org/semanticdesktop/aperture/mime/identifier/magic/
[10]
http://svn.apache.org/viewvc/incubator/tika/trunk/src/main/java/org/apache/tika/mime/
[11]
http://aperture.svn.sourceforge.net/viewvc/aperture/trunk/aperture/src/java/org/semanticdesktop/aperture/extractor/
[12]
http://svn.apache.org/viewvc/incubator/tika/trunk/src/main/java/org/apache/tika/parser/
[13] http://aperture.wiki.sourceforge.net/Documentation


--
[hidden email]

Aduna
Prinses Julianaplein 14-b
3817 CS Amersfoort
The Netherlands

[phone] phone
[phone] fax

http://www.aduna-software.com

[Jukka Zitting]

Hi,

On 10/5/07, Christiaan Fluit <[hidden email]> wrote:
> We have recently become aware of the existence of the Tika
> project, as several Tika developers/users brought it to our attention.

Likewise, Aperture surfaced on our radar a while ago [1], and it
certainly looks interesting!

Unless you've already seen it, you should check out the Tika proposal
at [2] for more background on where we are coming from and what the
goals of the Tika project are.

> It seems that you are trying to solve the same problems as we do. This
> mail is intended to give you an introduction to Aperture and the areas
> in which our projects overlap, so that you know we exist, what we do and
> how it relates to Tika. If you are interested, we can also explore
> various modes of cooperation.

Thanks for the introduction! I agree that we have similar goals and
would very much like see how and where we could work together. Tika is
still in an early stage of development, so I think we have lots of
options available, both technically and organizationally.

> As far as we can see, the scope of Aperture is currently broader than
> Tika, perhaps you can comment on this.

Yes. The scope of the Tika project is relatively tight on purpose, and
we're looking at ways to make the code as modular as possible to
support reuse in a wide variety of use cases. We want to make it easy
to use Tika for example with existing crawlers like in Apache Nutch
[3], with content repositories or databases like Apache Jackrabbit
[4], or with more advanced content analysis programs like Apache UIMA
[5].

> We believe that cooperation is better than competition. We could both
> benefit from our combined experience and ideas. We are looking forward
> to hear your view on this.

Agreed!

I see licensing as one major issue to be resolved for enabling better
cooperation. I see that your interfaces are licensed under AFL, which
seems to be in line with the Apache License, but your implementation
classes are under OSL, which makes it impossible for us to directly
use your code within an Apache project (see [6]).

One concrete thing that I think we could use as a starting point to
better understand the different design and licensing constraints would
be to implement an ApertureParser in Tika and a TikaExtractor in
Aperture. Such "cross-linking" would potentially allow Tika to use the
Aperture extractors and Aperture to use Tika parsers, and would
perhaps pave the way for more intimate integration in the future.

BR,

Jukka Zitting

[1] http://www.nabble.com/Aperture-tf4009924.html
[2] http://wiki.apache.org/incubator/TikaProposal
[3] http://lucene.apache.org/nutch/
[3] http://jacrkabbit.apache.org/
[5] http://incubator.apache.org/uima/
[6] http://people.apache.org/~rubys/3party.html