Lucene/Solr Developer content

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Lucene/Solr Developer content

Jan Høydahl / Cominvent
Hi devs,

Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):

* The websites. Very short instructions and linking to WIKI for in-depth
* The old Moin wikis at wiki.apache.org with more details

Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548

So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!


Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.

One idea is to create one or more Asciidoc guides in the source tree, e.g

* /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
* /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
* /solr/dev-guide : Solr-specific developer content. Publish in Solr web site

These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.

There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene/Solr Developer content

Joel Bernstein
+1 for more asciidoc guides. I find these to be extremely useful anytime I run across these on projects.

I'd be happy to add developer level docs in Streaming Expressions / Math Expressions.

On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl <[hidden email]> wrote:
Hi devs,

Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):

* The websites. Very short instructions and linking to WIKI for in-depth
* The old Moin wikis at wiki.apache.org with more details

Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548

So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!


Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.

One idea is to create one or more Asciidoc guides in the source tree, e.g

* /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
* /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
* /solr/dev-guide : Solr-specific developer content. Publish in Solr web site

These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.

There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene/Solr Developer content

david.w.smiley@gmail.com
Great plan, Jan!

A sticky bit of this I think is how to remove old stuff.  It's easy to keep content around forever but it gets stale and clutters things up with better content.  Maybe if I/someone wants to remove content, we send out a proposal to the list with links for easy peer review of what's at stake, and with a justification.

~ David Smiley
Apache Lucene/Solr Search Developer


On Mon, Jun 17, 2019 at 4:22 PM Joel Bernstein <[hidden email]> wrote:
+1 for more asciidoc guides. I find these to be extremely useful anytime I run across these on projects.

I'd be happy to add developer level docs in Streaming Expressions / Math Expressions.

On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl <[hidden email]> wrote:
Hi devs,

Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):

* The websites. Very short instructions and linking to WIKI for in-depth
* The old Moin wikis at wiki.apache.org with more details

Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548

So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!


Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.

One idea is to create one or more Asciidoc guides in the source tree, e.g

* /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
* /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
* /solr/dev-guide : Solr-specific developer content. Publish in Solr web site

These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.

There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene/Solr Developer content

Adrien Grand
In reply to this post by Jan Høydahl / Cominvent
+1

On Mon, Jun 17, 2019 at 10:18 PM Jan Høydahl <[hidden email]> wrote:

>
> Hi devs,
>
> Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):
>
> * The websites. Very short instructions and linking to WIKI for in-depth
> * The old Moin wikis at wiki.apache.org with more details
>
> Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548
>
> So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!
>
>
> Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.
>
> One idea is to create one or more Asciidoc guides in the source tree, e.g
>
> * /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
> * /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web site
>
> These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.
>
> There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


--
Adrien

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene/Solr Developer content

Alexandre Rafalovitch
In reply to this post by david.w.smiley@gmail.com
Does it need to go to Confluence? I know Apache built an export tool,
but is there a way to dump the whole thing into a text archive? Or
both :-)

I am wondering if this could be a good opportunity for dog-fooding.
Load the wiki export into Solr, cross-match against RefGuide, manually
inspect the differences, etc. I already have the code for pulling the
Ref-Guide into Solr split at the lowest-header level. A similar thing
could be done for Wiki and then we do "more like this" or "similarity"
or some such.

Also, how much non-Javadoc information actually exists for Lucene? How
many pages would a Lucene branch have. An honest question, I never
really paid attention to the documentation proportions.

Regards,
   Alex.


On Mon, 17 Jun 2019 at 17:32, David Smiley <[hidden email]> wrote:

>
> Great plan, Jan!
>
> A sticky bit of this I think is how to remove old stuff.  It's easy to keep content around forever but it gets stale and clutters things up with better content.  Maybe if I/someone wants to remove content, we send out a proposal to the list with links for easy peer review of what's at stake, and with a justification.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Jun 17, 2019 at 4:22 PM Joel Bernstein <[hidden email]> wrote:
>>
>> +1 for more asciidoc guides. I find these to be extremely useful anytime I run across these on projects.
>>
>> I'd be happy to add developer level docs in Streaming Expressions / Math Expressions.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl <[hidden email]> wrote:
>>>
>>> Hi devs,
>>>
>>> Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):
>>>
>>> * The websites. Very short instructions and linking to WIKI for in-depth
>>> * The old Moin wikis at wiki.apache.org with more details
>>>
>>> Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548
>>>
>>> So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!
>>>
>>>
>>> Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.
>>>
>>> One idea is to create one or more Asciidoc guides in the source tree, e.g
>>>
>>> * /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
>>> * /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
>>> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web site
>>>
>>> These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.
>>>
>>> There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?
>>>
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Lucene/Solr Developer content

Jan Høydahl / Cominvent
Moin wiki pages are stored in svn somewhere but I could not find the repo.
We could file an INFRA ticket to get a zip dump of all pages.

Looking at the Lucene wiki pages there are quite a few. Problem is most
of them are not maintained, but some are.

An alternative strategy could of course dump the source of Moin wiki pages
and not do a migration, then construct the necessary pages in new asciidoc
guide manually, and if there is still content from Wiki that we need in a wiki
format, put it in CWIKI. Problem with this strategy is that it needs some
(Lucene) volunteers who are willing to do the job during next two weeks
since old wiki will be shut down end of month. That's why I proposed to fist
move things over to cwiki and then start cleaning up, knowing how busy
all of us are :)

My gut feeling is that 80% of the content can be scrapped both in Lucene
and Solr spaces. For Solr we still have a bunch of reference pages pre-dating
the ref-guide and I think few of those add value beyond the ref-guide today.
What we need to do is find the pages that we need to retain and decide where
that content should live, and if it should be re-written etc.

Perhaps a good strategy is to migrate to Cwiki but put all the converted pages
under an "Old Wiki" top page, and then start moving things into a new Cwiki
page structure or to the asciidoc guides one by one, deleting the old content
as we go. This way we know what is left to do.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

18. jun. 2019 kl. 03:10 skrev Alexandre Rafalovitch <[hidden email]>:

Does it need to go to Confluence? I know Apache built an export tool,
but is there a way to dump the whole thing into a text archive? Or
both :-)

I am wondering if this could be a good opportunity for dog-fooding.
Load the wiki export into Solr, cross-match against RefGuide, manually
inspect the differences, etc. I already have the code for pulling the
Ref-Guide into Solr split at the lowest-header level. A similar thing
could be done for Wiki and then we do "more like this" or "similarity"
or some such.

Also, how much non-Javadoc information actually exists for Lucene? How
many pages would a Lucene branch have. An honest question, I never
really paid attention to the documentation proportions.

Regards,
  Alex.


On Mon, 17 Jun 2019 at 17:32, David Smiley <[hidden email]> wrote:

Great plan, Jan!

A sticky bit of this I think is how to remove old stuff.  It's easy to keep content around forever but it gets stale and clutters things up with better content.  Maybe if I/someone wants to remove content, we send out a proposal to the list with links for easy peer review of what's at stake, and with a justification.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Jun 17, 2019 at 4:22 PM Joel Bernstein <[hidden email]> wrote:

+1 for more asciidoc guides. I find these to be extremely useful anytime I run across these on projects.

I'd be happy to add developer level docs in Streaming Expressions / Math Expressions.


Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl <[hidden email]> wrote:

Hi devs,

Today we have mainly two sources of developer documentation (apart from Javadoc and refGuide):

* The websites. Very short instructions and linking to WIKI for in-depth
* The old Moin wikis at wiki.apache.org with more details

Soon the old Moin wiki is being discontinued and I plan to migrate that content to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 and https://issues.apache.org/jira/browse/SOLR-13548

So the first step will be to just start using Confluence instead of Moin. Help appreciated with the cleanup once the first migration is done in the two JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup once this is in Confluence is highly needed!


Someone has also suggested to move most developer resources found in the WIKI into the main GIT code tree, so you have it right there with your git clone. What I want to discuss here is more detailed how that would look like and what info to move over.

One idea is to create one or more Asciidoc guides in the source tree, e.g

* /dev-docs : Common info i.e. Git, Pull requests, building, doing releases etc. Publish in TLP site
* /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web site
* /solr/dev-guide : Solr-specific developer content. Publish in Solr web site

These will be built with Jekyll by Jenkins, into nice HTML guides and published to the web sites.

There may be other ways to do this as well, such as creating a new git repo for dev docs, but I think people have good experience from Solr's ref-guide with keeping code and docs in sync. What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, [hidden email]
For additional commands, [hidden email]