Documentation on SolrJ

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Documentation on SolrJ

Thomas L. Redman
Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Shawn Heisey-2
On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?

I don't have any specific details for that use case.

If you share the code that gives you NPE, somebody might be able to help
you get it working.

The best place to find documentation for SolrJ is actually SolrJ itself
-- the javadocs.  Much of that can be accessed pretty easily if you are
using an IDE to do your development.  Here is a link to the top level of
the SolrJ javadocs:

https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html

There's some documentation here, in the official reference guide:

https://lucene.apache.org/solr/guide/7_5/using-solrj.html

and here:

https://wiki.apache.org/solr/Solrj

If you get the PDF version of the reference guide, there are SolrJ
examples sprinkled throughout the entire document to cover specific
usages, although I do not think there's much for what you're asking
about specifically.  More SolrJ examples is something the reference
guide could definitely use.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Thomas L. Redman
In reply to this post by Thomas L. Redman
Hi Shawn, thanks for the prompt reply!

> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
>
> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
>
> I don't have any specific details for that use case.

Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.

Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.

>
> If you share the code that gives you NPE, somebody might be able to help you get it working.

I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.

>
> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
>
> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>

The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.

>
> There's some documentation here, in the official reference guide:
>
> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>

This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.

I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.

Also, if the JSON API supported by SolrJ, or is there any plan to support?
Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Cassandra Targett
Support for the JSON Facet API in SolrJ was very recently committed via https://issues.apache.org/jira/browse/SOLR-12965 <https://issues.apache.org/jira/browse/SOLR-12965>. This missed the cut-off for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able to use the patch there to see if there are gaps or bugs that could be fixed before 7.7 / 8.0.

Jason, who did the work on that issue, also presented on SolrJ at the Activate conference, you may find it interesting:
https://www.youtube.com/watch?v=ACPUR_GL5zM <https://www.youtube.com/watch?v=ACPUR_GL5zM>

If you do find the time to write some docs, I’d be happy to give you some editing help. Just open a Jira issue when/if you’ve got something and we can go from there.

> On Nov 30, 2018, at 9:53 AM, Thomas L. Redman <[hidden email]> wrote:
>
> Hi Shawn, thanks for the prompt reply!
>
>> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
>>
>> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
>>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
>>
>> I don't have any specific details for that use case.
>
> Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.
>
> Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.
>
>>
>> If you share the code that gives you NPE, somebody might be able to help you get it working.
>
> I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.
>
>>
>> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
>>
>> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>
>
> The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.
>
>>
>> There's some documentation here, in the official reference guide:
>>
>> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>
>
> This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.
>
> I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.
>
> Also, if the JSON API supported by SolrJ, or is there any plan to support?

Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Jason Gerlowski
Hi Thomas,

I recently added a first pass at JSON faceting support to SolrJ.  The
main classes are "JsonQueryRequest" and "DirectJsonQueryRequest" and
live in the package "org.apache.solr.client.solrj.request.json"
(https://github.com/apache/lucene-solr/tree/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/json).
I've also added examples of how to use this code on the "JSON
Faceting" page in the Solr ref guide.  Unfortunately, since this is a
recent addition it hasn't been released yet.  These classes will be in
the next 7x release (if there is one), or in 8.0 when that arrives.
This probably isn't super helpful for you.

Without this code, you have a few options:

1. If the facet requests you'd like to make are relatively
structured/similar, you can subclass QueryRequest and override
getContentWriter().  "ContentWriters" are the abstraction SolrJ is
using to write out the request body.  So you can trivially implement
getContentWriter to wrap a hardcoded string with some templated
variables. If interested, also checkout
"RequestWriter.StringPayloadContentWriter".  This'll be sufficient for
very cookie cutter facet requests, where maybe only a few parameters
change but nothing else.
2. If hardcoding a string JSON body is too inflexible, the JSON
faceting API is "just query params" like everything else.  You can
build your facet request and attach it to the request as a SolrParams
entry.  Doing this wouldn't be the most fun code to write, but it's
always possible.
3. You can copy-paste the unreleased JSON faceting helper classes I
mentioned above into your codebase.  They're not released in SolrJ but
you can still use them by copying them locally and using those copies
until you're able to use a SolrJ that contains these classes.  If you
go this route, please let me or someone else in the community know
your thoughts.  Their being unreleased makes them a bit more of a pain
to use, but it also gives us an opportunity to iterate and improve
them before a release comes and ties us to the existing (maybe awful)
interfaces.

> It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial.
We definitely need more "SolrJ Examples" coverage, though I'm not sure
the best way to expose/structure that.  Solr has a *ton* of API
surface area, and SolrJ is responsible for covering all of it.  Even
if I imagine a SolrJ version of the standard "Getting Started"
tutorial which shows users how to create a collection, index docs, do
a query, and do a faceting request...that'd only cover a fraction of
what's out there.  It might be easier to scale our SolrJ examples by
integrating them into the pages we already have for individual APIs
instead.  I'm all for a SolrJ tutorial, or SolrJ Cookbook sort of
thing if you like those ideas better though, and would also volunteer
to help edit or review things in that area.

Sorry, this got a little long.  But hope that helps.

Best,

Jason
On Fri, Nov 30, 2018 at 11:31 AM Cassandra Targett
<[hidden email]> wrote:

>
> Support for the JSON Facet API in SolrJ was very recently committed via https://issues.apache.org/jira/browse/SOLR-12965 <https://issues.apache.org/jira/browse/SOLR-12965>. This missed the cut-off for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able to use the patch there to see if there are gaps or bugs that could be fixed before 7.7 / 8.0.
>
> Jason, who did the work on that issue, also presented on SolrJ at the Activate conference, you may find it interesting:
> https://www.youtube.com/watch?v=ACPUR_GL5zM <https://www.youtube.com/watch?v=ACPUR_GL5zM>
>
> If you do find the time to write some docs, I’d be happy to give you some editing help. Just open a Jira issue when/if you’ve got something and we can go from there.
>
> > On Nov 30, 2018, at 9:53 AM, Thomas L. Redman <[hidden email]> wrote:
> >
> > Hi Shawn, thanks for the prompt reply!
> >
> >> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
> >>
> >> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
> >>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
> >>
> >> I don't have any specific details for that use case.
> >
> > Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.
> >
> > Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.
> >
> >>
> >> If you share the code that gives you NPE, somebody might be able to help you get it working.
> >
> > I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.
> >
> >>
> >> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
> >>
> >> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>
> >
> > The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.
> >
> >>
> >> There's some documentation here, in the official reference guide:
> >>
> >> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>
> >
> > This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.
> >
> > I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.
> >
> > Also, if the JSON API supported by SolrJ, or is there any plan to support?
>
Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Thomas L. Redman
In reply to this post by Cassandra Targett
Hi Jason. You Solr folks are really on top of things, I thank you Cassandra and Shawn for all the excellent support.

Short story, I can wait. I am building a 1.0 version of a new tool to query our very complex and large (100M docs) datastore, not to find individual documents, but to find subsets of the data suitable for end users (Social Science mostly) researchers. As soon as we get to 7.6/8.0, I will work toward a 1.1 release to include the improved grouping, nested faceting and so on.  To know this is even in the pipe makes my day.

You guys are in need of more documentation. I hope I’m not hurting any feelings, that is not my intention. Solr is a top shelf product, and I would not be one to minimize all the hard work. I think I agree with you Jason, some additions to the existing tutorial to cover more complex query capabilities would probably do the trick. I don’t think you need 600 pages like the Solr Ref Guide document. This will make more sense to do when we get to the 8.0 release (or the next release including JSON API support). I retire next year, may have some free time to build a more extensive query exemplar and document that. Is there a formal procedure I need to adhere to if I want to contribute?



> On Nov 30, 2018, at 10:40 AM, Jason Gerlowski <[hidden email]> wrote:
>
> Hi Thomas,
>
> I recently added a first pass at JSON faceting support to SolrJ.  The
> main classes are "JsonQueryRequest" and "DirectJsonQueryRequest" and
> live in the package "org.apache.solr.client.solrj.request.json"
> (https://github.com/apache/lucene-solr/tree/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/json).
> I've also added examples of how to use this code on the "JSON
> Faceting" page in the Solr ref guide.  Unfortunately, since this is a
> recent addition it hasn't been released yet.  These classes will be in
> the next 7x release (if there is one), or in 8.0 when that arrives.
> This probably isn't super helpful for you.
>
> Without this code, you have a few options:
>
> 1. If the facet requests you'd like to make are relatively
> structured/similar, you can subclass QueryRequest and override
> getContentWriter().  "ContentWriters" are the abstraction SolrJ is
> using to write out the request body.  So you can trivially implement
> getContentWriter to wrap a hardcoded string with some templated
> variables. If interested, also checkout
> "RequestWriter.StringPayloadContentWriter".  This'll be sufficient for
> very cookie cutter facet requests, where maybe only a few parameters
> change but nothing else.
> 2. If hardcoding a string JSON body is too inflexible, the JSON
> faceting API is "just query params" like everything else.  You can
> build your facet request and attach it to the request as a SolrParams
> entry.  Doing this wouldn't be the most fun code to write, but it's
> always possible.
> 3. You can copy-paste the unreleased JSON faceting helper classes I
> mentioned above into your codebase.  They're not released in SolrJ but
> you can still use them by copying them locally and using those copies
> until you're able to use a SolrJ that contains these classes.  If you
> go this route, please let me or someone else in the community know
> your thoughts.  Their being unreleased makes them a bit more of a pain
> to use, but it also gives us an opportunity to iterate and improve
> them before a release comes and ties us to the existing (maybe awful)
> interfaces.
>
>> It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial.
> We definitely need more "SolrJ Examples" coverage, though I'm not sure
> the best way to expose/structure that.  Solr has a *ton* of API
> surface area, and SolrJ is responsible for covering all of it.  Even
> if I imagine a SolrJ version of the standard "Getting Started"
> tutorial which shows users how to create a collection, index docs, do
> a query, and do a faceting request...that'd only cover a fraction of
> what's out there.  It might be easier to scale our SolrJ examples by
> integrating them into the pages we already have for individual APIs
> instead.  I'm all for a SolrJ tutorial, or SolrJ Cookbook sort of
> thing if you like those ideas better though, and would also volunteer
> to help edit or review things in that area.
>
> Sorry, this got a little long.  But hope that helps.
>
> Best,
>
> Jason
> On Fri, Nov 30, 2018 at 11:31 AM Cassandra Targett
> <[hidden email]> wrote:
>>
>> Support for the JSON Facet API in SolrJ was very recently committed via https://issues.apache.org/jira/browse/SOLR-12965 <https://issues.apache.org/jira/browse/SOLR-12965>. This missed the cut-off for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able to use the patch there to see if there are gaps or bugs that could be fixed before 7.7 / 8.0.
>>
>> Jason, who did the work on that issue, also presented on SolrJ at the Activate conference, you may find it interesting:
>> https://www.youtube.com/watch?v=ACPUR_GL5zM <https://www.youtube.com/watch?v=ACPUR_GL5zM>
>>
>> If you do find the time to write some docs, I’d be happy to give you some editing help. Just open a Jira issue when/if you’ve got something and we can go from there.
>>
>>> On Nov 30, 2018, at 9:53 AM, Thomas L. Redman <[hidden email]> wrote:
>>>
>>> Hi Shawn, thanks for the prompt reply!
>>>
>>>> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
>>>>
>>>> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
>>>>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
>>>>
>>>> I don't have any specific details for that use case.
>>>
>>> Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.
>>>
>>> Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.
>>>
>>>>
>>>> If you share the code that gives you NPE, somebody might be able to help you get it working.
>>>
>>> I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.
>>>
>>>>
>>>> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
>>>>
>>>> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>
>>>
>>> The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.
>>>
>>>>
>>>> There's some documentation here, in the official reference guide:
>>>>
>>>> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>
>>>
>>> This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.
>>>
>>> I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.
>>>
>>> Also, if the JSON API supported by SolrJ, or is there any plan to support?
>>

Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Erick Erickson
Thomas:

All contributions welcome! Opensource software lives and dies by
people stepping up and contributing when they see something they want
to improve, come join the club and help make it better.

Here's the basics of getting started:
https://wiki.apache.org/solr/HowToContribute

The part you care most about is probably getting the source code,
which includes the reference guide source (all the *.adoc files). In
essence, the process is
> create a logon for the Lucene/Solr JIRA, this should get you in the right vicinity: https://issues.apache.org/jira/projects/SOLR/issues/
> pull the source (including docs)
> make whatever changes you want
> create a JIRA describing your changes
> attach a patch (or a pull request if you're git-savvy) to the JIRA
> prompt for a committer to push it to the repo.

Here's a bit about documentation in particular:
https://lucene.apache.org/solr/guide/7_0/how-to-contribute.html

Don't be too worried about working with AsciiDoc, just download Atom.
Or if you use IntelliJ (and I assume Eclipse) or your favorite editor
supports an AsciiDoc plugin use that.

Best,
Erick
On Sat, Dec 1, 2018 at 9:31 AM Thomas L. Redman <[hidden email]> wrote:

>
> Hi Jason. You Solr folks are really on top of things, I thank you Cassandra and Shawn for all the excellent support.
>
> Short story, I can wait. I am building a 1.0 version of a new tool to query our very complex and large (100M docs) datastore, not to find individual documents, but to find subsets of the data suitable for end users (Social Science mostly) researchers. As soon as we get to 7.6/8.0, I will work toward a 1.1 release to include the improved grouping, nested faceting and so on.  To know this is even in the pipe makes my day.
>
> You guys are in need of more documentation. I hope I’m not hurting any feelings, that is not my intention. Solr is a top shelf product, and I would not be one to minimize all the hard work. I think I agree with you Jason, some additions to the existing tutorial to cover more complex query capabilities would probably do the trick. I don’t think you need 600 pages like the Solr Ref Guide document. This will make more sense to do when we get to the 8.0 release (or the next release including JSON API support). I retire next year, may have some free time to build a more extensive query exemplar and document that. Is there a formal procedure I need to adhere to if I want to contribute?
>
>
>
> > On Nov 30, 2018, at 10:40 AM, Jason Gerlowski <[hidden email]> wrote:
> >
> > Hi Thomas,
> >
> > I recently added a first pass at JSON faceting support to SolrJ.  The
> > main classes are "JsonQueryRequest" and "DirectJsonQueryRequest" and
> > live in the package "org.apache.solr.client.solrj.request.json"
> > (https://github.com/apache/lucene-solr/tree/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/json).
> > I've also added examples of how to use this code on the "JSON
> > Faceting" page in the Solr ref guide.  Unfortunately, since this is a
> > recent addition it hasn't been released yet.  These classes will be in
> > the next 7x release (if there is one), or in 8.0 when that arrives.
> > This probably isn't super helpful for you.
> >
> > Without this code, you have a few options:
> >
> > 1. If the facet requests you'd like to make are relatively
> > structured/similar, you can subclass QueryRequest and override
> > getContentWriter().  "ContentWriters" are the abstraction SolrJ is
> > using to write out the request body.  So you can trivially implement
> > getContentWriter to wrap a hardcoded string with some templated
> > variables. If interested, also checkout
> > "RequestWriter.StringPayloadContentWriter".  This'll be sufficient for
> > very cookie cutter facet requests, where maybe only a few parameters
> > change but nothing else.
> > 2. If hardcoding a string JSON body is too inflexible, the JSON
> > faceting API is "just query params" like everything else.  You can
> > build your facet request and attach it to the request as a SolrParams
> > entry.  Doing this wouldn't be the most fun code to write, but it's
> > always possible.
> > 3. You can copy-paste the unreleased JSON faceting helper classes I
> > mentioned above into your codebase.  They're not released in SolrJ but
> > you can still use them by copying them locally and using those copies
> > until you're able to use a SolrJ that contains these classes.  If you
> > go this route, please let me or someone else in the community know
> > your thoughts.  Their being unreleased makes them a bit more of a pain
> > to use, but it also gives us an opportunity to iterate and improve
> > them before a release comes and ties us to the existing (maybe awful)
> > interfaces.
> >
> >> It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial.
> > We definitely need more "SolrJ Examples" coverage, though I'm not sure
> > the best way to expose/structure that.  Solr has a *ton* of API
> > surface area, and SolrJ is responsible for covering all of it.  Even
> > if I imagine a SolrJ version of the standard "Getting Started"
> > tutorial which shows users how to create a collection, index docs, do
> > a query, and do a faceting request...that'd only cover a fraction of
> > what's out there.  It might be easier to scale our SolrJ examples by
> > integrating them into the pages we already have for individual APIs
> > instead.  I'm all for a SolrJ tutorial, or SolrJ Cookbook sort of
> > thing if you like those ideas better though, and would also volunteer
> > to help edit or review things in that area.
> >
> > Sorry, this got a little long.  But hope that helps.
> >
> > Best,
> >
> > Jason
> > On Fri, Nov 30, 2018 at 11:31 AM Cassandra Targett
> > <[hidden email]> wrote:
> >>
> >> Support for the JSON Facet API in SolrJ was very recently committed via https://issues.apache.org/jira/browse/SOLR-12965 <https://issues.apache.org/jira/browse/SOLR-12965>. This missed the cut-off for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able to use the patch there to see if there are gaps or bugs that could be fixed before 7.7 / 8.0.
> >>
> >> Jason, who did the work on that issue, also presented on SolrJ at the Activate conference, you may find it interesting:
> >> https://www.youtube.com/watch?v=ACPUR_GL5zM <https://www.youtube.com/watch?v=ACPUR_GL5zM>
> >>
> >> If you do find the time to write some docs, I’d be happy to give you some editing help. Just open a Jira issue when/if you’ve got something and we can go from there.
> >>
> >>> On Nov 30, 2018, at 9:53 AM, Thomas L. Redman <[hidden email]> wrote:
> >>>
> >>> Hi Shawn, thanks for the prompt reply!
> >>>
> >>>> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
> >>>>
> >>>> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
> >>>>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
> >>>>
> >>>> I don't have any specific details for that use case.
> >>>
> >>> Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.
> >>>
> >>> Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.
> >>>
> >>>>
> >>>> If you share the code that gives you NPE, somebody might be able to help you get it working.
> >>>
> >>> I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.
> >>>
> >>>>
> >>>> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
> >>>>
> >>>> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>
> >>>
> >>> The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.
> >>>
> >>>>
> >>>> There's some documentation here, in the official reference guide:
> >>>>
> >>>> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>
> >>>
> >>> This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.
> >>>
> >>> I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.
> >>>
> >>> Also, if the JSON API supported by SolrJ, or is there any plan to support?
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Documentation on SolrJ

Jason Gerlowski
> You guys are in need of more documentation. I hope I’m not hurting any feelings, that is not my intention.
I can't imagine anyone would.  In a sense that's what this mailing
list is for-  to share what works, what doesn't, and what's really
lacking.  Thanks for chiming in.

Jason
On Sat, Dec 1, 2018 at 12:56 PM Erick Erickson <[hidden email]> wrote:

>
> Thomas:
>
> All contributions welcome! Opensource software lives and dies by
> people stepping up and contributing when they see something they want
> to improve, come join the club and help make it better.
>
> Here's the basics of getting started:
> https://wiki.apache.org/solr/HowToContribute
>
> The part you care most about is probably getting the source code,
> which includes the reference guide source (all the *.adoc files). In
> essence, the process is
> > create a logon for the Lucene/Solr JIRA, this should get you in the right vicinity: https://issues.apache.org/jira/projects/SOLR/issues/
> > pull the source (including docs)
> > make whatever changes you want
> > create a JIRA describing your changes
> > attach a patch (or a pull request if you're git-savvy) to the JIRA
> > prompt for a committer to push it to the repo.
>
> Here's a bit about documentation in particular:
> https://lucene.apache.org/solr/guide/7_0/how-to-contribute.html
>
> Don't be too worried about working with AsciiDoc, just download Atom.
> Or if you use IntelliJ (and I assume Eclipse) or your favorite editor
> supports an AsciiDoc plugin use that.
>
> Best,
> Erick
> On Sat, Dec 1, 2018 at 9:31 AM Thomas L. Redman <[hidden email]> wrote:
> >
> > Hi Jason. You Solr folks are really on top of things, I thank you Cassandra and Shawn for all the excellent support.
> >
> > Short story, I can wait. I am building a 1.0 version of a new tool to query our very complex and large (100M docs) datastore, not to find individual documents, but to find subsets of the data suitable for end users (Social Science mostly) researchers. As soon as we get to 7.6/8.0, I will work toward a 1.1 release to include the improved grouping, nested faceting and so on.  To know this is even in the pipe makes my day.
> >
> > You guys are in need of more documentation. I hope I’m not hurting any feelings, that is not my intention. Solr is a top shelf product, and I would not be one to minimize all the hard work. I think I agree with you Jason, some additions to the existing tutorial to cover more complex query capabilities would probably do the trick. I don’t think you need 600 pages like the Solr Ref Guide document. This will make more sense to do when we get to the 8.0 release (or the next release including JSON API support). I retire next year, may have some free time to build a more extensive query exemplar and document that. Is there a formal procedure I need to adhere to if I want to contribute?
> >
> >
> >
> > > On Nov 30, 2018, at 10:40 AM, Jason Gerlowski <[hidden email]> wrote:
> > >
> > > Hi Thomas,
> > >
> > > I recently added a first pass at JSON faceting support to SolrJ.  The
> > > main classes are "JsonQueryRequest" and "DirectJsonQueryRequest" and
> > > live in the package "org.apache.solr.client.solrj.request.json"
> > > (https://github.com/apache/lucene-solr/tree/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/json).
> > > I've also added examples of how to use this code on the "JSON
> > > Faceting" page in the Solr ref guide.  Unfortunately, since this is a
> > > recent addition it hasn't been released yet.  These classes will be in
> > > the next 7x release (if there is one), or in 8.0 when that arrives.
> > > This probably isn't super helpful for you.
> > >
> > > Without this code, you have a few options:
> > >
> > > 1. If the facet requests you'd like to make are relatively
> > > structured/similar, you can subclass QueryRequest and override
> > > getContentWriter().  "ContentWriters" are the abstraction SolrJ is
> > > using to write out the request body.  So you can trivially implement
> > > getContentWriter to wrap a hardcoded string with some templated
> > > variables. If interested, also checkout
> > > "RequestWriter.StringPayloadContentWriter".  This'll be sufficient for
> > > very cookie cutter facet requests, where maybe only a few parameters
> > > change but nothing else.
> > > 2. If hardcoding a string JSON body is too inflexible, the JSON
> > > faceting API is "just query params" like everything else.  You can
> > > build your facet request and attach it to the request as a SolrParams
> > > entry.  Doing this wouldn't be the most fun code to write, but it's
> > > always possible.
> > > 3. You can copy-paste the unreleased JSON faceting helper classes I
> > > mentioned above into your codebase.  They're not released in SolrJ but
> > > you can still use them by copying them locally and using those copies
> > > until you're able to use a SolrJ that contains these classes.  If you
> > > go this route, please let me or someone else in the community know
> > > your thoughts.  Their being unreleased makes them a bit more of a pain
> > > to use, but it also gives us an opportunity to iterate and improve
> > > them before a release comes and ties us to the existing (maybe awful)
> > > interfaces.
> > >
> > >> It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial.
> > > We definitely need more "SolrJ Examples" coverage, though I'm not sure
> > > the best way to expose/structure that.  Solr has a *ton* of API
> > > surface area, and SolrJ is responsible for covering all of it.  Even
> > > if I imagine a SolrJ version of the standard "Getting Started"
> > > tutorial which shows users how to create a collection, index docs, do
> > > a query, and do a faceting request...that'd only cover a fraction of
> > > what's out there.  It might be easier to scale our SolrJ examples by
> > > integrating them into the pages we already have for individual APIs
> > > instead.  I'm all for a SolrJ tutorial, or SolrJ Cookbook sort of
> > > thing if you like those ideas better though, and would also volunteer
> > > to help edit or review things in that area.
> > >
> > > Sorry, this got a little long.  But hope that helps.
> > >
> > > Best,
> > >
> > > Jason
> > > On Fri, Nov 30, 2018 at 11:31 AM Cassandra Targett
> > > <[hidden email]> wrote:
> > >>
> > >> Support for the JSON Facet API in SolrJ was very recently committed via https://issues.apache.org/jira/browse/SOLR-12965 <https://issues.apache.org/jira/browse/SOLR-12965>. This missed the cut-off for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able to use the patch there to see if there are gaps or bugs that could be fixed before 7.7 / 8.0.
> > >>
> > >> Jason, who did the work on that issue, also presented on SolrJ at the Activate conference, you may find it interesting:
> > >> https://www.youtube.com/watch?v=ACPUR_GL5zM <https://www.youtube.com/watch?v=ACPUR_GL5zM>
> > >>
> > >> If you do find the time to write some docs, I’d be happy to give you some editing help. Just open a Jira issue when/if you’ve got something and we can go from there.
> > >>
> > >>> On Nov 30, 2018, at 9:53 AM, Thomas L. Redman <[hidden email]> wrote:
> > >>>
> > >>> Hi Shawn, thanks for the prompt reply!
> > >>>
> > >>>> On Nov 29, 2018, at 4:55 PM, Shawn Heisey <[hidden email]> wrote:
> > >>>>
> > >>>> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
> > >>>>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, and I can find no API for that. I see I can add a pivot field, I guess to a query in general, but that doesn’t seem to work at all, I get an NPE. The documentation on SolrJ is sorely lacking, the documentation I have found is less than a readme. Are there any books that provided a good tretise on SolrJ specifically? Does SolrJ support these more advanced features?
> > >>>>
> > >>>> I don't have any specific details for that use case.
> > >>>
> > >>> Check out page 498 of the PDF, that includes a brief but powerful discussion of the JSON Facet API. For just one example, I am interested in faceting a nominal field within a date range bucket. Example: I want to facet publication_date field into YEAR buckets, and within each YEAR bucket, facet on author to get the most prolific authors in that year, AND to also facet genre with the same bucket to find out how much scifi, adventure and so on was produced that year. From what I am seeing, beyond pivots(and pivots won’t support this specific use case), I don’t see this capability is supported by the SolrJ API, but this is a hugely powerful feature, and needs to be supported.
> > >>>
> > >>> Furthermore, I want to be able to support a vaste range of facets within a single query, perhaps including some collapse and expand, groupings and so on.
> > >>>
> > >>>>
> > >>>> If you share the code that gives you NPE, somebody might be able to help you get it working.
> > >>>
> > >>> I haven’t looked in to this enough to drop it in somebody elses' lap at this point, I suspect I am not using the API correctly. And since this won’t allow what I want, I’m not too worried about it.
> > >>>
> > >>>>
> > >>>> The best place to find documentation for SolrJ is actually SolrJ itself -- the javadocs.  Much of that can be accessed pretty easily if you are using an IDE to do your development.  Here is a link to the top level of the SolrJ javadocs:
> > >>>>
> > >>>> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html <https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html>
> > >>>
> > >>> The JavaDocs are limited. I surmise from tracing the code a bit though that I need to rely less on methods provided directly by SolrQuery, and add parameters using methods of the superclasses more frequently. Those superclass methods add simply key value pairs. Still not sure this will allow me the flexibility I need, particularly if the JSON Facet API is not supported.
> > >>>
> > >>>>
> > >>>> There's some documentation here, in the official reference guide:
> > >>>>
> > >>>> https://lucene.apache.org/solr/guide/7_5/using-solrj.html <https://lucene.apache.org/solr/guide/7_5/using-solrj.html>
> > >>>
> > >>> This is an excellent document. It would be wonderful if a document of this caliber was provided solely for SolrJ in the form of a tutorial. The existing online tutorial says nothing about how to do anything beyond a simple query. I notice in this document most of the examples of how to issue queries, for example, use curl to issue query. Simply put, this is not a practical approach for the typical user. That being the case, people need to build real UIs around applications that hide the intricacies of the search API. I would rather not build my own API, since SolrJ is already in place, and seems quite powerful. I have been using it for a few years, but really just to do queries.
> > >>>
> > >>> I might be interested in contributing to such a document, provided it is sufficiently succinct. I find myself quite busy these days. But I think I would really have to ramp up my understanding of SolrJ to be of any use. Is there any such document in the works, or any interested parties? I am NOT a good writer, I would need somebody to review my work for both accuracy and grammar.
> > >>>
> > >>> Also, if the JSON API supported by SolrJ, or is there any plan to support?
> > >>
> >