solrj (admin) requests

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

solrj (admin) requests

Arturas Mazeika
Hi Solr Team,

Short question:

How can I systematically explore the solrj functionality/API?

Long question:

I am discovering solrj functionality and I am pretty much impressed what
solrj can do. What I am less impressed is my knowledge how to find what I
am looking for. On the positive side, one can relatively quickly find ways
to insert/delete/update/query docs using solrj (basically using google [1]
or from [2]). It is a rather simple task to get exposed to some admin
functionality through [3]. From this on, things are getting more difficult:
I was able to query for some admin infos by trial and error using this java
code:

ArrayList<String> urls = new ArrayList<>();
urls.add(solrUrl);
CloudSolrClient client = new CloudSolrClient.Builder(urls)
    .withConnectionTimeout(10000)
    .withSocketTimeout(60000)
    .build();
client.setDefaultCollection("tph");

final SolrRequest request = new CollectionAdminRequest.ClusterStatus();
final NamedList<Object> response = client.request(request);
final NamedList<Object> cluster  = (NamedList<Object>)
response.get("cluster");
final ArrayList<Object> nodes    = (ArrayList<Object>)
cluster.get("live_nodes");
final NamedList<Object> cols       = (NamedList<Object>)
cluster.get("collections");
final LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
cols.get("tph");

and then looking at what the keys names are, etc. Things are becoming more
difficult here, as (1) lots of functionality hides behind generic "get"
methods and (2) The containers that are returned vary from NamedList, to
anything possible (arraylist, string, hashmap, etc.)

Getting the size of one of the index, e.g., with

ArrayList<String> urls = new ArrayList<>();
urls.add("http://localhost:8983/solr");
CloudSolrClient client = new CloudSolrClient.Builder(urls)
    .withConnectionTimeout(10000)
    .withSocketTimeout(60000)
    .build();

client.setDefaultCollection("trans");

CoreAdminRequest request = new CoreAdminRequest();
request.setAction(CoreAdminAction.STATUS);
request.setCoreName("trans_shard2_replica_n4");

request.setIndexInfoNeeded(true);
CoreAdminResponse resp = request.process(client);

NamedList<Object> coreStatus =
resp.getCoreStatus("trans_shard2_replica_n4");
NamedList<Object> indexStats = (NamedList<Object>) coreStatus.get("index");
System.out.println(indexStats.get("sizeInBytes"));

is even more challenging to learn (how to) as one needs to google deeper
and deeper [4].

I also looked at the following books for systematic ways to learn solrj:

* Apache Solr 4 Cookbook,
* Apache Solr Search Patterns

Is there a simple and systematic way to get exposed to solrj and available
API/functionality?

Cheers,
Arturas

[1] http://www.baeldung.com/apache-solrj
[2] https://lucene.apache.org/solr/guide/7_2/using-solrj.html
[3]
https://lucene.apache.org/solr/7_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/request/CollectionAdminRequest.ClusterStatus.html
[4]
https://www.programcreek.com/java-api-examples/?api=org.apache.solr.client.solrj.request.CoreAdminRequest
Reply | Threaded
Open this post in threaded view
|

Re: solrj (admin) requests

Erick Erickson
Yeah, that can be a pain. Unfortunately there's no official
"programming guide" for instance.

What there is, however, is an extensive suite of unit tests in
/Users/Erick/apache/solrJiras/master/solr/solrj/src/test/org/apache/solr/client/solrj.
From there it's often a hunt though.

Best,
Erick

On Thu, May 3, 2018 at 8:07 AM, Arturas Mazeika <[hidden email]> wrote:

> Hi Solr Team,
>
> Short question:
>
> How can I systematically explore the solrj functionality/API?
>
> Long question:
>
> I am discovering solrj functionality and I am pretty much impressed what
> solrj can do. What I am less impressed is my knowledge how to find what I
> am looking for. On the positive side, one can relatively quickly find ways
> to insert/delete/update/query docs using solrj (basically using google [1]
> or from [2]). It is a rather simple task to get exposed to some admin
> functionality through [3]. From this on, things are getting more difficult:
> I was able to query for some admin infos by trial and error using this java
> code:
>
> ArrayList<String> urls = new ArrayList<>();
> urls.add(solrUrl);
> CloudSolrClient client = new CloudSolrClient.Builder(urls)
>     .withConnectionTimeout(10000)
>     .withSocketTimeout(60000)
>     .build();
> client.setDefaultCollection("tph");
>
> final SolrRequest request = new CollectionAdminRequest.ClusterStatus();
> final NamedList<Object> response = client.request(request);
> final NamedList<Object> cluster  = (NamedList<Object>)
> response.get("cluster");
> final ArrayList<Object> nodes    = (ArrayList<Object>)
> cluster.get("live_nodes");
> final NamedList<Object> cols       = (NamedList<Object>)
> cluster.get("collections");
> final LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
> cols.get("tph");
>
> and then looking at what the keys names are, etc. Things are becoming more
> difficult here, as (1) lots of functionality hides behind generic "get"
> methods and (2) The containers that are returned vary from NamedList, to
> anything possible (arraylist, string, hashmap, etc.)
>
> Getting the size of one of the index, e.g., with
>
> ArrayList<String> urls = new ArrayList<>();
> urls.add("http://localhost:8983/solr");
> CloudSolrClient client = new CloudSolrClient.Builder(urls)
>     .withConnectionTimeout(10000)
>     .withSocketTimeout(60000)
>     .build();
>
> client.setDefaultCollection("trans");
>
> CoreAdminRequest request = new CoreAdminRequest();
> request.setAction(CoreAdminAction.STATUS);
> request.setCoreName("trans_shard2_replica_n4");
>
> request.setIndexInfoNeeded(true);
> CoreAdminResponse resp = request.process(client);
>
> NamedList<Object> coreStatus =
> resp.getCoreStatus("trans_shard2_replica_n4");
> NamedList<Object> indexStats = (NamedList<Object>) coreStatus.get("index");
> System.out.println(indexStats.get("sizeInBytes"));
>
> is even more challenging to learn (how to) as one needs to google deeper
> and deeper [4].
>
> I also looked at the following books for systematic ways to learn solrj:
>
> * Apache Solr 4 Cookbook,
> * Apache Solr Search Patterns
>
> Is there a simple and systematic way to get exposed to solrj and available
> API/functionality?
>
> Cheers,
> Arturas
>
> [1] http://www.baeldung.com/apache-solrj
> [2] https://lucene.apache.org/solr/guide/7_2/using-solrj.html
> [3]
> https://lucene.apache.org/solr/7_2_0/solr-solrj/index.html?org/apache/solr/client/solrj/request/CollectionAdminRequest.ClusterStatus.html
> [4]
> https://www.programcreek.com/java-api-examples/?api=org.apache.solr.client.solrj.request.CoreAdminRequest
Reply | Threaded
Open this post in threaded view
|

Re: solrj (admin) requests

Shawn Heisey-2
In reply to this post by Arturas Mazeika
On 5/3/2018 9:07 AM, Arturas Mazeika wrote:
> Short question:
>
> How can I systematically explore the solrj functionality/API?

As Erick said, there is not an extensive programming guide.  The
javadocs for SolrJ classes are pretty decent, but figuring out precisely
what the response objects actually contain does require experimentation.

> final NamedList<Object> cluster  = (NamedList<Object>)
> response.get("cluster");
> final ArrayList<Object> nodes    = (ArrayList<Object>)
> cluster.get("live_nodes");
> final NamedList<Object> cols       = (NamedList<Object>)
> cluster.get("collections");
> final LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
> cols.get("tph");

It's possible to replace all this code with one line:

LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
response.findRecursive("cluster", "collections", "tph");

I *did* make sure this really does work with a 7.3 cloud example, so use
with confidence!

> and then looking at what the keys names are, etc. Things are becoming more
> difficult here, as (1) lots of functionality hides behind generic "get"
> methods and (2) The containers that are returned vary from NamedList, to
> anything possible (arraylist, string, hashmap, etc.)

SolrJ (and Solr itself, because SolrJ is an integral part of the server)
uses the NamedList object for a LOT of things.  It's useful for encoding
a wide range of responses and configuration information.  It can, as you
noticed, hold any type of object, including additional NamedList
instances.  There is typically a NamedList object in all response
objects which contains the *entire* response.

Some of the information in a response is ALSO available via sugar
methods which translate parts of the full response to other data types. 
For instance, you can get numFound out of a query response without
ripping the NamedList apart:

  SolrQuery q = new SolrQuery("*:*");
  QueryResponse r = client.query(q);
  long numFound = r.getResults().getNumFound();

This value is also available from the NamedList object, but the code to
obtain it is very ugly:

  long numF = ((SolrDocumentList)
r.getResponse().get("response")).getNumFound();

If you make requests manually (possibly in a browser) with
wt=json&indent=true, it only takes a little practice before you'll be
able to translate what you see into the NamedList structure that the
SolrJ response object will contain.  You can also print the output of
the toString() method on the NamedList to see the structure, but that
output usually doesn't include the object types, only their values.

Javadocs are a primary source of information.  Exploring the responses
fills in the holes.

Using wt=xml instead of wt=json in manual requests actually yields more
information, but json is easier to read.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: solrj (admin) requests

Arturas Mazeika
Hi Erick, Shawn, et al,

Thanks a lot for a piece of wisdom Especially for unit tests,
findRecursive, as well as ideas how to get to infos that one is looking
for. Good job indeed.

Cheers,
Arturas

On Fri, May 4, 2018 at 12:22 AM, Shawn Heisey <[hidden email]> wrote:

> On 5/3/2018 9:07 AM, Arturas Mazeika wrote:
> > Short question:
> >
> > How can I systematically explore the solrj functionality/API?
>
> As Erick said, there is not an extensive programming guide.  The
> javadocs for SolrJ classes are pretty decent, but figuring out precisely
> what the response objects actually contain does require experimentation.
>
> > final NamedList<Object> cluster  = (NamedList<Object>)
> > response.get("cluster");
> > final ArrayList<Object> nodes    = (ArrayList<Object>)
> > cluster.get("live_nodes");
> > final NamedList<Object> cols       = (NamedList<Object>)
> > cluster.get("collections");
> > final LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
> > cols.get("tph");
>
> It's possible to replace all this code with one line:
>
> LinkedHashMap<String, Object> tph = (LinkedHashMap<String, Object>)
> response.findRecursive("cluster", "collections", "tph");
>
> I *did* make sure this really does work with a 7.3 cloud example, so use
> with confidence!
>
> > and then looking at what the keys names are, etc. Things are becoming
> more
> > difficult here, as (1) lots of functionality hides behind generic "get"
> > methods and (2) The containers that are returned vary from NamedList, to
> > anything possible (arraylist, string, hashmap, etc.)
>
> SolrJ (and Solr itself, because SolrJ is an integral part of the server)
> uses the NamedList object for a LOT of things.  It's useful for encoding
> a wide range of responses and configuration information.  It can, as you
> noticed, hold any type of object, including additional NamedList
> instances.  There is typically a NamedList object in all response
> objects which contains the *entire* response.
>
> Some of the information in a response is ALSO available via sugar
> methods which translate parts of the full response to other data types.
> For instance, you can get numFound out of a query response without
> ripping the NamedList apart:
>
>   SolrQuery q = new SolrQuery("*:*");
>   QueryResponse r = client.query(q);
>   long numFound = r.getResults().getNumFound();
>
> This value is also available from the NamedList object, but the code to
> obtain it is very ugly:
>
>   long numF = ((SolrDocumentList)
> r.getResponse().get("response")).getNumFound();
>
> If you make requests manually (possibly in a browser) with
> wt=json&indent=true, it only takes a little practice before you'll be
> able to translate what you see into the NamedList structure that the
> SolrJ response object will contain.  You can also print the output of
> the toString() method on the NamedList to see the structure, but that
> output usually doesn't include the object types, only their values.
>
> Javadocs are a primary source of information.  Exploring the responses
> fills in the holes.
>
> Using wt=xml instead of wt=json in manual requests actually yields more
> information, but json is easier to read.
>
> Thanks,
> Shawn
>
>