why don't all stored fields show up?

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

why don't all stored fields show up?

Hung Huynh-4
 

I have about 20 stored fields in string, text, and int, but only about 10
fields show up when I query for them, whether I do fl=*,score or list them
out. What's my problem? How do I retrieve all of fields? Thanks.

Reply | Threaded
Open this post in threaded view
|

Re: why don't all stored fields show up?

Yonik Seeley-2
On Fri, Apr 4, 2008 at 9:25 AM, Hung Huynh <[hidden email]> wrote:
>  I have about 20 stored fields in string, text, and int, but only about 10
>  fields show up when I query for them, whether I do fl=*,score or list them
>  out. What's my problem? How do I retrieve all of fields? Thanks.

You should be getting back all stored fields for every document.
Documents will only show fields they have (fields are sparse, it's not
like a DB table).

-Yonik
Reply | Threaded
Open this post in threaded view
|

RE: why don't all stored fields show up?

Hung Huynh-4
Do you think it might be a problem with my schema and data loading? I loaded
CSV with 39 fields and didn't get any error message. I have a total of 39
stored fields, but not all of them are reported back when I query for them.
Should I reload the Index? Is there a way for me to check if the Index has
all 39 fields?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Friday, April 04, 2008 10:48 AM
To: [hidden email]
Subject: Re: why don't all stored fields show up?

On Fri, Apr 4, 2008 at 9:25 AM, Hung Huynh <[hidden email]> wrote:
>  I have about 20 stored fields in string, text, and int, but only about 10
>  fields show up when I query for them, whether I do fl=*,score or list
them
>  out. What's my problem? How do I retrieve all of fields? Thanks.

You should be getting back all stored fields for every document.
Documents will only show fields they have (fields are sparse, it's not
like a DB table).

-Yonik

Reply | Threaded
Open this post in threaded view
|

Re: why don't all stored fields show up?

Yonik Seeley-2
On Fri, Apr 4, 2008 at 11:57 AM, Hung Huynh <[hidden email]> wrote:
> Do you think it might be a problem with my schema and data loading?

Maybe.

> I loaded
>  CSV with 39 fields and didn't get any error message. I have a total of 39
>  stored fields, but not all of them are reported back when I query for them.

Try to tackle it by getting more specific.
Look at a single row in the CSV, and query for the id of that document
in the index and see what's missing.  Check the schema for those
missing fields.  Try to replicate the problem with another CSV file
with just that single record.

If you still can't figure it out, give us the following info:
- the URL used to load the CSV data
- the single record CSV file
- the result of querying for that single record
- your schema

-Yonik
Reply | Threaded
Open this post in threaded view
|

RE: why don't all stored fields show up?

Hung Huynh-4
Thanks for spending time on this issue.

I removed most the fields, and it's still not working:

http://localhost:8983/solr/update/csv?commit=true&separator=|&escape=\&strea
m.file=exampledocs/test1.txt

test1.txt content
guid|sku
1|ABC001

Query:
http://localhost:8983/solr/select/?q=guid%3A1&version=2.2&start=0&rows=10&in
dent=on&fl=*,score

output:
<?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
 <int name="status">0</int>
 <int name="QTime">0</int>
 <lst name="params">
  <str name="fl">*,score</str>
  <str name="indent">on</str>
  <str name="start">0</str>
  <str name="q">guid:1</str>
  <str name="version">2.2</str>
  <str name="rows">10</str>
 </lst>
</lst>
<result name="response" numFound="1" start="0" maxScore="0.71231794">
 <doc>
  <float name="score">0.71231794</float>
  <long name="guid">1</long>
  <date name="timestamp">2008-04-04T19:35:44.427Z</date>
 </doc>
</result>
</response>

Schema:

   <field name="guid" type="slong" indexed="true" stored="true"
required="true" />
   <field name="sku" type="string" indexed="true" stored="true"/>

Guid is the unique numeric field.

Thanks,

Hung

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: Friday, April 04, 2008 12:02 PM
To: [hidden email]
Subject: Re: why don't all stored fields show up?

On Fri, Apr 4, 2008 at 11:57 AM, Hung Huynh <[hidden email]> wrote:
> Do you think it might be a problem with my schema and data loading?

Maybe.

> I loaded
>  CSV with 39 fields and didn't get any error message. I have a total of 39
>  stored fields, but not all of them are reported back when I query for
them.

Try to tackle it by getting more specific.
Look at a single row in the CSV, and query for the id of that document
in the index and see what's missing.  Check the schema for those
missing fields.  Try to replicate the problem with another CSV file
with just that single record.

If you still can't figure it out, give us the following info:
- the URL used to load the CSV data
- the single record CSV file
- the result of querying for that single record
- your schema

-Yonik

Reply | Threaded
Open this post in threaded view
|

Re: why don't all stored fields show up?

Yonik Seeley-2
Could you try with a recent solr nightly build?
I just tried this with the example schema (I just changed guid to id)
and it worked fine.

http://localhost:8983/solr/select/?q=id%3A1&version=2.2&start=0&rows=10&indent=on&fl=*,score
<?xml version="1.0" encoding="UTF-8"?>
<response>

<lst name="responseHeader">
 <int name="status">0</int>
 <int name="QTime">0</int>
 <lst name="params">
  <str name="rows">10</str>
  <str name="start">0</str>

  <str name="indent">on</str>
  <str name="q">id:1</str>
  <str name="fl">*,score</str>
  <str name="version">2.2</str>
 </lst>
</lst>
<result name="response" numFound="1" start="0" maxScore="3.0476928">
 <doc>

  <float name="score">3.0476928</float>
  <str name="id">1</str>
  <int name="popularity">0</int>
  <arr name="sku"><str>1</str><str>ABC001</str></arr>
  <date name="timestamp">2008-04-06T16:14:17.406Z</date>
 </doc>

</result>
</response>


Some other things to check:
- make sure that sku is not defined more than once in your schema
- check your line endings on the text file... I'm not sure if CSV can
handle mac-style endings... only unix or windows.
  - you can check the actual line endings by looking at the binary: od
-tx1 test1.txt

-Yonik


On Fri, Apr 4, 2008 at 3:40 PM, Hung Huynh <[hidden email]> wrote:

> Thanks for spending time on this issue.
>
>  I removed most the fields, and it's still not working:
>
>  http://localhost:8983/solr/update/csv?commit=true&separator=|&escape=\&strea
>  m.file=exampledocs/test1.txt
>
>  test1.txt content
>  guid|sku
>  1|ABC001
>
>  Query:
>  http://localhost:8983/solr/select/?q=guid%3A1&version=2.2&start=0&rows=10&in
>  dent=on&fl=*,score
>
>  output:
>  <?xml version="1.0" encoding="UTF-8"?>
>  <response>
>
>  <lst name="responseHeader">
>   <int name="status">0</int>
>   <int name="QTime">0</int>
>   <lst name="params">
>   <str name="fl">*,score</str>
>   <str name="indent">on</str>
>   <str name="start">0</str>
>   <str name="q">guid:1</str>
>   <str name="version">2.2</str>
>   <str name="rows">10</str>
>   </lst>
>  </lst>
>  <result name="response" numFound="1" start="0" maxScore="0.71231794">
>   <doc>
>   <float name="score">0.71231794</float>
>   <long name="guid">1</long>
>   <date name="timestamp">2008-04-04T19:35:44.427Z</date>
>   </doc>
>  </result>
>  </response>
>
>  Schema:
>
>    <field name="guid" type="slong" indexed="true" stored="true"
>  required="true" />
>    <field name="sku" type="string" indexed="true" stored="true"/>
>
>  Guid is the unique numeric field.
>
>  Thanks,
>
>  Hung
>
>
>  -----Original Message-----
>  From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
>
> Sent: Friday, April 04, 2008 12:02 PM
>  To: [hidden email]
>  Subject: Re: why don't all stored fields show up?
>
>
>
> On Fri, Apr 4, 2008 at 11:57 AM, Hung Huynh <[hidden email]> wrote:
>  > Do you think it might be a problem with my schema and data loading?
>
>  Maybe.
>
>  > I loaded
>  >  CSV with 39 fields and didn't get any error message. I have a total of 39
>  >  stored fields, but not all of them are reported back when I query for
>  them.
>
>  Try to tackle it by getting more specific.
>  Look at a single row in the CSV, and query for the id of that document
>  in the index and see what's missing.  Check the schema for those
>  missing fields.  Try to replicate the problem with another CSV file
>  with just that single record.
>
>  If you still can't figure it out, give us the following info:
>  - the URL used to load the CSV data
>  - the single record CSV file
>  - the result of querying for that single record
>  - your schema
>
>  -Yonik
>
>