Optimization taking days/weeks

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Optimization taking days/weeks

F Knudson
Optimization time on solr index has turned into days/weeks.
We are using solr 1.2.
We use one box to build/optimize indexes. This index is copied to another box for searching purposes.
We welcome suggestions/comments, etc.  We are a bit stumped on this.
Details are below.

Box details
Proc: 8 Dual Core 2.6GHz
Mem: 32 GB
OS: Red Hat Linux Enterprise 4
Kernel: 2.6.9-55.0.12.ELlargesmp

These are details from the index currently in use.  Search response time is very acceptable (searchers are very happy)
Optimization time - 10433  (12/11/07)
index size - 229486464
# of records - 84960570
index directory
flasher# ls -l
total 229486464
-rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
-rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
-rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
-rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
-rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
-rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
-rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
-rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
-rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
-rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
--------------------------

current directory listing
indexed new records - Jan 22 and Jan 27
# of records - 85032470
optimization time - 558188

There were no out of memory errors.  There was 800961792KB  left in the directory.  The files were not
collapsed as expected.  There are still files dated Jan 22 and Jan 27.  

A new optimization was started Feb. 11 and continues.
This is a snapshot of the index directory.

We have at least another million records to add.  Plus weekly updates of approximately 103K records.
We are using the direct indexing method.
java settings used - java -Xmx1024M -Xms1024M

The files continue to grow so work is progressing.
snapshot 2/21/08
-bash-3.00$ ls -ltr
total 205396680
-rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
-rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
-rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
-rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
-rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
-rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
-rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
-rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
-rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
-rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
-rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
-rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
-rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
-rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
-rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
-rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
-rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
-rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
-rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
-rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
-rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
-rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
-rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
-rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
-rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
-rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
-rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
-rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
-rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
-rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
-rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
-rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
-rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
-rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
-rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
-rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
-rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
-rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
-rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
-rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
-rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
-rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
-rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
-rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
-rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
-rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
-rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
-rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
-rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
-rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
-rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
-rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
-rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
-rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
-rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
-rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
-rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
-rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
-rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
-rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
-rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
-rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
-rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
-rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
-rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
-rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
-rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
-rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
-rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
-rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
-rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
-rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
-rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
-rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
-rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
-rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
-rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
-rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
-rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
-rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
-rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
-rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

Yonik Seeley-2
Have you checked if this is due to running out of heap memory?
When that happens, the garbage collector can start taking a lot of CPU.
If you are using a Java6 JVM, it should have management enabled by
default and you should be able to connect to it via jconsole and
check.

-Yonik

On Thu, Feb 28, 2008 at 9:54 AM, F Knudson <[hidden email]> wrote:

>
>  Optimization time on solr index has turned into days/weeks.
>  We are using solr 1.2.
>  We use one box to build/optimize indexes. This index is copied to another
>  box for searching purposes.
>  We welcome suggestions/comments, etc.  We are a bit stumped on this.
>  Details are below.
>
>  Box details
>  Proc: 8 Dual Core 2.6GHz
>  Mem: 32 GB
>  OS: Red Hat Linux Enterprise 4
>  Kernel: 2.6.9-55.0.12.ELlargesmp
>
>  These are details from the index currently in use.  Search response time is
>  very acceptable (searchers are very happy)
>  Optimization time - 10433  (12/11/07)
>  index size - 229486464
>  # of records - 84960570
>  index directory
>  flasher# ls -l
>  total 229486464
>  -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
>  -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
>  -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
>  -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
>  -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
>  -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
>  -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
>  -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
>  -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
>  -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
>  --------------------------
>
>  current directory listing
>  indexed new records - Jan 22 and Jan 27
>  # of records - 85032470
>  optimization time - 558188
>
>  There were no out of memory errors.  There was 800961792KB  left in the
>  directory.  The files were not
>  collapsed as expected.  There are still files dated Jan 22 and Jan 27.
>
>  A new optimization was started Feb. 11 and continues.
>  This is a snapshot of the index directory.
>
>  We have at least another million records to add.  Plus weekly updates of
>  approximately 103K records.
>  We are using the direct indexing method.
>  java settings used - java -Xmx1024M -Xms1024M
>
>  The files continue to grow so work is progressing.
>  snapshot 2/21/08
>  -bash-3.00$ ls -ltr
>  total 205396680
>  -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
>  -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
>  -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
>  -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
>  -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
>  -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
>  -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
>  -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
>  -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
>  -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
>  -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
>  -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
>  -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
>  -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
>  -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
>  -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
>  -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
>  -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
>  -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
>  -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
>  -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
>  -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
>  -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
>  -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
>  -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
>  -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
>  -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
>  -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
>  -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
>  -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
>  -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
>  -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
>  -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
>  -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
>  -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
>  -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
>  -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
>  -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
>  -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
>  -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
>  -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
>  -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
>  -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
>  -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
>  -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
>  -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
>  -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
>  -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
>  -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
>  -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
>  -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
>  -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
>  -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
>  -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
>  -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
>  -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
>  -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
>  -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
>  -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
>  -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
>  -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
>  -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
>  -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
>  -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
>  -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
>  -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
>  -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
>  -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
>  -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
>  -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
>  -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
>  -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
>  -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
>  --
>  View this message in context: http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
>  Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

Otis Gospodnetic-2
In reply to this post by F Knudson
That's a tiny little index there ;)  Circa 100GB?
 
What do you see if you run vmstat 2 while the optimization is happening? Non-idle CPU?  A pile of IO?  Is there a reason for such a small heap on a machine with 32GB of RAM?

Otis

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: F Knudson <[hidden email]>
> To: [hidden email]
> Sent: Thursday, February 28, 2008 9:54:50 AM
> Subject: Optimization taking days/weeks
>
>
> Optimization time on solr index has turned into days/weeks.
> We are using solr 1.2.
> We use one box to build/optimize indexes. This index is copied to another
> box for searching purposes.
> We welcome suggestions/comments, etc.  We are a bit stumped on this.
> Details are below.
>
> Box details
> Proc: 8 Dual Core 2.6GHz
> Mem: 32 GB
> OS: Red Hat Linux Enterprise 4
> Kernel: 2.6.9-55.0.12.ELlargesmp
>
> These are details from the index currently in use.  Search response time is
> very acceptable (searchers are very happy)
> Optimization time - 10433  (12/11/07)
> index size - 229486464
> # of records - 84960570
> index directory
> flasher# ls -l
> total 229486464
> -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
> -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
> -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
> -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
> -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
> -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
> -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
> -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
> -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
> -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
> --------------------------
>
> current directory listing
> indexed new records - Jan 22 and Jan 27
> # of records - 85032470
> optimization time - 558188
>
> There were no out of memory errors.  There was 800961792KB  left in the
> directory.  The files were not
> collapsed as expected.  There are still files dated Jan 22 and Jan 27.  
>
> A new optimization was started Feb. 11 and continues.
> This is a snapshot of the index directory.
>
> We have at least another million records to add.  Plus weekly updates of
> approximately 103K records.
> We are using the direct indexing method.
> java settings used - java -Xmx1024M -Xms1024M
>
> The files continue to grow so work is progressing.
> snapshot 2/21/08
> -bash-3.00$ ls -ltr
> total 205396680
> -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
> -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
> -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
> -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
> -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
> -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
> -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
> -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
> -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
> -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
> -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
> -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
> -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
> -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
> -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
> -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
> -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
> -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
> -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
> -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
> -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
> -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
> -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
> -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
> -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
> -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
> -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
> -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
> -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
> -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
> -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
> -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
> -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
> -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
> -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
> -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
> -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
> -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
> -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
> -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
> -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
> -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
> -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
> -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
> -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
> -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
> -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
> -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
> -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
> -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
> -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
> -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
> -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
> -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
> -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
> -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
> -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
> -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
> -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
> -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
> -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
> -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
> -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
> -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
> -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
> -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
> -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
> -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
> -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
> -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
> -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
> -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
> -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
> --
> View this message in context:
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


Reply | Threaded
Open this post in threaded view
|

RE: Optimization taking days/weeks

Alex Benjamen
In reply to this post by Yonik Seeley-2
This sounds too familiar...

>java settings used - java -Xmx1024M -Xms1024M  
Sounds like your settings are pretty low... if you're using 64bit JVM, you should be able to set
these much higher, maybe give it like 8gb.

Another thing, you may want to look at reducing the index size... is there any way you could
partition the index? Also only index fields which you need and do not store the values in the index.
I've originally had an index which was 50Gb in size, and after removing fields I do not need, I'm down
to 8Gb and not storing any values in the index.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

F Knudson
In reply to this post by Yonik Seeley-2
Yes indeed - it was spending all of its time in garbage collection.  We will be moving to Java6.
Thanks for your suggestion.

Frances

Yonik Seeley wrote
Have you checked if this is due to running out of heap memory?
When that happens, the garbage collector can start taking a lot of CPU.
If you are using a Java6 JVM, it should have management enabled by
default and you should be able to connect to it via jconsole and
check.

-Yonik

On Thu, Feb 28, 2008 at 9:54 AM, F Knudson <fknudson@lanl.gov> wrote:
>
>  Optimization time on solr index has turned into days/weeks.
>  We are using solr 1.2.
>  We use one box to build/optimize indexes. This index is copied to another
>  box for searching purposes.
>  We welcome suggestions/comments, etc.  We are a bit stumped on this.
>  Details are below.
>
>  Box details
>  Proc: 8 Dual Core 2.6GHz
>  Mem: 32 GB
>  OS: Red Hat Linux Enterprise 4
>  Kernel: 2.6.9-55.0.12.ELlargesmp
>
>  These are details from the index currently in use.  Search response time is
>  very acceptable (searchers are very happy)
>  Optimization time - 10433  (12/11/07)
>  index size - 229486464
>  # of records - 84960570
>  index directory
>  flasher# ls -l
>  total 229486464
>  -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
>  -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
>  -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
>  -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
>  -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
>  -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
>  -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
>  -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
>  -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
>  -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
>  --------------------------
>
>  current directory listing
>  indexed new records - Jan 22 and Jan 27
>  # of records - 85032470
>  optimization time - 558188
>
>  There were no out of memory errors.  There was 800961792KB  left in the
>  directory.  The files were not
>  collapsed as expected.  There are still files dated Jan 22 and Jan 27.
>
>  A new optimization was started Feb. 11 and continues.
>  This is a snapshot of the index directory.
>
>  We have at least another million records to add.  Plus weekly updates of
>  approximately 103K records.
>  We are using the direct indexing method.
>  java settings used - java -Xmx1024M -Xms1024M
>
>  The files continue to grow so work is progressing.
>  snapshot 2/21/08
>  -bash-3.00$ ls -ltr
>  total 205396680
>  -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
>  -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
>  -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
>  -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
>  -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
>  -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
>  -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
>  -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
>  -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
>  -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
>  -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
>  -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
>  -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
>  -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
>  -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
>  -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
>  -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
>  -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
>  -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
>  -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
>  -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
>  -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
>  -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
>  -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
>  -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
>  -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
>  -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
>  -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
>  -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
>  -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
>  -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
>  -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
>  -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
>  -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
>  -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
>  -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
>  -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
>  -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
>  -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
>  -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
>  -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
>  -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
>  -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
>  -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
>  -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
>  -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
>  -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
>  -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
>  -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
>  -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
>  -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
>  -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
>  -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
>  -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
>  -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
>  -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
>  -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
>  -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
>  -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
>  -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
>  -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
>  -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
>  -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
>  -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
>  -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
>  -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
>  -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
>  -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
>  -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
>  -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
>  -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
>  -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
>  -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
>  --
>  View this message in context: http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
>  Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

RE: Optimization taking days/weeks

F Knudson
In reply to this post by Alex Benjamen
We will review the java settings.  The current settings are a bit low - but the indexed typically does not reach even 50% of the allocated 1024MB Max Heap.

Yes the index is large - only 3 fields are stored - and I have set the positionIncrementGap to 50 (down from 100) in an attempt to reduce index size.  Would you suggest to build one index used only for searching and a second index used only for display?  Does that fit within your definition of "partition"?

Thanks
Frances

Alex Benjamen wrote
This sounds too familiar...

>java settings used - java -Xmx1024M -Xms1024M  
Sounds like your settings are pretty low... if you're using 64bit JVM, you should be able to set
these much higher, maybe give it like 8gb.

Another thing, you may want to look at reducing the index size... is there any way you could
partition the index? Also only index fields which you need and do not store the values in the index.
I've originally had an index which was 50Gb in size, and after removing fields I do not need, I'm down
to 8Gb and not storing any values in the index.
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

F Knudson
In reply to this post by Otis Gospodnetic-2
We are a bit concerned regarding the index size.  At least no response (so far) as indicated that the size is unmanagable.  We killed the process - will move to Java6 - and will use vmstat to monitor the new optimization process.
At what index size would you begin to worry?  Or is it a combination of index size, optimization time, and response time?
We are data rich here!
Thanks
Frances

Otis Gospodnetic wrote
That's a tiny little index there ;)  Circa 100GB?
 
What do you see if you run vmstat 2 while the optimization is happening? Non-idle CPU?  A pile of IO?  Is there a reason for such a small heap on a machine with 32GB of RAM?

Otis

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----
> From: F Knudson <fknudson@lanl.gov>
> To: solr-user@lucene.apache.org
> Sent: Thursday, February 28, 2008 9:54:50 AM
> Subject: Optimization taking days/weeks
>
>
> Optimization time on solr index has turned into days/weeks.
> We are using solr 1.2.
> We use one box to build/optimize indexes. This index is copied to another
> box for searching purposes.
> We welcome suggestions/comments, etc.  We are a bit stumped on this.
> Details are below.
>
> Box details
> Proc: 8 Dual Core 2.6GHz
> Mem: 32 GB
> OS: Red Hat Linux Enterprise 4
> Kernel: 2.6.9-55.0.12.ELlargesmp
>
> These are details from the index currently in use.  Search response time is
> very acceptable (searchers are very happy)
> Optimization time - 10433  (12/11/07)
> index size - 229486464
> # of records - 84960570
> index directory
> flasher# ls -l
> total 229486464
> -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
> -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
> -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
> -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
> -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
> -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
> -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
> -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
> -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
> -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
> --------------------------
>
> current directory listing
> indexed new records - Jan 22 and Jan 27
> # of records - 85032470
> optimization time - 558188
>
> There were no out of memory errors.  There was 800961792KB  left in the
> directory.  The files were not
> collapsed as expected.  There are still files dated Jan 22 and Jan 27.  
>
> A new optimization was started Feb. 11 and continues.
> This is a snapshot of the index directory.
>
> We have at least another million records to add.  Plus weekly updates of
> approximately 103K records.
> We are using the direct indexing method.
> java settings used - java -Xmx1024M -Xms1024M
>
> The files continue to grow so work is progressing.
> snapshot 2/21/08
> -bash-3.00$ ls -ltr
> total 205396680
> -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
> -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
> -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
> -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
> -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
> -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
> -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
> -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
> -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
> -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
> -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
> -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
> -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
> -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
> -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
> -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
> -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
> -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
> -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
> -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
> -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
> -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
> -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
> -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
> -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
> -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
> -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
> -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
> -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
> -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
> -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
> -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
> -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
> -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
> -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
> -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
> -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
> -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
> -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
> -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
> -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
> -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
> -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
> -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
> -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
> -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
> -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
> -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
> -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
> -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
> -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
> -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
> -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
> -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
> -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
> -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
> -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
> -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
> -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
> -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
> -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
> -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
> -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
> -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
> -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
> -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
> -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
> -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
> -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
> -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
> -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
> -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
> -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
> --
> View this message in context:
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

Otis Gospodnetic-2
In reply to this post by F Knudson
Note that moving to Java 6 alone will not save you.  You do need ti give this poor JVM more memory to play with when working with such a big index.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: F Knudson <[hidden email]>
> To: [hidden email]
> Sent: Friday, February 29, 2008 11:22:04 AM
> Subject: Re: Optimization taking days/weeks
>
>
> Yes indeed - it was spending all of its time in garbage collection.  We will
> be moving to Java6.
> Thanks for your suggestion.
>
> Frances
>
>
> Yonik Seeley wrote:
> >
> > Have you checked if this is due to running out of heap memory?
> > When that happens, the garbage collector can start taking a lot of CPU.
> > If you are using a Java6 JVM, it should have management enabled by
> > default and you should be able to connect to it via jconsole and
> > check.
> >
> > -Yonik
> >
> > On Thu, Feb 28, 2008 at 9:54 AM, F Knudson  wrote:
> >>
> >>  Optimization time on solr index has turned into days/weeks.
> >>  We are using solr 1.2.
> >>  We use one box to build/optimize indexes. This index is copied to
> >> another
> >>  box for searching purposes.
> >>  We welcome suggestions/comments, etc.  We are a bit stumped on this.
> >>  Details are below.
> >>
> >>  Box details
> >>  Proc: 8 Dual Core 2.6GHz
> >>  Mem: 32 GB
> >>  OS: Red Hat Linux Enterprise 4
> >>  Kernel: 2.6.9-55.0.12.ELlargesmp
> >>
> >>  These are details from the index currently in use.  Search response time
> >> is
> >>  very acceptable (searchers are very happy)
> >>  Optimization time - 10433  (12/11/07)
> >>  index size - 229486464
> >>  # of records - 84960570
> >>  index directory
> >>  flasher# ls -l
> >>  total 229486464
> >>  -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
> >>  -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
> >>  -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
> >>  -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
> >>  -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
> >>  -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
> >>  -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
> >>  -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
> >>  -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
> >>  -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
> >>  --------------------------
> >>
> >>  current directory listing
> >>  indexed new records - Jan 22 and Jan 27
> >>  # of records - 85032470
> >>  optimization time - 558188
> >>
> >>  There were no out of memory errors.  There was 800961792KB  left in the
> >>  directory.  The files were not
> >>  collapsed as expected.  There are still files dated Jan 22 and Jan 27.
> >>
> >>  A new optimization was started Feb. 11 and continues.
> >>  This is a snapshot of the index directory.
> >>
> >>  We have at least another million records to add.  Plus weekly updates of
> >>  approximately 103K records.
> >>  We are using the direct indexing method.
> >>  java settings used - java -Xmx1024M -Xms1024M
> >>
> >>  The files continue to grow so work is progressing.
> >>  snapshot 2/21/08
> >>  -bash-3.00$ ls -ltr
> >>  total 205396680
> >>  -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
> >>  -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
> >>  -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
> >>  -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
> >>  -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
> >>  -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
> >>  -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
> >>  -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
> >>  -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
> >>  -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
> >>  -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
> >>  -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
> >>  -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
> >>  -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
> >>  -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
> >>  -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
> >>  -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
> >>  -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
> >>  -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
> >>  -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
> >>  -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
> >>  -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
> >>  -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
> >>  -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
> >>  -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
> >>  -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
> >>  -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
> >>  -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
> >>  -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
> >>  -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
> >>  -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
> >>  -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
> >>  -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
> >>  -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
> >>  -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
> >>  -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
> >>  -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
> >>  -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
> >>  -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
> >>  -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
> >>  -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
> >>  -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
> >>  -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
> >>  -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
> >>  -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
> >>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
> >>  -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
> >>  -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
> >>  -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
> >>  -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
> >>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
> >>  -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
> >>  -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
> >>  -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
> >>  -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
> >>  -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
> >>  -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
> >>  -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
> >>  -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
> >>  -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
> >>  -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
> >>  -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
> >>  -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
> >>  -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
> >>  -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
> >>  -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
> >>  -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
> >>  -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
> >>  -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
> >>  -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
> >>  -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
> >>  -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
> >>  -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
> >>  -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
> >>  -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
> >>  -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
> >>  --
> >>  View this message in context:
> >> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
> >>  Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762017.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

Otis Gospodnetic-2
In reply to this post by F Knudson
You changed the positionIncrementGap to reduce index size....?  Hm, maybe I'm forgetting something at the moment, but I don't think this will make any difference with index size, just with token offsets noted at index time.  Are you referring to term index interval (normally 128 by default, I believe, but changeable, at least directly in Lucene)?  That is one thing that would have some effect on the index size (but unfortunately also on search performance).

Depending on how big those stored fields are, yes, perhaps it would make sense to keep those elsewhere (e.g. BDB, RDBMS).  This would make a difference for index optimization - fewer bytes would have to be read from and then written back to disk.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: F Knudson <[hidden email]>
> To: [hidden email]
> Sent: Friday, February 29, 2008 11:28:32 AM
> Subject: RE: Optimization taking days/weeks
>
>
> We will review the java settings.  The current settings are a bit low - but
> the indexed typically does not reach even 50% of the allocated 1024MB Max
> Heap.
>
> Yes the index is large - only 3 fields are stored - and I have set the
> positionIncrementGap to 50 (down from 100) in an attempt to reduce index
> size.  Would you suggest to build one index used only for searching and a
> second index used only for display?  Does that fit within your definition of
> "partition"?
>
> Thanks
> Frances
>
>
> Alex Benjamen wrote:
> >
> > This sounds too familiar...
> >
> >>java settings used - java -Xmx1024M -Xms1024M  
> > Sounds like your settings are pretty low... if you're using 64bit JVM, you
> > should be able to set
> > these much higher, maybe give it like 8gb.
> >
> > Another thing, you may want to look at reducing the index size... is there
> > any way you could
> > partition the index? Also only index fields which you need and do not
> > store the values in the index.
> > I've originally had an index which was 50Gb in size, and after removing
> > fields I do not need, I'm down
> > to 8Gb and not storing any values in the index.
> >  
> >  
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762156.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Optimization taking days/weeks

Otis Gospodnetic-2
In reply to this post by F Knudson
It's really about the combination of index size, hardware, required response rate, query rate and complexity.  You typically try to benchmark this stuff to see where the limit or where the sweet spot is for your hardware.  Unfortunately, I don't have an explanation for the sudden jump in increase time.  This might be something to take directly to java-user@lucene list, as optimization is really pure Lucene thing.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

----- Original Message ----

> From: F Knudson <[hidden email]>
> To: [hidden email]
> Sent: Friday, February 29, 2008 11:38:31 AM
> Subject: Re: Optimization taking days/weeks
>
>
> We are a bit concerned regarding the index size.  At least no response (so
> far) as indicated that the size is unmanagable.  We killed the process -
> will move to Java6 - and will use vmstat to monitor the new optimization
> process.
> At what index size would you begin to worry?  Or is it a combination of
> index size, optimization time, and response time?
> We are data rich here!
> Thanks
> Frances
>
>
> Otis Gospodnetic wrote:
> >
> > That's a tiny little index there ;)  Circa 100GB?
> >  
> > What do you see if you run vmstat 2 while the optimization is happening?
> > Non-idle CPU?  A pile of IO?  Is there a reason for such a small heap on a
> > machine with 32GB of RAM?
> >
> > Otis
> >
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> > ----- Original Message ----
> >> From: F Knudson
> >> To: [hidden email]
> >> Sent: Thursday, February 28, 2008 9:54:50 AM
> >> Subject: Optimization taking days/weeks
> >>
> >>
> >> Optimization time on solr index has turned into days/weeks.
> >> We are using solr 1.2.
> >> We use one box to build/optimize indexes. This index is copied to another
> >> box for searching purposes.
> >> We welcome suggestions/comments, etc.  We are a bit stumped on this.
> >> Details are below.
> >>
> >> Box details
> >> Proc: 8 Dual Core 2.6GHz
> >> Mem: 32 GB
> >> OS: Red Hat Linux Enterprise 4
> >> Kernel: 2.6.9-55.0.12.ELlargesmp
> >>
> >> These are details from the index currently in use.  Search response time
> >> is
> >> very acceptable (searchers are very happy)
> >> Optimization time - 10433  (12/11/07)
> >> index size - 229486464
> >> # of records - 84960570
> >> index directory
> >> flasher# ls -l
> >> total 229486464
> >> -rw-r--r--   1 flknud   staff    22197926593 Dec 12 08:07 _2bl6.fdt
> >> -rw-r--r--   1 flknud   staff    679684560 Dec 12 08:20 _2bl6.fdx
> >> -rw-r--r--   1 flknud   staff        208 Dec 12 08:23 _2bl6.fnm
> >> -rw-r--r--   1 flknud   staff    40176405625 Dec 12 09:28 _2bl6.frq
> >> -rw-r--r--   1 flknud   staff    594723994 Dec 12 09:41 _2bl6.nrm
> >> -rw-r--r--   1 flknud   staff    47616340310 Dec 12 12:07 _2bl6.prx
> >> -rw-r--r--   1 flknud   staff    76708079 Dec 12 12:25 _2bl6.tii
> >> -rw-r--r--   1 flknud   staff    6154384415 Dec 12 12:42 _2bl6.tis
> >> -rw-r--r--   1 flknud   staff         20 Dec 12 12:48 segments.gen
> >> -rw-r--r--   1 flknud   staff         44 Dec 12 12:48 segments_2c64
> >> --------------------------
> >>
> >> current directory listing
> >> indexed new records - Jan 22 and Jan 27
> >> # of records - 85032470
> >> optimization time - 558188
> >>
> >> There were no out of memory errors.  There was 800961792KB  left in the
> >> directory.  The files were not
> >> collapsed as expected.  There are still files dated Jan 22 and Jan 27.  
> >>
> >> A new optimization was started Feb. 11 and continues.
> >> This is a snapshot of the index directory.
> >>
> >> We have at least another million records to add.  Plus weekly updates of
> >> approximately 103K records.
> >> We are using the direct indexing method.
> >> java settings used - java -Xmx1024M -Xms1024M
> >>
> >> The files continue to grow so work is progressing.
> >> snapshot 2/21/08
> >> -bash-3.00$ ls -ltr
> >> total 205396680
> >> -rw-r--r--  1 flknud users         208 Jan 10 07:15 _2bm7.fnm
> >> -rw-r--r--  1 flknud users 22202159522 Jan 10 08:09 _2bm7.fdt
> >> -rw-r--r--  1 flknud users   679819760 Jan 10 08:09 _2bm7.fdx
> >> -rw-r--r--  1 flknud users 40184944027 Jan 16 18:16 _2bm7.frq
> >> -rw-r--r--  1 flknud users 47626230575 Jan 16 18:16 _2bm7.prx
> >> -rw-r--r--  1 flknud users  6155230704 Jan 16 18:16 _2bm7.tis
> >> -rw-r--r--  1 flknud users    76704158 Jan 16 18:16 _2bm7.tii
> >> -rw-r--r--  1 flknud users   594842294 Jan 16 18:18 _2bm7.nrm
> >> -rw-r--r--  1 flknud users         208 Jan 22 08:57 _2bpa.fnm
> >> -rw-r--r--  1 flknud users    10806426 Jan 22 08:57 _2bpa.fdt
> >> -rw-r--r--  1 flknud users      371200 Jan 22 08:57 _2bpa.fdx
> >> -rw-r--r--  1 flknud users    21114330 Jan 22 08:57 _2bpa.frq
> >> -rw-r--r--  1 flknud users    25683573 Jan 22 08:57 _2bpa.prx
> >> -rw-r--r--  1 flknud users     9225592 Jan 22 08:57 _2bpa.tis
> >> -rw-r--r--  1 flknud users      118660 Jan 22 08:57 _2bpa.tii
> >> -rw-r--r--  1 flknud users      324804 Jan 22 08:57 _2bpa.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 22 09:00 _2bpl.fnm
> >> -rw-r--r--  1 flknud users     1335931 Jan 22 09:00 _2bpl.fdt
> >> -rw-r--r--  1 flknud users       36800 Jan 22 09:00 _2bpl.fdx
> >> -rw-r--r--  1 flknud users     2646708 Jan 22 09:00 _2bpl.frq
> >> -rw-r--r--  1 flknud users     3781824 Jan 22 09:00 _2bpl.prx
> >> -rw-r--r--  1 flknud users     1429176 Jan 22 09:00 _2bpl.tis
> >> -rw-r--r--  1 flknud users       18582 Jan 22 09:00 _2bpl.tii
> >> -rw-r--r--  1 flknud users       32204 Jan 22 09:00 _2bpl.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 22 09:01 _2bpm.fnm
> >> -rw-r--r--  1 flknud users      121716 Jan 22 09:01 _2bpm.fdt
> >> -rw-r--r--  1 flknud users        3200 Jan 22 09:01 _2bpm.fdx
> >> -rw-r--r--  1 flknud users      205961 Jan 22 09:01 _2bpm.frq
> >> -rw-r--r--  1 flknud users      302114 Jan 22 09:01 _2bpm.prx
> >> -rw-r--r--  1 flknud users      233641 Jan 22 09:01 _2bpm.tis
> >> -rw-r--r--  1 flknud users        3036 Jan 22 09:01 _2bpm.tii
> >> -rw-r--r--  1 flknud users        2804 Jan 22 09:01 _2bpm.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:00 _2bpn.fnm
> >> -rw-r--r--  1 flknud users      227962 Jan 27 14:00 _2bpn.fdt
> >> -rw-r--r--  1 flknud users        7200 Jan 27 14:00 _2bpn.fdx
> >> -rw-r--r--  1 flknud users      437798 Jan 27 14:00 _2bpn.frq
> >> -rw-r--r--  1 flknud users      593858 Jan 27 14:00 _2bpn.prx
> >> -rw-r--r--  1 flknud users      516031 Jan 27 14:00 _2bpn.tis
> >> -rw-r--r--  1 flknud users        6814 Jan 27 14:00 _2bpn.tii
> >> -rw-r--r--  1 flknud users        6304 Jan 27 14:00 _2bpn.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpo.fnm
> >> -rw-r--r--  1 flknud users      231456 Jan 27 14:01 _2bpo.fdt
> >> -rw-r--r--  1 flknud users        7200 Jan 27 14:01 _2bpo.fdx
> >> -rw-r--r--  1 flknud users      448401 Jan 27 14:01 _2bpo.frq
> >> -rw-r--r--  1 flknud users      616557 Jan 27 14:01 _2bpo.prx
> >> -rw-r--r--  1 flknud users      587697 Jan 27 14:01 _2bpo.tis
> >> -rw-r--r--  1 flknud users        7801 Jan 27 14:01 _2bpo.tii
> >> -rw-r--r--  1 flknud users        6304 Jan 27 14:01 _2bpo.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpp.fnm
> >> -rw-r--r--  1 flknud users      229944 Jan 27 14:01 _2bpp.fdt
> >> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpp.fdx
> >> -rw-r--r--  1 flknud users      462865 Jan 27 14:01 _2bpp.frq
> >> -rw-r--r--  1 flknud users      652003 Jan 27 14:01 _2bpp.prx
> >> -rw-r--r--  1 flknud users      574096 Jan 27 14:01 _2bpp.tis
> >> -rw-r--r--  1 flknud users        7452 Jan 27 14:01 _2bpp.tii
> >> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpp.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpq.fnm
> >> -rw-r--r--  1 flknud users       53438 Jan 27 14:01 _2bpq.fdt
> >> -rw-r--r--  1 flknud users        1600 Jan 27 14:01 _2bpq.fdx
> >> -rw-r--r--  1 flknud users      103059 Jan 27 14:01 _2bpq.frq
> >> -rw-r--r--  1 flknud users      157149 Jan 27 14:01 _2bpq.prx
> >> -rw-r--r--  1 flknud users      179711 Jan 27 14:01 _2bpq.tis
> >> -rw-r--r--  1 flknud users        2489 Jan 27 14:01 _2bpq.tii
> >> -rw-r--r--  1 flknud users        1404 Jan 27 14:01 _2bpq.nrm
> >> -rw-r--r--  1 flknud users         198 Jan 27 14:01 _2bpr.fnm
> >> -rw-r--r--  1 flknud users      200777 Jan 27 14:01 _2bpr.fdt
> >> -rw-r--r--  1 flknud users        6400 Jan 27 14:01 _2bpr.fdx
> >> -rw-r--r--  1 flknud users      416658 Jan 27 14:01 _2bpr.frq
> >> -rw-r--r--  1 flknud users      570959 Jan 27 14:01 _2bpr.prx
> >> -rw-r--r--  1 flknud users      489593 Jan 27 14:01 _2bpr.tis
> >> -rw-r--r--  1 flknud users        6451 Jan 27 14:01 _2bpr.tii
> >> -rw-r--r--  1 flknud users        5604 Jan 27 14:01 _2bpr.nrm
> >> -rw-r--r--  1 flknud users         236 Jan 27 14:01 segments_2cap
> >> -rw-r--r--  1 flknud users          20 Jan 27 14:01 segments.gen
> >> -rw-r--r--  1 flknud users           0 Feb 11 09:56 write.lock
> >> -rw-r--r--  1 flknud users         208 Feb 11 09:56 _2bps.fnm
> >> -rw-r--r--  1 flknud users 22215367172 Feb 11 12:26 _2bps.fdt
> >> -rw-r--r--  1 flknud users   680259760 Feb 11 12:26 _2bps.fdx
> >> -rw-r--r--  1 flknud users    24416256 Feb 21 06:42 _2bps.tii
> >> -rw-r--r--  1 flknud users  1685057536 Feb 21 06:56 _2bps.tis
> >> -rw-r--r--  1 flknud users 29885433856 Feb 21 07:32 _2bps.frq
> >> -rw-r--r--  1 flknud users 38229336064 Feb 21 07:32 _2bps.prx
> >> --
> >> View this message in context:
> >> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15738090.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Optimization-taking-days-weeks-tp15738090p15762378.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>