[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

[jira] [Commented] (SOLR-11722) API to create a Time Routed Alias and first collection

JIRA jira@apache.org

    [ https://issues.apache.org/jira/browse/SOLR-11722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323658#comment-16323658 ]

ASF GitHub Bot commented on SOLR-11722:
---------------------------------------

Github user fsparv commented on a diff in the pull request:

    https://github.com/apache/lucene-solr/pull/304#discussion_r161155153
 
    --- Diff: solr/core/src/java/org/apache/solr/cloud/CreateAliasCmd.java ---
    @@ -68,34 +249,100 @@ public void call(ClusterState state, ZkNodeProps message, NamedList results)
         Thread.sleep(100);
       }
     
    +  private Map<String, String> buildAliasMap(String routedField, String routingType, String tz, String increment, String maxFutureMs, ZkNodeProps collectionProps) {
    +    Map<String, Object> properties = collectionProps.getProperties();
    +    Map<String,String> cleanMap = properties.entrySet().stream()
    +        .filter(stringObjectEntry ->
    +            !"fromApi".equals(stringObjectEntry.getKey())
    +                && !"stateFormat".equals(stringObjectEntry.getKey())
    +                && !"name".equals(stringObjectEntry.getKey()))
    +        .collect(Collectors.toMap((e) -> "collection-create." + e.getKey(), e -> String.valueOf(e.getValue())));
    +    cleanMap.put(ROUTING_FIELD, routedField);
    +    cleanMap.put(ROUTING_TYPE, routingType);
    +    cleanMap.put(ROUTING_INCREMENT, increment);
    +    cleanMap.put(ROUTING_MAX_FUTURE, maxFutureMs);
    +    cleanMap.put(TZ, tz);
    +    return cleanMap;
    +  }
    +
    +  private Instant validateStart(TimeZone zone, DateTimeFormatter fmt, String start) {
    +    // This is the normal/easy case, if we can get away with this great!
    +    TemporalAccessor startTime = attemptTimeStampParsing(start, zone.toZoneId());
    +    if (startTime == null) {
    +      // No luck, they gave us either date math, or garbage, so we have to do more work to figure out which and
    +      // to make sure it's valid date math and that it doesn't encode any millisecond precision.
    +      ZonedDateTime now = ZonedDateTime.now(zone.toZoneId()).truncatedTo(ChronoUnit.MILLIS);
    +      try {
    +        Date date = DateMathParser.parseMath(Date.from(now.toInstant()), start);
    +        String reformatted = fmt.format(date.toInstant().truncatedTo(ChronoUnit.MILLIS));
    +        Date reparse = Date.from(Instant.from(DATE_TIME_FORMATTER.parse(reformatted)));
    +        if (!reparse.equals(date)) {
    +          throw new SolrException(BAD_REQUEST,
    +              "Formatted time did not have the same milliseconds as original: " + date.getTime() + " vs. " +
    +                  reparse.getTime() + " This indicates that you used date math that includes milliseconds. " +
    +                  "(Hint: 'NOW' used without rounding always has this problem)" );
    +        }
    +        return date.toInstant();
    +      } catch (SolrException e) {
    +        throw new SolrException(BAD_REQUEST,
    +            "Start Time for the first collection must be a timestamp of the format yyyy-MM-dd_HH_mm_ss, " +
    +                "or a valid date math expression not containing specific milliseconds", e);
    +      }
    +    }
    +    return Instant.from(startTime);
    +  }
    +
    +  private TemporalAccessor attemptTimeStampParsing(String start, ZoneId zone) {
    +    try {
    +      DATE_TIME_FORMATTER.withZone(zone);
    +      return DATE_TIME_FORMATTER.parse(start);
    +    } catch (DateTimeParseException e) {
    +      return null;
    +    }
    +  }
    +
    +  private boolean anyRoutingParams(ZkNodeProps message) {
    +
    +    return message.containsKey(ROUTING_FIELD) || message.containsKey(ROUTING_TYPE) || message.containsKey(START)
    +        || message.containsKey(ROUTING_INCREMENT) || message.containsKey(TZ);
    +  }
    +
       private void validateAllCollectionsExistAndNoDups(List<String> collectionList, ZkStateReader zkStateReader) {
         final String collectionStr = StrUtils.join(collectionList, ',');
     
         if (new HashSet<>(collectionList).size() != collectionList.size()) {
    -      throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
    +      throw new SolrException(BAD_REQUEST,
               String.format(Locale.ROOT,  "Can't create collection alias for collections='%s', since it contains duplicates", collectionStr));
         }
         ClusterState clusterState = zkStateReader.getClusterState();
         Set<String> aliasNames = zkStateReader.getAliases().getCollectionAliasListMap().keySet();
         for (String collection : collectionList) {
           if (clusterState.getCollectionOrNull(collection) == null && !aliasNames.contains(collection)) {
    -        throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
    +        throw new SolrException(BAD_REQUEST,
                 String.format(Locale.ROOT,  "Can't create collection alias for collections='%s', '%s' is not an existing collection or alias", collectionStr, collection));
           }
         }
       }
    -  
    +
       /**
        * The v2 API directs that the 'collections' parameter be provided as a JSON array (e.g. ["a", "b"]).  We also
        * maintain support for the legacy format, a comma-separated list (e.g. a,b).
        */
       @SuppressWarnings("unchecked")
       private List<String> parseCollectionsParameter(Object colls) {
    -    if (colls == null) throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "missing collections param");
    +    if (colls == null) throw new SolrException(BAD_REQUEST, "missing collections param");
         if (colls instanceof List) return (List<String>) colls;
         return StrUtils.splitSmart(colls.toString(), ",", true).stream()
             .map(String::trim)
             .collect(Collectors.toList());
       }
     
    +  private ZkNodeProps selectByPrefix(String prefix, ZkNodeProps source) {
    --- End diff --
   
    Never really considered it a hack myself... it's just the standard way to work around the final variable restriction (now relaxed to "effectively final" of course) on anonymous inner classes. Lambdas (in java) are mostly just anon inner classes in disguise... but don't really care, changed to for loop in update.


> API to create a Time Routed Alias and first collection
> ------------------------------------------------------
>
>                 Key: SOLR-11722
>                 URL: https://issues.apache.org/jira/browse/SOLR-11722
>             Project: Solr
>          Issue Type: Sub-task
>      Security Level: Public(Default Security Level. Issues are Public)
>          Components: SolrCloud
>            Reporter: David Smiley
>         Attachments: SOLR-11722.patch, SOLR-11722.patch
>
>
> This issue is about creating a single API command to create a "Time Routed Alias" along with its first collection.  Need to decide what endpoint URL it is and parameters.
> Perhaps in v2 it'd be {{/api/collections?command=create-routed-alias}} or alternatively piggy-back off of command=create-alias but we add more options, perhaps with a prefix like "router"?
> Inputs:
> * alias name
> * misc collection creation metadata (e.g. config, numShards, ...) perhaps in this context with a prefix like "collection."
> * metadata for TimeRoutedAliasUpdateProcessor, currently: router.field
> * date specifier for first collection; can include "date math".
> We'll certainly add more options as future features unfold.
> I believe the collection needs to be created first (referring to the alias name via a core property), and then the alias pointing to it which demands collections exist first.  When figuring the collection name, you'll need to reference the format in TimeRoutedAliasUpdateProcessor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]