The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The default value is 1000.
index.mapping.depth.limit
The maximum depth for a field, which is measured as the number of inner objects. For instance, if all fields are defined at the root object level, then the depth is 1. If there is one object mapping, then the depth is 2, etc. The default is 20.
index.mapping.nested_fields.limit
The maximum number of nested fields in an index, defaults to 50. Indexing 1 document with 100 nested fields actually indexes 101 documents as each nested document is indexed as a separate hidden document.
index.mapping.nested_objects.limit
The maximum number of nested json objects within a single document across all nested fields, defaults to 10000. Indexing one document with an array of 100 objects within a nested field, will actually create 101 documents, as each nested object will be indexed as a separate hidden document.
혹시라도 문서 모델링 하시다가 field limit 에 대한 정보가 궁금 할 것 같아 일단 북마킹 합니다.
모델링을 flat 하게 펼치게 된 이유는 시간에 대한 다양한 요구사항이 발생을 하다 보니 부득이 이와 같이 펼치게 되었습니다.
As an alternative to requests being sent to copies of the data in a round robin fashion, you may enable adaptive replica selection. This allows the coordinating node to send the request to the copy deemed "best" based on a number of criteria:
Response time of past requests between the coordinating node and the node containing the copy of the data
Time past search requests took to execute on the node containing the data
The queue size of the search threadpool on the node containing the data
This can be turned on by changing the dynamic cluster setting cluster.routing.use_adaptive_replica_selection from false to true:
/** * Should a refresh be executed before this get operation causing the operation to * return the latest value. Note, heavy get should not set this to {@code true}. Defaults * to {@code false}. */ public GetRequest refresh(boolean refresh) { this.refresh = refresh; return this; }
public boolean refresh() { return this.refresh; }
public boolean realtime() { return this.realtime; }
@Override public RestChannelConsumer prepareRequest(final RestRequest request, final NodeClient client) throws IOException { final boolean includeTypeName = request.paramAsBoolean("include_type_name", true); final String type = request.param("type"); if (includeTypeName == false && MapperService.SINGLE_MAPPING_NAME.equals(type) == false) { throw new IllegalArgumentException("You may only use the [include_type_name=false] option with the get APIs with the " + "[{index}/_doc/{id}] endpoint."); } final GetRequest getRequest = new GetRequest(request.param("index"), type, request.param("id")); getRequest.refresh(request.paramAsBoolean("refresh", getRequest.refresh())); getRequest.routing(request.param("routing")); getRequest.preference(request.param("preference")); getRequest.realtime(request.paramAsBoolean("realtime", getRequest.realtime())); if (request.param("fields") != null) { throw new IllegalArgumentException("the parameter [fields] is no longer supported, " + "please use [stored_fields] to retrieve stored fields or [_source] to load the field from _source"); } final String fieldsParam = request.param("stored_fields"); if (fieldsParam != null) { final String[] fields = Strings.splitStringByCommaToArray(fieldsParam); if (fields != null) { getRequest.storedFields(fields); } }
GetResult result = indexShard.getService().get(request.type(), request.id(), request.storedFields(), request.realtime(), request.version(), request.versionType(), request.fetchSourceContext()); return new GetResponse(result); }
보시면 아시겠지만 refresh 와 realtime 의 동작에는 차이가 있습니다.
이해 하기 쉽게 정리 하면,
- refresh 는 get 수행 전에 relevant shard 를 대상으로 refresh 동작을 먼저 하기 때문에 성능 저하가 있을 수 있습니다.
- realtime 은 수행 시점에 refresh 동작을 수행 하게 됩니다. (이 경우 refresh searcher 의 대상은 internal searcher 가 됩니다.)
- 여기서 translog 에서 데이터를 읽어야 하는 경우가 있는데 이 경우는 update API 에서 사용 되게 됩니다.
[InternalEngine.java]
@Override public GetResult get(Get get, BiFunction<String, SearcherScope, Searcher> searcherFactory) throws EngineException { assert Objects.equals(get.uid().field(), IdFieldMapper.NAME) : get.uid().field(); try (ReleasableLock ignored = readLock.acquire()) { ensureOpen(); SearcherScope scope; if (get.realtime()) { VersionValue versionValue = null; try (Releasable ignore = versionMap.acquireLock(get.uid().bytes())) { // we need to lock here to access the version map to do this truly in RT versionValue = getVersionFromMap(get.uid().bytes()); } if (versionValue != null) { if (versionValue.isDelete()) { return GetResult.NOT_EXISTS; } if (get.versionType().isVersionConflictForReads(versionValue.version, get.version())) { throw new VersionConflictEngineException(shardId, get.type(), get.id(), get.versionType().explainConflictForReads(versionValue.version, get.version())); } if (get.isReadFromTranslog()) { // this is only used for updates - API _GET calls will always read form a reader for consistency // the update call doesn't need the consistency since it's source only + _parent but parent can go away in 7.0 if (versionValue.getLocation() != null) { try { Translog.Operation operation = translog.readOperation(versionValue.getLocation()); if (operation != null) { // in the case of a already pruned translog generation we might get null here - yet very unlikely TranslogLeafReader reader = new TranslogLeafReader((Translog.Index) operation, engineConfig .getIndexSettings().getIndexVersionCreated()); return new GetResult(new Searcher("realtime_get", new IndexSearcher(reader)), new VersionsAndSeqNoResolver.DocIdAndVersion(0, ((Translog.Index) operation).version(), reader, 0)); } } catch (IOException e) { maybeFailEngine("realtime_get", e); // lets check if the translog has failed with a tragic event throw new EngineException(shardId, "failed to read operation from translog", e); } } else { trackTranslogLocation.set(true); } } refresh("realtime_get", SearcherScope.INTERNAL); } scope = SearcherScope.INTERNAL; } else { // we expose what has been externally expose in a point in time snapshot via an explicit refresh scope = SearcherScope.EXTERNAL; }
// no version, get the version from the index, we know that we refresh on flush return getFromSearcher(get, searcherFactory, scope); } }
