[BUG] org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]

[reformit]

What is the bug?

When I attempt to start OpenSearch 2.11.0 with data from our ElasticSearch 7.5.2 setup I get the following exception in the logs:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]

How can one reproduce the bug?

Use the following docker-compose.yml to startup elasticsearch with kibana:

services:
  elasticsearch-node:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.5.2
    container_name: elasticsearch-node
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - ./volumes/data/:/usr/share/elasticsearch/data
      - ./volumes/logs/:/usr/share/elasticsearch/logs
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - elasticsearch-net

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.5.2
    environment:
      ELASTICSEARCH_HOSTS: '["http://elasticsearch-node:9200"]'
    ports:
      - 5600:5601
    networks:
      - elasticsearch-net

networks:
  elasticsearch-net:

bring down ElasticSearch and Kabana (docker-compose down)

Use the following docker-compose to start OpenSearch making sure that the volumes mapped on opensearch-node are the same folders that were used when starting ElasticSearch previously

version: "3"
services:
  opensearch-node:
    image: opensearchproject/opensearch:2.11.0
    container_name: opensearch-node
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "DISABLE_SECURITY_PLUGIN=true"
      - action.auto_create_index=false
    ports:
      - 9201:9200
      - 9601:9600 # required for Performance Analyzer
    volumes:
      - ./volumes/data/:/usr/share/opensearch/data
      - ./volumes/logs/:/usr/share/opensearch/logs
    networks:
      - opensearch-net

  opensearch-dashboard:
    image: opensearchproject/opensearch-dashboards:2.11.0
    container_name: opensearch-dashboard
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["http://opensearch-node:9200"]'
    networks:
      - opensearch-net

networks:
  opensearch-net:

The exception should show in the logs.

What is the expected behavior?

No exception and OpenSearch starts successfully (at the moment it doesn't)

What is your host/environment?

POP_OS 22.04 LTS

Do you have any screenshots?

Nothing of value

Do you have any additional context?

Here's the stacktrace in the OpenSearch logs

opensearch-node         | [2023-11-17T23:03:55,449][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [0fdcca6d2139] uncaught exception in thread [main]
opensearch-node         | org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         | Caused by: java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:252) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | Caused by: org.opensearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [local_metadata]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:440) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | Caused by: org.opensearch.index.mapper.MapperParsingException: No handler for type [flattened] declared on field [local_metadata]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:381) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:265) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:389) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:183) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:144) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:133) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:438) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
opensearch-node         |       ... 6 more
opensearch-node         | uncaught exception in thread [main]
opensearch-node         | java.lang.IllegalStateException: unable to upgrade the mappings for the index [[.kibana_1/am9gfVSeQ_OuTpWmFLVWFQ]]
opensearch-node         | Likely root cause: MapperParsingException[No handler for type [flattened] declared on field [local_metadata]]
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:381)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:265)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:389)
opensearch-node         |       at org.opensearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:296)
opensearch-node         |       at org.opensearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:183)
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:144)
opensearch-node         |       at org.opensearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:133)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:438)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.internalMerge(MapperService.java:426)
opensearch-node         |       at org.opensearch.index.mapper.MapperService.merge(MapperService.java:412)
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.checkMappingsCompatibility(MetadataIndexUpgradeService.java:248)
opensearch-node         |       at org.opensearch.cluster.metadata.MetadataIndexUpgradeService.upgradeIndexMetadata(MetadataIndexUpgradeService.java:121)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadata(GatewayMetaState.java:289)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.upgradeMetadataForNode(GatewayMetaState.java:271)
opensearch-node         |       at org.opensearch.gateway.GatewayMetaState.start(GatewayMetaState.java:167)
opensearch-node         |       at org.opensearch.node.Node.start(Node.java:1368)
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:339)
opensearch-node         |       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:413)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
opensearch-node         |       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
opensearch-node         |       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
opensearch-node         |       at org.opensearch.cli.Command.main(Command.java:101)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
opensearch-node         |       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
opensearch-node         | For complete error details, refer to the log at /usr/share/opensearch/logs/docker-cluster.log

[sumobrian]

Hi @reformit - Have you tried updating to OpenSearch 1.x before moving on to OpenSearch 2.x? It's important to note that multi-version upgrades are not supported in both Elasticsearch and OpenSearch. Could you please wrap your *.yml configuration in triple backticks in the bug description? This will help preserve the indentation in the YAML files, which is crucial for parsing and building with docker-compose.

[reformit]

Hi @sumobrian, thanks for your reply! I've updated with backticks as requested. I don't completely understand your question about moving from OS 1.x to 2.x? I'm trying to migrate from our ElasticSearch 7.5.2 stack to OpenSearch. Are you suggesting I migrate to OS 1.x from 7.5.2 and then upgrade OS 1.x to 2.x? If that's the case I have tried it with OS 1.x and still had the same issue. Appreciate any suggestions/guidance you can give.

[wbeckler]

The flattened data type doesn't exist in OpenSearch. Here is the equivalent: https://opensearch.org/docs/latest/field-types/supported-field-types/flat-object/

[ComBin]

Anybody have any suggestion how to perform upgrade without removing Kibana index?

[sumobrian]

Our solution to migration from ES 7.5.2 to OS 1/2.x is documented here. The documentation and feature set has been greatly enhanced since this issue was opened. If you have any questions please open another issue. Although migrations are currently not available on the migration console, it is available within this repo and noted in the referenced wiki documentation. Here is the internal documentation. Please reach out to me or open another issue if you run into issues with your upgrade/migration process.