Optimizing “index.mapping.total_fields.limit” Parameter in Elasticsearch

Problem:

The client seeks guidance on determining the appropriate value for the “index.mapping.total_fields.limit” parameter in Elasticsearch version 7.15.2.

Process:

1. Initial Query and Analysis: The client sought advice on determining the appropriate value for the index.mapping.total_fields.limit Elasticsearch parameter based on their data characteristics.
2. Script Execution: An expert provided scripts (both in Bash and Python) to analyze the maximum and minimum number of fields in the client’s indexes.
3. Output Analysis: The scripts provided the client with information on the minimum and maximum number of fields in their indexes. In this case, the maximum number of fields was found to be 981.
4. Parameter Recommendation: Based on the observed maximum number of fields, the expert suggested setting the index.mapping.total_fields.limit parameter slightly higher, at 1100. This recommendation allows for some flexibility while not excessively increasing the limit beyond what is necessary.
5. Continuous Monitoring and Adjustment: The client was advised to continuously monitor the number of fields in their indexes and adjust the total_fields.limit parameter accordingly. This process could be automated by scheduling the provided scripts to run at regular intervals.
6. Client Request for Meeting: The client expressed interest in arranging a meeting with the expert to address questions more efficiently.

Solution:

Upon analysis of the customer’s Elasticsearch environment, including the maximum and minimum number of fields and nested fields in their collections, it was determined that the index “actimize_truistactone_index” had a minimum of 981 fields and a maximum of 981 fields. Based on this data, it was recommended to set the value of “index.mapping.total_fields.limit” slightly higher than the observed maximum, such as 1100. However, it’s important to note that this recommendation is not a one-size-fits-all solution, and adjustments should be made based on ongoing monitoring and analysis.

The process involved the creation and execution of shell and Python scripts to determine the field counts in the indexes. Continuous monitoring of field counts and performance metrics is crucial for making informed decisions about adjusting the parameter value.

Conclusion:

The collaboration between the client and the expert resulted in a deeper understanding of Elasticsearch tuning, specifically regarding the “index.mapping.total_fields.limit” parameter. The client was provided with recommendations on how to determine an appropriate value based on their specific data model, indexing patterns, and resource availability. Additionally, standard operating procedures (SOP) were outlined for continuous monitoring and adjustment of the parameter value as needed.