[es] Hot Threads API

Elasticsearch의 hot_threads API는 클러스터 내에서 CPU가 높게 사용되는 스레드 정보를 제공하는 엔드포인트입니다. 이를 통해 어떤 작업이 CPU를 많이 사용하고 있는지를 식별하고 성능 이슈를 해결할 수 있음

 

 

Hot Threads API 사용 방법:

cURL을 사용한 기본 사용법

curl -X GET "localhost:9200/_nodes/hot_threads?pretty"

위 명령은 로컬 호스트의 9200번 포트에서 실행 중인 Elasticsearch에 연결하고, 각 노드의 hot threads 정보를 출력.

 

 

위 명령은 각 노드에서 상위 5개의 hot threads만 출력합니다. threads 파라미터를 사용하여 출력할 스레드의 수를 조절할 수 있음

curl -X GET "localhost:9200/_nodes/hot_threads?threads=5&pretty"

 

함 해보자

GET _nodes/node-prd-totalsearch-01-data/hot_threads?pretty
GET _nodes/node-prd-totalsearch-02-data/hot_threads?pretty
GET _nodes/node-prd-totalsearch-03-data/hot_threads?pretty
GET _nodes/node-prd-totalsearch-04-data/hot_threads?pretty
GET _nodes/node-prd-totalsearch-05-data/hot_threads?pretty
GET _nodes/node-prd-totalsearch-06-data/hot_threads?pretty

 

 

Hot Threads API 출력 예시:

Hot Threads API의 출력은 다음과 비슷한 형태를 가질 수 있습니다:

 

::: {node_name}{node_id} 
   Hot threads at 2022-01-01T00:00:00Z, ...

각 노드의 hot threads는 해당 노드의 이름과 ID로 시작합니다. 그 후, 각 스레드에 대한 정보가 표시됩니다.

 

 

   1.0% (4ms out of 500ms) cpu usage by thread 'elasticsearch[...][[search][[example-index][2]:]'
     9/10 snapshots sharing following 4 elements
       app//org.apache.lucene.util.packed.MonotonicBlockPackedReader.fillBulk(MonotonicBlockPackedReader.java:77)
       app//org.apache.lucene.util.packed.AbstractBlockPackedReader.readValues(AbstractBlockPackedReader.java:266)
       app//org.apache.lucene.search.DocIdSetBuilder$BulkBuffer.add(DocIdSetBuilder.java:235)
       app//org.apache.lucene.search.DocIdSetBuilder$Bulk.add(DocIdSetBuilder.java:276)
   0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[...][[search][[example-index][0]:]'
     7/10 snapshots sharing following 5 elements
       app//org.apache.lucene.util.packed.MonotonicBlockPackedReader.fillBulk(MonotonicBlockPackedReader.java:77)
       app//org.apache.lucene.util.packed.AbstractBlockPackedReader.readValues(AbstractBlockPackedReader.java:266)
       app//org.apache.lucene.search.DocIdSetBuilder$BulkBuffer.add(DocIdSetBuilder.java:235)
       app//org.apache.lucene.search.DocIdSetBuilder$Bulk.add(DocIdSetBuilder.java:276)
       app//org.elasticsearch.search.aggregations.bucket.terms.PrefixFieldMapper$PrefixFieldData.build(FixedBitSet.java:813)

 

출력 내용의 해석:

• 각 스레드의 CPU 사용량 및 작업 정보가 표시됩니다.
• cpu usage by thread 부분에서는 각 스레드의 CPU 사용량이 얼마나 되는지가 나타납니다.
• Thread의 이름이나 식별자, 해당 스레드가 어떤 작업을 수행 중인지 등의 정보가 표시됩니다.
• 각 스레드의 스택 트레이스를 통해 스레드가 어떤 메서드에서 시간을 소비하는지 확인할 수 있습니다.

Hot Threads API를 사용하면 Elasticsearch 클러스터에서 가장 CPU가 많이 사용되는 스레드를 식별하고, 성능 이슈를 파악하는 데 도움이 됩니다.

 

 

   87.0% (434.8ms out of 500ms) cpu usage by thread 'elasticsearch[node-prd-totalsearch-04-data][write][T#6]'
     4/10 snapshots sharing following 27 elements
       org.apache.lucene.analysis.ko.UnitWordFilter.incrementToken(UnitWordFilter.java:87)
       app//org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
       app//org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:888)
       app//org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:501)
       app//org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:465)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1333)
       app//org.apache.lucene.index.IndexWriter.softUpdateDocument(IndexWriter.java:1661)
       app//org.elasticsearch.index.engine.InternalEngine.updateDocs(InternalEngine.java:1260)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1091)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)

 

 

 

81.7% (408.3ms out of 500ms) cpu usage by thread 'elasticsearch[node-prd-totalsearch-04-data][write][T#7]'
     2/10 snapshots sharing following 38 elements
       java.base@11.0.8/java.util.regex.Pattern$GroupHead.match(Pattern.java:4804)
       java.base@11.0.8/java.util.regex.Pattern$Branch.match(Pattern.java:4749)
       java.base@11.0.8/java.util.regex.Pattern$Start.match(Pattern.java:3619)
       java.base@11.0.8/java.util.regex.Matcher.search(Matcher.java:1729)
       java.base@11.0.8/java.util.regex.Matcher.find(Matcher.java:746)
       org.apache.lucene.analysis.ko.morph.MorphAnalyzer.tokenize(MorphAnalyzer.java:142)
       org.apache.lucene.analysis.ko.morph.MorphAnalyzer.analyze(MorphAnalyzer.java:205)
       org.apache.lucene.analysis.ko.morph.MorphAnalyzer.analyze(MorphAnalyzer.java:188)
       org.apache.lucene.analysis.ko.KoreanFilter.analysisKorean(KoreanFilter.java:197)
       org.apache.lucene.analysis.ko.KoreanFilter.incrementToken(KoreanFilter.java:119)
       org.apache.lucene.analysis.ko.UnitWordFilter.incrementToken(UnitWordFilter.java:87)
       app//org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
       app//org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:888)
       app//org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:501)
       app//org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:465)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1333)
       app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
       app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1262)
       app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1158)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1095)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)
     2/10 snapshots sharing following 32 elements
       org.apache.lucene.analysis.ko.morph.MorphAnalyzer.analyze(MorphAnalyzer.java:205)
       org.apache.lucene.analysis.ko.morph.MorphAnalyzer.analyze(MorphAnalyzer.java:188)
       org.apache.lucene.analysis.ko.KoreanFilter.analysisKorean(KoreanFilter.java:197)
       org.apache.lucene.analysis.ko.KoreanFilter.incrementToken(KoreanFilter.java:119)
       org.apache.lucene.analysis.ko.UnitWordFilter.incrementToken(UnitWordFilter.java:87)
       app//org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
       app//org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:888)
       app//org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:501)
       app//org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:465)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1333)
       app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
       app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1262)
       app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1158)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1095)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)
     2/10 snapshots sharing following 23 elements
       app//org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:240)
       app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:485)
       app//org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:615)
       app//org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:424)
       app//org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:392)
       app//org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:109)
       app//org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:68)
       app//org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:266)
       app//org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:799)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:776)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)
     2/10 snapshots sharing following 34 elements
       java.base@11.0.8/java.util.regex.Pattern.<init>(Pattern.java:1428)
       java.base@11.0.8/java.util.regex.Pattern.compile(Pattern.java:1068)
       java.base@11.0.8/java.util.regex.Pattern.matches(Pattern.java:1173)
       java.base@11.0.8/java.lang.String.matches(String.java:2024)
       org.apache.lucene.analysis.ko.UnitWordFilter.is_number(UnitWordFilter.java:64)
       org.apache.lucene.analysis.ko.UnitWordFilter.capture(UnitWordFilter.java:40)
       org.apache.lucene.analysis.ko.UnitWordFilter.incrementToken(UnitWordFilter.java:114)
       app//org.apache.lucene.analysis.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:49)
       app//org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:888)
       app//org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:501)
       app//org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:465)
       app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:233)
       app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:419)
       app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1333)
       app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1619)
       app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1262)
       app//org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1158)
       app//org.elasticsearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1095)
       app//org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:935)
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)
     2/10 snapshots sharing following 15 elements
       app//org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:819)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:791)
       app//org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:748)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:274)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:164)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:209)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:116)
       app//org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:82)
       app//org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:161)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@11.0.8/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@11.0.8/java.lang.Thread.run(Thread.java:834)