はじめに
くどうです
HDInsightをモニタリングしたい場合、Ambari UIを利用すると非常に便利です。
しかし、KafkaについてはAmbari UIでモニタリング出来る項目が少なく別の方法を検討する必要があります。
その一つとして、Jolokiaを利用した方法を紹介します。
Jolokiaとは
Jolokia
https://jolokia.org/index.html
JolokiaはJMXの情報を収集し、HTTP REST APIとして返答できるAgentです。
Agentは一つではなく下記が用意されています。
-
1,WAR Agent for deployment as web application in a JEE Server.
2,OSGi Agent for deployment in an OSGi container. This agent is packaged as a bundle and comes in two flavors (minimal, all-in-one).
3,JVM Agent which can be used with any Oracle/Sun based JVM, Version 6 or later.
4,Mule Agent for usage within a Mule ESB.
それぞれのAgentの詳細は下記で確認できます。
https://jolokia.org/agent.html
KafkaではJVM Agentを利用し、REST APIにてモニタリングします。
構成
今回の構成図
Hdinsight はクラスターの種類はStorm、OSはLinux、バージョンはStorm 0.10.0(HDI 3.4)です。
ログインしてKafkaを確認
-
hdinsight-kafka 2.4
kafka-2-4-2-0-258 0.9.0.2.4.2.0-258
Hdinsight とモニタ用仮想マシン（Linux）は同Vnetの必要があります。
図中の、StormについてあStorm UIを利用して、REST APIが利用できます。
KafkaについてはJolokiaを利用します。
Jolokiaを設定
最初にHDInsight側にJolokiaを設置する必要があります。
AmbariUIを利用し、Kafkaが動作しているIPアドレスを確認します。
Summaryを見るとIPを確認できます。
まず、Hdinsight にSSHでログインします。
その後、Summaryで確認したIPへログインします。
|
1 2 |
# wget http://search.maven.org/remotecontent?filepath=org/jolokia/jolokia-jvm/1.3.3/jolokia-jvm-1.3.3-agent.jar -O jolokia-jvm-1.3.3-agent.jar # mv /root/jolokia-jvm-1.3.3-agent.jar /usr/sbin/ |
以上で設置は完了です、
次に、KAFKA_OPTSにてjavaagent動作させます。
直接、Configを書き換えでも大丈夫ですがAmbari UIから設定してみましょう。
Advanced kafka-envを開きます。
kafka-env templateに下記を追記します。
追記する場所は
|
1 |
export KAFKA_KERBEROS_PARAMS={{kafka_kerberos_params}} |
の直下
|
1 |
export KAFKA_OPTS=-javaagent:/usr/sbin/jolokia-jvm-1.3.3-agent.jar=host=0.0.0.0 |
host=0.0.0.0はすべてのIPアドレスからREST APIのGETリクエストを受け付けることを意味します。
Saveします。
Saveします。
Restartします。
以上でHdinsight 側に準備ができました。
REST API
実際にモニタ用仮想マシンから確認してみましょう。
モニタ用仮想マシンにログインします。
事前にjqをインストールします。
デフォルトポートは8778です
確認できるメトリクスを確認する場合のsearch書式は
http://[サーバーIP]:8778/jolokia/search/[メトリクス]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 |
[root@kafkamonitoring ~]# curl -s http://10.1.0.13:8778/jolokia/search/kafka.*:* | jq . { "value": [ "kafka.server:delayedOperation=Fetch,name=NumDelayedOperations,type=DelayedOperationPurgatory", "kafka.server:delayedOperation=Produce,name=PurgatorySize,type=DelayedOperationPurgatory", "kafka.network:name=RequestsPerSec,request=Heartbeat,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=StopReplica,type=RequestMetrics", "kafka.server:name=UnderReplicatedPartitions,type=ReplicaManager", "kafka.network:name=RemoteTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=StopReplica,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=ListGroups,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.server:name=TotalProduceRequestsPerSec,type=BrokerTopicMetrics", "kafka.network:name=ResponseSendTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=RequestQueueSize,type=RequestChannel", "kafka.network:name=LocalTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.log:name=max-buffer-utilization-percent,type=LogCleaner", "kafka.network:name=RequestsPerSec,request=StopReplica,type=RequestMetrics", "kafka.server:broker-id=1001,type=controller-channel-metrics", "kafka.network:name=RequestQueueTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=Offsets,type=RequestMetrics", "kafka.network:name=NetworkProcessorAvgIdlePercent,type=SocketServer", "kafka.controller:name=OfflinePartitionsCount,type=KafkaController", "kafka.network:name=ResponseQueueTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=6,type=Processor", "kafka.network:name=ResponseSendTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=0,type=Processor", "kafka.network:name=RequestQueueTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=7,type=Processor", "kafka.network:name=TotalTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.controller:name=UncleanLeaderElectionsPerSec,type=ControllerStats", "kafka.network:name=TotalTimeMs,request=StopReplica,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.server:networkProcessor=6,type=socket-server-metrics", "kafka.network:name=TotalTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.server:type=Produce", "kafka.network:name=ResponseQueueSize,processor=0,type=RequestChannel", "kafka.network:name=ResponseQueueTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.log:name=max-clean-time-secs,type=LogCleaner", "kafka.network:name=ThrottleTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=7,type=RequestChannel", "kafka.network:name=IdlePercent,networkProcessor=2,type=Processor", "kafka.network:name=ResponseQueueTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=StopReplica,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=3,type=RequestChannel", "kafka.server:name=PartitionCount,type=ReplicaManager", "kafka.network:name=RequestsPerSec,request=OffsetCommit,type=RequestMetrics", "kafka.server:networkProcessor=3,type=socket-server-metrics", "kafka.server:name=BrokerState,type=KafkaServer", "kafka.server:delayedOperation=Heartbeat,name=PurgatorySize,type=DelayedOperationPurgatory", "kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool", "kafka.network:name=RequestQueueTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=2,type=RequestChannel", "kafka.network:name=RemoteTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=StopReplica,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics", "kafka.network:name=LocalTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=ListGroups,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=OffsetFetch,type=RequestMetrics", "kafka.controller:name=PreferredReplicaImbalanceCount,type=KafkaController", "kafka.network:name=RequestsPerSec,request=LeaveGroup,type=RequestMetrics", "kafka.server:name=IsrShrinksPerSec,type=ReplicaManager", "kafka.network:name=TotalTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=StopReplica,type=RequestMetrics", "kafka.server:networkProcessor=1,type=socket-server-metrics", "kafka.server:networkProcessor=5,type=socket-server-metrics", "kafka.network:name=TotalTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.server:name=FailedProduceRequestsPerSec,type=BrokerTopicMetrics", "kafka.network:name=ResponseQueueTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=Produce,type=RequestMetrics", "kafka.server:name=IsrExpandsPerSec,type=ReplicaManager", "kafka.network:name=IdlePercent,networkProcessor=1,type=Processor", "kafka.network:name=RemoteTimeMs,request=ListGroups,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.server:delayedOperation=Produce,name=NumDelayedOperations,type=DelayedOperationPurgatory", "kafka.network:name=RequestsPerSec,request=Produce,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=4,type=Processor", "kafka.network:name=RemoteTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=5,type=Processor", "kafka.network:name=ThrottleTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=SyncGroup,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=5,type=RequestChannel", "kafka.network:name=RequestsPerSec,request=JoinGroup,type=RequestMetrics", "kafka.server:delayedOperation=Rebalance,name=PurgatorySize,type=DelayedOperationPurgatory", "kafka.network:name=RemoteTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=FetchFollower,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=ListGroups,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=4,type=RequestChannel", "kafka.server:networkProcessor=4,type=socket-server-metrics", "kafka.network:name=RemoteTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.server:delayedOperation=Fetch,name=PurgatorySize,type=DelayedOperationPurgatory", "kafka.network:name=RemoteTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=IdlePercent,networkProcessor=3,type=Processor", "kafka.network:name=ResponseQueueTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=Fetch,type=RequestMetrics", "kafka.server:name=BytesRejectedPerSec,type=BrokerTopicMetrics", "kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics", "kafka.network:name=TotalTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=Fetch,type=RequestMetrics", "kafka.server:networkProcessor=7,type=socket-server-metrics", "kafka.server:name=LeaderCount,type=ReplicaManager", "kafka.network:name=ThrottleTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=GroupCoordinator,type=RequestMetrics", "kafka.server:type=Fetch", "kafka.network:name=TotalTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=StopReplica,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=ListGroups,type=RequestMetrics", "kafka.server:delayedOperation=Heartbeat,name=NumDelayedOperations,type=DelayedOperationPurgatory", "kafka.network:name=LocalTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=ListGroups,type=RequestMetrics", "kafka.server:clientId=Replica,name=MaxLag,type=ReplicaFetcherManager", "kafka.network:name=LocalTimeMs,request=JoinGroup,type=RequestMetrics", "kafka.server:networkProcessor=0,type=socket-server-metrics", "kafka.network:name=TotalTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.coordinator:name=NumGroups,type=GroupMetadataManager", "kafka.network:name=RequestQueueTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,type=RequestChannel", "kafka.network:name=TotalTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.server:name=FailedFetchRequestsPerSec,type=BrokerTopicMetrics", "kafka.log:name=max-dirty-percent,type=LogCleanerManager", "kafka.network:name=RemoteTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=LeaveGroup,type=RequestMetrics", "kafka.server:delayedOperation=Rebalance,name=NumDelayedOperations,type=DelayedOperationPurgatory", "kafka.network:name=ResponseSendTimeMs,request=Metadata,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=Offsets,type=RequestMetrics", "kafka.utils:name=cleaner-io,type=Throttler", "kafka.network:name=ResponseSendTimeMs,request=Heartbeat,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.network:name=ResponseQueueTimeMs,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=Metadata,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=Metadata,type=RequestMetrics", "kafka.server:networkProcessor=2,type=socket-server-metrics", "kafka.network:name=RequestQueueTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=UpdateMetadata,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=Fetch,type=RequestMetrics", "kafka.network:name=ResponseSendTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.network:name=LocalTimeMs,request=OffsetFetch,type=RequestMetrics", "kafka.controller:name=ActiveControllerCount,type=KafkaController", "kafka.network:name=LocalTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.server:name=BytesOutPerSec,type=BrokerTopicMetrics", "kafka.network:name=ResponseQueueTimeMs,request=ListGroups,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.controller:name=LeaderElectionRateAndTimeMs,type=ControllerStats", "kafka.network:name=ResponseQueueSize,processor=6,type=RequestChannel", "kafka.network:name=ResponseQueueTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.log:name=cleaner-recopy-percent,type=LogCleaner", "kafka.network:name=LocalTimeMs,request=Produce,type=RequestMetrics", "kafka.network:name=ResponseQueueSize,processor=1,type=RequestChannel", "kafka.server:clientId=Replica,name=MinFetchRate,type=ReplicaFetcherManager", "kafka.network:name=ResponseSendTimeMs,request=SyncGroup,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=GroupCoordinator,type=RequestMetrics", "kafka.server:name=TotalFetchRequestsPerSec,type=BrokerTopicMetrics", "kafka.network:name=RemoteTimeMs,request=FetchFollower,type=RequestMetrics", "kafka.network:name=ThrottleTimeMs,request=FetchConsumer,type=RequestMetrics", "kafka.coordinator:name=NumOffsets,type=GroupMetadataManager", "kafka.server:id=1001,type=app-info", "kafka.network:name=RemoteTimeMs,request=Offsets,type=RequestMetrics", "kafka.network:name=RequestQueueTimeMs,request=OffsetCommit,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=LeaderAndIsr,type=RequestMetrics", "kafka.network:name=RemoteTimeMs,request=DescribeGroups,type=RequestMetrics", "kafka.network:name=RequestsPerSec,request=ControlledShutdown,type=RequestMetrics", "kafka.network:name=TotalTimeMs,request=ListGroups,type=RequestMetrics" ], "request": { "type": "search", "mbean": "kafka.*:*" }, "status": 200, "timestamp": 1465974055 } |
メトリクスを確認する場合のread書式は
http://[サーバーIP]:8778/jolokia/read/[メトリクス]
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
[root@kafkamonitoring ~]# curl -s http://10.1.0.13:8778/jolokia/read/kafka.server:networkProcessor=4,type=socket-server-metrics | jq . { "value": { "response-rate": 0, "outgoing-byte-rate": 0, "io-time-ns-avg": 4206.140350877193, "io-wait-time-ns-avg": 300293983.64912283, "incoming-byte-rate": 0, "connection-count": 0, "io-wait-ratio": 1.0025923015375604, "io-ratio": 1.4043051691316445e-05, "request-rate": 0, "request-size-avg": 0, "request-size-max": null, "select-rate": 3.338702591887538, "connection-creation-rate": 0, "connection-close-rate": 0, "network-io-rate": 0 }, "request": { "type": "read", "mbean": "kafka.server:networkProcessor=4,type=socket-server-metrics" }, "status": 200, "timestamp": 1465980199 } |
すべてを確認する場合は
curl -s http://10.1.0.13:8778/jolokia/read/kafka.*:* | jq .
にて確認できます。
これらをCronなどで定期的に取集することなども行えます。jqやawkなど利用してログに残すこともできます。
まとめ
Hdinsight をモニタリングする場合、ポータルで出来るのが理想ですが、さすがに無理なので今回はjolokiaを利用したモニタリングの方法を紹介しました。
これはHdinsight やkafkaに限った方法ではありません。JMXを利用してモニタリングを行いたい場合は非常に便利です。
ではでは
この記事へのコメントはありません。