The company that I recently joined uses honeycomb.io, versus ELK at my previous job. Maybe I haven't played with it enough, or maybe we have it badly configured, but I find it a huge step backwards. As far as I know, there's no full text search and the types of reports and aggregation you can do are extremely primitive. The UI is also barely usable..huge horizontal scrolling, can't click on a specific value to apply that sub-filter. The list goes on and on. I wish devops team would focus on these fundamentals (proper centralized logging) before playing with all the "cool" and trendy toys.
As for logging, Our `log_format` directive looks like:
log_format stats '$time_iso8601\t$host\t$request_method\t$request_uri\t$request_length\t$status\t$upstream_cache_status\t$bytes_sent\t$request_time\t$upstream_response_time\t$upstream_http_x_route\t$client_id';
This captures most of the generic data you'd associate with a request (time, host, method, uri, length) as well as the generic response data you'd care about (status, length, time to reply). Note that `$request_time` measures the full time to first byte until time to last byte that nginx spends serving the request, vs $upstream_response_time which is more specific to the upstream's response time.
If you're using nginx as a cache, $upstream_cache_status tells you the cache hit/miss status.
A list of all variables is available at http://nginx.org/en/docs/varindex.html
All of our services can set an "x-route" header, which helps canonicalize URIs into something more meaningful. It's up the the service to decide what to do..but /v1/users/ID could be called "users:show". A more complex route might use a different name based on arguments. For example, depending on the parameters we expect some hits to /v1/reports to be very fast, and some to be very slow, so we'll set the name to "reports:list:fast" or "reports:list:slow" so that we can get more accurate statistics.
Finally, we use OpenRestry to do some initial global authentication (along with some other stuff). This is where $client_id comes from. All requests go through something like:
location / {
set $client_id ''
set $upstream ''
access_by_lua_block {
require('execute')()
}
proxy_pass http://$upstream;
}
A piece of that execute code will possibly set $client_id to something which will then get logged.
As for logging, Our `log_format` directive looks like:
This captures most of the generic data you'd associate with a request (time, host, method, uri, length) as well as the generic response data you'd care about (status, length, time to reply). Note that `$request_time` measures the full time to first byte until time to last byte that nginx spends serving the request, vs $upstream_response_time which is more specific to the upstream's response time.If you're using nginx as a cache, $upstream_cache_status tells you the cache hit/miss status.
A list of all variables is available at http://nginx.org/en/docs/varindex.html
All of our services can set an "x-route" header, which helps canonicalize URIs into something more meaningful. It's up the the service to decide what to do..but /v1/users/ID could be called "users:show". A more complex route might use a different name based on arguments. For example, depending on the parameters we expect some hits to /v1/reports to be very fast, and some to be very slow, so we'll set the name to "reports:list:fast" or "reports:list:slow" so that we can get more accurate statistics.
Finally, we use OpenRestry to do some initial global authentication (along with some other stuff). This is where $client_id comes from. All requests go through something like:
location / { set $client_id '' set $upstream '' access_by_lua_block { require('execute')() } proxy_pass http://$upstream; }
A piece of that execute code will possibly set $client_id to something which will then get logged.