{"id":138,"date":"2015-04-26T17:46:17","date_gmt":"2015-04-26T16:46:17","guid":{"rendered":"https:\/\/zorq.net\/b\/?p=138"},"modified":"2015-04-26T17:46:17","modified_gmt":"2015-04-26T16:46:17","slug":"getting-started-with-graylog-on-macos","status":"publish","type":"post","link":"https:\/\/zorq.net\/b\/2015\/04\/26\/getting-started-with-graylog-on-macos\/","title":{"rendered":"Getting Started with Graylog on MacOS"},"content":{"rendered":"

Graylog<\/a> is an amazing open source tool for recording, viewing and analysing application logs. The performance and full text indexing of the underlying elasticsearch database<\/a> means you probably won’t need any other logging tools throughout your whole organisation.<\/p>\n

Application developers writing to Graylog may need a local instance to retrieve diagnostic information during development. This post gives some quick instructions for setting up a local logging server using docker.<\/p>\n

Docker<\/a> is used for managing lightweight operating system containers with isolated process and network namespaces – each container can have a process of PID 10 using TCP port 80. Unfortunately, Mac OS X’s kernel doesn’t contain the necessary APIs to run a Linux-based container and so we need to use a Virtual Machine running Linux itself. Boot2docker<\/a>, installed via Homebrew, comes to our rescue to make the whole process as simple as possible:<\/p>\n

\r\nbrew install Caskroom\/cask\/virtualbox\r\nbrew install Caskroom\/cask\/boot2docker\r\nbrew install docker\r\n<\/pre>\n
Now all the tools are installed we need to download and start up the Linux VM:<\/p>\n
\r\nboot2docker download\r\nboot2docker init\r\nboot2docker start\r\n<\/pre>\n
Another complexity of the docker host being in a virtual machine is that the docker command line interface needs to connect across a virtual network interface. Boot2docker helps us here by telling us the required settings. I put the ‘export’ lines in my ~\/.profile<\/tt> so that the docker<\/tt> command will work without any special setup in the future.<\/p>\n
\r\nboot2docker shellinit\r\n# outputs:\r\nWriting \/Users\/djb\/.boot2docker\/certs\/boot2docker-vm\/ca.pem\r\nWriting \/Users\/djb\/.boot2docker\/certs\/boot2docker-vm\/cert.pem\r\nWriting \/Users\/djb\/.boot2docker\/certs\/boot2docker-vm\/key.pem\r\n    export DOCKER_HOST=tcp:\/\/192.168.59.103:2376\r\n    export DOCKER_CERT_PATH=\/Users\/djb\/.boot2docker\/certs\/boot2docker-vm\r\n    export DOCKER_TLS_VERIFY=1\r\n<\/pre>\n
To test your new docker installation you can run the hello-world<\/tt> container:<\/p>\n
\r\ndocker run hello-world\r\n<\/pre>\n
Assuming that went okay we can now get started with Graylog! Run the following command to download the image for containers we’ll create. This may take some time as there are a few hundred MB of Linux distribution and Java libraries to fetch.<\/p>\n
\r\ndocker pull graylog2\/allinone\r\n<\/pre>\n
You can now create a new container. The numeric arguments are the TCP ports that will be exposed by the container.<\/p>\n
\r\ndocker run -t -p 9000:9000 -p 12201:12201 graylog2\/allinone\r\n<\/pre>\n
Once the flurry of activity in your console comes to a stop the container is fully up and running. You can visit the admin page by visiting http:\/\/ip-of-server:9000<\/tt>. The IP address is that of your boot2docker virtual machine, which you found earlier using boot2docker shellinit<\/tt>. The default username and password to the web interface are admin<\/tt> and admin<\/tt>.<\/p>\n
Once you’re inside the web interface Graylog will warn you that there are no inputs running. To fix this, browse to http:\/\/ip-of-server:9000\/system\/inputs<\/tt> and add an input of type GELF HTTP<\/tt> running on port 12201<\/tt>. The name of the input isn’t important unless you have plans for hundreds of them so I’ve unimaginatively called mine GELF HTTP<\/tt>.<\/p>\n
Now we’ve got a running server configured enough for testing, but no logs! A real application will produce logs of its own but for the purpose of demonstration we can write a script to read in your local computer’s system log and send the messages to Graylog for indexing. Note that you’d never do this in production as Graylog is perfectly capable of using the syslog UDP protocol, which would avoid the need to write any code.<\/p>\n
\r\n#!\/usr\/bin\/env python\r\n\r\nimport urllib2\r\nimport json\r\n\r\npath = \"\/var\/log\/system.log\"\r\nwith open(path) as log:\r\n    log = f.readlines()\r\n\r\nfor line in f:\r\n    message = {}\r\n\r\n    message['version'] = '1.1'\r\n    message['host'] = 'localhost'\r\n    message['short_message'] = line\r\n\r\n    req = urllib2.Request('http:\/\/192.168.59.103:12201\/gelf')\r\n    req.add_header('Content-Type', 'application\/json')\r\n    response = urllib2.urlopen(req, json.dumps(message))\r\n<\/pre>\n
Once your logs are in the server you can start searching them. Try querying for kernel<\/tt> to find all the messages logged from the Mac OS X kernel. Having a full text index of our logs is useful but belies the promise of elasticsearch’s structured storage. A more useful implementation would log application-specific information as fields in the GELF message. For example, with the following code a single click a complete history of logs for a particular client could be retrieved:<\/p>\n
\r\nmessage['client_ip'] = ipFromApplicationVariable\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"
Graylog is an amazing open source tool for recording, viewing and analysing application logs. The performance and full text indexing of the underlying elasticsearch database means you probably won’t need any other logging tools throughout your whole organisation. Application developers writing to Graylog may need a local instance to retrieve diagnostic information during development. This […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17],"tags":[20,19],"class_list":["post-138","post","type-post","status-publish","format-standard","hentry","category-macos","tag-devops","tag-logging"],"_links":{"self":[{"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/posts\/138","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/comments?post=138"}],"version-history":[{"count":18,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/posts\/138\/revisions"}],"predecessor-version":[{"id":156,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/posts\/138\/revisions\/156"}],"wp:attachment":[{"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/media?parent=138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/categories?post=138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zorq.net\/b\/wp-json\/wp\/v2\/tags?post=138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}