https://dchua.com/tags/sysadmin/ Recent content in sysadmin on davidchua Hugo -- gohugo.io en-us Mon, 06 Mar 2017 00:00:00 +0000 https://dchua.com/posts/2017-03-06-forwarding-vault-audit-logs-to-a-remote-syslog-server/ Mon, 06 Mar 2017 00:00:00 +0000 https://dchua.com/posts/2017-03-06-forwarding-vault-audit-logs-to-a-remote-syslog-server/ Using Vault’s Audit Backend to send logs to a remote Syslog server. Objective Send audit_logs from Hashicorp’s Vault to an Graylog instance Prerequisite Setup a Syslog TCP/UDP Input on Graylog (if you’re using graylog) Has a remote syslog server running Steps Important Notice: Vault has a Syslog Audit Backend as part of its suite but it currently does not allow remote forwarding. In order to do that we will have to make use of rsyslog’s rules forwarding. https://dchua.com/posts/2017-02-28-alerting-with-prometheus-and-alertmanager/ Tue, 28 Feb 2017 00:00:00 +0000 https://dchua.com/posts/2017-02-28-alerting-with-prometheus-and-alertmanager/ How to setup Prometheus AlertManager and get a whole alerting pipeline setup. Objectives and Goals Write and Deploy Prometheus Alert Rules Configure Prometheus to send Alerts to Alert Manager Setup AlertManager to receive Prometheus Alert Send a Slack message on Alert Prerequisite Prometheus is already setup and running Alert Rules To begin writing and deploying alerts, you’ll need to modify your prometheus config file. Usually, its located at /etc/prometheus/prometheus. https://dchua.com/posts/2017-02-03-deleting-files-and-getting-your-diskspace-back-without-rebooting/ Fri, 03 Feb 2017 00:00:00 +0000 https://dchua.com/posts/2017-02-03-deleting-files-and-getting-your-diskspace-back-without-rebooting/ When you run out of diskspaces and need to delete files quickly to recover them, most of the time, your deleted files will not free up the diskspace until the process that is using it is restarted or deleted. To force the filesystem to free up the lock to the file so that the files can be cleared up, you’d need to find the process that is using the file and truncate it. https://dchua.com/posts/2015-12-25-using-graylog-extractors/ Fri, 25 Dec 2015 00:00:00 +0000 https://dchua.com/posts/2015-12-25-using-graylog-extractors/ If you’re new to Graylog, Graylog Extractors are a great way to pull out information from your logs for easier storing and manipulation. If like me, you sometimes experience a “String fields longer than 32kb” indexing error on one of your fields, a good way to help mitigate it is to use extractors to split your field into two. I wouldn’t recommend this all the time as obviously the way to solve this is to set your particular field to be non_indexable but in certain cases when you need the full data and still want to be able to search for it, this might be the better solution. https://dchua.com/posts/2014-09-17-figure-out-which-dns-server-you-are-using/ Wed, 17 Sep 2014 00:00:00 +0000 https://dchua.com/posts/2014-09-17-figure-out-which-dns-server-you-are-using/ If you need to find out what DNS you are currently using (for troubleshooting purposes, this snippet might help): $ tcpdump udp and src port 53 $ tcpdump udp and dst port 53 $ tcpdump -n -s 1500 -i eth0 udp port 53 These 3 commands will sniff your packets for port 53 interactions and will let you know what is the ip address of the DNS server that you’re interacting with.