# CloudWatch logs shell utilities

This package provides two simple, but useful utilities: 

* cwlogs-cat - display CloudWatch logs from single Log Group
* cwlogs-tail - display / follow logs tail

Both support simple filtering on server-side (by CloudWatch), therefore speeding up work with big log files in some scenarios. Still standard, shell utilities can be used to do final work, without need to fetch complete logs.

Moreover, multiple log streams can be combined into single call, and entries from all log streams are returned interleaved, in logging order. Convenient regexp matching of log streams have been implemented too.

### Installation

Package can be installed from LCloud PyPI repository:

    python3 -m pip install -i https://pypi.lcloud.pl/ aws-cloudwatch-logs-utils

### Usage

```
$ cwlogs-cat [-h] [--startTime STARTTIME] [--endTime ENDTIME] [--debug DEBUG] [--logStreamNamePrefix LOGSTREAMNAMEPREFIX] [--logStreamNames LOGSTREAMNAMES] [--logStreamNames_regexp LOGSTREAMNAMES_REGEXP]
                [--filterPattern FILTERPATTERN] [-p] [-t]
                logGroupName

    positional arguments:
    logGroupName          CloudWatch logs group name

    options:
    -h, --help            show this help message and exit
    --startTime STARTTIME
                            Start time in format "YYYY-MM-DDTHH:MM:SS+HHMM" or timestamp
    --endTime ENDTIME     End time time in format "YYYY-MM-DDTHH:MM:SS+HHMM" or timestamp
    --debug DEBUG         Debug level: from 1 to 10
    --logStreamNamePrefix LOGSTREAMNAMEPREFIX
                            Filter logs by the LogStream name prefix
    --logStreamNames LOGSTREAMNAMES
                            Filter logs by the LogStream names, comma-separated
    --logStreamNames_regexp LOGSTREAMNAMES_REGEXP
                            Filter logs by the LogStream names matching regular expression
    --filterPattern FILTERPATTERN
                            Filter logs by filter pattern
    -p, --printLogStreamNames
                            Print LogStream names
    -t, --no-timestamp    Do not print event timestamp
```

```
$ cwlogs-tail [-h] [-f] [-n TAIL] [-s TAIL_LAST_HOURS] [--debug DEBUG] [--logStreamNamePrefix LOGSTREAMNAMEPREFIX] [--logStreamNames LOGSTREAMNAMES] [--logStreamNames_regexp LOGSTREAMNAMES_REGEXP] [--filterPattern FILTERPATTERN]
                    [-p] [-t]
                    logGroupName

    positional arguments:
    logGroupName          CloudWatch logs group name

    options:
    -h, --help            show this help message and exit
    -f, --follow          Follow log tail
    -n TAIL               Log tail length
    -s TAIL_LAST_HOURS    Log tail since <s> hours
    --debug DEBUG         Debug level: from 1 to 10
    --logStreamNamePrefix LOGSTREAMNAMEPREFIX
                        Filter logs by the LogStream name prefix
    --logStreamNames LOGSTREAMNAMES
                        Filter logs by the LogStream names, comma-separated
    --logStreamNames_regexp LOGSTREAMNAMES_REGEXP
                        Filter logs by the LogStream names matching regular expression
    --filterPattern FILTERPATTERN
                        Filter logs by filter pattern
    -p, --printLogStreamNames
                        Print LogStream names
    -t, --no-timestamp    Do not print event timestamp
```

### Time format

Log time window (startTime : endTime) can be specified in ISO format:
YYYY-MM-DDTHH:MM:SS+HHMM, i.e.:

    --startTime 2018-02-19T20:25:00+0000


### FilterPattern

Syntax: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html

### Example usage scenarios:

* follow apache log, searching for 404 errors. Preliminary filtering is done with `filterPattern` (for speed), accurate filtering done with standard `grep`

    $ cwlogs-tail apacheLogGroup -f --logStreamNames=/var/log/httpd/ access_log --filterPattern ' 404 ' | grep -P 'HTTP/\d\.\d" 404 '

* get apache requests from single IP: 10.1.1.1, done during single day. Again - preliminary search is done with `filterPattern`. Then `grep` does accurate filtering (in this case filtering out 10.1.1.11, 10.1.1.12 etc) 

    $ cwlogs-cat apacheLogGroup --logStreamNames=/var/log/httpd/ access_log --filterPattern '10.1.1.1' --startTime 2018-02-19T00:00:00+0000 --endTime 2018-02-20T00:00:00+0000 | grep '^10\.1\.1\.1 '

* follow all log streams from workflow (log group "Staging"), log entries combined into single log flow:

    $ cwlogs-tail -f -p Staging --logStreamNames_regexp=^workflow-

    $ cwlogs-tail -f -p Staging --logStreamNamePrefix=workflow-


### Limitations:

`filterPattern` does not implement regular expressions, so we have to combine it with other utils (like `grep`) to achieve speed and accuracy.

Due to AWS logs filter_log_events() API limitations, getting log tail is simulated: we retrieve last 15 minutes of relevant logs and display just a few last of them. This can have unexpected results: long call duration (if log size is big), less than expected TAIL entries.
