commit a4ce90a0b8acae89ccdbd551f29007a5ec260142
parent ece4c8df3850a1c6a5578a3bf30917cc020c29aa
Author: Erik Loualiche <eloualic@umn.edu>
Date: Tue, 24 Mar 2026 21:29:30 -0500
docs: update README and logger guide for v0.11.0
Document all 6 log formats with examples, cascading_loglevels,
thread safety, and deprecation of :log4j symbol.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Diffstat:
3 files changed, 158 insertions(+), 79 deletions(-)
diff --git a/README.md b/README.md
@@ -13,13 +13,13 @@ It is a more mature version of [`Prototypes.jl`](https://github.com/louloulibs/P
The package provides:
- - [`custom_logger`](#custom-logging): configurable logging with per-level file output, module filtering, and multiple format options (`pretty`, `log4j`, `syslog`)
+ - [`custom_logger`](#custom-logging): configurable logging with per-level file output, module filtering, and multiple format options (`:pretty`, `:oneline`, `:json`, `:logfmt`, `:syslog`, `:log4j_standard`)
- ~~`read_jsonl` / `stream_jsonl` / `write_jsonl`~~: **deprecated** — use [`JSON.jl`](https://github.com/JuliaIO/JSON.jl) v1 with `jsonlines=true` instead
## Installation
-`BazerUtils.jl` is a registered package.
+`BazerUtils.jl` is a registered package.
You can install from the my julia registry [`loulouJL`](https://github.com/LouLouLibs/loulouJL) via the julia package manager:
```julia
> using Pkg, LocalRegistry
@@ -38,30 +38,64 @@ If you don't want to add a new registry, you can install it directly from github
### Custom Logging
-This one is a little niche.
-I wanted to have a custom logger that would allow me to filter messages from specific modules and redirect them to different files, which I find useful to monitor long jobs in a format that is easy to read and that I can control.
-The formatter is hard-coded to what I like but I guess I could change it easily and make it an option.
+A configurable logger that lets you filter messages from specific modules and redirect them to different files, with a format that is easy to read and control.
-Here is an example where you can create a custom logger and redirect logging to different files.
-See the doc for more [examples](https://louloulibs.github.io/BazerUtils.jl/dev/man/logger_guide)
```julia
custom_logger(
- "./log/build_stable_sample_multiplier"; # prefix of log-file being generated
- file_loggers=[:warn, :debug], # which file logger to deploy
+ "./log/build_stable_sample_multiplier";
+ file_loggers=[:warn, :debug], # which file loggers to deploy
- filtered_modules_all=[:HTTP], # filtering messages across all loggers from specific modules
- filtered_modules_specific=[:TranscodingStreams], # filtering messages for stdout and info from specific modules
+ filtered_modules_all=[:HTTP], # filter across all loggers
+ filtered_modules_specific=[:TranscodingStreams], # filter for stdout and info only
- displaysize=(50,100), # how much to show
- log_format=:log4j, # how to format the log for files
- log_format_stdout = :pretty, # how to format the log for the repl
+ displaysize=(50,100), # how much to show for non-string messages
+ log_format=:oneline, # format for files (see formats below)
+ log_format_stdout=:pretty, # format for REPL
- create_log_files=true, # if false all logs are written to a single file
- overwrite=true, # overwrite old logs
-
+ cascading_loglevels=false, # false = each file gets only its level
+ # true = each file gets its level and above
+
+ create_log_files=true, # separate file per level
+ overwrite=true,
);
```
+#### Log Formats
+
+| Format | Symbol | Description |
+|--------|--------|-------------|
+| **Pretty** | `:pretty` | Box-drawing + ANSI colors — default for stdout |
+| **Oneline** | `:oneline` | Single-line with timestamp, level, module, file:line — default for files |
+| **JSON** | `:json` | One JSON object per line — for log aggregation (ELK, Datadog, Loki) |
+| **logfmt** | `:logfmt` | `key=value` pairs — grep-friendly, popular with Splunk/Heroku |
+| **Syslog** | `:syslog` | RFC 5424 syslog format |
+| **Log4j Standard** | `:log4j_standard` | Apache Log4j PatternLayout — for Java tooling interop |
+
+Example output for each:
+
+```
+# :pretty (stdout default)
+┌ [08:28:08 2025-02-12] Info | @ Main[script.jl:42]
+└ Processing batch 5 of 10
+
+# :oneline (file default)
+[/home/user/project] 2025-02-12 08:28:08 INFO Main[./script.jl:42] Processing batch 5 of 10
+
+# :json
+{"timestamp":"2025-02-12T08:28:08","level":"INFO","module":"Main","file":"script.jl","line":42,"message":"Processing batch 5 of 10"}
+
+# :logfmt
+ts=2025-02-12T08:28:08 level=info module=Main file=script.jl line=42 msg="Processing batch 5 of 10"
+
+# :syslog
+<14>1 2025-02-12T08:28:08 hostname julia 12345 - - Processing batch 5 of 10
+
+# :log4j_standard
+2025-02-12 08:28:08,000 INFO [1] Main - Processing batch 5 of 10
+```
+
+> **Note:** `:log4j` still works as a deprecated alias for `:oneline` and will be removed in a future version.
+
### JSON Lines (deprecated)
@@ -77,7 +111,7 @@ JSON.json("out.jsonl", data; jsonlines=true) # write
## Other stuff
-See my other package
+See my other package
- [BazerData.jl](https://github.com/louloulibs/BazerData.jl) which groups together data wrangling functions.
- [FinanceRoutines.jl](https://github.com/louloulibs/FinanceRoutines.jl) which is more focused and centered on working with financial data.
- [TigerFetch.jl](https://github.com/louloulibs/TigerFetch.jl) which simplifies downloading shape files from the Census.
diff --git a/docs/src/index.md b/docs/src/index.md
@@ -4,7 +4,7 @@ Utility functions for everyday Julia.
## Features
-- **[Custom Logger](@ref Logging)**: Configurable logging with per-level file output, module filtering, and multiple format options (`pretty`, `log4j`, `syslog`).
+- **[Custom Logger](@ref Logging)**: Configurable logging with per-level file output, module filtering, thread safety, and six format options (`pretty`, `oneline`, `json`, `logfmt`, `syslog`, `log4j_standard`).
- **[JSON Lines](@ref "Working with JSON Lines Files")** *(deprecated)*: Read/write JSONL files. Use [`JSON.jl`](https://github.com/JuliaIO/JSON.jl) v1 with `jsonlines=true` instead.
## Installation
diff --git a/docs/src/man/logger_guide.md b/docs/src/man/logger_guide.md
@@ -1,16 +1,18 @@
# Logging
The function `custom_logger` is a wrapper over the `Logging.jl` and `LoggingExtras.jl` libraries.
-I made them such that I could fine tune the type of log I use repeatedly across projects.
+I made them such that I could fine tune the type of log I use repeatedly across projects.
The things I find most useful:
1. four different log files for each different level of logging from *error* to *debug*
-2. pretty (to me) formatting for stdout but also an option to have `log4j` style formatting in the files
-3. filtering out messages of verbose packages (`TranscodingStreams`, etc...) which sometimes slows down julia because of excessive logging.
+2. six output formats: `:pretty` for the REPL, `:oneline`, `:json`, `:logfmt`, `:syslog`, and `:log4j_standard` for files
+3. filtering out messages of verbose packages (`TranscodingStreams`, etc...) which sometimes slows down julia because of excessive logging
+4. exact-level filtering so each file gets only its own level (or cascading for the old behavior)
+5. thread-safe writes via per-stream locking
There are still a few things that might be useful down the line:
-(1) a catch-all log file where filters do not apply; (2) filtering out specific functions of packages;
+(1) a catch-all log file where filters do not apply; (2) filtering out specific functions of packages;
Overall this is working fine for me.
@@ -19,107 +21,150 @@ Overall this is working fine for me.
Say at the beginning of a script you would have something like:
```julia
using BazerUtils
-custom_logger("/tmp/log_test";
- filtered_modules_all=[:StatsModels, :TranscodingStreams, :Parquet2],
- create_log_files=true,
- overwrite=true,
- log_format = :log4j);
-
-┌ Info: Creating four different files for logging ...
-│ ⮑ /tmp/log_test_error.log
-│ /tmp/log_test_warn.log
-│ /tmp/log_test_info.log
-└ /tmp/log_test_debug.log
+custom_logger("/tmp/log_test";
+ filtered_modules_all=[:StatsModels, :TranscodingStreams, :Parquet2],
+ create_log_files=true,
+ overwrite=true,
+ log_format=:oneline);
+
+┌ Info: Creating 4 log files:
+│ ⮑ /tmp/log_test_error.log
+│ /tmp/log_test_warn.log
+│ /tmp/log_test_info.log
+└ /tmp/log_test_debug.log
```
The REPL will see all messages above debug level:
```julia
> @error "This is an error level message"
-┌ [08:28:08 2025-02-12] ERROR | @ Main[REPL[17]:1]
+┌ [08:28:08 2025-02-12] Error | @ Main[REPL[17]:1]
└ This is an error level message
> @warn "This is an warn level message"
-┌ [08:28:08 2025-02-12] WARN | @ Main[REPL[18]:1]
+┌ [08:28:08 2025-02-12] Warn | @ Main[REPL[18]:1]
└ This is an warn level message
> @info "This is an info level message"
-┌ [08:28:08 2025-02-12] INFO | @ Main[REPL[19]:1]
+┌ [08:28:08 2025-02-12] Info | @ Main[REPL[19]:1]
└ This is an info level message
> @debug "This is an debug level message"
```
-Then each of the respective log-levels will be redirected to the individual files and if the log4j option was specified they will look like something like this
-```log4j
-2025-02-12 08:28:08 ERROR Main[REPL[17]:1] - This is an error level message
-2025-02-12 08:28:08 WARN Main[REPL[18]:1] - This is an warn level message
-2025-02-12 08:28:08 INFO Main[REPL[19]:1] - This is an info level message
-2025-02-12 08:28:08 DEBUG Main[REPL[20]:1] - This is an debug level message
+Then each of the respective log-levels will be redirected to the individual files. With the `:oneline` format they will look like:
+```
+[/home/user] 2025-02-12 08:28:08 ERROR Main[./REPL[17]:1] This is an error level message
+[/home/user] 2025-02-12 08:28:08 WARN Main[./REPL[18]:1] This is an warn level message
+[/home/user] 2025-02-12 08:28:08 INFO Main[./REPL[19]:1] This is an info level message
+[/home/user] 2025-02-12 08:28:08 DEBUG Main[./REPL[20]:1] This is an debug level message
```
## Options
-### Formatting
+### Log Formats
-The `log_format` is `log4j` by default (only for the files).
-The only other option for now is `pretty` which uses the format I wrote for the REPL; note that it is a little cumbersome for files especially since you have to make sure your editor has the ansi interpreter on.
+The `log_format` kwarg controls how file logs are formatted. Default is `:oneline`.
+The `log_format_stdout` kwarg controls REPL output. Default is `:pretty`.
-### Files
+All formats are available for both file and stdout.
-The default is to create one file for each level.
-There is an option to only create one file for each level and keep things a little tidier in your directories:
-```julia
-> custom_logger("/tmp/log_test";
- create_log_files=false, overwrite=true, log_format = :log4j);
+| Format | Symbol | Best for |
+|--------|--------|----------|
+| Pretty | `:pretty` | Human reading in the REPL. Box-drawing characters + ANSI colors. |
+| Oneline | `:oneline` | File logs. Single line with timestamp, level, module, file:line, message. |
+| JSON | `:json` | Structured log aggregation (ELK, Datadog, Loki). One JSON object per line, zero external dependencies. |
+| logfmt | `:logfmt` | Grep-friendly structured logs. `key=value` pairs, popular with Splunk/Heroku. |
+| Syslog | `:syslog` | RFC 5424 syslog collectors. |
+| Log4j Standard | `:log4j_standard` | Java tooling interop. Actual Apache Log4j PatternLayout with thread ID and milliseconds. |
-> @error "This is an error level message"
-> @warn "This is an warn level message"
-> @info "This is an info level message"
-> @debug "This is an debug level message"
+Example:
+```julia
+# JSON logs for a data pipeline
+custom_logger("/tmp/pipeline";
+ log_format=:json,
+ create_log_files=true,
+ overwrite=true)
+
+# logfmt for grep-friendly output
+custom_logger("/tmp/pipeline";
+ log_format=:logfmt,
+ overwrite=true)
```
-And then the file `/tmp/log_test` has the following:
-```log4j
-2025-02-12 08:37:29 ERROR Main[REPL[22]:1] - This is an error level message
-2025-02-12 08:37:29 WARN Main[REPL[23]:1] - This is an warn level message
-2025-02-12 08:37:29 INFO Main[REPL[24]:1] - This is an info level message
-2025-02-12 08:37:29 DEBUG Main[REPL[25]:1] - This is an debug level message
-```
+> **Deprecation note:** `:log4j` still works as an alias for `:oneline` but emits a deprecation warning. Use `:oneline` for the single-line format or `:log4j_standard` for the actual Apache Log4j format.
-Now imagine you want to keep the same log file but for a different script.
-You can use the same logger option with the `overwrite=false` option:
-```julia
-> custom_logger("/tmp/log_test";
- create_log_files=false, overwrite=false, log_format = :log4j);
-> @error "This is an error level message from a different script and new logger"
-```
+### Level Filtering: `cascading_loglevels`
-### Filtering
+By default (`cascading_loglevels=false`), each file gets only messages at its exact level:
+- `app_error.log` — only errors
+- `app_warn.log` — only warnings
+- `app_info.log` — only info
+- `app_debug.log` — only debug
-- `filtered_modules_specific::Vector{Symbol}=nothing`: which modules do you want to filter out of logging (only for info and stdout)
- Some packages just write too much log ... filter them out but still be able to check them out in other logs
-- `filtered_modules_all::Vector{Symbol}=nothing`: which modules do you want to filter out of logging (across all logs)
- Examples could be TranscodingStreams (noticed that it writes so much to logs that it sometimes slows down I/O)
+With `cascading_loglevels=true`, each file gets its level **and everything above**:
+- `app_error.log` — only errors
+- `app_warn.log` — warnings + errors
+- `app_info.log` — info + warnings + errors
+- `app_debug.log` — everything
+```julia
+# Old cascading behavior
+custom_logger("/tmp/log_test";
+ create_log_files=true,
+ cascading_loglevels=true,
+ overwrite=true)
+```
-## Other
-For `log4j` the message string is modified to fit on one line: `\n` is replaced by ` | `.
+### Files
+
+The default is to write all levels to a single file.
+Set `create_log_files=true` to create one file per level:
-There is also a path shortener (`shorten_path_str`) that reduces file paths to a fixed size.
-The cost is that paths will no longer be clickable, but log messages will start at the same column.
+```julia
+# Single file (default)
+custom_logger("/tmp/log_test"; overwrite=true)
+
+# Separate files per level
+custom_logger("/tmp/log_test";
+ create_log_files=true, overwrite=true)
+```
+You can also select only specific levels:
+```julia
+custom_logger("/tmp/log_test";
+ create_log_files=true,
+ file_loggers=[:warn, :debug], # only warn and debug files
+ overwrite=true)
+```
+Use `overwrite=false` (the default) to append to existing log files across script runs.
+### Filtering
+- `filtered_modules_specific::Vector{Symbol}`: filter modules from stdout and info-level file logs only.
+ Some packages write too much — filter them from info but still see them in debug.
+- `filtered_modules_all::Vector{Symbol}`: filter modules from ALL logs.
+ Use for extremely verbose packages like `TranscodingStreams` that can slow down I/O.
+```julia
+custom_logger("/tmp/log_test";
+ filtered_modules_all=[:TranscodingStreams],
+ filtered_modules_specific=[:HTTP],
+ overwrite=true)
+```
+### Thread Safety
+All file writes are wrapped in per-stream `ReentrantLock`s. Multiple threads can log concurrently without interleaving output. In single-file mode, all levels share one lock. In multi-file mode, each file has its own lock so writes to different files don't block each other.
+## Other
+For single-line formats (`:oneline`, `:logfmt`, `:syslog`, `:log4j_standard`), multi-line messages are collapsed to a single line: `\n` is replaced by ` | `. The `:json` format escapes newlines as `\n` in the JSON string.
+There is also a path shortener (`shorten_path`) that reduces file paths. Options: `:relative_path` (default), `:truncate_middle`, `:truncate_to_last`, `:truncate_from_right`, `:truncate_to_unique`, `:no`.
