Parsing rules seems not applied to logs

Support: Telemetry Data Platform (TDP) Log Management
#1

Hi,
I’m trying to parse logs from postgres. I configured newrelic infra and lags are properly pushed to newrelic but log parsing rules seems not working at all. I created parsing rule, set proper parsing regex (it is working in grok debugger):

%{DATESTAMP:timestamp} %{TZ} \[%{DATA:user_id}\](?:\s%{WORD}@%{WORD})? %{WORD:level}:%{GREEDYDATA:message}

which is working in “Parsed log messages” preview (all attributes are parsed and visible in preview):

Parsing results
Message

2020-11-24 09:51:51.660 UTC [3472558] LOG:  listening on IPv4 address "127.0.0.1", port 5432
Parsing results

{
  "level": "LOG",
  "message": "  listening on IPv4 address \"127.0.0.1\", port 5432",
  "DATE_EU": "20-11-24",
  "user_id": "3472558",
  "timestamp": "20-11-24 09:51:51.660"
}

But newly received logs are not parsed (attributes are not visible in log table).

Also noticed that predefined logtype=nginx-error is not working when upstream is used in nginx.
message: 2020/11/24 08:38:36 [error] 3532#3532: *53025 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 00.00.00.000, server: some.example.net, request: "GET /.git HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.3-fpm.sock", host: "some.example.net", referrer: "https://www.some.example.net/product" is not parsed but when i remove upstream: "fastcgi://unix:/var/run/php/php7.3-fpm.sock", part from log then parsing is working correctly.

Should I do some additional steps to make it work?

1 Like
#2

@stat Sorry you have been waiting awhile for a response from our community. I’m going to bring this back to the attention of our support team. Thanks for your patience!

Neal Mc

#3

Hi stat!

Our engineering team found that the service that publishes the parsing rules lost connection with our database which caused the issue you saw. They have re-established the connection and it should be working as expected.

If you are still experiencing unexpected behavior, please try refreshing affected pages and let me know what you are continuing to see.

1 Like
#4

I believe I am also experiencing this issue. Can you confirm it is still working?

#5

actually it appears to be working, it takes a while and only works on new incoming logs, not old ones. Sorry for the false alarm

1 Like
#6

Thanks for letting us know @pboyd!

#7

I appear to be having the same problem. I started with a built-in ruleset and parsing worked fine (I was able to see the parsed columns thru “Add Attribute”). Then I went to Manage Parsing and added a custom parser. The “Test Grok” worked great so I enabled the rule. But now I’m not seeing any of the parsed columns when I try to “Add Attribute”.

Any thoughts?

#8

Hi @pete14, maybe @hzurek could take a look into this for you?

1 Like