Missing Dimensions from Custom Events

I am using the Event API to send custom events for POC as described here. I’ve formatted my events accordingly with an event type, timestamp and custom attributes in camel case, and I’ve managed to get them POSTed to the Insights API. I can find the events in the ‘Browse data’ section under my account, however the different attributes/dimensions in the events are not available to query, even though I can see they are registered correctly in the ‘Raw data’ tab (screenshots attached)

What can I do to have the attributes recognised so that I can, for example, slice the chart by ‘Process code’?


Hi, @davidcereijo: This is odd, I would expect your event’s attributes to be available as dimensions. Will you please post the code you are using to create the events (replacing your insert key with xxxx)?

Hello @philweber,

Sure. Here is the code and a sample of the raw events that I use as input

Code

import requests
import json
import gzip
from datetime import datetime
from inflection import camelize

ACCOUNT_ID = 'XXXX'
KEY = 'XXXX'

FILENAME = 'events.json'

lines = []
with open(FILENAME) as events_file:
  for line in events_file.readlines():
    lines.append(
      json.loads(line)
    )

events = []
for line in lines:
  event = {}
  for key, value in line.items():
    event[camelize(key, uppercase_first_letter=False)] = value
  event['eventType'] = 'LandingCompletedV4'
  ts = datetime.fromisoformat(line['audit_insert_datetime'])
  event['timestamp'] = int(ts.timestamp())
  events.append(event)
  print(event)

payload = json.dumps(events)
compressed = gzip.compress(payload.encode('utf-8'))

response = requests.post(
  headers={
    'X-Insert-Key': KEY,
    'Content-Encoding': 'gzip',
    'Content-Type': 'application/json'
  },
  data=compressed,
  url=f"https://insights-collector.eu01.nr-data.net/v1/accounts/{ACCOUNT_ID}/events"
)

print(response)
print(response.reason)

Sample events (events.json)

{"audit_id":17457584, "audit_insert_datetime":"2021-05-15 00:03:05", "process_id":248, "process_code":"LPDAU", "process_schedule":"2021-05-14 05:30:00", "process_run_id":"scheduled__2021-05-14T05:30:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:03:05"}
{"audit_id":17457916, "audit_insert_datetime":"2021-05-15 00:04:17", "process_id":250, "process_code":"LPDGB", "process_schedule":"2021-05-14 22:50:00", "process_run_id":"scheduled__2021-05-14T22:50:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:04:17"}
{"audit_id":17459182, "audit_insert_datetime":"2021-05-15 00:12:19", "process_id":323, "process_code":"LA", "process_schedule":"2021-05-14 23:00:00", "process_run_id":"scheduled__2021-05-14T23:00:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:12:19"}
{"audit_id":17459190, "audit_insert_datetime":"2021-05-15 00:12:22", "process_id":439, "process_code":"LA_QA", "process_schedule":"2021-05-14 23:00:00", "process_run_id":"scheduled__2021-05-14T23:00:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:12:22"}
{"audit_id":17460617, "audit_insert_datetime":"2021-05-15 00:20:30", "process_id":248, "process_code":"LPDAU", "process_schedule":"2021-05-14 06:30:00", "process_run_id":"scheduled__2021-05-14T06:30:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:20:30"}
{"audit_id":17462266, "audit_insert_datetime":"2021-05-15 00:30:33", "process_id":58, "process_code":"LPFLGB", "process_schedule":"2021-05-14 05:00:00", "process_run_id":"scheduled__2021-05-14T05:00:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 00:30:33"}
{"audit_id":17469700, "audit_insert_datetime":"2021-05-15 01:04:44", "process_id":250, "process_code":"LPDGB", "process_schedule":"2021-05-14 23:50:00", "process_run_id":"scheduled__2021-05-14T23:50:00+00:00", "event_type":"COMPLETED", "event_update_datetime":"2021-05-15 01:04:44"}

Hi, @davidcereijo: I finally had a chance to look into this; I apologize for the delay. Unfortunately, I am not able to reproduce the behavior you are seeing; when I run your code to insert events into my account, I see the attributes as available dimensions in the Data Explorer:

I did make a few small changes to your code:

  • I changed the eventType to MyCustomEvent
  • My account is in the US region, so I changed the URL to https://insights-collector.newrelic.com/v1/accounts/{ACCOUNT_ID}/events
  • I commented out the timestamp assignment code, since the API only allows custom timestamps within the past 24 hours, and yours are from several days ago

Can you take another look at your data, and see if you are still seeing this behavior?

Hey @philweber, thanks for checking.

I just did a couple of tests more I believe I know what’s causing this.

I did two tests:

  • On the first one I removed timestamp assignment and submitted the events without this field. The dimensions where correctly recognised.
  • On the second test (I’ve changed the event name to have a clean test), I did include the timestamp assignment. In this case the dimensions where not recognised as expected and I got the message that “This event doesn’t have any dimensions…” in the UI

So, I am not 100% sure, but I’m going to guess that timestamp assignment is somehow messing with dimension recognition.

We have the same issue. Was working fine, now we get no dimensions, although the data we are sending has the data (and has not changed).