Feature Idea: Vert.x 3.6 and Kotlin coroutines support

Hi everyone,
Unfortunately, and this was confirmed by the New Relic support team, the Java agent doesn’t currently support Vert.x 3.6 and, based on past experience, the basic support for Kotlin coroutines is also quite bad.

Does anyone know of any workaround for still creating and tracking requests between multiple handlers? In particular when using https://vertx.io/docs/vertx-web-api-contract/java/?


New Relic Edit

  • I want this too
  • I have more info to share (reply below)
  • I have a solution for this

0 voters

We take feature ideas seriously and our product managers review every one when plotting their roadmaps. However, there is no guarantee this feature will be implemented. This post ensures the idea is put on the table and discussed though. So please vote and share your extra details with our team.

Hey @guido.mariotti Thanks for posting here in the Feature Ideas space after working with our support team.

I’ve gone ahead adding a poll to your post so that others can vote on this feature idea too.

Hi, while I am hoping for an official support for Kotlin Coroutines, is there any idea on how we would implement custom instrumentation for coroutine code? We noticed our coroutines migration make our traces disappear (previously we’re using ScheduledExecutor).

Would the Java Async API be helpful here?
https://docs.newrelic.com/docs/agents/java-agent/async-instrumentation/java-agent-api-asynchronous-applications

Unfortunately, the only solution I can think of is to link the token after you create a coroutine and after each suspending method call. However, this increase the verbosity of the entire codebase without any real benefit. It would be better if the agent could be able to link a transaction/token to the continuation associated to each suspending method without the need of any custom instrumentation.

Thanks @guido.mariotti - You’re right when you say: It would be better if the agent could be able to link a transaction/token to the continuation associated to each suspending method without the need of any custom instrumentation.

The feature request we filed for you will go to the product management team, so that is in front of the right people. Hopefully this will be something they can work on soon.

I’ve been playing with integrating the New Relic agent API with Kotlin coroutines and ktor at https://bitbucket-------.org/marshallpierce/ktor-new-relic/src/master/ (intentionally broken domain – the forum wasn’t allowing me to post a link for some reason).

So far, the project is in a “mostly working” state. There’s a demo you can run (with your own license key and agent jar) to see how transactions show up in the web UI.

  • It can track ktor requests, mostly:
    • Transaction names are set from the matched route
    • Response elapsed time is correct
    • If a segment is used in another coroutine, that does get tracked correctly
      • Multiple segments get summed, however, even if the coroutines are running in parallel, which is misleading: waiting for the longer of 700 and 500ms does not take 1200ms.
    • 404s end up attached to /NettyDispatcher, and have a transaction trace. Oddly, normal (non 404) requests do not end up with any traces.
    • Crossing thread boundaries leads to the agent not being able to figure anything out, even when using segments or tokens
    • If blocking i/o is done in the request coroutine context directly (an antipattern), it can at least allocate all of the response time to code (instead of having mostly untracked time), but it can’t figure out that the i/o is a JDBC call

The transaction (and segment and/or token, if applicable) are exposed via coroutine context, so if you wanted to get distributed tracing payloads in some child coroutine, etc, that is easily doable.

Questions:

  • Why are segments that are started/ended in a different thread from the transaction not being tracked at all? The javadoc makes it sound like that’s valid:

    […] a Segment’s timed duration may encompass arbitrary application work; it is not limited to a single method or thread.

  • Also, the Segment docs say:

    A Segment will show up in the Transaction Breakdown table, as well as the Transaction Trace page in APM.

    That doesn’t work, in fact – even when segments are being tracked correctly, they aren’t showing up in the Transaction Trace section of the APM UI.

  • Why don’t tokens seem to do anything, at least how I’m using them? See https://bitbucket-------.org/marshallpierce/ktor-new-relic/src/73111f00bcf75f5a8ece84c26892f84dd6d4bc78/coroutine-new-relic/src/main/kotlin/org/mpierce/kotlin/coroutine/newrelic/NewRelicCoroutineContext.kt?at=master#lines-62. I’m creating a token and immediately calling link() inside a function with @Trace(async = true) on it, and yet that appears to have no effect, in that I can’t see anything different when I use or don’t use a token.

Anyway, probably some simple mistakes I’m making, so please let me know if I’m using the API wrong, etc. Feedback welcome!

P.S. In case improving the agent API is on the roadmap, reading the docs hasn’t proven terribly illuminating. An especially curious example is Trace.async’s “Tells the agent that this method is asynchronous” which doesn’t really mean anything. I’m sure the author of those words had thoughts about what “this method is asynchronous” means, but it’s not exactly a common phrase, and arguably self-contradictory: how can a method run at a different time than … when it’s running? A project overall can be structured in an asynchronous way (whether via offloading work to other threads, or nonblocking i/o, or …), but a method, not so much. Loosely, it could mean “it returns a Future/CompletableFuture”, or “it starts a thread”, or “it should be run in another thread”, or maybe something else?

2 Likes

Hi @marshall_nr - This is great!

I’m on our Java support team here at New Relic and definitely appreciate the effort it’d take to put this together. That being said…I am certainly nowhere near an expert with Kotlin so the information I provide here will be more generic and hopefully you can translate that into what you need.

First a quick “paradigm” review of the Java Agent: A Transaction can be thought of as our highest-level data structure and typically represents a full code-path such as: Receiving/acting on an incoming message, an incoming web request that’s followed by a response, etc. However, a majority of the data in a Transaction comes from the Segments(eg: method calls) that are attached to it. When a Segment is created on any thread it will automagically get attached to that thread’s Transaction, if one exists. A Transaction initially only exists on the thread in which it’s created but the context of that Transaction can be passed to other threads by using the Token.link() method.

A Token is basically just an object with some information about the Transaction that created it. When you use Token.link() on a new Thread it lets the Agent know to assign any Segment that gets created on that thread to the Token’s originating Transaction. Segments created outside of the scope of a Transaction will always end up getting ignored/not reported.

Now to your questions:

Why are segments that are started/ended in a different thread from the transaction not being tracked at all?

I mostly covered this above but we initially limit a Transaction’s scope to the thread that created it to allow for multiple Transactions to occur at the same time. If the Transaction scope was global we’d only be able to have one at a time without running into issues finding the right Transaction to assign a segment to. But, this is where Token.link() comes in handy.

Also, the Segment docs say: “A Segment will show up in the Transaction Breakdown table, as well as the Transaction Trace page in APM.”

If you happen to have a permalink to your test application I may be able to give more precise information here but I’m guessing this is just confusion on the verbiage being used. The Transaction Trace section of the UI will show data for slower Transactions and the Segments are expected to be shown inside of those Transaction Traces after clicking on one. The Segments won’t have their own entries in the Transaction Trace table.

Why don’t tokens seem to do anything, at least how I’m using them? See https://bitbucket-------.org/marshallpierce/ktor-new-relic/src/73111f00bcf75f5a8ece84c26892f84dd6d4bc78/coroutine-new-relic/src/main/kotlin/org/mpierce/kotlin/coroutine/newrelic/NewRelicCoroutineContext.kt?at=master#lines-62. I’m creating a token and immediately calling link() inside a function with @Trace(async = true) on it, and yet that appears to have no effect, in that I can’t see anything different when I use or don’t use a token.

This is where my lack of Kotlin knowledge is going to show. I’m not positive what’s happening in this chunk of code but basically the Token should be created on the same thread where the Transaction was created(or on a thread that has already been linked using a Token) and Token.link() should be called in the new thread. It sounds like you’re calling link() in the new thread so my guess is that the issue is coming from how/when the Token is being created, is it possible you’re attempting to create the Token on a thread that doesn’t have an existing Transaction?

And finally, you’re right that our verbiage could be improved on that docs page regarding @Trace(async=true). The best interpretation here would be: “Tells the Agent that this method is being executed on a different thread than where the Transaction began.” The async=true portion is only required for the first method being traced on a new thread. After that, any new segments created on the linked thread will automatically know which Transaction they belong to since that thread has already been linked to a Transaction.

It’s great to see somebody digging in to the Agent at this level so let me know if there’s anything I can clarify from the above or if you run into any other issues! Also, if anybody with better knowledge about Kotlin sees this and wants to chime in with more code-specific answers please feel free!

3 Likes

Hi Matt,

Thanks for taking a look, and for the helpful comments!

Let me see if I have the right mental model by explaining the implementation approach that comes to mind when reading your paradigm section:

  • Transactions are created either by Agent#getTransaction or by the agent automatically when it finds known bytecode patterns for common libraries. They exist perhaps in some in-memory collection managed by the agent until they’re shipped off to NR servers.
  • Tokens are threadlocals. When creating a transaction, that by default assigns a token to the current thread, but tokens can be set on other threads as well. When code is instrumented (because the agent has rewritten bytecode in a @Trace(async) method, for instance), the instrumentation looks for the current threadlocal token value, if any, and uses that to determine what transaction to assign the measurements to.
  • Segments are maybe created automatically (or maybe the automagic stuff isn’t a segment?) when the agent recognizes known bytecode patterns, like JDBC calls, or created by hand with Transaction#startSegment().
    withContext(anotherThreadDispatcher) {
        withNewRelicSegment("anotherThread") {
            delay(500)
        }
    }

If you happen to have a permalink to your test application I may be able to give more precise information here but I’m guessing this is just confusion on the verbiage being used.

https://rpm.newrelic.com/accounts/2371032/applications/298014167

However, I think the issue is explained by segments in other threads not being mapped to the originating transaction. (I think it’s a bit unintuitive API wise that I can certainly pass a Transaction object to another thread and use it and yet it is effectively a no-op, while Tokens have basically the same usage pattern and do have an effect – why even make it possible to use segments that way? Why have the Transaction object have a startSegment() method at all, if really the ability to create segments is tied to a thread, not a Transaction object?)

Anyway, let’s unpack the Kotlin code a little bit. The coroutine token code is saying: grab the current transaction (not from the agent but from the coroutine context, which will propagate across threads) and use that Transaction object in another thread to create a token.

The equivalent with plain old threads would be (in Java 11, probably – I didn’t try to compile this but hopefully the intent is clear):

// on thread T1
var transaction = NewRelic.getAgent().getTransaction();

// normally you should use an executor, not start threads directly, but this is just
// an example
new Thread(new Runnable() {
    @Override
    @Timed(async=true)
    void run() {
        // on a different thread T2 here, but referring to the same transaction object
        var token = transaction.getToken();
        try {
            token.link();
            
            // business logic here
            // ...
        } finally {
            token.expire();
        }
    }
}).start();

Now, for the Kotlin:

suspend fun <T> withNewRelicToken(block: suspend () -> T): T {
    // 1
    val context = coroutineContext[NewRelicTransaction]
        ?.let { NewRelicToken(it.txn.token) }
    return if (context != null) {
        // 2
        withContext(context) {
            try {
                // 3
                runInAsyncTrace {
                    // 4 
                    context.token.link()
                    // 5
                    block()
                }
            } finally {
                // 6
                context.token.expire()
            }
        }
    } else {
        // 7
        block()
    }
}


@Trace(async = true)
private suspend fun <T> runInAsyncTrace(block: suspend () -> T): T = block()

  1. If the coroutine context has a transaction, get it, and use it to create a token, and a coroutine context element to hold that token
  2. Add the token context element to the coroutine context for the scope of the provided block
  3. Call a function annotated with @Trace(async=true)
  4. Call link
  5. Run the block of code provided originally to withNewRelicToken
  6. Expire the token
  7. If no transaction was available, just run the block – this makes it possible to run the same code without having any new relic bits available

(The short version of Kotlin coroutines: when a suspending point in the code – like delay() in the demo service – is hit, the compiler effectively does the continuation-passing-style transform to wrap the remaining code in a callback and schedules that callback for execution later.)

Using that code to achieve the same effect as the above Java:

withContext(anotherThreadDispatcher) {
    // now we're running on a different thread, but the coroutine context has carried over
    withNewRelicToken {
        // token has been linked
        slowQuery()
    }
}

the Token should be created on the same thread where the Transaction was created(or on a thread that has already been linked using a Token) and Token.link() should be called in the new thread

To test that, I tried a modified version:

suspend fun <T> withNewRelicToken(dispatcher: CoroutineDispatcher, block: suspend () -> T): T {
    // create token and put it in a context element
    val context = coroutineContext[NewRelicTransaction]?.let { NewRelicToken(it.txn.token) }
    return if (context != null) {
        // switch to another thread (via including the dispatcher in the context)
        withContext(context + dispatcher) {
            try {
                runInAsyncTrace {
                    // link
                    context.token.link()
                    block()
                }
            } finally {
                context.token.expire()
            }
        }
    } else {
        withContext(dispatcher) {
            block()
        }
    }
}

The equivalent Java:

// on thread T1
var transaction = NewRelic.getAgent().getTransaction();

// same thread
var token = transaction.getToken();

new Thread(new Runnable() {
    @Override
    @Timed(async=true)
    void run() {
        try {
            // linked in T2
            token.link();
            
            // business logic here
            // ...
        } finally {
            token.expire();
        }
    }
}).start();

And that does work (work done in other threads is attached to the txn), so that’s good!

So, to confirm: it is necessary to call transaction.getToken() on thread T1, even though the transaction on T1 can be legally (by Java rules) passed to T2? In other words, is it the case that I can access a Transaction object across threads, but it has no effect when I do so, and the only thing that can move across threads and be useful is a Token?

If that’s the case, that’s unintuitive to me (which could be an issue with my mental model above). The docs don’t give that impression, and it again calls into question what the point of the Transaction object is if its methods are that “special”: one would hope that calling getTransaction() would do all the necessary thread local lookups, and once you have a transaction it’s just a plain old object that you can treat like any other Java object. If I have a ThreadLocal and get the Foo object out of it for my current thread, I can use that Foo on other threads if I want to without fear that the Foo will secretly revert to a no-op.

Anyway, API questions notwithstanding, thanks so much for taking the time! Hopefully I’ve also clarified the Kotlin bits. My remaining questions:

  • Why is time spent in JDBC not being recognized as such? I figured the agent should automagically figure that out.
  • Why is the time taken for two segments running parallel summed? https://bitbucket------------.org/marshallpierce/ktor-new-relic/src/2042c1252e340cf61546076a1409047faa268e5e/demo/src/main/kotlin/org/mpierce/ktor/newrelic/KtorDemoMain.kt?at=master#lines-66 waits for 700ms but NR shows it as taking 1200 (500 + 700). (What is with not letting me post links to bitbucket?)
  • Why does the 404 endpoint have a “transaction trace” in NR (at the bottom right of the per transaction view) and nothing else does?

Hi Marshall,

You’re very welcome! For the most part you’ve got everything down correctly there are just a couple of misunderstandings I want to clarify.

Transactions are created either by Agent#getTransaction or by the agent automatically when it finds known bytecode patterns for common libraries…

Agent#getTransaction shouldn’t actually be creating Transactions, just returning the current, existing Transaction if one exists. The only way to create a new Transaction in code would be to use the @Trace(dispatcher=true) annotation. Of course, our out-of-the-box instrumentation also starts Transactions at certain points such as incoming HTTP Requests. You’re right that Transactions are basically in-memory collections with some additional data.

We do have threadlocals but I’m not exactly sure where they exist. One important point is that we also have a helper class called TransactionActivity for Transactions and I believe it’s the existence of this class that is the reason why you can’t pass just the Transaction between threads with full functionality. This is more of a behind-the-scenes worker class so we don’t call it out in the docs to avoid confusion since the other API calls automatically interact with it as needed.

Segments do get created automatically when our Agent recognizes code that we’ve set it to instrument. We have a lot of out-of-box instrumentation but there are only specific points where we also create a new Transaction. This is important because a Segment needs to have a parent Transaction in order to be reported. For instance, we will instrument a JDBC call as a Segment but if that call wasn’t triggered by something like an incoming HTTP request(which would create a Transaction) then there will likely not be a current Transaction to store the segment data under so it gets dropped. It sounds like you’re right about why your PostgreSQL calls aren’t getting reported if they don’t have any Transaction to report under.

Interestingly I don’t see too many cases of folks using Transaction#startSegment() since the @Trace() annotation will tell the Agent to trace the annotated method as a segment without any need to call Segment#end(). There’s nothing wrong with startSegment() I think it’s just an ease-of-use reason why most people use the @Trace() annotation.

Your callout regarding the bytecode is spot-on also; our out-of-the-box instrumentation does not just look at method signatures but also verifies that the bytecode within those methods is expected. This is so that, if a new version of a framework is rolled out that changes a method which we automatically instrument, the modified method won’t get instrumented until our Engineers have been able to test against it, verify there are no issues, and let the Agent know it’s acceptable.

I can’t speak too much to the “why” of these API calls being the way they are but I know a lot of work has been done around optimization as well as some necessary fiddling to handle the wide variety of environments available with Java. That being said, I’m absolutely going to share the feedback with our Engineering team. They’re always wanting to improve usability in this area.

I am glad to hear you were able to get the multi-threaded work reporting correctly after adjusting that code though! You’ve also intrigued me with the quick kotlin code rundown, I’m sure I’ll end up spending some time learning a bit more about that.

I tried to address your various points in a somewhat chronological order based on your post but let me know if I missed anything important. For your final questions:

  • JDBC time tracking should happen automatically, especially if you’re seeing a Transaction around the JDBC call. My only suspicion would be that a Transaction doesn’t exist when the JDBC call is made. You could try annotating the calling method with @Trace(dispatcher=true) to guarantee there’s a Transaction and if the JDBC timing starts showing up you’ll know that was the issue. I checked some backend metrics for the APM app you linked and I do see our postgresql instrumentation was loaded so the Agent should be attempting to instrument those calls.
  • For the timing question about parallel segment times being added together, this sounds like the difference between our reported "Response Time" and “Total Time”. The charts will typically show the Total Time as a summed result of all of the segments so that you can see the overall effect of the Transaction in regards to total processing time. We should also be showing a “Response Time” for that same Transaction in the form of a blue bar. This is the direct time between the start and end of a Transaction without taking into account any asynchronous work. This document provides a little more information on that.
  • Transaction Traces are generally collected based on response time. If a Transaction takes too much longer than your configured Apdex value then the Agent will save a full Transaction Trace for that specific Transaction. However, you can prioritize other Transactions for Traces by making them Key Transactions

Let us know what else we can help with!

I think a totally reasonable outcome re: the cross-thread usability of a Transaction object would be to simply state in the class-level docs that a Transaction object may only be used in the same thread in which it was acquired via Agent#getTransaction(), and that the only cross-thread interaction that is supported is to get a Token (in the same thread) and then link() the token (in another thread). It’s a perfectly reasonable constraint as long as you know it’s there. :slight_smile: I’m guessing the underlying reason is that it allows the per-txn collections of measurements to use non-thread-safe (i.e. faster) data structures.

In Java, I wonder if people aren’t as accustomed to using lambdas (which, having no name or corresponding syntactic structure, can’t be annotated AFAIK) for trace-able work, especially since lambdas can’t throw checked exceptions like IOException, limiting their usefulness for your typical service call. Perhaps in Kotlin segments might be more useful since all exceptions are unchecked and therefore lambdas are perfectly reasonable to use to express i/o work, especially if people use my library. :wink:

Anyway, idle musings aside, now we’re down to just the JDBC measurement issue. It’s quite possible that I’m still misinterpreting these charts (I’d missed the Response Time line, for instance…), but see the attached screenshots for the slowQuery endpoint. The code:

// a query that takes 100ms
fun slowQuery() {
    dsFuture.get().connection.use { conn ->
        conn.prepareStatement("SELECT pg_sleep(0.1);").use { stmt ->
            stmt.execute()
        }
        conn.commit()
    }
}

// ...

get("/slowQuery") {
    // time allocated to callInTxn wrapper, not JDBC
    slowQuery()
    call.respond(HttpStatusCode.OK, "done")
}

Pretty straightforward JDBC executing in the same thread, and yet as you can see, all the time is recorded for the callInTxn (which has @Trace(dispatch = true)) that’s part of the ktor integration, not the JDBC call. It seems like that must be in a transaction already because that’s in the Transactions view…


Sorry for the late reply here! That sounds like a great recommendation for our docs, I’m going to run it (and the rest of this conversation) past our Engineers and Docs team so they’re aware.

As far as that JDBC call goes, I don’t see anything obvious that would prevent our Agent from reporting the JDBC timing. It also sounds like your expectations are correct in that we’d call out the ~100ms as reported in JDBC, not in callInTxn.

I’m running through our instrumentation and it looks like we do provide instrumentation on the execute() method but one thing I should verify…is the stmt object a java.sql.PreparedStatement? The code makes me expect that to be the case but I wanted to double-check that.

Cheers, Matt

Correct, stmt is that type.

If you feel that motivated and want to poke around in your IDE and see all the types, etc, just clone https://bitbucket.org/marshallpierce/kotlin-new-relic/src (git clone https://bitbucket.org/marshallpierce/kotlin-new-relic.git) and open it in IntelliJ or your Kotlin-capable IDE of choice. See the ktor-demo subproject. Or, check its README if you want to actually run the demo to see how it behaves in your new relic account.

1 Like

Hi @RyanVeitch, @MrMatt any update on native New Relic coroutines support? Spring Framework 5.2 is now officially supporting them and I guess more and more people will start using them in their services

5 Likes

Hi @marshall_nr. Thanks for the great insights here. I have been using your code for a little while in KTor now and it seems to be working fine.

I do have some slow requests though that I am starting to launch async with withContext(Dispatchers.IO) { … } . I am getting a token from the Transaction before, and then calling token.link() inside withContext .

But I am not getting the JDBC info etc. that I was getting before adding withContext . Any idea? Any pointers is highly appreciated. :slight_smile:

Never mind. I found the withNewRelicToken wrapper and now it works again. :slight_smile:

1 Like

Glad you were able to find that solution @anders2 :smiley: Thanks for following up to confirm.

Hi, I have been trying to correctly track transactions in a spark + kotlin server. I Used the idea in the @marshall_nr code, but found that transaction segments where beeing lost. I realised that this happens because a single coroutine may be excecuted in several threads, and “withNewRelicToken” only links the firs one. When a coroutine is suspended it may resume in another thread. For this I implemented a continuationInterceptor:

    class NewRelicTokenHolder : AbstractCoroutineContextElement(NewRelicTokenHolder) {
    companion object Key : CoroutineContext.Key<NewRelicTokenHolder>

    private val token: Token = NewRelic.getAgent().transaction.token

    fun link(): Boolean {
        if (!token.isActive) {
            RuntimeException("[${Date().toInstant()}] [${Thread.currentThread().name}] [${NewRelic.getAgent().transaction.tracedMethod.metricName}] Tried to link inactive NR Token").printStackTrace()
        }
        return token.link()
    }

    fun expire() {
        if (!token.isActive) {
            RuntimeException("[${Date().toInstant()}] [${Thread.currentThread().name}] [${NewRelic.getAgent().transaction.tracedMethod.metricName}] Tried to expire inactive NR Token").printStackTrace()
        }
        token.expire()
    }
}

class NewRelicContinuationInterceptor(
    private val dispatcher: ContinuationInterceptor = Dispatchers.Default
) : AbstractCoroutineContextElement(ContinuationInterceptor), ContinuationInterceptor {
    override fun <T> interceptContinuation(continuation: Continuation<T>): Continuation<T> =
        dispatcher.interceptContinuation(NewRelicWrapper(continuation))
}

class NewRelicWrapper<T>(private val continuation: Continuation<T>) : Continuation<T> {

    override val context: CoroutineContext get() = continuation.context
    val token: NewRelicTokenHolder? = context[NewRelicTokenHolder]

    @Trace(async = true)
    private fun linkContinuation(block: () -> Unit) {
        token?.link()
        block()
    }

    override fun resumeWith(result: Result<T>) = linkContinuation {
        continuation.resumeWith(result)
    }
}

With this code I was able to track te complete transaction, I know this because I am reporting external services done in a transaction, and I found that in many cases there was no transaction when reporting the external service. After implementing this code, all external services found the transaction!

I did found a problem with my implementation, I had to issue a new token each time I linked a thread to the transaction, but I do not know a way of knowing when the coroutine is finalized, so there are some tokens that are issued, but never linked… Can this cause any problem?

Thanks,
Tomás

Hi Tomas,
Could you give us an example on how to use this implementation?
Thanks!

1 Like

Hi Pablo! sure!

Notice that I updated the code to the latest version I am using, I modified it so that does not use that many tokens…

val newRelicToken = NewRelicTokenHolder()
val dispatcher = context[ContinuationInterceptor] ?: Dispatchers.Default
val newRelicContext = newRelicToken + NewRelicContinuationInterceptor(dispatcher)

try {
    withContext(context + newRelicContext, block)
} finally {
    newRelicToken.expire()
}

This would be if you already are inside a coroutine…

In case u need to do some async work:

val newRelicToken = NewRelicTokenHolder()
val dispatcher = context[ContinuationInterceptor] ?: this.coroutineContext[ContinuationInterceptor] ?: Dispatchers.Default
val newRelicContext = newRelicToken + NewRelicContinuationInterceptor(dispatcher)

val deferredJob = this.async(context + newRelicContext, start, block)
deferredJob.invokeOnCompletion {
    newRelicToken.expire()
}
2 Likes