This is the recording of yesterday’s webinar From Symptom to Root Cause: Tracking Issues with io.Insights Traces.
In this session, Velko Nikolov, lead for io.Insights, shows how traces help you follow what happened across connected applications and find where a delay or error started. The examples cover platform, application and Workspace startup, unresponsive web apps, slow context updates and a business workflow that fails across several applications.
Jump to a section
Choose a section below to play it in the video above.
Understanding and reading traces
- 02:12 - Traces, metrics and logs: what each tells you - What a trace records, how spans connect operations, and how to read waterfall and time-sequence views.
- 08:06 - What io.Insights traces out of the box - OpenTelemetry, propagation between applications, API instrumentation, web requests and user actions.
- 09:56 - One Click, Four Processes - Follow a click through an interop call, a backend request and a shared context update to see who reacted to what and where the time went.
- 12:34 - Using traces and logs together - Move from a span to the relevant log entries, or from a log entry back to its trace.
- 13:48 - Diagnostics overview and measuring application readiness - The examples ahead, startup versus runtime performance, and why the application itself needs to indicate when it is ready for use.
Performance and error investigation
- 15:18 - Why is the platform slow to start? - Compare a slow startup with a healthy baseline, then use the critical path to investigate a delay in starting the gateway.
- 20:32 - Why is an individual application slow to start? - Separate platform initialization from page loading and identify a JavaScript function that holds up startup.
- 23:29 - Why is a Workspace slow to open? - Follow frame creation, frame initialization and application startup to locate a slow script download.
- 26:37 - Why is a web application unresponsive? - Investigate long animation frames, delayed user inputs and a grid-rendering function that repeatedly blocks the application.
- 29:30 - Why does a button click take so long? - Follow a user action to a large context update and use the recorded payload information to explain the delay.
- 30:59 - Where did a workflow fail across applications? - Trace a call-handling workflow through client identification, Workspace opening and verification, then find why the verification intent was unavailable.
Extending the trace and Q&A
- 35:15 - Adding your own business context to traces - Add custom spans and attributes, connect business operations to the automatically recorded API calls, and make user actions easier to identify.
- 38:26 - Recap
- 39:13 - Audience Q&A - Data volume, personal information and performance overhead.
Questions and answers
The questions and Velko’s answers below have been edited for readability from the live Q&A.
39:42 - How much data does tracing publish?
Question: With so much metadata being captured, particularly in a large platform with many applications, does tracing produce a lot of data?
Answer: Velko described a typical user publishing the default traces as generating tens of megabytes per day. The volume scales with the number of operations being traced, so a system performing a very large number of operations will produce more data.
Bandwidth is something to assess during a pilot. If it becomes a concern, traces can also be written to local files alongside the application logs and retrieved when needed, avoiding the need to continuously send that tracing data over the network.
41:02 - Can tracing expose personally identifiable information (PII)?
Question: Is there a risk of exposing personal information through tracing, including the information captured out of the box?
Answer: Velko said the risk is low, but it is not zero. The default instrumentation aims to be conservative about what it publishes, recording identifiers such as context and Workspace names rather than business data such as client names, telephone numbers or email addresses.
If you choose to publish more information, those decisions need to be reviewed. The same care you apply to error messages and log files also applies to telemetry: check what is being recorded and avoid including sensitive information. Velko also mentioned work on tools to help filter information captured inadvertently, while stressing that teams still need to check their own data.
Follow-up: If we add custom tracing or metrics, could we capture something we did not intend to expose?
Answer: Yes. Custom instrumentation can include whatever data the implementation adds, just as a log message can. The same discipline around reviewing and sanitizing logs applies to traces.
43:18 - What is the performance overhead of tracing?
Question: What effect do capturing and sending tracing data have on application or platform performance?
Answer: Velko estimated the runtime overhead at a couple of percent for typical tracing. Tracing a very large amount of custom data, or an operation that runs thousands of times per second, can increase the cost, particularly when publishing the data.
For tracing events associated with normal user actions, he said the delay should not be noticeable. These were estimates discussed in the session; the amount and frequency of data being recorded matter.