Collecting User-Initiated Bug Reports: What would the ideal experience look like?

[bruno-garcia]

Some problems don’t manifest in runtime errors and crashes; however, they still can degrade your user’s experience and cause user confusion. For example: a broken link, typo, or a user is not sure why a button is disabled.

We’re looking at improving the concept of User Feedback at Sentry. Instead of prompting an end-user for feedback once an error happens, we want to allow a user to initiate a feedback report at any time.

Do you collect, and if so how, user reports/feedback today?

Some initial thoughts we had for this new and improved User Feedback experience, are….

• With Session Replay a user report becomes a lot more valuable. You can see what the user was struggling with in addition to their written report. Do you agree with this assumption?
• Include a widget that’s part of our Browser SDK and can easily be added to your website
  • This widget allows the user to type their email and description of the issue.
    • If a user’s email is set on the scope, through Sentry.setUser, it’ll auto-populate it
  • Always save with the feedback the same useful context you get with Sentry errors, such as browser information, URL, release, etc.
  • Also capture a Session Replay (including sometime leading up to the user triggering the User Feedback)
• Alerting: This has been in our backlog in this GitHub issue for over 5 years. Would you want to be alerted on each user report submission?
• Triaging flow:
  • Sentry has a User Feedback menu item already. Is that where you’d want to work in triaging user reports?
  • Alternatively, would you expect these to be in the Issues list, categorized as User Feedback instead?
    • If so, how would you expect them grouped? By URL? By user?
    • Would you expect the same triaging mechanisms (e.g., assign, resolve, ignore, etc.)
    • Are there other mechanisms you’d expect in terms of triaging User Feedback that Issues doesn’t have today?

We want to hear from you! What are you looking for in a User Feedback/Report tool? Please comment below.

Thank you!

[pimterry]

With Session Replay a user report becomes a lot more valuable.

I can see how people might want that, but personally I'd still typically disable this for privacy reasons, so it definitely shouldn't be required. I've had quite a few users concerned about full session recording tools in the past, it just feels weird, so I avoid them.

Would you want to be alerted on each user report submission?

Yes, always always always!

Instead of prompting an end-user for feedback once an error happens, we want to allow a user to initiate a feedback report at any time.

This is interesting as an option, but personally I already have feedback mechanisms for users elsewhere, more focused on feature suggestions (public lists of ideas, voting) and general bug reports (specific format, pushing users to describe what they're trying to do, steps to repro, why it's not doing what they expect, etc). Here I'm mainly interested in error-integrated feedback - I can imagine this could replace the bug reports, but I think I'd always want feedback for feature suggestions handled very differently to error/bug reporting.

Sentry has a User Feedback menu item already. Is that where you’d want to work in triaging user reports?

Yes, I think so - I'd consider these as separate submissions, which may include a reference to a specific error event that happened just beforehand. I certainly wouldn't consider it just another type of issue, so mixing the two is seems odd.

[bruno-garcia]

With Session Replay a user report becomes a lot more valuable.

I can see how people might want that, but personally I'd still typically disable this for privacy reasons, so it definitely shouldn't be required. I've had quite a few users concerned about full session recording tools in the past, it just feels weird, so I avoid them.

Even though we do our best to avoid anything that could identify a user or any text in the screen, that's fair. I'm sure you're not alone with this. Sentry really goes out of its way to mask everything, by default replacing all text with *** (see more: https://docs.sentry.io/product/session-replay/protecting-user-privacy/#how-session-replay-approaches-data-capture ) there's no need to force a replay to User Feedback submissions. But including it would greatly help understand what difficulties the user had.

Would you want to be alerted on each user report submission?

Yes, always always always!

Do you have concerns about how noisy this can be? At what point would you decide to turn this off? Or rather, what controls would you expect in terms of alerting? For example, 1 alert per user, per day? One per page?

Instead of prompting an end-user for feedback once an error happens, we want to allow a user to initiate a feedback report at any time.

This is interesting as an option, but personally I already have feedback mechanisms for users elsewhere, more focused on feature suggestions (public lists of ideas, voting) and general bug reports (specific format, pushing users to describe what they're trying to do, steps to repro, why it's not doing what they expect, etc).

What feedback mechanisms/tools do you currently use? What do you like about them?

Here I'm mainly interested in error-integrated feedback - I can imagine this could replace the bug reports, but I think I'd always want feedback for feature suggestions handled very differently to error/bug reporting.

What way you present the widget would differ? Or could it be a category instead? Like Type: Bug, Feature Request?
Could you expand on what you expect in terms of error-integrated feedback? Sounds a bit with what Sentry offers today?

Sentry has a User Feedback menu item already. Is that where you’d want to work in triaging user reports?

Yes, I think so - I'd consider these as separate submissions, which may include a reference to a specific error event that happened just beforehand. I certainly wouldn't consider it just another type of issue, so mixing the two is seems odd.

We have Performance Issues also in the Sentry Issues list. I guess that's what I had in mind when I suggested "another type of issue". Also, an issue can be handled or not, etc. But in a world where we can have User Feedback outside of errors, would it be odd if they landed on the issues list?

[souredoutlook]

One thing I'd love to see out of changes in this space is seeing User Feedback without an error context reported as occasionally there may be a relevant transaction, Replay or both.

[pimterry]

Do you have concerns about how noisy this can be? At what point would you decide to turn this off? Or rather, what controls would you expect in terms of alerting? For example, 1 alert per user, per day? One per page?

I'd always want an immediate alert for every message. If it got too noisy, I would turn off the feedback widget before I turned off the alerts.

In practice, I've repeatedly seen the current feedback widget (mis)used as a general "send a message to support" input box after something goes wrong, where users ask direct questions, or even ask to talk to the sales team. I'd expect the same notifications for this as for a contact form on the front page of the website.

I've had major customers specifically say they didn't get in touch via other support channels about critical issues they were facing, because they had already sent it via Sentry's feedback form, and assumed that would immediately be received and handled.

Accepting freeform customer messages and not immediately notifying somebody to handle them is basically unacceptable (that's my main issue with the current implementation!).

What feedback mechanisms/tools do you currently use? What do you like about them?

Primarily GitHub, as my audience is quite technical. It's widely understood and many users already have accounts so it's low-friction, it has tools to push different types of feedback into different formats (adding a 'vote for this' footer to feature requests, requiring certain fields on bug reports) or direct users elsewhere (sending non-paying user general questions to stack overflow), using 👍 vote on issues is widespread & well understood, it's easy to search through & sort existing issues accordingly, and to link issues to development elsewhere.

I've seen quite a few other products using Canny too, which seems to work well in less technical circles.

I already have feedback mechanisms for users elsewhere, more focused on feature suggestions

What way you present the widget would differ? Or could it be a category instead? Like Type: Bug, Feature Request?

It's not so much about a difference in input - it's about the very different experience afterwards. Feature requests should be public, searchable, linkable to a roadmap, voteable, groupable/de-dupeable etc. Individual user error reports shouldn't be most of those things. Bug reports maybe should, but those too are quite different to raw error reports in my mind, as they're at a different level of granularity (many bits of user feedback or many quite different error reports may collectively represent facets of a single bug).

Could you expand on what you expect in terms of error-integrated feedback? Sounds a bit with what Sentry offers today?

Yes, to be honest the core of Sentry's current approach today is sensible for this part. It's just the issues around subsequent notification & UX that cause issues (imo it needs a very different model to error reports - e.g. I will always want to at least skim every sent message, while I never want to look at every single instance of an error).

The changes described here to separate feedback from errors, to allow non-error feedback and allow attaching other info like recordings are interesting and nice to have certainly, but otherwise not critical imo.

[shaedrich]

With Session Replay a user report becomes a lot more valuable.

I agree with @pimterry here. Although, this is highly value and could be encouraged, the user should have a choice. The default setting can be determined by the DNT for example.

To better pinpoint the element in question,

• users should have the ability to attach a screenshot (browsers offer the possibility to capture HTML elements. Maybe this would be a helpful option)
• if feature flags are enabled, the feature could be explicitly linked

In return, the user would get more information on the release:

• Version number
• Release notes/changelog? - This might be aid suspect commits

Triaging flow:

Especially, when user feedback is going to be not only for errors, issues would be the wrong place. However, there could be a link to an issue in case of an error, and more generally, the possibility to turn a feedback into an issue depending on the type, the user might be able to submit. Some types might be forwarded to certain integrations.

Will there be an option to (auto) respond to users when they provide their email address? Auto responses could be tied to triaging (e.g. "feedback is turned into an issue"). What about handling feature requests?

GitHub offers issue templates—are you planning to offer something similar?

Would you want to be alerted on each user report submission?

This alert rule could simply be configurable, couldn't it?

I also see the opportunity to integrate that closer to with parts of the [Performance Monitoring](Performance Monitoring).

[jas-kas]

Hey @shaedrich,

I have some updates to share after discussing this initiative further with the team this month:

Although, this is highly value and could be encouraged, the user should have a choice.

Session Replay will be an optional choice (i.e., the Sentry user can decide if they want to capture replays for their user-initiated bug reports or not)

users should have the ability to attach a screenshot (browsers offer the possibility to capture HTML elements. Maybe this would be a helpful option)

Our first pass for the alpha will focus on Session Replay; however attaching a screenshot and annotating it is a capability we're considering for the future.

However, there could be a link to an issue in case of an error

This is how we plan to build it. :)

Will there be an option to (auto) respond to users when they provide their email address? Auto responses could be tied to triaging (e.g. "feedback is turned into an issue").

Not for our alpha release, but we've given thought to how this would work.

What about handling feature requests?

This product is oriented around bugs that could happen in your product (not necessarily triggering Sentry errors) and areas where your users get stuck/confused. If a user does submit a feature request, then it can be handled the same way as a bug report (you can respond via email), but there won't be specifically tooling built for feature requests (e.g., upvoting/downvoting feature requests from users).

GitHub offers issue templates—are you planning to offer something similar?

Our first iteration of the widget to collect bug reports will ask for email and a description as mentioned above, but as we collect more feedback, we'll iterate on what we capture and offer customization abilities to our Sentry users.

This alert rule could simply be configurable, couldn't it?

100%, it would be configurable.

I hope this helps and let me know if you have any more questions! If you'd like to partake in our alpha release, you can fill out this short form. :)

[shaedrich]

Hey @jas-kas and thanks for your reply :)

I'm happy, we are mostly on the same page and you take the rest into consideration for a later stage :)

[ueman]

I'm coming from a background in Flutter.
I wrote a user feedback tool myself: https://pub.dev/packages/feedback. this in particular allows you to take a screenshot, scribble on it, and attach text to it. The screenshot is particularly helpful for design flaws.

On Sentry I would like to have a dedicated view for those issues. Each user feedback should have a list of breadcrumbs attached to it, just like crash events.

If so, how would you expect them grouped? By URL? By user?

I would like to attach some kind of tags or categories to those feedback entries. That allows me group those issues by that tag. Tags could be for example: UX, a11y, performance, etc

Would you expect the same triaging mechanisms (e.g., assign, resolve, ignore, etc.)

Yes

[jas-kas]

We haven't considered attaching screenshots (with annotations from the user) to feedback entries, since we were focusing on tying them to Session Replay videos, but we will keep that in mind. :)

I would like to attach some kind of tags or categories to those feedback entries. That allows me group those issues by that tag. Tags could be for example: UX, a11y, performance, etc

So you'd be okay with manual tagging here of the feedback entries, rather than auto-grouping?

Thanks for the feedback, Jonas!

[ueman]

I would like the user to add a category, not me. Either from a list of predefined categories or via free form text field.

[jas-kas]

Makes sense, thanks for clarifying @ueman!

[foundry-markf]

I'm coming from a background in C/C++, for the desktop platforms Windows, Linux, Mac (Intel and Apple Silicon). While I think your discussion might be more web-based technologies oriented, I do have feedback to give on giving feedback in the Sentry ecosystem.

The options for getting an eventID, and giving feedback, at the point of crash is pretty limited in native land. Using crashpad, even with the preferred out-of-process handling so that you aren't reliant on doing work in a crashed process, you are limited with what you can do.

Out of the box, from Sentry Native 0.6.3(I believe), Windows and Linux at least had some detection of the EventID, in the __sentry-event file in the database on disk. Also, the on_crash callback in the application. Mac doesn't have either of these - from what I gather, it's just plain hard to do on that platform. So immediately out of the box, we can't do a cross-platform consistent solution.

crashpad itself doesn't offer any feedback mechanisms. We've tried modifying it to show a UI, but it's invasive non-trivial threading code, since the upload occurs in a worker thread, and you need to synchronise between that and its main thread to show a UI.

The nice part about giving feedback is that you can do it with CURL (or similar). You just need the org and product slugs, eventID, and can use DSN authentication. So, as long as you can get that eventID, job's a goodun. It's getting the eventID for a crash that's difficult; at least consistently across all platforms.

The surprising part of giving feedback is that it's limited to under 30 minutes after the crash occurred. (Described https://docs.sentry.io/api/projects/submit-user-feedback/.) So in our investigations, we've needed to work around that so that our users can provide feedback in their own time, meaning we must control when the initial report upload occurs.

From studying what other products do with just crashpad, the way to proceeds appears to be to request feedback post-crash. I've had success in rolling my own solution, by always disabling auto-uploads at the point of crash (sentry_options_set_require_user_consent), and using crashpad specific APIs to get unprocessed crash report details. Subsequently the application can show those reports in a UI (we're in full control of this so it's easy to maintain, unlike patches in crashpad for instance), and for the user to decide whether to both send and provide feedback.

Initially, we want to gather feedback with a message, but being able to upload files is on the roadmap. These could be log files, images, project files, text or binary, compressed or uncompressed, almost anything to help reproduce the crashing scenario. We've not explored this in detail yet, although am aware of https://docs.sentry.io/platforms/native/enriching-events/attachments/. Being able to configure the server to accept/reject file types, and to configure the size of each attachment (rather than as a sum total) would be useful in this instance - I don't know if that's already possible.

[georgiosd]

I would be in favor of seeing errors and user feedback in one stream, with the appropriate search filters so that people can customize. We're a small team and even establishing a process to click around to see if there's something that deserves our immediate attention takes too much time in the every-day release hustle.

DEFINITELY want the option to receive alerts for user feedback. Grouping by URL makes sense to me but it may become valuable to see a single user's feedback in one view too. Surely that's easily done with the search filters.

On the topic of privacy - sure, I get that, but don't make it automatically based on any mechanism (like DNT suggested). The dialog should make an effort to demonstrate that this will drastically improve the chances of resolving the problem (as a user benefit). The biggest problem we have with the sentry dialog is that most people won't fill it in because they believe it's being worked on anyway - once we've changed the text to reflect the usefulness of filling in the dialog, we saw an increase in submissions.

[jas-kas]

Thank you for this feedback, Georgios and your experience with the existing User Feedback tool!

[lucas-zimerman]

Other case that I see is business logic errors, for example, user cannot checkout on an app due to different timezone, this doesn't generate an error since it's expected behaviour, but it's a wrong behaviour that the user expects to work. So Ideally, we could link user feedbacks to also transactions or replay ids (Or even not link to any event at all?, maybe developers could use the user feedback as a way of collecting feedbacks from the users on what they like or not on t he application)

[jas-kas]

Hey Lucas, thanks for the feedback!

So Ideally, we could link user feedbacks to also transactions or replay ids (Or even not link to any event at all?, maybe developers could use the user feedback as a way of collecting feedbacks from the users on what they like or not on t he application)

The idea is to link user feedback reports to any debugging context that is available - so that would include transactions and replays. In the event that there is minimal debugging context, we intend to still show the written feedback report in Sentry. :) Let me know if you have any questions.

[dbainbridge]

We are using the User Feedback through the SDK for mobile/desktop apps. The current issue is that the "Sentry.UserFeedback" api does not support attaching files so we have to create an event with the logs then attach the event to user feedback with "Sentry.UserFeedback(eventId:eventId)". This is useful for being able to send detailed log files if user wishes (there is checkbox to include them in our UI when sending feedback).

[jas-kas]

Thanks for the feedback David!

How often would you say you need the detailed log file to successfully debug an issue, as opposed to the typical Sentry error report and what cases is this the most common?

[amanv8060]

Some way to reply users from user feedback without actually creating a new mail trail would be nice.

[jas-kas]

Hey Aman, could you provide more detail around what you mean by replying to users without creating a new mail trail? How would you expect that to work? And would you like the ability to respond to a user within Sentry?

Thank you for the feedback!

[amanv8060]

Currently, the only option is to copy their email and then start a separate email conversation with the user. As you correctly mentioned, the ability to respond to a user within Sentry is exactly what I am looking for.

[cbrwizard]

Alerting: This has been in our backlog in #5055 for over 5 years. Would you want to be alerted on each user report submission?

Yes, that's exactly what we need at Streamline. We want to ensure that each user's feedback is processed by our customer support. Knowing that a user has left feedback will help developers to prioritize resolving the issue. We need integration of such alerts with Slack and maybe Intercom too.

Session replays sound useful but we don't use them right now.

[jas-kas]

Hey @cbrwizard,

Any particular reason you don't use Session Replay at the moment?

[jas-kas]

Hey folks 👋

Thanks for all of your detailed feedback over the last month!

I'm excited to announce that we're gearing up for an alpha launch of user-initiated bug reports in October. 🎉

If you'd like to be an alpha partner, influence how this product is built, and get insightful feedback from your end-users, please fill out this short form.

If you have any questions, let me know in this thread.👇 Thank you!

[cbrwizard]

"This form can only be viewed by users in the owner's organisation."

[jas-kas]

@cbrwizard Sorry about that! The form should be public now. Let me know if you encounter other issues.

[joshacheson]

One small thing that would be nice to see (and apologies if it's already captured or implied) would be to use the client (like the one found with Sentry.getCurrentHub().getClient() to create user feedback easily if we want to implement custom UI. Currently, I have to capture an event manually (like just Sentry.captureEvent()) to get an event id to associate user feedback with, spin up a network client for sentry (or otherwise figure out how to orchestrate an ad hoc call to sentry), figure out auth to sentry, make sure my project name is correct, etc. More or less all of this is already configured when doing Sentry.init, but it has to be duplicated more or less to make a direct user feedback api call. Being able to do this with one convenient method call on the client (so that I can implement my own UI to actually capture the feedback) would be great.

[bruno-garcia]

In the version 7.47.0 we added the a JS API to capture User Feedback: https://docs.sentry.io/platforms/javascript/enriching-events/user-feedback/#user-feedback-api

copying the snippet from the docs for convenience:

const eventId = Sentry.captureMessage('User Feedback');
// OR: const eventId = Sentry.lastEventId();

const userFeedback = {
  event_id: eventId;
  name: 'John Doe',
  email: '[email protected]',
  comments: 'I really like your App, thanks!'
}
Sentry.captureUserFeedback(userFeedback);

[goodones-mac]

Make something like instabug unnecessary on the mobile side, which means creating a UX for users to fill in bug reports with that we wouldn't have to implement ourselves. So shake to report, add screenshots, write descriptions, etc. Then send these bug reports to something like zendesk, intercom or an email, etc with a link (for the company, not the customer) to the new issue generated in sentry to we can see all the background eng info. Hooks for zapier would also be useful.

[jas-kas]

Hey folks 👋🏻

We just launched the improved User Feedback feature (i.e., 'user-initiated bug reports') to all Early Adopters (you can turn on being an Early Adopter in your org settings). 🎉 This is currently in beta, with more improvements lined up in the future, but would love to hear everyone's first impressions and feedback.

To learn more, please see...

• Set up docs. If that page appears blank, check our README.
• Blog post
• Quick video demo by me!

Let me know if you have any questions or concerns.

[jas-kas]

Follow-up GH discussion post: #60134

[joshacheson]

@jas-kas this is super exciting! Congrats to you and others involved on what are obviously positive moves for the User Feedback feature.

I have one question, referencing the following snippet from the blog post

If you are currently using our existing User Feedback product where we attach written user messages to Sentry issues, rest assured that this functionality will not be going away. New incoming submissions from this modal will live in the new UI we built for User Feedback shown above.

I'm curious whether it will be possible to use the new User Feedback programmatically (i.e. with our own modal rather than the one you provide)? If not immediately, is that anywhere on the roadmap?

Congrats again! Overall this is awesome. We rely a lot on User Feedback and these enhancements matter a lot to us.

[jas-kas]

@joshacheson Thank you 🎉 I'm glad to hear you're hyped about the new and improved User Feedback and how this is valuable to your team.

I'm curious whether it will be possible to use the new User Feedback programmatically (i.e. with our own modal rather than the one you provide)? If not immediately, is that anywhere on the roadmap?

Yes, absolutely! Follow the steps outlined here: https://www.npmjs.com/package/@sentry-internal/feedback#bring-your-own-widget

LMK if you have any follow up questions.

[nokibsarkar]

Why not make all the fields configurable when passing to the showFeedbackDialog? More specifically, I don't want my users to enter their email address for privacy reason.

[bruno-garcia]

You can set showEmail to false and the widget won't show it: https://www.npmjs.com/package/@sentry-internal/feedback#userform-related-configuration

If there's any customization you can't make through options such as this though, you can also roll your own form and just call our JS API to capture the values you care about.

[asif2k]

I am using sentry-go and want to know how to prevent it sending sdk information and packages list to my sentry with each event?

[bruno-garcia]

Not sure we have this capability. Please raise a ticket here, maintainers will be able to help you

[klemmchr]

It seems that currently there is no way to get in touch with the user reporting an issue. We have their email address, but there is no direct way to communicate with the author directly in Sentry. This means that whenever I have a question regarding the issue I would need to write the user an email using my own mail client, using my user specific mail address and none of the conversation will be included in the user feedback. It would be great to have the ability to write the reporting user right away within Sentry.

[jas-kas]

Hey @klemmchr 👋

Thanks for sharing your feedback! We're hoping to improve the communication flow to end-users in the coming months. I do have some broad questions for you around this:

none of the conversation will be included in the user feedback.

Why is this important for your workflow? Is it because you want to convey to your team members that progress has been made on a ticket?

This means that whenever I have a question regarding the issue I would need to write the user an email using my own mail client

Are you often asking follow up questions to understand the feedback submission, or are you more so conveying the status of it ("we're working on it", "team is investigating", "this is resolved now", etc.)?

I'd like to share some ideas with you on what we're looking at building next, if you could send me an email at jasmin . kassas @ sentry.io, I'll schedule a quick 30m call if you're open to it!

[klemmchr]

Hey @jas-kas
Thanks for the fast response.

Why is this important for your workflow? Is it because you want to convey to your team members that progress has been made on a ticket?

Not really because I want to convey team members about progress made, it's more about transparency in communication and the ability for multiple persons to help out each other. A very simple example could be vacation or sick leave. All the conversations with the reporting user would be stuck in my mailbox and nobody could hop on to represent me in my period of absence. Additionally it's never a bad idea to have multiple people being able to read what the user actually said in the follow up to prevent misunderstandings.

Are you often asking follow up questions to understand the feedback submission, or are you more so conveying the status of it ("we're working on it", "team is investigating", "this is resolved now", etc.)?

Both would be highly interesting for us. Currently we are using a ticketing system for bug reports with the ability to respond to the user directly. We would need both, a status update mechanism and individual communication purposes to fill the gaps of information we might need to reproduce the error. While Sentry is of good help already by including relevant metadata, there are some scenarios where we need more info that can only be provided by the user.

I'd like to share some ideas with you on what we're looking at building next, if you could send me an email at jasmin . kassas @ sentry.io, I'll schedule a quick 30m call if you're open to it!

That would be great. Gonna drop you a mail.

[jas-kas]

Hey @klemmchr 👋

Just responded to your email. Looking forward to our chat!

[klemmchr]

Thank you again for the nice chat. It was a pleasure to talk to you all. I cannot state enough how great Sentry is doing, not just in terms of adding value for their customers but also communication wise.