Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inline images #1466

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

Inline images #1466

wants to merge 4 commits into from

Conversation

jubalh
Copy link
Member

@jubalh jubalh commented Jan 11, 2021

PR for #1301

So far this is just a test. Displaying an image ~/test.png when Profanitys "about" is printed.
So far in ASCII art.

libchafa can also return colourful images using escape codes. But not sure how to use them inside ncurses yet.

I assume later we want a setting that either displays all images (URLs which we assume to be images based on ending) or we want it only explicitly with somethine like /image / /imagepreview / /inlineimage where one can tab through the latest URLs with image endings.

First tests with libchafa for displaying pictures inline.
We choose to use ASCII art for now. Reason is that the regular method
uses color escape codes and ncurses will just print those codes and not
use the colors.

Regards #1301
@jubalh jubalh added the feature label Jan 11, 2021
@jubalh jubalh added this to the 0.11.0 milestone Jan 11, 2021
@jubalh
Copy link
Member Author

jubalh commented Jun 28, 2021

chafa has a test branch with added functionality for us:
https://github.com/hpjansson/chafa/tree/wip/cell-access
example: https://github.com/hpjansson/chafa/blob/wip/cell-access/tests/ncurses.c

Need to build this chafa version and see whether this helps progressing on this task.

@jubalh jubalh modified the milestones: 0.11.0, 0.12.0 Jun 29, 2021
@jubalh jubalh self-assigned this Jun 29, 2021
@ghost
Copy link

ghost commented Jul 5, 2021

i'm pretty sure using viu would work too, no? It displays stuff with colored blocks, or uses Kitty's image protocol if available, which i guess would be similar to what you're looking for, plus Kitty users could get stuff in high quality

@jubalh
Copy link
Member Author

jubalh commented Jul 5, 2021

viu is an cli image viewer. There are plenty of those. It outputs things on the terminal we can't use that to display a picture inside our chat windows. viuer is the library that viu uses. It is written in Rust. Profanity is written in C. So we we settled on libchafa.

@hpjansson
Copy link

I merged the wip/cell-access branch into master some time ago. I don't expect it to change very much between now and release, so it should be safe to start using.

I'm also planning Kitty and iTerm2 support, but I'm not sure how you'd get ncurses to cooperate...

@jubalh jubalh removed this from the 0.12.0 milestone Feb 23, 2022
@jubalh
Copy link
Member Author

jubalh commented May 31, 2022

@binex-dsk a little similar to the QR code task. Do you think that would be something for you? ;)

@ghost
Copy link

ghost commented May 31, 2022

@binex-dsk a little similar to the QR code task. Do you think that would be something for you? ;)

I could possibly do basic block art in the same vain as the QR code. Not sure if colorful images would be my forte, but I'll look into it.

Also: are shell color codes agnostic to operating systems/shell standards?

Also, I'm currently on a band trip without access to a computer. I'll see if I want to tackle this after next Sunday/Monday.

@jubalh
Copy link
Member Author

jubalh commented May 31, 2022

I could possibly do basic block art in the same vain as the QR code. Not sure if colorful images would be my forte, but I'll look into it.

In this PR I tried to use the library "chafa" actually it should do most of the work. We just need set the correct settings (what it will return) and then print this properly in the window.

Like chafa_symbol_map_add_by_tags (symbol_map, CHAFA_SYMBOL_TAG_ASCII); and there are various thing to set there. I also think @hpjansson added new options since I tried this last time.

Also: are shell color codes agnostic to operating systems/shell standards?

I'm not entirely sure. But I don't think so.

Also, I'm currently on a band trip without access to a computer. I'll see if I want to tackle this after next Sunday/Monday.

Enjoy!

@hpjansson
Copy link

I recommend reading the ncurses example @jubalh mentioned before. I've tried to comment it liberally, but if anything is unclear I'll gladly answer questions and review code.

As an aside, I think it's possible to support Kitty and sixel graphics in ncurses too, but you'd have to print raw character codes, so it probably wouldn't work inline (ncurses repaints would likely mess it up). But e.g. vifm uses a frame in a fixed location, and that seems to work (using fileviewer *.png,*.jpg chafa -f kitty -s %pwx%ph %c:p %pd %N 2>/dev/null).

@ghost
Copy link

ghost commented Jun 2, 2022

In this PR I tried to use the library "chafa" actually it should do most of the work. We just need set the correct settings (what it will return) and then print this properly in the window.

Like chafa_symbol_map_add_by_tags (symbol_map, CHAFA_SYMBOL_TAG_ASCII); and there are various thing to set there. I also think @hpjansson added new options since I tried this last time.

Okay, I see. In that case I'll mess around with the settings and see what I can do. What is it you're looking for that this PR does not yet do (other than deciding on what to do for displaying them, of course)

And should I test this on the cell-access thing mentioned? Or on master?

EDIT: cell-access is old and merged, so I'll test this on master. And also I won't have a huge amount of time again (Robotics, summer classes, sports, etc...) but depending on what's needed I'm confident I'll be able to get it done quickly.

@hpjansson
Copy link

If your package manager has Chafa 1.8.0 or later, the cell access API will be there, so you can build against the devel package instead of master. In case it saves time.

@ghost
Copy link

ghost commented Jun 15, 2022

If no one else takes this up within a few weeks I'll go ahead and do it... right now I've got a boatload of stuff to do: summer classes, driving practice, and soccer. I might experiment with the ncurses examples at some point though :)

@mrusme
Copy link

mrusme commented Aug 30, 2022

Having inline images rendered via Chafa would really be awesome! I have only scratched the surface here, but this also requires integration with OMEMO in case of aesgcm:// URLs, correct?

@jubalh
Copy link
Member Author

jubalh commented Oct 17, 2022

@binex-dsk how are your driving classes? Do you have time for this feature now maybe?

@ghost
Copy link

ghost commented Oct 17, 2022

@binex-dsk how are your driving classes? Do you have time for this feature now maybe?

Finished them up. I have my driver's test this week and soccer (should) end this week as well, so I will begin work for real (like, actually for real this time I swear) next week.

@jubalh
Copy link
Member Author

jubalh commented Oct 17, 2022

That's great news!

@jubalh jubalh assigned ghost Oct 18, 2022
@jubalh jubalh added this to the 0.14.0 milestone Oct 18, 2022
@ghost
Copy link

ghost commented Nov 3, 2022

Quick update, I have been doing some basic libchafa testing (on its own, not with profanity) and although I haven't made that much progress, I AM getting there. I would be farther but, I had to go to urgent care, which (unfortunately) is more important than programming...

@jubalh
Copy link
Member Author

jubalh commented Nov 3, 2022

Thanks for the update!
Take your time and take care!

@ghost
Copy link

ghost commented Dec 7, 2022

I've been messing with this for a while and so far I've not been able to figure out several things.

Here's the point I've reached:

image

I have yet to figure out:

  • True color; term info IS properly detected but truecolor mode is not yet working. In my previous tests I never managed to get this to work either...
  • Kitty pixel support; Also never managed to figure this out in previous tests.
  • Wrong aspect ratio; Never a problem in my previous tests.

You can also notice that the pixels don't look right. It's a QR code so it should be all squares.

Commit is here:

https://github.com/binex-dsk/profanity/commit/8e5aa52c72fbb99b8ca4b50b8a7278521ad56b83

@ghost
Copy link

ghost commented Dec 7, 2022

Looked into the ncurses example a bit more and I'm confused.

@hpjansson Would you mind explaining the canvas_to_ncurses function in the ncurses example? I believe this is the function we need for truecolor but I haven't managed to decipher it.

Also, how would we go about actual Kitty support? It seems like it would be really hard to make it work with kitty, though I don't know the internals of the Kitty protocol so maybe I'm missing something.

@hpjansson
Copy link

hpjansson commented Dec 8, 2022

@hpjansson Would you mind explaining the canvas_to_ncurses function in the ncurses example? I believe this is the function we need for truecolor but I haven't managed to decipher it.

Sure! It iterates over the ChafaCanvas character cells, row by row and column by column. For each cell, it:

  • Extracts the Unicode character with chafa_canvas_get_char_at() and turns it into a wchar_t, which is what ncurses wants.
  • Extracts the cell's BG and FG colors using chafa_canvas_get_colors_at(), which yields signed ints that are either -1 for transparency or else an RGB triplet in the lower three bytes (truecolor mode -- other modes use the _raw equivalent, which gets -1 or indexes from a fixed palette).
  • Initializes a corresponding "color pair" for ncurses using init_extended_pair().
  • Initializes an ncurses cchar_t structure combining the character, color pair and default attributes using setcchar().
  • Puts the resulting cchar_t to the ncurses window using mvadd_wch().

The ncurses functions init_extended_pair, setcchar and mvadd_wch can be looked up with man in a shell. The Chafa functions have documentation pages on the web.

Note that you need to link with the correct build of ncurses (e.g. ending in w6 for wide characters), and your TERM variable must be set to a terminal type that supports directcolor for this to work (e.g. TERM=xterm-direct).

Also, how would we go about actual Kitty support? It seems like it would be really hard to make it work with kitty, though I don't know the internals of the Kitty protocol so maybe I'm missing something.

I think this would be impossible, or at least not worth the effort, to support in an ncurses program unless there's a non-scrolling preview rectangle where you can just bypass ncurses and overwrite the contents using your own escape codes. I think vifm does this (see: hpjansson/chafa#77, although it's calling chafa externally).

@ghost
Copy link

ghost commented Dec 8, 2022

Note that you need to link with the correct build of ncurses (e.g. ending in w6 for wide characters)

Aha, that's why the function didn't end up working after I plugged it in!

An interesting thing I'll need to look into is getting it to work with the standard win_println-style of printing--that is, with the time and indented as other messages are. Although it might be better to end up displaying it without the indent/time, and instead, image messages have text content "Image" or "Media Message" or similar, and then in the next line the image is displayed.

I'm unfamiliar with the internal workings of the profanity window thingy so I have no clue if it'll properly scroll if printed with mvadd_wch. Although a brief look at the print functions does suggest it will.

I suppose I'll look into all of this, perhaps more, tomorrow. Depends how much free time I've got. Which might not be a lot, because God knows if I'll end up needing to go to the ER again.

I think this would be impossible, or at least not worth the effort, to support in an ncurses program unless there's a non-scrolling preview rectangle where you can just bypass ncurses and overwrite the contents using your own escape codes.

This is what I suspected. I have an idea on how that could work, but it would be clunky, awkward, perhaps user-unfriendly, and not to mention a HUGE pain to program.

Thanks for the assistance @hpjansson; this makes a lot more sense now!

@ghost
Copy link

ghost commented Dec 9, 2022

After a test using canvas_to_ncurses, mvadd_ch does not properly print the image to the window... I think what we should do is add a way to print a wchar array, and in the canvas_to_ncurses function, we construct a wchar array and use that to print. But the former is beyond my area of expertise...

Also, if ncursesw isn't found we probably have to do basic ASCII/block art

@ghost
Copy link

ghost commented Dec 9, 2022

@jubalh perhaps you could help with the wchar printing? I have no idea how to implement this, as I'm not familiar with the internal print functions.

@jubalh
Copy link
Member Author

jubalh commented Dec 12, 2022

@binex-dsk ok I will try to look at it! :)

@ghost
Copy link

ghost commented Dec 27, 2022

@jubalh how has the wchar printing progress coming? Perfectly okay if you haven't done anything yet

@jubalh
Copy link
Member Author

jubalh commented Dec 31, 2022

@binex-dsk sorry I haven't yet. I'm having some health issues. I will try to look at it when all is okay again :)

@hpjansson
Copy link

hpjansson commented Dec 31, 2022

Just from a quick look, it looks like buffer.c would be a good place to start. You could add code there to queue messages with images in them. Then window.c has a redraw function and print functions -- functionality could be added there to paint the contents of images from the buffer object.

I'd store the original raster image (after decoding from the network) in the buffer entry, and call Chafa every time it needs to redraw. That lets you deal with window resizes etc. gracefully after the message is received.

@hpjansson
Copy link

Also -- @jubalh, take care and get well soon!

@ghost
Copy link

ghost commented Dec 31, 2022

I'm having some health issues

Lmao same

Hope you get well! :)

@jubalh jubalh modified the milestones: 0.14.0, next Feb 15, 2023
@jubalh jubalh removed this from the next milestone Jul 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants